A data security storage method and system in a big data transaction

Abstract

The application relates to the technical field of data security, in particular to a data security storage method and system in big data transaction. The method maps the file volume to the CPU estimated time consumption by extracting physical characteristics such as the data block quantity, the key segment quantity and the deletion entry quantity, and combining the hardware benchmark parameter and the environment correction factor; the latest completion time and the thread load imbalance ratio of the system are predicted by constructing a virtual scheduler in the memory and simulating the task allocation of the background thread pool; the recommended delay coefficient is generated based on the prediction result, and the lock-free sleep operation is executed before the write request acquires the mutual exclusion lock resource. By analyzing the implicit CPU computing power overhead, simulating task allocation with the virtual scheduler, predicting the thread load state in advance, implementing flow control before the write request acquires the storage engine mutual exclusion lock through the lock-free delay execution, the dead lock risk is reduced, and the stability and throughput capacity of the system in the compliance encryption scene are improved.

Description

Technical Field

[0001] This invention relates to the field of data security technology, specifically to a data security storage method and system for big data transactions. Background Technology

[0002] With the widespread adoption of high-speed non-volatile storage media such as NVMe SSDs in financial big data transaction systems, the performance bottleneck of database storage engines is shifting from traditional disk I / O bandwidth to CPU computing power. In financial compliance scenarios, data storage must meet requirements for transparent data encryption and frequent key rotation. This results in the underlying SSTable files often being encrypted with multiple different historical key segments throughout their lifecycle, containing a large number of data records marked for deletion but still requiring integrity verification. This computationally intensive scenario means that even two data files of the same size can have significantly different CPU consumption due to differences in the number of internal key segments and the proportion of invalid data.

[0003] However, existing storage engine flow control mechanisms typically only monitor the number of bytes written or the I / O rate, and cannot detect the nonlinear CPU computing power overhead caused by the differences in key topology. They are prone to misjudging "small" files with high computational complexity as lightweight tasks, causing the background compression thread to be blocked by such tasks for a long time, exhausting CPU resources, and thus triggering cascading write pauses in the front-end trading system, affecting the stability of high-frequency trading. Summary of the Invention

[0004] To address the technical problem in existing technologies that lack the ability to detect the nonlinear CPU computational overhead caused by differences in key topology, easily misclassifying computationally complex "small-volume" files as lightweight tasks, leading to prolonged blocking of background compression threads and exhaustion of CPU resources, the present invention aims to provide a data security storage method and system for big data transactions. The specific technical solution adopted is as follows: This invention provides a method for secure data storage in big data transactions, the method comprising: During the data writing process, monitor the memory buffer freeze event to obtain the physical characteristics of the set of tasks to be processed; the physical characteristics include: the number of data blocks, the number of key segments, and the number of deleted entries; Based on preset hardware baseline parameters and environmental correction factors, the physical characteristics of the tasks to be processed are mapped to the estimated CPU time. In the virtual scheduler, the task allocation process of the background thread pool is simulated based on the estimated CPU time to determine the latest completion time and the thread load imbalance ratio. Based on the latest completion time and the current write rate, determine the time safety margin of the memory buffer; determine the suggested latency coefficient of the current write request based on the time safety margin and the thread load imbalance ratio; wherein, when the thread load imbalance ratio exceeds the preset balancing threshold, a suggested latency coefficient is generated based on a non-linear penalty strategy. Before the write request acquires the storage engine's mutex lock resource, a sleep operation is performed according to the recommended latency factor.

[0005] Furthermore, the method for obtaining the estimated CPU time includes: Obtain hardware baseline parameters, including baseline decryption time, key reload time, and authentication time; For each task in the set of tasks to be processed, the product of the number of data blocks and the baseline decryption time is used as the basic decryption overhead; the product of the number of key segments exceeding the baseline value and the key reload time is used as the context switching overhead; and the product of the number of deleted entries and the authentication time is used as the invalid authentication overhead. The sum of the basic decryption overhead, context switching overhead, and invalid authentication overhead is calculated, and the sum is multiplied by the environment correction factor to obtain the estimated CPU time.

[0006] Furthermore, the method for obtaining the latest completion time includes: Initialize the current cumulative load time of each background thread on the virtual timeline; sort the tasks in the set of tasks to be processed in descending order according to the estimated CPU consumption time; Iteratively execute the allocation operation: Select the unallocated task with the largest estimated CPU time, assign it to the background thread with the smallest current cumulative load time, and add the estimated CPU time of the task to the cumulative load time of the background thread, until all tasks are allocated. The maximum value among the cumulative load times of all background threads after the simulation allocation is completed is determined as the latest completion time.

[0007] Furthermore, the method for obtaining the thread load imbalance ratio includes: After the simulation allocation is completed, the average cumulative load time of all background threads is obtained, and the ratio between the latest completion time and the average cumulative load time is used as the thread load imbalance ratio.

[0008] Furthermore, the method for obtaining the time safety margin includes: The estimated filling time required to fill the remaining space of the memory buffer is analyzed based on the current write rate; the time safety margin is obtained based on the deviation between the estimated filling time and the latest completion time.

[0009] Furthermore, determining the suggested latency coefficient for the current write request based on the time safety margin and thread load imbalance ratio includes: If the time safety margin is not less than zero, the suggested delay coefficient is set to zero, and no flow control is applied; if the time safety margin is less than zero, an overflow risk is identified, and flow control decision is initiated. When there is an overflow risk and the thread load imbalance ratio does not exceed the preset balancing threshold, a linear strategy is adopted to generate a suggested delay coefficient that is linearly proportional to the absolute value of the time safety margin.

[0010] Furthermore, the generation of the proposed delay coefficient based on the nonlinear penalty strategy includes: Based on the time safety margin, the suggested delay coefficient obtained under the linear strategy is used as the base delay coefficient; the penalty increment is analyzed according to the degree to which the thread load imbalance ratio exceeds the preset balance threshold; the base delay coefficient and the penalty increment are superimposed to obtain the suggested delay coefficient.

[0011] Further, the step of performing the sleep operation according to the suggested delay coefficient includes: Obtain the batch size of the write request data; combine the suggested latency coefficient with the batch size of the data to obtain the target sleep duration; Before requesting a mutex lock on the storage engine for a write request, a lock-free sleep operation is performed for a duration equal to the target sleep duration.

[0012] Furthermore, it also includes updates to environmental correction factors: When the background task is completed, the actual processing time and the theoretical estimated time without considering the environmental correction factor are obtained. Based on the ratio of the actual processing time to the theoretical estimated time, a smooth iterative algorithm is used to periodically update the environmental correction factor.

[0013] The present invention also provides a data security storage system for big data transactions, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the data security storage method for big data transactions described above.

[0014] The present invention has the following beneficial effects: This invention first extracts physical characteristics such as the number of data blocks, the number of key segments, and the number of deleted entries, and combines these with hardware baseline parameters and environmental correction factors to achieve an accurate mapping from file size to estimated CPU time consumption. This allows for the identification and quantification of the actual system computing power consumption of "small but computationally heavy" encryption tasks. Second, by constructing a virtual scheduler in memory and simulating task allocation in a background thread pool, it proactively predicts the system's latest completion time and thread load imbalance ratio, providing data support for accurately assessing the existence of overall bottlenecks or structural blocking risks. Finally, based on the prediction results, a suggested latency coefficient is generated, and a lock-free sleep operation is performed before write requests acquire mutex lock resources. Furthermore, for structural load imbalances, a non-linear penalty strategy is used to quickly suppress the write rate of malformed data. This mechanism avoids the risk of system deadlock caused by directly blocking lock-holding threads while effectively smoothing foreground business traffic, ensuring the continuous availability and stability of the high-frequency trading system under extreme compliance loads. This invention analyzes the implicit overhead of CPU computing power, uses a virtual scheduler to simulate task allocation, predicts thread load status in advance, and implements flow control before write requests acquire storage engine mutex locks through lock-free delayed execution, thereby reducing the risk of deadlock and improving the stability and throughput of the system in compliant encryption scenarios. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a data security storage method in big data transactions, as provided in one embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a data security storage method and system for big data transactions proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details a specific solution for a data security storage method and system in big data transactions provided by this invention.

[0020] Please see Figure 1 The diagram illustrates a flowchart of a data security storage method in big data transactions according to an embodiment of the present invention. The method includes the following steps: S1: During the data writing process, listen for the freeze event of the memory buffer and obtain the physical characteristics of the set of tasks to be processed; the physical characteristics include: the number of data blocks, the number of key segments, and the number of deleted entries.

[0021] As front-end services continuously submit key-value pairs to the database, the storage engine maintains an active memory buffer in memory to handle these write operations. When the capacity of the active memory buffer reaches a preset threshold or a specific trigger condition is met, such as when the used memory size of the active memory buffer grows to the configured upper limit, such as 64MB or 128MB, the system automatically triggers a freeze event. The buffer is then marked as immutable, indicating that a freeze event has been triggered. Simultaneously, a new empty buffer is initialized to receive subsequent writes. The freeze event also marks the beginning of memory data preparation for disk writes or merging. In this embodiment of the invention, these immutable memory components and associated SSTable files that have not yet completed background processing constitute a set of tasks to be processed, reflecting the current system backlog of data debt that must be digested by consuming CPU computing power.

[0022] Then, the SSTable file corresponding to each task in the set of tasks to be processed is traversed, the attribute blocks resident in the memory cache are parsed, and the physical features that characterize the encryption computation complexity of each task are extracted to quantify the actual CPU computing power requirements of the task in the subsequent encryption storage process.

[0023] In this embodiment of the invention, the physical characteristics include: the number of data blocks, representing the total number of data blocks physically stored in the SSTable file, reflecting the basic linear computation workload required to perform AES-GCM standard decryption and verification operations, which can be directly obtained by reading the data block counter field in the attribute block; the number of key segments, representing the number of independent encryption key IDs contained in the file, reflecting the frequency with which the CPU needs to perform instruction set context reset and key expansion operations when processing the file, which can be obtained by parsing the key identifier list in the attribute block and counting the number of deduplicated IDs; and the number of deleted entries, representing the number of data records in the file marked for deletion but still requiring integrity verification, reflecting the mandatory authentication overhead that still consumes CPU resources even if the data logic is invalid, which can be obtained by reading the Tombstone statistics in the attribute block.

[0024] Specifically, if the key information is not recorded in the attribute block, such as in a legacy version file, the default key segment count is 1 to ensure the compatibility of the computation logic.

[0025] By using discrete physical characteristics, abstract file attributes are decoupled into key dimensions that determine computational complexity, providing a data foundation for subsequent load assessment from a computing power perspective.

[0026] S2: Based on preset hardware baseline parameters and environmental correction factors, the physical characteristics of the task to be processed are mapped to the estimated CPU time. In the virtual scheduler, the task allocation process of the background thread pool is simulated based on the estimated CPU time to determine the latest completion time and the thread load imbalance ratio.

[0027] Since physical characteristics can only reflect the static attributes of data and cannot directly reflect the computing power consumption during CPU processing, and different hardware has different encryption and verification performance, and there are interference factors such as I / O jitter and CPU frequency fluctuation in the actual operating environment, it is necessary to combine preset hardware benchmark parameters when analyzing the hidden computing power of the CPU. Hardware benchmark parameters characterize the basic performance indicators of hardware in encryption, key switching and data authentication operations, and provide a unified reference standard for computing power optimization. Specifically, they include benchmark decryption time, key reload time and authentication time.

[0028] The baseline decryption time represents the average time it takes for the CPU to complete AES-GCM decryption and integrity verification of a standard data block, reflecting the baseline computational cost of basic encryption processing. The key reload time represents the average time it takes for the CPU to perform a key context reset and extended key loading once, reflecting the additional computational cost of key switching. The authentication time represents the average time it takes for the CPU to perform only GCM integrity verification on an invalid data block, reflecting the computational cost of invalid data processing.

[0029] In this embodiment of the invention, hardware benchmark parameters can be obtained through offline benchmark testing or trial sampling during the system cold start phase. For example, during the system initialization phase, a set of standard AES decryption, key expansion, and GCM verification instructions are run, and their average execution time is used as the initial benchmark value. It should be noted that the specific values ​​of the benchmark parameters depend on the CPU model and instruction set (such as AES-NI) support, and are not limited here.

[0030] Therefore, by combining hardware baseline parameters with physical characteristics, in this embodiment of the invention, the computational cost of each task in the set of tasks to be processed is reconstructed through linear superposition. The product of the number of data blocks and the baseline decryption time is used as the basic decryption overhead, which represents the linear cost of performing standard decryption operations on the main data of a file under full CPU pipeline load, corresponding to the basic computing power requirement for core data processing.

[0031] The product of the number of key segments exceeding the base value and the key reload time is then used as the context switching overhead. Specifically, since the loading of the first key is already included in the initialization process, there is no need to calculate the reload cost separately. The switching overhead is only calculated for key segments exceeding one. That is, the difference between the number of key segments minus one is multiplied by the key reload time to obtain the context switching overhead, which represents the non-linear overhead caused by key rotation. The more key segments there are, the greater this overhead becomes.

[0032] The product of the number of deleted entries and the authentication time is then used as the invalid authentication overhead, representing the cost of the CPU's forced verification of logically deleted data. This overhead is often ignored in traditional volume-based flow control, but it may become a dominant factor in high deletion rate scenarios.

[0033] Finally, an environmental correction factor is introduced. This factor reflects the dynamic interference level of the current operating environment and is used to absorb the impact of non-computational factors such as I / O wait times and system scheduling delays on the time estimation, making the theoretical estimation more consistent with actual engineering scenarios. Its value is dynamically learned through the ratio of actual processing time to theoretical processing time in historical tasks. In this embodiment, when a background task completes, the actual processing time and the theoretically estimated processing time without incorporating the environmental correction factor are obtained. Based on the ratio of actual processing time to theoretically estimated processing time as an environmental deviation reference, a smooth iterative algorithm is used to periodically update the environmental correction factor. The initial value of the environmental correction factor can be set to 1.

[0034] In this embodiment of the invention, the environmental correction factor is updated using an exponentially weighted moving average algorithm. The periodic update is triggered once after each background merging task is completed, ensuring that the factor reflects the current system's I / O jitter and scheduling overhead in real time. Simultaneously, to prevent the factor value from diverging, the updated environmental correction factor is limited to a preset upper and lower bound range, such as 0.5 to 3.0. The implementer can adjust this range as needed; no restrictions are imposed here.

[0035] Therefore, the sum of the basic decryption overhead, context switching overhead, and invalid authentication overhead is calculated to obtain the theoretical estimated CPU time without environmental interference. The sum is then multiplied by the environmental correction factor to obtain the estimated CPU time, which reflects the actual CPU time required by the task under the current hardware and environment.

[0036] After mapping is complete, tasks can be managed from a computing power perspective. By constructing a virtual time and task space in memory—a virtual scheduler—future load distribution can be simulated without interfering with the operation of physical threads. By synchronizing the real-time running status of the physical thread pool, such as reading the task scoreboard, the starting point of the virtual timeline is established, allowing the simulation to be based on the actual backlog. Subsequently, a greedy algorithm is used to simulate the future task allocation process. In this embodiment, the Longest Processing Time First (LPT) greedy algorithm is used. This algorithm prioritizes processing time-consuming tasks, thereby simulating the long-tail blocking effect in the worst case, ensuring the conservatism and safety of flow control decisions.

[0037] In this embodiment of the invention, the current cumulative load time of each background thread on the virtual timeline is initialized, that is, the sum of the remaining estimated CPU time of the tasks already undertaken by each thread, as the initial load basis for simulation allocation. Since tasks with high computational complexity are more likely to cause long-tail blocking of single threads, the tasks in the set of tasks to be processed are sorted in descending order of estimated CPU time, giving priority to tasks with high computing power requirements, and ensuring that the simulation process can capture key blocking points.

[0038] Subsequently, the allocation operation is performed iteratively: the unallocated task with the longest estimated CPU time is selected and assigned to the background thread with the shortest current cumulative load time, i.e., the currently idle thread. This ensures that high-time-consuming tasks are prioritized for allocation to idle resources, avoiding excessive backlog in a single thread, and ensuring that tasks always flow to the lightest-loaded computing unit, simulating an ideal, efficient scheduling strategy. After allocation, the estimated CPU time of the task is added to the cumulative load time of the background thread, updating its completion time, until all tasks are allocated. This achieves a virtual allocation of all pending tasks, representing a panoramic preview of the background thread pool's working state over a future period.

[0039] Finally, the maximum value of the cumulative load time of all background threads after the simulation allocation is completed is determined as the latest completion time. This time is the theoretical latest moment when all threads complete all allocated tasks. It represents the theoretical computational limit time required for the system to process all current backlog tasks, including running and queued tasks. It reflects the total throughput capacity of the background system and can be used to judge whether the foreground write is too fast.

[0040] Simultaneously, the dispersion of load among different threads is quantified to identify structural problems. In this embodiment of the invention, after the simulated allocation is completed, the average cumulative load time of all background threads is obtained. The ratio between the latest completion time and the average cumulative load time is used as the thread load imbalance ratio, which directly reflects the degree of thread load balance. Under ideal load balancing conditions, this ratio is close to 1, indicating that the load of each thread is uniform and there is no obvious risk of blocking. If the ratio is significantly greater than 1, it indicates that there is structural blocking due to excessive load on a single thread, reflecting a performance bottleneck in the system caused by uneven task allocation.

[0041] This completes the quantitative mapping from physical characteristics to CPU time consumption, as well as the forward-looking simulation of background thread load status, providing core indicator support for subsequent flow control decisions.

[0042] S3: Determine the time safety margin of the memory buffer based on the latest completion time and the current write rate; determine the suggested delay coefficient of the current write request based on the time safety margin and the thread load imbalance ratio; wherein, when the thread load imbalance ratio exceeds the preset balancing threshold, a suggested delay coefficient is generated based on a non-linear penalty strategy.

[0043] Considering the supply and demand balance between foreground writes and background processing, the matching degree between the foreground write rate and the background processing capacity directly determines whether the memory buffer will overflow. The system monitors the current write rate of data written to the memory buffer by the foreground application in real time, reflecting the instantaneous pressure of data generated by the foreground business. At the same time, based on the remaining available space of the memory buffer and the write rate, the risk of buffer saturation is predicted, and the time safety margin is analyzed.

[0044] In this embodiment of the invention, the estimated filling time required to fill the remaining space of the memory buffer is analyzed based on the current write rate. Specifically, it is calculated by dividing the remaining available space of the memory buffer by the current write rate. A very small non-zero constant is introduced during the calculation to avoid division-by-zero anomalies caused by a write rate of zero, such as 1KB / s, to ensure computational robustness. This estimated filling time reflects the theoretical time it takes for the buffer to be exhausted due to foreground writes. Based on the deviation between the estimated filling time and the latest completion time, a time safety margin is obtained. In one specific embodiment of the invention, the difference between the estimated filling time and the latest completion time is used as the time safety margin, reflecting the safe buffer time of the system under the current load.

[0045] Considering the system's adaptive adjustment capability, when the time safety margin is not less than zero, it indicates that the current write rate is within a safe range, the background has sufficient capacity to process the data in a timely manner, and the system does not impose any delay; that is, the recommended latency coefficient is set to zero. Conversely, if the time safety margin is less than zero, it is determined that there is an overflow risk, and flow control decisions must be made to reduce the foreground write rate through a backpressure mechanism to avoid system crash.

[0046] Based on the system's adaptive adjustment capabilities, when the time safety margin is not less than zero, it indicates that the current write rate is within a safe range, the background has sufficient capacity to process the data in a timely manner, and the system does not impose any delay, i.e., the suggested latency coefficient is set to zero. Conversely, if the time safety margin is less than zero, it is determined that there is an overflow risk, and flow control decisions must be made to reduce the foreground write rate through a backpressure mechanism to avoid system crash.

[0047] When flow control is required, different control strategies are needed to address the different causes of load imbalance. The bottleneck type is determined based on the thread load imbalance ratio, and a suggested latency coefficient is obtained, which represents the recommended sleep time per unit of data volume. Dynamic balancing of background load can be achieved by adjusting the write rhythm.

[0048] In this embodiment of the invention, under the condition of overflow risk, when the load imbalance ratio does not exceed the preset balancing threshold, it indicates that the system only has a total computing power gap and the load distribution is relatively even. A linear strategy is adopted to generate a suggested latency coefficient that is linearly proportional to the absolute value of the time safety margin, thereby achieving gentle capacity matching and avoiding the impact of drastic rate limiting on services. As an example, the expression for the suggested latency coefficient under the linear strategy is: In the formula, This represents the suggested delay coefficient under a linear strategy. Represented as time safety margin, This indicates the latest completion time. Represented as linear gain, This is expressed as the reference delay constant per unit data. This reflects the severity of the spillover risk. The linear gain and the reference delay constant represent the sensitivity and reference step size of the adjustment. Their values ​​can be calibrated according to the number of CPU cores and I / O capabilities of the system. For example, the reference delay constant can be 50-100 microseconds, and the linear gain can be 1, etc. There are no restrictions here.

[0049] When the thread load imbalance ratio exceeds a preset balancing threshold, it indicates a structural blockage in the system. This means some threads are becoming performance bottlenecks due to excessive backlog of computationally complex tasks. In this case, strong intervention is needed to quickly suppress the generation of abnormal tasks. A suggested delay coefficient is generated based on a non-linear penalty strategy to quickly suppress the generation of high-entropy data and prevent single-threaded bottlenecks from dragging down the entire system. The penalty increment is analyzed based on the degree to which the thread load imbalance ratio exceeds the preset balancing threshold, where the preset balancing threshold is set to 1.2 to distinguish between normal fluctuations and structural imbalances. The suggested delay coefficient is obtained by superimposing the base delay coefficient and the penalty increment. As an example, the expression for the suggested delay coefficient under the non-linear penalty strategy is: In the formula, This is expressed as the suggested delay coefficient under a nonlinear penalty policy. This is expressed as the thread load imbalance ratio. It reflects the degree to which the load imbalance deviates from the equilibrium state, and amplifies the impact of the deviation through a square relationship to achieve a nonlinear penalty effect. This represents the quadratic penalty delay increment, reflecting the strong inhibitory effect on structural blockage; the more severe the imbalance, the more significant the penalty.

[0050] With a dual-mode strategy, the system exhibits mild linear rate limiting under normal load, while when a key fragmentation attack or abnormal load is detected, it can quickly "stop" foreground writes with high latency to protect the backend storage engine from being damaged.

[0051] S4: Perform a hibernation operation based on the recommended latency factor before the write request acquires the storage engine's mutex lock resource.

[0052] To avoid the risk of deadlocks or cascading failures caused by a write thread holding a storage engine mutex lock and then performing a sleep operation in a high-concurrency scenario, which could lead to long-term lock resource occupation and block other threads, the sleep operation is moved to before the mutex lock acquisition. By adjusting the flow through the delay in the lock-free state, the system stability problems caused by holding a lock and then sleeping can be avoided.

[0053] In this embodiment of the invention, the data batch size of the write request is obtained to measure the resource consumption of the request. Then, the target sleep duration is obtained by combining the suggested delay coefficient and the data batch size. Specifically, it is calculated by multiplying the suggested delay coefficient by the data batch size. The target sleep duration reflects the total time that the current write request needs to be delayed, ensuring that the delay level is accurately matched with the computational complexity of the data and the system load status.

[0054] Finally, before a write request acquires the storage engine's mutex lock, a lock-free sleep operation is performed for the target sleep duration. This operation achieves traffic shaping through thread sleep, neither consuming lock resources nor interfering with the normal execution of other write tasks, thus achieving flexible control over the foreground write rate. By employing the "sleep first, then hold lock" logic, the risk of system deadlock is avoided, and the write rhythm can be dynamically adjusted based on the background load status, ensuring the continuous availability and data storage security of the high-frequency trading system in encrypted compliance scenarios.

[0055] In summary, this invention first extracts physical characteristics such as the number of data blocks, the number of key segments, and the number of deleted entries, and combines these with hardware baseline parameters and environmental correction factors to achieve an accurate mapping from file size to estimated CPU time consumption. This allows for the identification and quantification of the actual computational burden on the system by "small but computationally heavy" encryption tasks. Secondly, by constructing a virtual scheduler in memory and simulating task allocation in a background thread pool, it proactively predicts the system's latest completion time and thread load imbalance ratio, providing data support for accurately assessing the existence of overall bottlenecks or structural blocking risks. Finally, based on the prediction results, a suggested latency coefficient is generated, and a lock-free sleep operation is performed before write requests acquire mutex lock resources. Furthermore, for structural load imbalances, a non-linear penalty strategy is used to quickly suppress the write rate of malformed data. This mechanism avoids the risk of system deadlock caused by directly blocking lock-holding threads while effectively smoothing foreground business traffic, ensuring the continuous availability and stability of the high-frequency trading system under extreme compliance loads. This invention analyzes the implicit overhead of CPU computing power, uses a virtual scheduler to simulate task allocation, predicts thread load status in advance, and implements flow control before write requests acquire storage engine mutex locks through lock-free delayed execution, thereby reducing the risk of deadlock and improving the stability and throughput of the system in compliant encryption scenarios.

[0056] The present invention also provides a data security storage system for big data transactions, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the data security storage method for big data transactions described above.

[0057] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0058] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for secure data storage in big data transactions, characterized in that, The method includes: During the data writing process, the system monitors for the freezing event of the memory buffer and obtains the physical characteristics of the set of tasks to be processed. The physical characteristics include: the number of data blocks, the number of key segments, and the number of deleted entries. When the used memory size of the active memory buffer reaches a preset threshold, the buffer will be marked as immutable by the system, indicating that a freezing event has been triggered. Based on preset hardware baseline parameters and environmental correction factors, the physical characteristics of the tasks to be processed are mapped to the estimated CPU time. In the virtual scheduler, the task allocation process of the background thread pool is simulated based on the estimated CPU time to determine the latest completion time and the thread load imbalance ratio. The method for obtaining the estimated CPU time is as follows: Obtain hardware baseline parameters, including baseline decryption time, key reload time, and authentication time; for each task in the set of tasks to be processed, multiply the number of data blocks by the baseline decryption time as the basic decryption overhead; multiply the number of key segments exceeding the baseline value by the key reload time as the context switching overhead; multiply the number of deleted entries by the authentication time as the invalid authentication overhead; calculate the sum of the basic decryption overhead, context switching overhead, and invalid authentication overhead, and multiply the sum by the environment correction factor to obtain the estimated CPU time; where the baseline decryption time represents the average time for the CPU to complete AES-GCM decryption and integrity verification of a standard data block; the key reload time represents the average time for the CPU to perform one key context reset and extended key loading; the authentication time represents the average time for the CPU to perform only GCM integrity verification on an invalid data block; the baseline value is 1. The method for obtaining the thread load imbalance ratio includes: after the simulation allocation is completed, obtaining the average cumulative load time of all background threads, and taking the ratio between the latest completion time and the average cumulative load time as the thread load imbalance ratio. Based on the latest completion time and the current write rate, a time safety margin for the memory buffer is determined. A suggested latency coefficient for the current write request is determined based on the time safety margin and the thread load imbalance ratio, including: if the time safety margin is not less than zero, the suggested latency coefficient is set to zero, and no flow control is applied; if the time safety margin is less than zero, an overflow risk is identified, and flow control decision is initiated: when an overflow risk exists and the thread load imbalance ratio does not exceed a preset balancing threshold, a linear strategy is adopted to generate a suggested latency coefficient that is linearly proportional to the absolute value of the time safety margin; when the thread load imbalance ratio exceeds the preset balancing threshold, a suggested latency coefficient is generated based on a non-linear penalty strategy. Before the write request acquires the storage engine's mutex lock resource, a sleep operation is performed based on the recommended latency factor.

2. The data security storage method in big data transactions according to claim 1, characterized in that, The methods for obtaining the latest completion time include: Initialize the current cumulative load time of each background thread on the virtual timeline; sort the tasks in the set of tasks to be processed in descending order according to the estimated CPU consumption time; Iteratively execute the allocation operation: Select the unallocated task with the largest estimated CPU time, assign it to the background thread with the smallest current cumulative load time, and add the estimated CPU time of the task to the cumulative load time of the background thread, until all tasks are allocated. The maximum value among the cumulative load times of all background threads after the simulation allocation is completed is determined as the latest completion time.

3. The data security storage method in big data transactions according to claim 1, characterized in that, The method for obtaining the time safety margin includes: The estimated filling time required to fill the remaining space of the memory buffer is analyzed based on the current write rate; the time safety margin is obtained based on the deviation between the estimated filling time and the latest completion time.

4. The data security storage method in big data transactions according to claim 1, characterized in that, The generation of suggested delay coefficients based on a nonlinear penalty strategy includes: Based on the time safety margin, the suggested delay coefficient obtained under the linear strategy is used as the base delay coefficient; the penalty increment is analyzed according to the degree to which the thread load imbalance ratio exceeds the preset balance threshold; the base delay coefficient and the penalty increment are superimposed to obtain the suggested delay coefficient.

5. The data security storage method in big data transactions according to claim 1, characterized in that, The step of performing a sleep operation according to the recommended latency factor includes: Obtain the batch size of the write request data; combine the suggested latency coefficient with the batch size of the data to obtain the target sleep duration; Before requesting a mutex lock on the storage engine for a write request, a lock-free sleep operation is performed for a duration equal to the target sleep duration.

6. The data security storage method in big data transactions according to claim 1, characterized in that, This also includes updates to environmental correction factors: When the background task is completed, the actual processing time and the theoretical estimated time without considering the environmental correction factor are obtained. Based on the ratio of the actual processing time to the theoretical estimated time, a smooth iterative algorithm is used to periodically update the environmental correction factor.

7. A data security storage system for big data transactions, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data security storage method in big data transactions as described in any one of claims 1 to 6.

Images