A memory resident data management method based on a big data architecture

Abstract

The application discloses a memory resident data management method based on a big data architecture, relates to the technical field of data management, and realizes dynamic management of cache data by establishing metadata for cache data blocks and maintaining a global cache list in the memory of a big data computing framework in combination with a timing cleaning and a forced cleaning mechanism based on memory occupancy; when the memory occupancy is lower than a preset safety threshold, a judgment index is further calculated and compared with a judgment threshold to judge whether a safety state is truly entered or an alarm is triggered. The method can dynamically adapt system load when task quantity suddenly increases or memory pressure fluctuates, avoids resource idling or memory overflow, and improves cache retention efficiency and reduces the risk of mistaken deletion through a multi-level cleaning strategy. Meanwhile, the intelligent alarm mechanism is introduced to guarantee the operation stability of the system in a high-pressure environment and improve the safety and intelligent level of overall memory management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, and particularly relates to a memory resident data management method based on a big data architecture. BACKGROUND

[0002] With the wide application of big data technology, distributed computing frameworks such as Apache Spark are widely used in real-time computing and batch processing tasks in the financial, communication, Internet and other industries. In order to improve computing efficiency, these frameworks usually use a caching mechanism to keep key data in memory for a long time to reduce frequent disk I / O access, thereby significantly improving task execution speed. The management of memory resident data is therefore an important link in system performance optimization.

[0003] However, in existing big data architectures, the cache management strategy is mostly based on fixed time cleaning or single threshold triggering. Such methods can maintain basic operation when the load is stable, but in actual application, when the task size suddenly increases or the memory pressure changes rapidly, the existing mechanism often appears rigid and extensive, leading to the following problems: first, it cannot dynamically adapt to load fluctuations, and is prone to resource idling or out-of-memory (OOM); second, the cleaning strategy is too single, which may cause data deletion or insufficient cleaning; third, there is a lack of effective emergency and alarm mechanism, when the memory pressure rises sharply, the system stability has a great risk. SUMMARY

[0004] The purpose of the present application is to solve the above-mentioned problems, and to provide a memory resident data management method based on a big data architecture.

[0005] The present application provides a memory resident data management method based on a big data architecture, which comprises:

[0006] In the memory of the big data computing framework, metadata is established for the cache data block and a global cache list is maintained;

[0007] Performing a timing cleaning operation on the global cache list to delete cache data blocks that exceed the preset survival time;

[0008] Monitoring the memory occupancy rate of the big data computing framework, when reaching the preset gradient threshold, performing a forced cleaning operation based on the global cache list, and re-detecting the memory occupancy rate after the cleaning is completed;

[0009] When the memory occupancy rate is lower than the preset safety threshold, the judgment index is calculated, and the judgment index is compared with the preset judgment threshold to determine whether to enter the safety zone or trigger the alarm.

[0010] The present application has the following advantages:

[0011] The application provides a memory resident data management method based on a big data architecture. Metadata is established for a cache data block and a global cache list is maintained in the memory of a big data computing framework. A timing cleaning operation is performed on the global cache list to delete cache data blocks exceeding a preset survival time. The memory occupancy rate of the big data computing framework is monitored. When the preset gradient threshold is reached, a forced cleaning operation is performed based on the global cache list, and the memory occupancy rate is re-detected after the cleaning is completed. When the memory occupancy rate is lower than a preset safety threshold, a judgment index is calculated, and the judgment index is compared with a preset judgment threshold to determine whether to enter a safety zone or trigger an alarm. In this way, in actual application, when the task size suddenly increases or the memory pressure rapidly changes, the load fluctuation can be dynamically adapted, resource idling or memory overflow can be reduced, the cleaning strategy is various, and data misdeletion or insufficient cleaning will not be caused. Meanwhile, an effective emergency and alarm mechanism is set, and when the memory pressure suddenly rises, the system stability has a small risk. BRIEF DESCRIPTION OF DRAWINGS

[0012] The application will be further described below in combination with the drawings.

[0013] Figure 1 A flowchart of the memory resident data management method based on the big data architecture. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the application will be clearly and completely described below in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0015] Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0016] The application provides a memory resident data management method based on a big data architecture. Referring to Figure 1 , Figure 1 A flowchart of the memory resident data management method based on the big data architecture. The method comprises the following steps:

[0017] Metadata is established for a cache data block and a global cache list is maintained in the memory of a big data computing framework.

[0018] A timing cleaning operation is performed on the global cache list to delete cache data blocks exceeding a preset survival time.

[0019] The memory occupancy of a big data computing framework is monitored, when reaching a preset gradient threshold, a forced cleaning operation is performed based on a global cache list, and after the cleaning is completed, the memory occupancy is re-detected;

[0020] When the memory occupancy is lower than a preset safety threshold, a judgment index is calculated, and the judgment index is compared with a preset judgment threshold to determine whether to enter a safety zone or trigger an alarm.

[0021] The memory resident data management method based on the big data architecture provided by the embodiment of the present application, through the above-mentioned manner, in actual application, when the task scale suddenly increases or the memory pressure rapidly changes, the load fluctuation can be dynamically adapted, the resource idling or memory overflow can be reduced, the cleaning strategy is various, and the data misdeletion or insufficient cleaning will not be caused; at the same time, an effective emergency and alarm mechanism is set, when the memory pressure suddenly rises, the system stability has a small risk.

[0022] In one embodiment, metadata is established for a cache data block in the memory of a big data computing framework, and a global cache list is maintained;

[0023] Specifically, metadata is established for each cache data block, the metadata includes a unique identifier, a creation time and an expiration time, and the cache data block is registered in the global cache list, so as to be uniformly managed;

[0024] In the memory of a big data computing framework, establishing metadata for a cached data block and maintaining a global cache list are basic steps for implementing dynamic management of memory-resident data. Specifically, when a task loads data into memory and performs a cache operation, the system creates a corresponding metadata structure for each cached data block. The metadata includes at least a unique identifier (blockId) of the cache block, a creation time (create_time), and an expiration time (expire_time). The expiration time is usually set based on a default survival period policy, for example, set to 12 hours. All created cache data blocks and their metadata are registered in the global cache list (GlobalCacheList), which is in the form of a data structure and resides in memory, used to centrally store, find, and operate all active cache block information. Taking the Apache Spark framework as an example, when multiple Executor nodes cache different data partitions, the system can maintain a local cache list for each Executor and synchronize it to the Driver node or central scheduling module to form a unified global cache list. This allows the management module to obtain the life cycle status and distribution information of all cache blocks in real time without relying on the underlying scheduling thread, thereby providing complete and reliable data support for subsequent timed cleaning, forced cleaning, and alarm mechanisms. In this way, not only is the metadata of the cached data managed throughout its life cycle, but a structural foundation is also laid for the system to dynamically assess cache load, accurately locate cleaning objects, and reasonably schedule memory resources.

[0025] In one embodiment, the step of performing a timed cleaning operation on the global cache list to delete cache data blocks that exceed a preset survival time includes:

[0026] Initializing the global cache list and setting a preset survival time for each cache data block;

[0027] Starting a timed cleaning task, which runs at a fixed time period; preferably, it is executed every hour;

[0028] In each execution of the timed cleaning task, the global cache list is traversed to identify cache data blocks whose creation time has exceeded the survival time;

[0029] The identified expired cache data blocks are removed from the global cache list, and the corresponding memory resources are released;

[0030] After completing the deletion operation, the structure state of the global cache list is updated.

[0031] It should be noted that initializing the global cache list and setting the preset time to live for each cache data block is a preparation step when the memory resident data management mechanism is started. The system creates a corresponding cache data block object when the cache data is first written into the memory, and allocates unique identifiers, creation time, and expiration time, etc. metadata information to it. The time to live can be uniformly set based on the default TTL policy, for example, set to 12 hours. After initialization, the system starts a timing cleaning task, which is run as a background daemon thread at a fixed period, preferably once every hour, to ensure that the cache data is periodically maintained without interfering with the execution of the main task. During the execution of the timing cleaning task, the system iterates through the currently maintained global cache list, compares the creation time of each cache data block with the current system time, and identifies cache data blocks that have exceeded the preset time to live. For example, if the creation time of a cache block is 9:00 am and the current time is 10:00 pm, the cache block is determined to be expired data under the premise that the TTL is 12 hours. Subsequently, the system removes the identified expired cache data block from the global cache list and synchronously releases the space occupied by it in the Executor memory. After the deletion operation is completed, to ensure the structural consistency of the global cache list and the efficient execution of subsequent operations, the system also updates the global cache list, such as re-adjusting the internal data index, clearing the invalid references, and rebuilding the hash mapping relationship, to maintain the stability of the data structure and access performance, thereby providing a complete and accurate cache information basis for the next period of cleaning and subsequent forced cleaning operations.

[0032] In one embodiment, the memory occupancy of the big data computing framework is monitored, and when a preset threshold is reached, a forced cleaning operation is performed based on the global cache list, and after the cleaning is completed, the memory occupancy is re-detected. The steps include:

[0033] Periodically obtain the memory usage of each execution node (Executor) in the big data computing framework, and aggregate the overall memory occupancy;

[0034] Compare the memory occupancy with a preset memory occupancy threshold, and when the memory occupancy is greater than or equal to the first threshold, start the forced cleaning process;

[0035] When performing forced cleaning, based on the global cache list, cache data blocks with a creation time earlier than the current time by more than six hours are preferentially identified and removed from the global cache list;

[0036] If the memory occupancy is still higher than the second threshold after cleaning is completed, cache data blocks with a creation time earlier than four hours and one hour are continuously identified and deleted to release memory resources in turn;

[0037] After each batch of cache data blocks is cleaned up, the current memory occupancy is recalculated, and it is determined whether it has fallen below the second threshold value;

[0038] After each round of cleaning operation, the state of the global cache list is updated to reflect the current distribution of cached data.

[0039] It should be noted that the memory occupancy of the big data computing framework is monitored, and when the preset threshold is reached, a forced cleaning operation is performed based on the global cache list, and the memory occupancy is re-detected after cleaning is completed, which is a key link to ensure that the system can still maintain stable operation under high load. Specifically, the system periodically obtains the real-time memory usage data of each execution node (Executor) and aggregates it through the scheduler or monitoring module to calculate the overall memory occupancy of the entire big data computing framework; when the detection result shows that the memory occupancy is greater than or equal to the preset first threshold (such as 60%), the forced cleaning process is immediately started. During the cleaning process, the system first identifies cache data blocks created more than six hours ago based on the global cache list, and removes them from the cache list, while releasing the corresponding memory space. For example, if the current time is 9 pm, then any cache block created before 3 pm will be considered a "priority cleaning object". After the cleaning operation is completed, the system re-detects the memory occupancy at this time; if it is still higher than the second threshold (such as 40%), further cleaning operations are performed to delete cache data blocks created four hours and one hour ago, respectively, to form a step-by-step recycling strategy from old to new, ensuring that the memory release process is gradual and fault-tolerant. After each round of cleaning of cache data blocks, the system updates the state of the global cache list in real time, removes the cleaned cache records, and synchronously refreshes the index structure or time sorting logic, so that the subsequent cleaning steps do not rely on repeated traversal, thereby improving the efficiency and accuracy of the entire cleaning mechanism. Through the above mechanism, even in extreme situations such as sudden increase in tasks and intensive data writing, the memory can be prevented from being quickly filled, avoiding Out of Memory (OOM) errors, and providing a stable memory occupancy state input for determining whether to enter the safe zone or triggering the subsequent alarm mechanism.

[0040] In one embodiment, when the memory occupancy is below the preset safety threshold, a judgment index is calculated and compared with a preset judgment threshold to determine whether to enter the safe zone or trigger an alarm;

[0041] Specifically, the step of calculating the judgment index is:

[0042] When the memory occupancy rate is lower than the second threshold, a judgment index is calculated, the judgment index comprising a memory rebound rate index and a memory fragmentation degree index, the memory rebound rate index and the memory fragmentation degree index being normalized and weighted and summed to obtain the judgment index, the formula being: , wherein is the judgment index, and are the normalized memory rebound rate index and the normalized memory fragmentation degree index, respectively, respectively represent preset weight coefficients of the normalized memory rebound rate index and the normalized memory fragmentation degree index, and are both greater than 0;

[0043] It should be noted that the above normalization method removes the dimension method, including Min-Max normalization, Z-Score standardization, etc., which will not be described here; According to the actual situation, generally are equal and the sum is 1, for example, may be 0.5, 0.5.

[0044] In one embodiment, the calculation step of the memory rebound rate index comprises:

[0045] After the memory cleaning operation is completed, a total observation time is set, and memory occupancy rates are sampled multiple times at fixed time intervals to form an ordered memory occupancy rate sampling sequence, the first data point of the sequence being the initial memory occupancy rate when the cleaning is completed, and the remaining data points being subsequent memory occupancy rate values collected every fixed time;

[0046] The growth rate between adjacent sampling points is calculated in turn, specifically: the memory occupancy rate of the latter sampling point is subtracted from the memory occupancy rate of the former sampling point, and divided by the sampling interval time between the two points to obtain the local growth rate value in the corresponding time period, until all adjacent sampling points are calculated to form a set of local growth rate sequences;

[0047] The maximum value and the minimum value are extracted from the growth rate sequence, and the difference value divided by the maximum value is calculated to obtain a burst degree ratio, the burst degree ratio representing the fluctuation intensity of the local memory rebound speed, the greater the burst degree ratio, the more unstable the memory rebound;

[0048] The growth rate sequence is traversed, the first sampling point whose growth rate exceeds a preset growth threshold is identified, and the time interval between the corresponding sampling point and the initial sampling point is recorded, the corresponding interval time is divided by the total observation time to obtain a lag release coefficient, the smaller the lag release coefficient, the earlier the rebound, the greater the risk;

[0049] The average of all local growth rates is calculated as the memory growth average, and the larger the average, the faster the overall memory rebound speed;

[0050] The burst degree ratio, the lag release coefficient, and the memory growth average are normalized respectively to make them all between 0 and 1, and are combined to obtain the memory rebound rate index as follows:

[0051] The burst degree ratio is multiplied by the lag release coefficient, and then multiplied by the memory growth average, and the result is the memory rebound rate index, which is a non-dimensional quantity between 0 and 1, and the larger the value, the more intense and unstable the memory rebound after cleaning, and the system needs to trigger an alarm.

[0052] It should be noted that in the calculation process of the memory rebound rate index, all the data involved are derived from real-time running monitoring data of the system after the memory cleaning operation is completed, which can be collected and recorded through the memory monitoring module or scheduler interface deployed in the big data computing framework (such as Apache Spark). Specifically, after the system executes a forced memory cleaning process, the memory occupancy rate at the current time is recorded as the initial value, and a periodic monitoring program is started to continuously record the current memory usage of the system at a fixed sampling interval (such as every 5 seconds) within a set total observation time period (such as 60 seconds), forming a set of time-ordered occupancy rate data points. Each data point can be obtained by reading the memory monitoring interface exposed by the Executor node in the big data framework, such as accessing memory usage indicators or JVM heap occupancy rate through the Metrics system in Spark; the time interval required for calculating the growth rate is the fixed sampling interval value preset by the system, which can be set uniformly by the scheduling module and passed into the analysis module. The growth rate sequence is obtained by calculating the two memory occupancy rate data points, and then the maximum value, the minimum value, and the average value can be extracted from it; when identifying the "lag release" time point, the system compares the continuous growth rate value with the set threshold value in real time, the threshold value can be set by experience or dynamically adjusted based on historical data, and once the first slope value exceeding the threshold value appears, the corresponding time point can be recorded and the time interval can be obtained by subtracting the initial time; the total observation time is a system preset configuration parameter, which is usually provided as a global variable when initializing the analysis process. All original data do not depend on external input and do not need to introduce user intervention, and can be automatically collected and generated by the system itself during running, ensuring the objectivity and repeatability of the judgment result.

[0053] It should be noted that the memory rebound rate index is a kind of index for quantitatively judging whether there is a "rapid rebound" and "high instability" risk of system memory occupancy rate after performing memory cleaning operation, and the core purpose is to measure the persistence and effectiveness of the cleaning behavior on the memory release effect in a short time. The index continuously monitors the memory occupancy rate after cleaning, combines the burst degree ratio, the lag release coefficient and the average growth rate and other factors to comprehensively evaluate whether there is hidden short-term backfill pressure under the condition that the "surface memory has been released". When the index value is large, it means that although the system memory release is successful once, it rebounds rapidly in a very short time, and the elastic growth is uneven (the burst degree is large), the start is too early (the release lag coefficient is small), and the overall growth trend is strong (the growth rate is large), which means that the system currently has a large number of high-concurrency tasks, short-period data inflow, or a large number of high-activity caches are retained, resulting in subsequent memory pressure difficult to continue to alleviate through simple deletion. At this time, if the data deletion is still continued, it is very likely to touch the cache data block that is still actively used, thereby destroying the computing task that the system is currently performing, and even causing result errors or task interruption. Therefore, the larger the memory rebound rate index, the more likely the system is in a "false release, true impact" critical state, which cannot be fundamentally solved by data cleaning, but will trigger a new round of high-frequency cleaning and misdeletion risk cycle, so the automatic cleaning operation should be suspended, and manual intervention should be guided immediately through the alarm mechanism, and higher-level resource regulation means such as task rescheduling, node expansion, task speed limit, cache priority intervention should be taken to ensure the stability of system operation. For example, after memory cleaning, the memory occupancy rate decreases to 39%, but rebounds to 52% in 30 seconds, and the growth is mainly concentrated in the 10th to 20th second, which indicates that the current system pressure cannot be alleviated by cache cleaning, and manual intervention means must be used, otherwise OOM or cache invalidation exception may be triggered again in a few minutes. As can be seen, the memory rebound rate index is not to reflect "whether it is cleaned now", but to predict "whether it is really safe after cleaning now", and once the index is high, blind deletion will push the system into a more unstable state.

[0054] In one embodiment, the calculation steps of the memory fragmentation degree index are:

[0055] After completing the memory cleaning operation, the current memory space of the system is scanned, all memory blocks in the idle state are identified, and the starting address and corresponding memory block size of each idle block are obtained, which are arranged in order according to the address order of the idle block to construct an ordered idle memory block list;

[0056] In the list of free memory blocks, for each pair of adjacent free blocks, the distance difference between the end address of the previous free block and the start address of the next free block is calculated. The distance difference represents the memory jump distance between the two free blocks. The corresponding operations are executed in sequence to form a jump distance sequence, which is used to characterize the space jump situation of the current free space.

[0057] Traverse the jump distance sequence, extract the maximum jump distance value and the minimum jump distance value, calculate the difference between the two, and then divide the corresponding difference by the sum of all jump distances in the jump distance sequence to obtain the jump perturbation degree. The jump perturbation degree is used to measure the volatility of the spatial distribution of free memory blocks in the system. The larger the value, the more uneven the jump and the more fragmented the structure.

[0058] For each pair of memory blocks in the list of free memory blocks, count the number of times the first block is larger than the second block in their size relationship, and record the number of such "reversals". Then divide the corresponding number of reversals by the total number of all pairwise combinations to obtain the density reversal rate, which is used to measure the degree of abnormal arrangement of free memory block sizes in the sequential structure. The higher the reversal rate, the more unbalanced the size distribution.

[0059] Multiplying the jump perturbation degree by the density inversion rate yields the memory fragmentation index. The index ranges from 0 to 1. A higher value indicates that the memory structure is more discontinuous, irregular, and unsuitable for carrying large blocks of continuous data. The system has a high risk of fragmentation, requiring an alarm to be triggered and further automatic cleanup operations to be paused.

[0060] It should be noted that in the calculation process of the memory fragmentation degree index, the data involved in each calculation step is derived from the real-time memory space structure snapshot of the system after completing the memory cleaning operation, which is extracted through the memory management interface of the runtime environment in the big data computing framework or the memory allocator information at the operating system level. Specifically, after the cleaning operation is completed, the system scans the usage state of the current memory space through the scheduling module or the memory monitoring module, obtains all memory blocks in the "free" state, and extracts the starting address and corresponding size information of each free block. These data can be directly parsed from the heap address mapping table, page table, or object allocation interval recorded by the memory management module. In a big data computing framework such as Apache Spark, this information can be indirectly obtained through the JVM memory monitoring interface bound to the Executor node, or it can be sampled and calculated through the memory block allocation records provided by the built-in memory allocator (such as UnifiedMemoryManager or TaskMemoryManager). The calculation of the jump distance relies on the order of the starting addresses of the free blocks, which can be completed by traversing the extracted address list; the size of the free block is used to calculate the number of reversals of the block size sequence, which needs to record and compare the actual allocation size of each free block. This information can also be obtained from the released block pool or object heap release record of the memory manager. The collection process of all these data can be automatically completed by the system without relying on manual input, and it has repeatability and real-time nature, which can provide complete data support for the accurate evaluation of the memory fragmentation degree.

[0061] It should be noted that the memory fragmentation degree index is a kind of index for measuring the spatial continuity and structural integrity of the current idle memory of the system after the memory cleaning operation is performed, and its core significance lies in reflecting whether the distribution of the idle memory has been seriously fragmented, that is, whether the idle space has been cut into a large number of discontinuous small blocks, resulting in actual inefficient utilization. The index comprehensively evaluates the jumping interval fluctuation degree (i.e., whether there are a large number of irregular holes between the idle blocks) and the disorder degree of size arrangement (i.e., whether the large blocks and small blocks are mixed in disorder) by analyzing the address distribution law and size arrangement structure of all idle memory blocks after cleaning, so as to judge whether the current memory state can still support the normal loading of large tasks or large objects. When the index value is small, it means that the idle block distribution is relatively concentrated and the size is ordered, the memory structure is healthy, and the strategy-based data cleaning can still be continued; but when the index gradually increases, it means that the idle blocks are highly cut and lack of continuous contiguous areas, and a large number of small fragments cannot meet the subsequent allocation requirements, so even if more data is cleaned, it is difficult to release enough continuous memory space, which may lead to frequent memory rearrangement, fragment consolidation or task interruption, and even unpredictable memory allocation failure in severe cases. Therefore, when the index exceeds the set threshold, the automatic data cleaning should be stopped in priority, and manual intervention should be performed, such as task scheduling delay, dynamic re-partitioning, node re-allocation or idle block merging for structure optimization, rather than continuing to clean data. For example, a system releases 200MB of memory after cleaning, but the 200MB is distributed in 30 non-contiguous small blocks below 1MB. At this time, although the total idle amount seems to be available, any object exceeding 5MB cannot be successfully loaded, which finally leads to repeated task failures. In such a case, if the cleaning is continued, it cannot alleviate the problem, but may destroy the mergability of the existing memory distribution, causing waste of system resources and efficiency decline, so an alarm mechanism must be introduced to guide manual intervention to optimize the memory structure. As can be seen, the memory fragmentation degree index does not measure "how much space is released", but is used to accurately judge "whether the released space is really available". The larger the value is, the more the system needs to take immediate structure reorganization intervention measures.

[0062] In one embodiment, the step of comparing the judgment index with the preset judgment threshold to determine whether to enter the safe area or trigger the alarm is:

[0063] The judgment index is compared with the preset judgment threshold, and if the judgment index is not less than the preset judgment threshold, an alarm is directly triggered, and manual intervention is handled;

[0064] If the judgment index is less than the preset judgment threshold, it is determined that the current memory state enters the safe area and the cleaning process is ended.

[0065] It should be noted that in the step of comparing the judgment index with the preset judgment threshold, the system accurately evaluates whether the current memory state is truly safe according to the calculated judgment index value after cleaning: when the judgment index is greater than or equal to the set judgment threshold, it is indicated that although the memory occupancy rate has been reduced below the safe zone on the surface, through comprehensive analysis of the memory rebound trend, fragment structure and other multi-dimensional factors, it is found that there is still a potential instability risk, such as the memory may rebound rapidly in a short time, the released space is highly fragmented, etc. At this time, continuing to perform data cleaning cannot effectively reduce the risk, and even may exacerbate the misdeletion or task exception, so the system no longer blindly cleans, but immediately triggers the alarm mechanism to notify the operation and maintenance personnel or the upper layer scheduling system for manual intervention, such as delaying task scheduling, reconstructing cache strategy or adjusting resource allocation, etc. On the contrary, if the judgment index is less than the judgment threshold, it is indicated that the overall state after memory cleaning is stable, and there is no obvious rebound sign or structure abnormality, the system can determine that the current state has entered the safe zone, terminate the cleaning process, and release resources. Through the above mechanism setting, the system not only avoids the extensive strategy of taking occupancy rate as the only cleaning judgment standard, but also realizes more fine and dynamic identification of the risk state after cleaning, avoids unnecessary resource consumption or wrong operation, and improves the intelligence and stability of the overall memory management. Such design not only can guarantee the system running safety in the environment of high concurrency and high dynamic load, but also can maximize the cache effectiveness and reduce the performance rollback caused by misdeletion behavior, which is an important embodiment of the evolution of memory regulation strategy from static index to comprehensive behavior model.

[0066] The above detailed one embodiment of the present application, but the content is only the preferred embodiment of the present application, can not be considered for limiting the scope of the present application. Any equivalent changes and improvements made in the scope of the present application should still belong to the scope of the present application.

Claims

1. A method for managing memory-resident data based on a big data architecture, characterized in that, Includes the following steps: In the memory of the big data computing framework, metadata is created for cached data blocks and a global cache list is maintained; Perform a scheduled cleanup operation on the global cache list to delete cache data blocks that have exceeded the preset lifespan; Monitor the memory usage of the big data computing framework. When the preset gradient threshold is reached, perform a forced cleanup operation based on the global cache list, and re-detect the memory usage after the cleanup is completed. When the memory usage rate is lower than the preset safety threshold, a judgment index is calculated and compared with the preset judgment threshold to determine whether to enter the safe zone or trigger an alarm. The steps for calculating the judgment index are as follows: When the memory utilization rate is lower than the second threshold, the judgment index is calculated. The judgment index includes the memory bounce rate index and the memory fragmentation index. The memory bounce rate index and the memory fragmentation index are normalized, and the normalized memory bounce rate index and the memory fragmentation index are weighted and summed to obtain the judgment index. The steps for calculating the memory bounce rate index include: After the memory cleanup operation is completed, a total observation period is set, and memory usage is sampled multiple times at fixed time intervals to form an ordered memory usage sampling sequence. The first data point in the sequence is the initial memory usage when the cleanup is completed, and the remaining data points are the subsequent memory usage values ​​collected at fixed time intervals. The growth rate between two adjacent sampling points is calculated sequentially. Specifically, the memory occupancy rate of the next sampling point is subtracted from the memory occupancy rate of the previous sampling point, and then divided by the sampling interval between the two points to obtain the local growth rate value within the corresponding time period. This process continues until all adjacent sampling points have been calculated, forming a set of local growth rate sequences. Extract the maximum and minimum values ​​from the growth rate sequence, and calculate the difference between them divided by the maximum value to obtain the suddenness ratio. Traverse the growth rate sequence, identify the first sampling point where the growth rate exceeds the preset growth threshold, and record the time interval between the corresponding sampling point and the initial sampling point. Divide the corresponding interval time by the total observation time to obtain the hysteresis release coefficient. Calculate the average of all local growth rates as the average memory growth rate; The burst ratio, the hysteresis release coefficient, and the average memory growth rate are normalized respectively. The normalized burst ratio is multiplied by the hysteresis release coefficient and then by the average memory growth rate. The result is the memory bounce rate index. The steps for calculating the memory fragmentation index are as follows: After completing the memory cleanup operation, the system's current memory space is scanned to identify all free memory blocks, and the starting address and corresponding memory block size of each free block are obtained. The free blocks are then arranged in order of their addresses to build an ordered list of free memory blocks. In the list of free memory blocks, for each pair of adjacent free blocks, the distance difference between the end address of the previous free block and the start address of the next free block is calculated. The distance difference represents the memory jump distance between the two free blocks. The corresponding operations are executed in sequence to form a jump distance sequence. Traverse the jump distance sequence, extract the maximum jump distance value and the minimum jump distance value, calculate the difference between the two, and then divide the corresponding difference by the sum of all jump distances in the jump distance sequence to obtain the jump perturbation degree. For each pair of memory blocks in the list of free memory blocks, count the number of times the first block is larger than the second block in their size relationship, and record the number of such "reversals". Then divide the corresponding number of reversals by the total number of all pairwise combinations to obtain the density reversal rate. Multiplying the jump perturbation degree by the density inversion rate yields the memory fragmentation index.

2. The memory-resident data management method based on a big data architecture according to claim 1, characterized in that, Metadata is created for each cached data block, including a unique identifier, creation time, and expiration time. The cached data blocks are then registered in a global cache list for unified management.

3. The memory-resident data management method based on a big data architecture according to claim 1, characterized in that, The steps for performing a scheduled cleanup of the global cache list to delete cached data blocks that have exceeded a preset lifespan include: Initialize the global cache list and set a preset time-to-live for each cached data block; Start a scheduled cleanup task, which runs at a fixed time interval; it executes once every hour. During each scheduled cleanup task, the global cache list is traversed to identify cached data blocks whose creation time has exceeded their lifespan. Remove the identified expired cached data blocks from the global cache list and release the corresponding memory resources; After the deletion operation is completed, update the structure state of the global cache list.

4. The memory-resident data management method based on a big data architecture according to claim 1, characterized in that, The steps for monitoring the memory usage of a big data computing framework, performing a forced cleanup operation based on the global cache list when a preset threshold is reached, and then re-monitoring the memory usage after the cleanup is complete include: Periodically obtain the memory usage of each execution node in the big data computing framework, and summarize and calculate the overall memory occupancy rate; The memory usage rate is compared with a preset memory usage rate threshold. When the memory usage rate is greater than or equal to the first threshold, a forced cleanup process is initiated. When performing forced cleanup, based on the global cache list, cached data blocks whose creation time is more than six hours earlier than the current time are identified first and removed from the global cache list; If the memory usage rate is still higher than the second threshold after the cleanup is completed, continue to identify and delete cached data blocks created earlier than four hours and one hour, and release memory resources in turn; After each batch of cached data blocks is cleared, the current memory usage is recalculated, and it is determined whether it has fallen below the second threshold. After each round of cleanup operations, the state of the global cache list is updated to reflect the current distribution of cached data.

5. The memory-resident data management method based on a big data architecture according to claim 1, characterized in that, The steps for comparing the judgment index with a preset judgment threshold to determine whether to enter the safe zone or trigger an alarm are as follows: The judgment index is compared with the preset judgment threshold. If the judgment index is not less than the preset judgment threshold, an alarm is triggered directly, and manual intervention is required. If the judgment index is less than the preset judgment threshold, the current memory state is determined to enter the safe zone and the cleanup process ends.

Images