Cache management system based on data monitoring and analysis

Abstract

The invention relates to the technical field of cache management, in particular to a cache management system based on data monitoring and analysis. The system comprises a data monitoring module, a cache strategy configuration module, a cache execution module, a primary correction module, a performance monitoring module and a secondary correction module. According to the method, the corresponding initial strategy identifier is determined for each task instance, the initial priority and the initial cache sequence of the single task instance are determined according to the single historical loading duration of the single task instance in combination with the corresponding initial strategy identifier, and the initial cache set is generated; and performing primary correction on each initial cache sequence and the initial cache set according to the real-time behavior data and the real-time change data, and collecting a performance monitoring result in a cache execution process so as to update the initial strategy identifier to realize secondary correction on each initial cache sequence. The cache content can be dynamically adjusted, the waste of cache resources is avoided, and the load of the database is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of cache management, and in particular to a cache management system based on data monitoring and analysis. Background Art

[0002] With the continuous development of technology, the requirements for service performance are getting higher and higher; and in the case of high concurrency, it will also bring huge pressure to the database. In this case, caching is an indispensable solution to these problems.

[0003] Chinese patent application publication number CN103493029A discloses a general cache management system, which allocates caches to data processing instances in a volatile memory of a computing device, wherein each of the caches is exclusively allocated to a single one of the data processing instances. A public cache is allocated in a volatile memory of a computing device, wherein the public cache is shared by the data processing instances. Each instance of the data processing instance is configured to: identify a data block in a specific cache allocated to the instance, wherein the data block has not changed since the data block was last persistently written to one or more storage devices; cause the data block to be stored in the public cache; and eliminate the data block from the specific cache.

[0004] This shows that the current cache management technology cannot customize the cache strategy, which will bring huge pressure to the database. Summary of the invention

[0005] To this end, the purpose of the present invention is to provide a cache management system based on data monitoring and analysis, which is used to overcome the problem that the current cache management technology cannot customize the cache strategy, which brings huge pressure to the database.

[0006] To achieve the above object, the present invention provides a cache management system based on data monitoring and analysis, comprising: A data monitoring module, which is used to obtain historical monitoring data of each task instance and obtain identification key information based on the historical monitoring data; A cache strategy configuration module, which determines a corresponding initial strategy identifier for each task instance based on the identification key information, stores each identified task instance in a to-be-cached task queue, and determines an initial priority of a single task instance based on a single historical loading duration of the single task instance in combination with the corresponding initial strategy identifier; A cache execution module, which is used to determine the initial cache order of each task instance and generate an initial cache set according to the remaining resource memory of the database when caching the task instance and the initial priority; A primary correction module, which is used to collect the real-time behavior data of the user and the real-time change data of each task instance during the cache execution process, generate the real-time key information of the user according to the real-time behavior data, compare the real-time key information of the user with the identification key information, obtain the comparison result, and perform a primary correction on each initial cache sequence and the initial cache set according to the comparison result combined with the real-time change data; A performance monitoring module, which is used to collect performance data during cache execution and analyze the performance data to obtain performance monitoring results; A secondary correction module is used to update the initial strategy identifier according to the performance monitoring result to achieve secondary correction of each initial cache sequence.

[0007] Further, the data monitoring module includes: a data collection unit, a data identification unit and an extraction unit; For any task instance, the data collection unit is used to integrate the historical data of different data sources of the task instance to obtain the historical monitoring data; The data identification unit is used to identify key indicators according to the historical monitoring data, and analyze the identification results in different modes to obtain a number of data analysis results; The extraction unit is used to extract features from each data analysis result to obtain the key identification information.

[0008] Further, the data identification unit includes: a key indicator identification subunit, a first analysis subunit and a second analysis subunit; The key indicator identification subunit is used to identify the key indicator to obtain the historical change trend of the key indicator; The first analysis subunit is used to analyze the historical change trend in a numerical sequence mode to obtain a first data analysis result; The second analysis subunit is used to analyze the historical change trend in a time series mode to obtain a second data analysis result.

[0009] Further, the extraction unit performs numerical feature extraction on the first data analysis result to obtain historical key statistics and historical trend turning points; The extraction unit extracts time features from the second data analysis result to obtain historical fluctuations and historical turning points; Among them, the key identification information includes: numerical identification key information and time identification key information; the numerical identification key information includes: the historical key statistics and the historical trend turning points; the time identification key information includes: the historical fluctuations and the historical turning periods.

[0010] Further, the cache strategy configuration module includes: a first judgment unit, a second judgment unit and an overall determination unit; The first judgment unit is used to determine the actual loading status of the single task instance according to the comparison result between the single historical loading duration and the preset duration; The second determination unit is used to determine the historical identification status of the single task instance according to the initial strategy identifier; The overall determination unit is used to determine the initial priority of the single task instance as the first initial priority, or the second initial priority, or the third initial priority according to the actual loading situation and the historical identification situation; The priority of the first initial priority task instance is higher than the second initial priority task instance, and the priority of the second initial priority task instance is higher than the third initial priority task instance.

[0011] Further, the cache execution module includes: an acquisition unit, an overall comparison unit and a partial comparison unit; The acquisition unit is used to acquire the memory required for the entire to-be-executed program and the memory required for the single-class to-be-executed program corresponding to different initial priorities; The overall comparison unit is used to determine whether to start the hierarchical comparison mode according to the comparison result of the overall required memory to be executed and the remaining resource memory, or to output a first execution signal; The partial comparison unit determines to output a second execution signal, or a third execution signal, or a fourth execution signal based on the hierarchical comparison mode and the comparison result between each single type of memory required for execution and the remaining resource memory.

[0012] Further, the cache execution module further includes: an execution determination unit; The execution determination unit is used to receive each execution signal and determine the actual generation method of the initial cache sequence according to different types of the execution signal.

[0013] Further, the partial comparison unit includes: a first comparison subunit and a second comparison subunit; The first comparison subunit is used to obtain the first single type of memory required for execution corresponding to each first initial priority, and determine to output the second execution signal according to the comparison result of the first single type of memory required for execution and the remaining resource memory, or to start the sum comparison mode; The second comparison sub-unit obtains the second single type of memory required for execution and part of the memory required for execution of the first single type of memory required for execution corresponding to each second initial priority level based on the sum comparison mode, and determines to output the third execution signal, or the fourth execution signal, according to the comparison result of the part of the memory required for execution and the remaining resource memory.

[0014] Further, the primary correction module includes: a comparison unit, a primary update unit and a primary cleaning unit; The comparison unit is used to compare the real-time key information with the identification key information to determine the actual state change, and to compare the real-time change data with the real-time change data according to the actual state change. Determine whether to send an update signal or a cleanup signal to the task instance; The primary updating unit updates each initial cache sequence of the task instance based on the primary updating signal; The primary cleaning unit adds or removes task instances to the initial cache set accordingly based on the cleaning signal to obtain an actual cache set.

[0015] Further, the secondary correction module includes: a result analysis unit and a secondary updating unit; The result analysis unit is used to obtain the cache hit rate, the single actual loading time and the actual response time of each task instance according to the performance monitoring result, determine the number of actual task instances that need to perform initial policy identifier update according to the cache hit rate, the single actual loading time and the actual response time, and determine whether to send a secondary update signal according to the actual number of task instances; The secondary updating unit updates the initial strategy identifier of each task instance based on the secondary updating signal, and rearranges the order of the task instances in the cache queue according to the updated actual strategy identifier.

[0016] Compared with the prior art, the beneficial effect of the present invention lies in that, by analyzing the historical monitoring data and real-time behavior data of the product details page, it can more accurately predict and respond to the user's browsing needs, thereby providing a fast and smooth page loading experience, and correcting the cache strategy in real time to ensure that users can get a good experience even during peak access periods. The determination of the initial strategy identifier and the initial priority enables cache resources to be reasonably allocated according to the importance and access frequency of the page. Through the primary correction module and the secondary correction module, the cache content can be dynamically adjusted to avoid wasting cache resources. By caching the product details data in advance, the number of database queries and loads during peak hours is reduced, and the database load is reduced. Real-time monitoring and correction of cache strategies help prevent database overload and maintain system stability. Performance data during the execution process can be collected. By analyzing these data, the cache strategy can be adjusted according to the performance monitoring results to further improve the overall performance, which can effectively improve the operating efficiency of the e-commerce platform.

[0017] By integrating historical data from different data sources, the comprehensiveness and accuracy of monitoring data are ensured. The identification and analysis of key indicators can reveal the access patterns and trends of product detail pages, provide data support for caching strategies, and formulate more reasonable caching strategies based on the changing trends of historical visit volumes. By identifying historical trend turning points and historical fluctuations, it is possible to predict and cache in advance product detail pages that may be affected by seasonal fluctuations or promotional activities, thereby improving user access speed and satisfaction. The extraction of numerical features and time features helps to identify which pages require higher priority caching and which pages can reduce cache priority or be removed from the cache, thereby reducing unnecessary resource consumption. The caching strategy can be adjusted according to historical turning points and fluctuations, allowing e-commerce platforms to flexibly respond to market changes and changes in user needs.

[0018] It can perform refined cache management based on the page loading speed and product type to ensure the most effective use of cache resources. It can assign different initial priorities based on the actual loading situation of the page to ensure that users can get faster loading speeds when visiting product details pages of popular products or high-traffic periods, thereby improving user experience. By giving higher cache priority to slowly loading pages, the server load during peak hours can be reduced. It can identify products of interest during high-traffic periods, so that the cache strategy can be adjusted within a specific time period to cope with traffic fluctuations. Comprehensive consideration of loading conditions and product types helps to reasonably allocate cache resources, avoid excessive caching of unpopular product pages, and save resources. The introduction of initial policy identifiers ensures that each product details page has clear cache rules, which improves the efficiency and accuracy of the cache. Through effective cache management, unnecessary server resource consumption is reduced, thereby reducing operating costs.

[0019] By evaluating the memory requirements of cache tasks as a whole and comparing them with the remaining resource memory of the database, reasonable resource allocation decisions can be made. Cache tasks are managed in different levels according to different priorities to ensure that high-priority tasks are given priority when resources are limited. The cache strategy can be flexibly adjusted according to real-time resource conditions and task priorities to adapt to the ever-changing environment, ensuring that when resources are tight, pages that have the greatest impact on user experience can be cached first, thereby improving cache efficiency. By giving priority to caching high-priority task instances with smaller memory requirements, users can get a quick response when visiting popular product detail pages, thereby improving user experience. Refined management can avoid excessive resource consumption while ensuring the caching of key tasks, thereby enhancing the stability and reliability of the cache. When overall resources are sufficient, all product detail pages can be cached in order of priority to ensure that all pages can be cached and maximize the user experience. When resources are limited, first-priority pages with smaller memory requirements are cached first to ensure fast access to key pages. When resources are further limited, the first two priority pages are cached in order of priority, and the third-priority page is selectively cached to balance user experience and resource usage. When resources are very tight, only the first two priority pages are cached, and the second-priority pages are sorted by memory requirements to ensure the caching of key pages.

[0020] By comparing real-time key information with identified key information in real time, it is possible to quickly identify status changes of product details pages and make corresponding adjustments, thereby improving real-time responsiveness. It is possible to dynamically update the cache order of task instances based on an update signal to ensure that the cache strategy always matches current business needs and optimizes cache efficiency. It is possible to add or remove cache collections accordingly based on cleanup signals, effectively utilize cache resources, and avoid resource waste. By adjusting the cache location of popular product details pages in real time, it is possible to ensure that users get faster loading speeds when accessing these pages, thereby improving user experience. By cleaning up pages with low traffic, cache space can be freed up to cache more popular product pages and optimize overall traffic distribution. It is possible to adjust the cache queue based on real-time data to ensure that popular product pages are always cached, while pages with low traffic can be eliminated, thereby improving the flexibility of the cache strategy.

[0021] By monitoring cache hit rates, loading times, and response times, we can identify task instances with poor performance, thereby making targeted optimizations and continuously monitoring performance data to ensure that the system can respond to performance changes in real time and maintain service stability and reliability. Automated processes reduce manual intervention and improve management efficiency. Through timely performance adjustments, we can better cope with sudden traffic and changes in user behavior, improve the robustness of the system, ensure the stable operation of the e-commerce platform under high traffic, and optimize the user experience based on real-time data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the structure of a cache management system based on data monitoring and analysis according to an embodiment of the present invention; Figure 2 It is a structural diagram of a data monitoring module in a cache management system based on data monitoring and analysis according to an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a cache policy configuration module in a cache management system based on data monitoring and analysis according to an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of a cache execution module in a cache management system based on data monitoring and analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0025] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0026] See also Figure 1-Figure 4 As shown, Figure 1 A schematic diagram of the structure of a cache management system based on data monitoring and analysis according to an embodiment of the present invention; Figure 2 It is a structural diagram of a data monitoring module in a cache management system based on data monitoring and analysis according to an embodiment of the present invention; Figure 3A schematic diagram of the structure of a cache policy configuration module in a cache management system based on data monitoring and analysis according to an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of a cache execution module in a cache management system based on data monitoring and analysis according to an embodiment of the present invention.

[0027] The present invention provides a cache management system based on data monitoring and analysis, comprising: A data monitoring module, which is used to obtain historical monitoring data of each task instance and obtain identification key information based on the historical monitoring data; A cache strategy configuration module, which determines a corresponding initial strategy identifier for each task instance based on the identification key information, stores each identified task instance in a to-be-cached task queue, and determines an initial priority of a single task instance based on a single historical loading duration of the single task instance in combination with the corresponding initial strategy identifier; A cache execution module, which is used to determine the initial cache order of each task instance and generate an initial cache set according to the remaining resource memory of the database when caching the task instance and the initial priority; A primary correction module, which is used to collect the real-time behavior data of the user and the real-time change data of each task instance during the cache execution process, generate the real-time key information of the user according to the real-time behavior data, compare the real-time key information of the user with the identification key information, obtain the comparison result, and perform a primary correction on each initial cache sequence and the initial cache set according to the comparison result combined with the real-time change data; A performance monitoring module, which is used to collect performance data during cache execution and analyze the performance data to obtain performance monitoring results; A secondary correction module is used to update the initial strategy identifier according to the performance monitoring result to achieve secondary correction of each initial cache sequence.

[0028] In this embodiment, each task instance is a product details page in the e-commerce platform. The product details page is an important entry point for users to browse and purchase products. In order to improve user experience and reduce database pressure, the product details data is usually cached in advance and the cache is flexibly managed based on real-time monitoring data.

[0029] Specifically, the embodiments of the present invention can more accurately predict and respond to users' browsing needs by analyzing historical monitoring data and real-time behavior data of product details pages, thereby providing a fast and smooth page loading experience, and correcting cache strategies in real time to ensure that users can have a good experience even during peak access periods. The determination of the initial strategy identifier and the initial priority enables cache resources to be reasonably allocated according to the importance and access frequency of the page. Through the primary correction module and the secondary correction module, the cache content can be dynamically adjusted to avoid wasting cache resources. By caching product details data in advance, the number of database queries and loads during peak hours is reduced, and the database load is reduced. Real-time monitoring and correction of cache strategies help prevent database overload and maintain system stability. Performance data during the execution process can be collected. By analyzing these data, the cache strategy can be adjusted according to the performance monitoring results to further improve the overall performance, which can effectively improve the operating efficiency of the e-commerce platform.

[0030] Specifically, the data monitoring module in this embodiment includes: a data collection unit, a data identification unit and an extraction unit; For any task instance, the data collection unit is used to integrate the historical data of different data sources of the task instance to obtain the historical monitoring data; The data identification unit is used to identify key indicators according to the historical monitoring data, and analyze the identification results in different modes to obtain a number of data analysis results; The extraction unit is used to extract features from each data analysis result to obtain the key identification information.

[0031] The data identification unit includes: a key indicator identification subunit, a first analysis subunit and a second analysis subunit; The key indicator identification subunit is used to identify the key indicator to obtain the historical change trend of the key indicator; The first analysis subunit is used to analyze the historical change trend in a numerical sequence mode to obtain a first data analysis result; The second analysis subunit is used to analyze the historical change trend in a time series mode to obtain a second data analysis result.

[0032] The extraction unit extracts numerical features from the first data analysis result to obtain historical key Statistics and historical trend turning points; The extraction unit extracts time features from the second data analysis result to obtain historical fluctuations and historical turning points; Among them, the key identification information includes: numerical identification key information and time identification key information; the numerical identification key information includes: the historical key statistics and the historical trend turning points; the time identification key information includes: the historical fluctuations and the historical turning periods.

[0033] In this embodiment, for any product details page, the data collection unit collects historical data from different data sources, including the historical page access log and historical page content data corresponding to the product details page, obtains the historical page views through the historical page access log, and obtains the single historical loading time through the historical page content data to obtain the historical monitoring data, and ensures that the data obtained by each data source contains a timestamp to facilitate subsequent time series analysis. The data identification unit obtains the key indicator corresponding to each product details page, namely, the historical page views, by identifying the historical monitoring data; The key indicator identification subunit identifies the historical visits at different historical moments to obtain the historical change trend. The first analysis subunit analyzes the historical change trend of the key indicator in a numerical sequence mode, for example, obtaining the average value, maximum value, and minimum value of the visits by a statistical analysis method. The second analysis subunit analyzes the historical change trend of the key indicator in a time series mode, for example, identifying the change pattern of the visits over time, such as seasonal fluctuations or the impact of promotional activities. The extraction unit extracts numerical features from the first data analysis result. The specific extraction process is to obtain the historical access standard deviation and the timestamp of a sudden increase or decrease in the number of visits from the average, maximum and minimum values ​​of the number of visits. In the specific implementation process, the historical key statistics are the historical access standard deviation, and the historical trend turning point is the timestamp of a sudden increase or decrease in the number of visits. The extraction unit extracts time features from the second data analysis result. The specific extraction process is to extract time features from the second data analysis result to obtain the fluctuation of the number of visits during holidays or promotions and the time period of a sudden increase or decrease in the number of visits. In the specific implementation process, the historical fluctuation is the fluctuation of the number of visits during holidays or promotions, and the historical turning period is the time period of a sudden increase or decrease in the number of visits.

[0034] Specifically, the embodiments of the present invention ensure the comprehensiveness and accuracy of monitoring data by integrating historical data from different data sources. The identification and analysis of key indicators can reveal the access patterns and trends of product details pages, provide data support for caching strategies, and formulate more reasonable caching strategies based on the changing trends of historical access volume. By identifying historical trend turning points and historical fluctuations, it can predict and cache in advance product details pages that may be affected by seasonal fluctuations or promotional activities, thereby improving user access speed and satisfaction. The extraction of numerical features and time features helps to identify which pages require higher priority caching and which pages can reduce cache priority or be removed from the cache, thereby reducing unnecessary resource consumption. The caching strategy can be adjusted according to historical turning points and fluctuations, so that the e-commerce platform can flexibly respond to market changes and changes in user needs.

[0035] Specifically, in this embodiment, the cache strategy configuration module includes: a first judgment unit, a second judgment unit and an overall determination unit; The first judgment unit is used to determine the actual loading status of the single task instance according to the comparison result between the single historical loading duration and the preset duration; The second determination unit is used to determine the historical identification status of the single task instance according to the initial strategy identifier; The overall determination unit is used to determine the initial priority of the single task instance as the first initial priority, or the second initial priority, or the third initial priority according to the actual loading situation and the historical identification situation; The priority of the first initial priority task instance is higher than the second initial priority task instance, and the priority of the second initial priority task instance is higher than the third initial priority task instance.

[0036] In the embodiment of the present invention, determining the corresponding initial policy identifier for each task instance includes using a UUID library to generate a globally unique identifier. In the specific implementation process, a hash function can also be used to hash the key identification information of each task instance to generate a unique identifier. In this embodiment, the specific method for generating the initial policy identifier is not specifically limited and will not be repeated. The initial policy identifier includes information about the page cache policy, specifically including: expiration time, update frequency and product type. The product type is used to measure whether the product details page is a popular product page or a product that is popular during high traffic periods.

[0037] In this embodiment, the first judgment unit compares a single historical loading time of the product details page with a preset time, and determines that the actual loading situation is slow loading if the single historical loading time is greater than the preset time, determines that the actual loading situation is normal loading if the single historical loading time is equal to the preset time, and determines that the actual loading situation is fast loading if the single historical loading time is less than the preset time; The second judgment unit determines, based on the initial strategy identifier of the product details page, that its historical identification is a popular product page, or a product of interest during a high traffic period, or a common product, or a rare product; The overall determination unit combines the results of the first determination unit and the second determination unit to determine the initial priority of the product details page. The specific determination process is as follows: First initial priority: The page loads slowly or is a popular product; Second initial priority: the page loads normally or during high traffic periods to focus on the product; Third initial priority: The page loads quickly or is a niche product.

[0038] In the specific implementation process, based on the identification key information obtained by the data monitoring module, the cache strategy configuration module assigns a strategy identifier to each product details page. For example, the product details page corresponding to the popular product is marked as "high priority", that is, the first initial priority, and the promotional product details page corresponding to the product of high traffic period is marked as "temporary high priority", that is, the second initial priority. In the cache management process of the product details page of the e-commerce platform, the longer the loading time of the product details web page, the higher the priority of caching the web page to optimize the user experience and improve the system performance. Since pages with long loading times may increase the user waiting time, thereby affecting the user experience, caching these pages in advance can reduce the loading time and improve the user experience. Pages that load for a long time may consume more server resources. Caching in advance can reduce the server load.

[0039] Specifically, the embodiments of the present invention can perform refined cache management according to the page loading speed and product type to ensure the most effective use of cache resources. Different initial priorities are assigned according to the actual loading situation of the page to ensure that users can obtain faster loading speeds when accessing product details pages of popular products or high-traffic periods, thereby improving user experience. By giving higher cache priorities to pages that load slowly, the server load during peak hours can be reduced. Products of interest during high-traffic periods can be identified, so that the cache strategy can be adjusted within a specific time period to cope with traffic fluctuations. Comprehensive consideration of loading conditions and product types helps to reasonably allocate cache resources, avoid excessive caching of unpopular product pages, and save resources. The introduction of the initial policy identifier ensures that each product details page has clear cache rules, which improves the efficiency and accuracy of the cache. Through effective cache management, unnecessary server resource consumption is reduced, thereby reducing operating costs.

[0040] Specifically, the cache execution module in this embodiment includes: an acquisition unit, an overall comparison unit, a partial comparison unit and an execution determination unit; The acquisition unit is used to acquire the memory required for the entire to-be-executed program and the memory required for the single-class to-be-executed program corresponding to different initial priorities; The overall comparison unit is used to determine whether to start the hierarchical comparison mode according to the comparison result of the overall required memory to be executed and the remaining resource memory, or to output a first execution signal; The partial comparison unit determines to output a second execution signal, or a third execution signal, or a fourth execution signal based on the hierarchical comparison mode and according to the comparison result of each single type of memory required for execution and the remaining resource memory; The execution determination unit is used to receive each execution signal and determine the actual generation method of the initial cache sequence and the actual set capacity of the initial cache set according to different types of the execution signal; The partial comparison unit includes: a first comparison subunit and a second comparison subunit; The first comparison subunit is used to obtain the first single type of memory required for execution corresponding to each first initial priority, and determine to output the second execution signal according to the comparison result of the first single type of memory required for execution and the remaining resource memory, or to start the sum comparison mode; The second comparison sub-unit obtains the second single type of memory required for execution and part of the memory required for execution of the first single type of memory required for execution corresponding to each second initial priority level based on the sum comparison mode, and determines to output the third execution signal, or the fourth execution signal, according to the comparison result of the part of the memory required for execution and the remaining resource memory.

[0041] In this embodiment, the overall comparison unit compares the overall memory required for execution with the remaining resource memory of the database, and outputs a first execution signal if the overall memory required for execution is less than or equal to the remaining resource memory; if the overall memory required for execution is greater than the remaining resource memory, a hierarchical comparison mode is turned on; In the hierarchical comparison mode, the partial comparison unit compares the memory required for execution of single categories of different initial priorities with the remaining resource memory. The first comparison subunit obtains the required memory of each product detail page corresponding to the first initial priority, that is, the first single category of memory required for execution, and compares the first single category of memory required for execution with the remaining resource memory. If the first single category of memory required for execution is greater than the remaining resource memory, a second execution signal is output. If the first single category of memory required for execution is less than or equal to the remaining resource memory, a sum comparison mode is turned on. In the sum comparison mode, the second comparison subunit obtains the required memory of each product detail page corresponding to the first initial priority, that is, the first single category of memory required for execution, obtains the required memory of each product detail page corresponding to the second initial priority, that is, the second single category of memory required for execution, and sums the first single category of memory required for execution and the second single category of memory required for execution to obtain partial memory required for execution. The memory required for some pending execution = the memory required for the first category to be executed + the memory required for the second category to be executed. If the memory required for the part to be executed is less than or equal to the remaining resource memory, the third execution signal is output; if the memory required for the part to be executed is greater than the remaining resource memory, the fourth execution signal is output; Based on the output of the first execution signal, the corresponding actual generation method is: cache all product details pages in the order of initial priority; based on the output of the second execution signal, the corresponding actual generation method is: start caching each product details page of the first initial priority in the order of memory to be executed from small to large; based on the output of the third execution signal, the corresponding actual generation method is: cache the product details pages corresponding to the first initial priority and the second initial priority in the order of initial priority, and start caching each product details page of the third initial priority in the order of memory to be executed from small to large; based on the output of the fourth execution signal, cache the product details pages corresponding to the first initial priority and the second initial priority in the order of initial priority, and start caching each product details page of the second initial priority in the order of memory to be executed from small to large.

[0042] In the specific process, the cache execution module gives priority to caching the task instances with the highest priority. When resources are limited, it gives priority to caching those task instances with smaller memory resource requirements.

[0043] Specifically, the embodiments of the present invention make reasonable resource allocation decisions by evaluating the memory requirements of cache tasks as a whole and comparing them with the remaining resource memory of the database, hierarchically managing cache tasks according to different priorities to ensure that high-priority tasks are given priority when resources are limited, and flexibly adjusting cache strategies according to real-time resource conditions and task priorities to adapt to the ever-changing environment, ensuring that when resources are tight, pages that have the greatest impact on user experience are cached first, thereby improving cache efficiency. By giving priority to caching task instances with high priorities and low memory requirements, it is possible to ensure that users get a quick response when accessing popular product details pages, thereby improving user experience. The refined management of task instances can avoid excessive resource consumption while ensuring the caching of key tasks, thereby enhancing the stability and reliability of the cache. When overall resources are sufficient, all product detail pages can be cached in order of priority to ensure that all pages can be cached and maximize the user experience. When resources are limited, first-priority pages with smaller memory requirements are cached first to ensure fast access to key pages. When resources are further limited, the first two priority pages are cached in order of priority, and the third-priority page is selectively cached to balance user experience and resource usage. When resources are very tight, only the first two priority pages are cached, and the second-priority pages are sorted by memory requirements to ensure the caching of key pages.

[0044] Specifically, the primary correction module in this embodiment includes: a comparison unit, a primary update unit and a primary cleaning unit; The comparison unit is used to compare the real-time key information with the identification key information to determine the actual state change, and to compare the real-time change data with the real-time change data according to the actual state change. Determine whether to send an update signal or a cleanup signal to the task instance; The primary updating unit updates each initial cache sequence of the task instance based on the primary updating signal; The primary cleaning unit adds or removes task instances to the initial cache set accordingly based on the cleaning signal to obtain an actual cache set.

[0045] In this embodiment, in the product details page, the real-time behavior data is the real-time visit volume, and the real-time change data is the update status of the product details; If the product details page is determined to have changed based on the product details update situation, the comparison unit sends an update signal; Compare the real-time page views with the minimum and maximum historical page views obtained through the identification key information; If the real-time access volume is less than or equal to the minimum historical access volume, a cleanup signal is issued; if the real-time access volume is greater than or equal to the maximum historical access volume, an update signal is issued; After receiving an update signal, the primary update unit re-evaluates the cache priority of the product details page to adjust the position of the page in the cache queue in real time. For example, if a page becomes popular, it is moved to the front of the cache queue to ensure that popular pages can be loaded faster. For example, real-time data shows that user visits to promotional products have not met expectations, but visits to some regular product pages have increased. Based on this finding, the primary correction module adjusts the cache order and caches some regular product details pages in advance.

[0046] Specifically, the embodiments of the present invention can quickly identify the status changes of the product details page and make corresponding adjustments through real-time comparison of real-time key information and identification key information, thereby improving real-time responsiveness, and can dynamically update the cache order of the task instance according to an update signal to ensure that the cache strategy always matches the current business needs and optimize the cache efficiency. It can perform corresponding addition or removal operations on the cache collection according to the cleanup signal to effectively utilize cache resources and avoid resource waste. By adjusting the cache position of popular product details pages in real time, it can ensure that users get faster loading speeds when accessing these pages, thereby improving user experience. By cleaning up pages with low traffic, cache space can be freed up for caching more popular product pages, optimizing overall traffic distribution, and can adjust the cache queue according to real-time data to ensure that popular product pages are always in a cached state, while pages with low traffic can be eliminated, thereby improving the flexibility of the cache strategy.

[0047] Specifically, the secondary correction module in this embodiment includes: a result analysis unit and a secondary updating unit; The result analysis unit is used to obtain the cache hit rate, the single actual loading time and the actual response time of each task instance according to the performance monitoring result, determine the number of actual task instances that need to perform initial policy identifier update according to the cache hit rate, the single actual loading time and the actual response time, and determine whether to send a secondary update signal according to the actual number of task instances; The secondary updating unit updates the initial strategy identifier of each task instance based on the secondary updating signal, and rearranges the order of the task instances in the cache queue according to the updated actual strategy identifier.

[0048] In this embodiment, the cache performance, loading performance and response performance of the product details page are monitored and recorded to obtain the cache hit rate, the actual loading time of a single page and the actual response time. The performance monitoring results include: the cache hit rate, the actual loading time of a single page and the actual response time. The cache hit rate of each product details page is checked. If the hit rate is lower than a preset threshold (for example, lower than 50%), it is marked as requiring update. The actual loading time of each page is checked. If the loading time exceeds the preset maximum acceptable time (for example, more than 3 seconds), it is marked as requiring update. The actual response time of each page is checked. If the response time exceeds the preset maximum acceptable time (for example, more than 3 seconds), it is marked as requiring update. If the maximum acceptable duration is set, it is marked as needing to be updated. The number of task instances that need to be updated determines whether to send a secondary update signal. If the number of instances that need to be updated exceeds a preset ratio threshold, a secondary update signal is sent. After receiving the secondary update signal, the secondary update unit updates the initial policy identifier of the product details page that needs to be updated. According to the updated policy identifier, the order of the task instances in the cache queue is rearranged. The specific update process is to obtain more historical data from the data source to obtain updated identification key information, and generate an updated actual policy identifier by using the UUID library. For example: Assume that monitoring finds that the cache hit rate of three product detail pages is less than 50%, and the loading time of two pages exceeds 3 seconds. Determine the actual number of task instances that need to perform the initial policy identifier update: a total of five pages need to be updated. Since the number of pages that need to be updated exceeds the preset threshold (for example, more than 10% of the total pages), it is determined to send a secondary update signal.

[0049] Specifically, the embodiments of the present invention can identify task instances with poor performance by monitoring cache hit rate, loading time and response time, so as to perform targeted optimization and continuously monitor performance data to ensure that the system can respond to performance changes in real time and maintain service stability and reliability. The automated process reduces manual intervention and improves management efficiency. Through timely performance adjustment, it can better cope with sudden traffic and changes in user behavior, improve the robustness of the system, ensure the stable operation of the e-commerce platform under high traffic, and optimize the user experience based on real-time data.

[0050] The calculation compensation parameters and the calculation adjustment parameters described in the present invention have two functions: one is to balance the left and right dimensions of the formula, and the other is to adjust the numerical results. No specific assignment is performed in this embodiment. In addition, the calculation formulas in this embodiment are used to intuitively reflect the adjustment relationship between the numerical values, such as positive correlation and negative correlation. Unless otherwise specified, the parameter values ​​that are not specifically limited to numerical values ​​are all positive.

[0051] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cache management system based on data monitoring and analysis, characterized in that: include: A data monitoring module, which is used to obtain historical monitoring data of each task instance and obtain identification key information based on the historical monitoring data; A cache strategy configuration module, which determines a corresponding initial strategy identifier for each task instance based on the identification key information, stores each identified task instance in a to-be-cached task queue, and determines an initial priority of a single task instance based on a single historical loading duration of the single task instance in combination with the corresponding initial strategy identifier; A cache execution module, which is used to determine the initial cache order of each task instance and generate an initial cache set according to the remaining resource memory of the database when caching the task instance and the initial priority; A primary correction module, which is used to collect the real-time behavior data of the user and the real-time change data of each task instance during the cache execution process, generate the real-time key information of the user according to the real-time behavior data, compare the real-time key information of the user with the identification key information, obtain the comparison result, and perform a primary correction on each initial cache sequence and the initial cache set according to the comparison result combined with the real-time change data; A performance monitoring module, which is used to collect performance data during cache execution and analyze the performance data to obtain performance monitoring results; A secondary correction module, which is used to update the initial strategy identifier according to the performance monitoring result to achieve secondary correction of each initial cache sequence; The performance data includes: cache performance, loading performance and response performance.

2. The cache management system based on data monitoring and analysis according to claim 1 is characterized in that: The data monitoring module includes: a data collection unit, a data identification unit and an extraction unit; For any task instance, the data collection unit is used to integrate the historical data of different data sources of the task instance to obtain the historical monitoring data; The data identification unit is used to identify key indicators according to the historical monitoring data, and analyze the identification results in different modes to obtain a number of data analysis results; The extraction unit is used to extract features from each data analysis result to obtain the key identification information.

3. The cache management system based on data monitoring and analysis according to claim 2 is characterized in that: The data identification unit includes: a key indicator identification subunit, a first analysis subunit and a second analysis subunit; The key indicator identification subunit is used to identify the key indicator to obtain the historical change trend of the key indicator; The first analysis subunit is used to analyze the historical change trend in a numerical sequence mode to obtain a first data analysis result; The second analysis subunit is used to analyze the historical change trend in a time series mode to obtain a second data analysis result.

4. The cache management system based on data monitoring and analysis according to claim 3 is characterized in that: The extraction unit extracts numerical features from the first data analysis result to obtain historical key statistics and historical trend turning points; The extraction unit extracts time features from the second data analysis result to obtain historical fluctuations and historical turning points; Among them, the key identification information includes: numerical identification key information and time identification key information; the numerical identification key information includes: the historical key statistics and the historical trend turning points; the time identification key information includes: the historical fluctuations and the historical turning periods.

5. The cache management system based on data monitoring and analysis according to claim 4 is characterized in that: The cache strategy configuration module includes: a first judgment unit, a second judgment unit and an overall determination unit; The first judgment unit is used to determine the actual loading status of the single task instance according to the comparison result between the single historical loading duration and the preset duration; The second determination unit is used to determine the historical identification status of the single task instance according to the initial strategy identifier; The overall determination unit is used to determine the initial priority of the single task instance as the first initial priority, or the second initial priority, or the third initial priority according to the actual loading situation and the historical identification situation; The priority of the first initial priority task instance is higher than the second initial priority task instance, and the priority of the second initial priority task instance is higher than the third initial priority task instance.

6. The cache management system based on data monitoring and analysis according to claim 5, characterized in that: The cache execution module includes: an acquisition unit, an overall comparison unit and a partial comparison unit; The acquisition unit is used to acquire the memory required for the entire to-be-executed program and the memory required for the single-class to-be-executed program corresponding to different initial priorities; The overall comparison unit is used to determine whether to start the hierarchical comparison mode according to the comparison result of the overall required memory to be executed and the remaining resource memory, or to output a first execution signal; The partial comparison unit determines to output a second execution signal, or a third execution signal, or a fourth execution signal based on the hierarchical comparison mode and the comparison result between each single type of memory required for execution and the remaining resource memory.

7. The cache management system based on data monitoring and analysis according to claim 6 is characterized in that: The cache execution module further includes: an execution determination unit; The execution determination unit is used to receive each execution signal and determine the actual generation method of the initial cache sequence according to different types of the execution signal.

8. The cache management system based on data monitoring and analysis according to claim 7 is characterized in that: The partial comparison unit includes: a first comparison subunit and a second comparison subunit; The first comparison subunit is used to obtain the first single type of memory required for execution corresponding to each first initial priority, and determine to output the second execution signal according to the comparison result of the first single type of memory required for execution and the remaining resource memory, or to start the sum comparison mode; The second comparison sub-unit obtains the second single type of memory required for execution and part of the memory required for execution of the first single type of memory required for execution corresponding to each second initial priority level based on the sum comparison mode, and determines to output the third execution signal, or the fourth execution signal, according to the comparison result of the part of the memory required for execution and the remaining resource memory.

9. The cache management system based on data monitoring and analysis according to claim 8, characterized in that: The primary correction module includes: a comparison unit, a primary update unit and a primary cleaning unit; The comparison unit is used to compare the real-time key information with the identification key information to determine the actual state change, and to compare the real-time change data with the real-time change data according to the actual state change. Determine whether to send an update signal or a cleanup signal to the task instance; The primary updating unit updates each initial cache sequence of the task instance based on the primary updating signal; The primary cleaning unit adds or removes task instances to the initial cache set accordingly based on the cleaning signal to obtain an actual cache set.

10. The cache management system based on data monitoring and analysis according to claim 9, characterized in that: The secondary correction module includes: a result analysis unit and a secondary updating unit; The result analysis unit is used to obtain the cache hit rate, the single actual loading time and the actual response time of each task instance according to the performance monitoring result, determine the number of actual task instances that need to perform initial policy identifier update according to the cache hit rate, the single actual loading time and the actual response time, and determine whether to send a secondary update signal according to the actual number of task instances; The secondary updating unit updates the initial strategy identifier of each task instance based on the secondary updating signal, and rearranges the order of the task instances in the cache queue according to the updated actual strategy identifier.

Images