Data processing method, apparatus, device, medium, product

Abstract

The application is suitable for the computer technology field, and particularly relates to a data processing method, device, equipment, medium and product, which comprises the following steps: through a cache middleware, the storage states of the cache and the database are adaptively adjusted in combination with the running state of a current data processing system, so that the storage states of the business data in the cache and the database meet different business demands, the access pressure of the database is reduced, and the data processing efficiency of the data processing system is improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a data processing method, apparatus, device, medium, or product. Background Technology

[0002] In business processing scenarios, hot data is usually stored in the cache, while non-hot data is usually stored in the database. When a business request is received, the cache is usually accessed first. If the business data corresponding to the business request is not found in the cache, the database is then accessed to retrieve the business data corresponding to the business request.

[0003] In related technologies, a fixed caching strategy is usually set in advance. For example, a hit rate threshold is set in advance, and business data in the cache that is lower than the hit rate threshold is evicted at certain time intervals to free up cache memory so that new hot data can be stored.

[0004] However, a single caching strategy cannot cope with complex business processing scenarios. When business data is deprecated, there will be a need to access that business data. At this time, it is necessary to access the database to retrieve the business data, which leads to a surge in database access and increases the pressure on database data processing. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, medium, and product. By adding a caching middleware to the data processing system, the caching strategy is dynamically adjusted based on the access characteristics of the target business data determined by the caching middleware, thereby improving the flexibility of data caching and enhancing the adaptability of data caching to different application scenarios.

[0006] In a first aspect, embodiments of this application provide a data processing method, the method being applied to a computer device, the computer device running a data processing system, the data processing system including a database, a cache, and cache middleware, the cache storing first business data, the database storing second business data, the access frequency of the first business data being lower than the access frequency of the second business data; the method includes: The cache middleware is used to obtain target business data corresponding to business requests received within a preset time range, wherein the target business data is one of the first business data or the second business data. The caching middleware determines a data storage strategy based on the access characteristics corresponding to the target business data and in combination with preset rules. The data storage strategy is used to determine at least one of the storage methods of the first business data or the storage methods of the second business data. Update at least one of the first business data in the cache or the second business data in the database based on the data storage strategy.

[0007] In some embodiments, the preset rules include cache hit rate rules; The step of determining a data storage strategy based on the access characteristics corresponding to the target business data and in combination with preset rules through the caching middleware includes: The cache middleware determines a first hit rate of the first business data based on the access characteristics. The first hit rate is used to indicate the number of times the first business data is the target business data. The first business data includes the i-th business data, where i is a positive integer. If the first hit rate corresponding to the i-th service data is lower than the first hit rate threshold, the i-th service data is removed from the cache.

[0008] In some embodiments, the method further includes: If the i-th business data is not stored in the database, the i-th business data will be stored in the database.

[0009] In some embodiments, the preset rules include database load rules; The step of determining a data storage strategy based on the access characteristics corresponding to the target business data and in combination with preset rules through the caching middleware includes: If the target business data does not belong to the first business data, the caching middleware determines a second hit rate for the second business data based on the access characteristics. The second hit rate indicates the number of times the second business data is the target business data, wherein the second business data includes the nth business data, where n is a positive integer. If the second hit rate of the nth service data is higher than the second hit rate threshold, and the nth service data belongs to the first service data in the cache, the storage time of the nth service data in the cache shall be extended.

[0010] In some embodiments, the method further includes: If the second hit rate of the nth service data is higher than the second hit rate threshold, and the nth service data does not belong to the first service data in the cache, the nth service data is stored in the cache.

[0011] In some embodiments, the cache includes a first cache node, which stores the m-th business data, where m is a positive integer; The step of determining a data storage strategy based on the access characteristics corresponding to the target business data and in combination with preset rules through the caching middleware includes: The cache middleware determines the third hit rate corresponding to the m-th business data based on the access characteristics, and the third hit rate is used to indicate the number of times the m-th business data is the target business data; If the memory corresponding to the first cache node is lower than a preset memory threshold, and the third hit rate is lower than a third hit rate threshold, the m-th service data is removed from the cache.

[0012] In some embodiments, the method further includes: Acquire hotspot data within a historical time range and acquire historical business data within the historical time range. Hotspot data refers to business data whose access frequency is higher than a preset access threshold, and historical business data refers to business data that does not belong to at least one of the first business data or the second business data. The hot data is matched with the historical business data, and the historical data with a matching degree higher than a preset matching degree threshold is cached in the cache or the database, or at least one of them.

[0013] In some embodiments, the cache includes multiple cache nodes; The method further includes: Based on the distribution of time periods within the historical time range, the start and stop results of the multiple cache nodes are adjusted.

[0014] Secondly, embodiments of this application provide a data processing apparatus, including: The acquisition module is used to acquire target business data corresponding to business requests received within a preset time range through a caching middleware. The cache stores first business data, and the database stores second business data. The access frequency of the first business data is lower than that of the second business data. The target business data is one of the first business data or the second business data. The determination module is used to determine a data storage strategy based on the access characteristics corresponding to the target business data through the caching middleware and in combination with preset rules. The data storage strategy is used to determine at least one of the storage methods of the first business data or the storage methods of the second business data. The update module is used to update at least one of the first business data in the cache or the second business data in the database based on the data storage strategy.

[0015] Thirdly, this application also provides a computer device. The computer device includes a memory, one or more processors, and a computer program stored in the memory and executable on the processor. When the computer device executes the computer program, it implements a method for any of the implementations of the first aspect described above.

[0016] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method of any of the implementations of the first aspect described above.

[0017] Fifthly, this application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute any of the implementation methods of the first aspect described above.

[0018] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here.

[0019] The beneficial effects corresponding to the embodiments of this application are as follows: By adding a caching middleware to the data processing system, the middleware retrieves the target business data corresponding to business requests received within a preset time range. Based on the access characteristics of the target business data and preset rules, the data storage strategy is dynamically adjusted. This allows for real-time adjustment of the storage status of the first business data in the cache and the storage status of the second business data in the database. In other words, the caching middleware adaptively adjusts the storage status of the cache and database in conjunction with the current operation of the data processing system, ensuring that the storage status of business data in the cache and database meets different business needs, reducing database access pressure, and improving the data processing efficiency of the data processing system. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the overall solution provided in one embodiment of this application; Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application; Figure 3 This is a flowchart of a data processing method provided in an embodiment of this application; Figure 4This is a structural block diagram of a data processing apparatus provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0026] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] First, the terms used in this application will be explained: Hot data refers to data that is frequently accessed and read with high concurrency in a system. In other words, it is accessed frequently by a large number of users or requests in a short period of time. Examples include popular product details on e-commerce platforms and trending news on news websites.

[0029] Cache hit: This means that when the requested data exists in the cache, the result is returned directly from the cache without accessing the backend storage (such as a database).

[0030] Cache penetration refers to a situation where the requested data does not exist in either the cache or the database, causing each request to bypass the cache and directly access the database.

[0031] Cache breakdown occurs when a large number of concurrent requests access frequently accessed data in the cache, causing a sudden surge in database pressure.

[0032] In business scenarios, frequently accessed data is typically stored in the cache first. Therefore, when a business request is received, the cache is accessed first. If the data corresponding to the request is not found in the cache, the database is then accessed to retrieve the relevant data. Related technologies usually pre-define a fixed caching strategy. For example, if the hit rate of certain business data exceeds a certain threshold, it is identified as frequently accessed data and cached. At regular intervals, business data in the cache with a hit rate below the threshold is evicted to free up cache memory for storing new frequently accessed data. However, a single caching strategy cannot handle complex business processing scenarios. For instance, after evicting certain business data, there may be a need to access that data again. In this case, the database needs to be accessed again to retrieve the data, leading to a surge in database access and increasing the pressure on database processing.

[0033] To systematically address the aforementioned problems, this application provides a data processing method. By adding a caching middleware to the data processing system, the method utilizes the caching middleware to obtain the target business data corresponding to the business requests received within a preset time range. Based on the access characteristics of the target business data and preset rules, the data storage strategy is dynamically adjusted. This allows for real-time adjustment of the storage status of the first business data in the cache and the storage status of the second business data in the database. In other words, through the caching middleware, the storage status of the cache and the database is adaptively adjusted in conjunction with the current operating status of the data processing system. This ensures that the storage status of business data in the cache and the database meets different business needs, reduces the access pressure on the database, and improves the data processing efficiency of the data processing system.

[0034] Figure 1 This is a schematic diagram of a data processing system architecture provided in an embodiment of this application, wherein the data processing system 100 includes a web server, a caching middleware, a cache, and a database.

[0035] The website server receives business requests from clients (e.g., requests to access product details). After receiving the request, the website server forwards it to the caching middleware. The caching middleware temporarily caches multiple frequently accessed hot data. It determines whether the business data corresponding to the request exists among these hot data. If not, it accesses the cache and database sequentially to determine the business data corresponding to the request. After determining the business data, it sends the data back to the website server, which then forwards it to the client.

[0036] In addition to the data processing scenarios mentioned above, the caching middleware will also determine the storage strategy for business data in the cache and database based on business needs. For example, it will store frequently accessed hot data in a designated node in the cache and remove infrequently accessed non-hot data from the cache and store it in the database.

[0037] from Figure 1 As can be seen, adding caching middleware can balance the data processing between various servers and dynamically adjust the data storage strategies in the cache and database to meet different business needs.

[0038] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application, which includes the following steps.

[0039] S201: Obtain the target business data corresponding to the business requests received within a preset time range through the caching middleware.

[0040] The computer equipment runs a data processing system, which includes a database, a cache, and a cache middleware. The cache stores first business data, and the database stores second business data. The access frequency of the first business data is lower than that of the second business data. The target business data is either the first business data or the second business data.

[0041] In illustrative terms, a cache, also known as a cache server, refers to a software server deployed in a data processing system. It caches frequently accessed data, allowing for faster retrieval of the target business data corresponding to a business request when processing that request. The purpose of caching is to place frequently accessed data in a medium closest to the user or with faster access speeds, thereby accelerating data access and reducing response time.

[0042] To illustrate, a database is a persistent storage system used to store structured data for a long time and supports complex queries and transaction operations. Therefore, a database can store more business data, but the response time of a database is longer and the data access speed is slower.

[0043] Therefore, frequently accessed data is usually stored in the cache, while less frequently accessed data is usually stored in the database.

[0044] It is worth noting that in this embodiment, all terms such as "hot data", "non-hot data", "data" or "a certain data" refer to business data, and will not be repeated in subsequent embodiments.

[0045] To illustrate, since the cache stores hot data (first business data) and the database stores non-hot data (second business data), when there is a business request, the hit rate of the first business data is higher than that of the second business data. Therefore, the access frequency of the first business data is higher than that of the second business data.

[0046] The access frequency refers to the number of times the business data (including the first business data and the second business data) is accessed, and "access" refers to using the business data as the target business data corresponding to the business request.

[0047] Optionally, the first business data and the second business data are two independent types of business data. That is, under the current circumstances, the first business data and the second business data are completely different. Alternatively, the first business data and the second business data may contain the same business data, meaning that there is a specific business data that is stored both in the cache and in the database. In this case, the storage method is "hit-through". If the business data is only stored in the database and not in the cache, then the storage method is "hit-writeback".

[0048] Indicatively, a preset time range refers to a pre-defined time range, such as within 24 hours before the current time.

[0049] Optionally, the target business data can be implemented as first business data, in which case the target business data belongs to "hot data". Alternatively, the target business data can be implemented as second business data. Or, the target business data can belong to neither first business data nor second business data. This application embodiment does not limit this.

[0050] Optionally, a layered architecture design can be adopted in the cache, including two cache structures: local cache and distributed cache.

[0051] Local caching is deployed within the application process (e.g., JVM heap memory) and is directly managed by the application code. Distributed caching is a cache cluster independent of the application process (e.g., Redis) and is accessed over the network. Local cache access time is shorter than distributed cache access time; therefore, the access frequency of business data stored in local cache is higher than that of business data stored in distributed cache. This allows for more refined caching of multiple frequently accessed data points.

[0052] In one example, for a read request, the following logic can be used: The first step is to query the local cache: if a match is found, return the business data directly.

[0053] The second step is to query the distributed cache: if the local cache is not found, query the distributed cache; if it is found, write the business data to the local cache and return the business data.

[0054] The third step is to query the database: if the distributed cache misses, query the database; write the results to the local cache and the distributed cache, and return the business data.

[0055] S202 determines the data storage strategy based on the access characteristics corresponding to the target business data and in combination with preset rules through the caching middleware.

[0056] The data storage strategy is used to determine at least one of the storage methods for the first business data or the storage methods for the second business data.

[0057] In illustrative terms, access characteristics are used to characterize the access behavior corresponding to the target business data.

[0058] Optionally, the access characteristics include at least one of the following characteristic types: 1. Access frequency.

[0059] This indicates the number of times or rate at which data is accessed.

[0060] 2. Temporal locality.

[0061] This indicates that data is repeatedly accessed within a short period of time. For example, when a user browses a webpage, multiple resources on the same page may be requested consecutively.

[0062] 3. Spatial locality.

[0063] This indicates that accessing certain data may also access nearby data.

[0064] 4. Access order.

[0065] Indicates whether data access follows a specific order. For example: sequential access: streaming log files (from beginning to end); random access: querying user information (no fixed order).

[0066] 5. Popularity distribution of visits.

[0067] This indicates the distribution of the frequency of access to business data within the dataset.

[0068] 6. Data lifecycle.

[0069] This indicates the lifespan of data from creation to expiration.

[0070] The illustrative, preset rule refers to a pre-set basis for determining which data storage strategy to select from multiple data storage strategies, which will be described in detail in subsequent embodiments.

[0071] To illustrate, taking the storage method of the first business data as an example, the data storage strategy includes at least one of the following strategies: 1. Remove the specified first business data from the cache. Evicting means not storing the first business data in the cache. 2. Store the specified business data in the cache; 3. Extend the cache duration of the first business data in the cache. For example, if the storage time of business data 1 in the cache is specified to be 2 hours before it is automatically evicted, extend it to 4 hours before it is evicted.

[0072] 4. Change the storage location of the first business data. For example, if there are cache node 1 and cache node 2 in the cache, and cache node 1 and cache node 2 are located in different cache locations, store business data 2 from cache node 1 to cache node 2.

[0073] 5. Adjust the activation status of cache nodes. For example, if there are cache node 1 and cache node 2 in the cache, both cache node 1 and cache node 2 should be enabled during the daytime. During the nighttime, only cache node 1 should be enabled and cache node 2 should not be enabled.

[0074] It is worth noting that the above-described data storage strategies are merely illustrative examples, and the embodiments of this application do not limit them.

[0075] As an illustration, the storage method for the second business data is the same as that for the first business data, and will not be repeated here.

[0076] S203, based on the data storage strategy, update at least one of the first business data in the cache or the second business data in the database.

[0077] Indicatively, after determining the data storage strategy, the first business data stored in the cache is updated according to the data storage strategy, or the second business data stored in the database is updated according to the data storage strategy, or the first business data stored in the cache and the second business data stored in the database are updated synchronously according to the data storage strategy. This application embodiment does not limit this.

[0078] The data processing method provided in this application embodiment adds a caching middleware to the data processing system. The caching middleware retrieves the target business data corresponding to business requests received within a preset time range. Based on the access characteristics of the target business data and preset rules, the data storage strategy is dynamically adjusted. This allows for real-time adjustment of the storage status of the first business data in the cache and the storage status of the second business data in the database. In other words, the caching middleware adaptively adjusts the storage status of the cache and database in conjunction with the current operation of the data processing system, ensuring that the storage status of business data in the cache and database meets different business needs, reducing database access pressure, and improving the data processing efficiency of the data processing system.

[0079] In some embodiments, the selection method of data storage strategy is described in detail.

[0080] The first type includes cache hit rate rules in the preset rules.

[0081] In some embodiments, a first hit rate of the first business data is determined by a caching middleware based on access characteristics. The first hit rate is used to indicate the number of times the first business data is the target business data. The first business data includes the i-th business data, where i is a positive integer. If the first hit rate corresponding to the i-th business data is lower than the first hit rate threshold, the i-th business data is removed from the cache.

[0082] For illustrative purposes, the first hit rate refers to the number of times the first business data is used as the target business data.

[0083] For illustration, if the first hit rate corresponding to the i-th business data is lower than the preset first hit rate threshold, it indicates that the access frequency of the i-th business data is low. Therefore, the i-th business data can be removed from the cache.

[0084] In some embodiments, if the i-th business data is not stored in the database, the i-th business data is stored in the database.

[0085] To illustrate, if the i-th business data is only stored in the cache, and the cache of the i-th business data is removed, then it is stored in the database.

[0086] The second type includes database load rules in the preset rules.

[0087] In some embodiments, when the target business data does not belong to the first business data, the second hit rate of the second business data is determined by the caching middleware based on the access characteristics. The second hit rate is used to indicate the number of times the second business data is the target business data. The second business data includes the nth business data, where n is a positive integer. If the second hit rate of the nth business data is higher than the second hit rate threshold, the nth business data is stored in the cache.

[0088] To illustrate, if the target business data does not belong to the first business data, then it is necessary to access the second business data stored in the database to determine the second hit rate of the second business data.

[0089] For illustration, if the second hit rate of the nth business data is higher than the preset second hit rate threshold, it means that the access frequency of the nth business data is high. In this case, the nth business data is stored in the cache, and the storage time of the nth business data in the cache is extended.

[0090] In some embodiments, if the second hit rate of the nth service data is higher than the second hit rate threshold and the nth service data does not belong to the first service data in the cache, the storage time of the nth service data in the cache is extended.

[0091] The third type includes node storage adjustment rules in the preset rules.

[0092] In some embodiments, the cache includes a first cache node, which stores the m-th business data, where m is a positive integer; the cache middleware determines the third hit rate corresponding to the m-th business data based on access characteristics, and the third hit rate is used to indicate the number of times the m-th business data is used as target business data; if the memory corresponding to the first cache node is lower than a preset memory threshold and the third hit rate is lower than the third hit rate threshold, the m-th business data is removed from the cache.

[0093] For illustration, if the third hit rate of the m-th business data is lower than the preset third hit rate threshold, it means that the access frequency of the m-th business data is low. At this time, the amount of the first business data cached in the first cache node reaches the preset data amount threshold, which means that the memory corresponding to the first cache node is lower than the preset memory threshold. Therefore, the m-th business data is removed from the cache, and the memory space of the first cache node is released.

[0094] In some embodiments, hot data within a historical time range and historical business data within a historical time range are obtained. Hot data refers to business data whose access frequency is higher than a preset access threshold, and historical business data refers to business data that does not belong to at least one of the first business data or the second business data. Hot data is matched with historical business data, and historical data with a matching degree higher than a preset matching degree threshold is cached in a cache or in a database.

[0095] In some embodiments, the cache includes multiple cache nodes; the start and stop results of multiple cache nodes are adjusted based on the distribution of corresponding time periods within a historical time range.

[0096] In this embodiment, a review thread is started during the period of low access volume in the early morning: comparing the hot data of the previous day with the newly generated uncached data, and caching similar data; dynamically adjusting the number of nodes according to the access time of the previous day so that resources can be put away when idle and the number of nodes can be started during hot periods. For example, if there are cache node 1 and cache node 2 in the cache, cache node 1 and cache node 2 are started simultaneously during the daytime, and only cache node 1 is started during the nighttime, and cache node 2 is not started.

[0097] The data processing method provided in this application embodiment adds a caching middleware to the data processing system. The caching middleware retrieves the target business data corresponding to business requests received within a preset time range. Based on the access characteristics of the target business data and preset rules, the data storage strategy is dynamically adjusted. This allows for real-time adjustment of the storage status of the first business data in the cache and the storage status of the second business data in the database. In other words, the caching middleware adaptively adjusts the storage status of the cache and database in conjunction with the current operation of the data processing system, ensuring that the storage status of business data in the cache and database meets different business needs, reducing database access pressure, and improving the data processing efficiency of the data processing system.

[0098] This is illustrative; please refer to it. Figure 3 It illustrates a schematic diagram of a data processing system architecture provided in an exemplary embodiment of this application, such as... Figure 3As shown, the system includes a client 301, a website server 302, a caching middleware 303, a cache 304, and a database 305.

[0099] Website server 302 receives a business request from client 301 and forwards the business request to cache middleware 303. Cache middleware 303 obtains the target business data according to the data storage strategy, obtains the first business data from cache 304, and obtains the second business data from database 305. It determines the storage method of the first business data and the storage method of the second business data according to the access characteristics of the target business data, dynamically adjusts the data storage strategy during the early morning period, and preloads new business data.

[0100] In this application, the core features of the adaptive caching middleware revolve around "dynamic adaptation, intelligent decision-making, high availability, and low intrusion".

[0101] I. Core Features: Dynamic Perception and Adaptation This is the most essential feature of adaptive caching middleware: it completely breaks away from the "static configuration dependency" of traditional caching and achieves an automated closed loop of "system state → strategy adjustment".

[0102] Automatic collection of full-link metrics for the system includes: request QPS, cache hit rate, number of penetrations / breakdowns, backend database CPU / memory load, cache node resource utilization (memory / bandwidth), data update frequency, etc.

[0103] Supports multi-dimensional data linkage analysis: for example, it can associate "decreased cache hit rate" with "surge in database load" to accurately pinpoint whether the problem lies in the caching strategy or changes in data access characteristics.

[0104] Adaptive caching algorithm: Automatically switch eviction algorithms based on access characteristics (e.g., switch from LRU to LFU). For example, in the event of a sudden hotspot, switch to LFU to prioritize the retention of frequently accessed data, while in the event of a low-frequency update, use LRU to improve cache utilization.

[0105] TTL Smart Optimization: Dynamically adjust the expiration time based on data popularity and update frequency (e.g., extend the TTL of hot data to 5 minutes, shorten the TTL of low-frequency data to 30 seconds, and set the TTL of high-frequency update data to 10 seconds).

[0106] Dynamic resource allocation: In distributed scenarios, hot data is automatically migrated to high-performance cache nodes, or cache nodes are expanded according to traffic peaks and scaled down during off-peak hours to optimize resource utilization.

[0107] II. Intelligent Decision-Making: Precise Optimization Based on Data and Algorithms By using a built-in intelligent engine to replace manual decision-making, the caching strategy is optimized and automated. Its core features include: By combining algorithms such as sliding window and heat decay, it automatically identifies stable hotspots and sudden hotspots without the need for manual labeling.

[0108] Hot data is given "special treatment": dual-caching storage (local cache + distributed cache), extended TTL, and priority allocation of cache resources to avoid hotspot penetration.

[0109] Proactively intercept risks: Identify cache penetration (Bloom filter automatically intercepts invalid keys), cache avalanche (randomly delays expiration time), and cache breakdown (hot data never expires + asynchronous updates).

[0110] Self-healing: When a cache node fails, it automatically switches to a backup node; when the cache cluster has uneven load, it automatically migrates data to achieve load balancing without manual intervention.

[0111] The threshold is not a fixed value: for example, the hotspot judgment threshold (such as 100 accesses per second) will be dynamically adjusted according to the overall system traffic (the threshold is raised during peak hours and lowered during off-peak hours) to avoid misjudgment or missed judgment.

[0112] Supports machine learning optimization (advanced features): Some solutions can train models based on historical data to predict traffic changes and data access characteristics, and adjust caching strategies in advance (such as preloading potential hot data before major promotions).

[0113] III. High Availability: Ensuring Stable System Operation Adaptive caching middleware acts as a system "performance buffer layer," featuring a highly available design while enhancing the stability of the entire chain. It supports master-slave replication and sharded clusters (such as Redis Cluster), and automatic failover in case of node failure to avoid single points of failure.

[0114] Data replication storage: Core data is backed up on multiple cache nodes to prevent data loss.

[0115] When the backend service / database is overloaded, the circuit breaker is automatically triggered, and cached data is returned first (even if it is expired data, a "degradation strategy" can be configured) to avoid cascading failures.

[0116] Supports traffic shaping: Buffers sudden traffic surges and intercepts most requests through caching, preventing the backend from being overwhelmed by instantaneous peak traffic.

[0117] Adaptive cache update strategy selection: Use "Write-Through" for frequently updated data and "Write-Back" for infrequently updated data to balance consistency and performance.

[0118] Avoid dirty data: Ensure that cached data is consistent with the source data through version number, timestamp verification, or asynchronous update mechanisms.

[0119] IV. Low Intrusion and Compatibility: Reducing Implementation Costs Adaptive caching middleware exists as "middleware," eliminating the need to refactor business logic and offering exceptional compatibility and ease of use. It supports mainstream development frameworks (Spring Boot, Dubbo, etc.) and can be integrated through annotations and configuration files without modifying business logic code.

[0120] (1) Data acquisition module The built-in monitoring module collects real-time metrics such as request traffic, cache hit / penetration / breakdown counts, database response time, and cache server CPU / memory usage.

[0121] (2) Strategy Decision Engine Based on preset rules and intelligent algorithms, the collected metrics are analyzed to determine the current system status. Common decision-making logic includes switching the cache eviction algorithm when the cache hit rate is below a threshold; extending the caching time for hot data when the database load is too high; and prioritizing the eviction of low-frequency access data when the cache node memory is insufficient.

[0122] (3) Execute the adjustment module Based on the decision results, the caching strategy is dynamically adjusted, and the adjustment results are fed back to the monitoring layer to form a closed loop.

[0123] (4) Post-mortem strategy: The review process was initiated during the early morning hours when traffic was low. Compare the previous day's hot data with newly generated, uncached data, and cache similar data; The number of nodes is dynamically adjusted based on the access time period of the previous day, so that resources are put away when idle and nodes are activated during peak periods.

[0124] (5) Seamlessly integrate with existing mainstream frameworks It supports mainstream development frameworks (Spring Boot, Dubbo, etc.) and can be integrated through annotations and configuration files without modifying business logic code.

[0125] Corresponding to the robot control method in the above embodiments, Figure 4A structural block diagram of a robot control device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0126] Reference Figure 4 The device 400 includes the following: The acquisition module 410 is used to acquire the target business data corresponding to the business requests received within a preset time range through the caching middleware. The cache stores the first business data and the database stores the second business data. The access frequency of the first business data is lower than the access frequency of the second business data. The determination module 420 is used to determine a data storage strategy based on the access characteristics corresponding to the target business data through the caching middleware and in combination with preset rules. The data storage strategy is used to determine at least one of the storage methods of the first business data or the storage methods of the second business data. The update module 430 is used to update at least one of the first business data in the cache or the second business data in the database based on the data storage strategy.

[0127] The data processing apparatus provided in this application embodiment adds a caching middleware to the data processing system. This middleware acquires the target business data corresponding to business requests received within a preset time range. Based on the access characteristics of the target business data and preset rules, it dynamically adjusts the data storage strategy, thereby adjusting the storage status of the first business data in the cache and the storage status of the second business data in the database in real time. In other words, through the caching middleware, the storage status of the cache and database is adaptively adjusted according to the current operating status of the data processing system, ensuring that the storage status of business data in the cache and database meets different business needs, reducing database access pressure, and improving the data processing efficiency of the data processing system. It should be noted that the information interaction and execution process between the above-mentioned devices / modules are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section, and will not be repeated here.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0129] To implement the above embodiments, this application also proposes an electronic device. Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0130] like Figure 5 As shown, the above-mentioned electronic device 500 includes: The system includes a memory 510 and at least one processor 520, and a bus 530 connecting the different components (including the memory 510 and the processor 520). The memory 510 stores a computer program, and when the processor 520 executes the program, it implements the method for controlling the robot according to the embodiments of this application.

[0131] Bus 530 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0132] Electronic device 500 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 500, including volatile and non-volatile media, removable and non-removable media.

[0133] Memory 510 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 540 and / or cache memory 550. Electronic device 500 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 560 can be used to read and write non-removable, non-volatile magnetic media (… Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 530 via one or more data media interfaces. Memory 510 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0134] A program / utility 580 having a set (at least one) of program modules 570 may be stored, for example, in memory 510. Such program modules 570 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 570 typically perform the functions and / or methods described in the embodiments of this application.

[0135] Electronic device 500 can also communicate with one or more external devices 590 (e.g., keyboard, pointing device, display 591, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 595. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 593. As shown, network adapter 593 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0136] The processor 520 performs various functional applications and data processing by running programs stored in the memory 510.

[0137] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the data processing method in the embodiments of this application, and will not be repeated here.

[0138] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0139] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some regions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0141] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0142] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0143] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] In the foregoing, specific details such as particular system architectures and techniques have been set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted to avoid unnecessary detail from obscuring the description of this application.

[0146] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0147] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0148] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0149] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0150] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0151] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method, characterized by, The method is applied to a computer device, which runs a data processing system. The data processing system includes a database, a cache, and cache middleware. The cache stores first business data, and the database stores second business data. The access frequency of the first business data is lower than the access frequency of the second business data. The method includes: The cache middleware is used to obtain the target business data corresponding to the business requests received within a preset time range. The caching middleware determines a data storage strategy based on the access characteristics corresponding to the target business data and in combination with preset rules. The data storage strategy is used to determine at least one of the storage methods of the first business data or the storage methods of the second business data. Update at least one of the first business data in the cache or the second business data in the database based on the data storage strategy.

2. The method of claim 1, wherein, The preset rules include cache hit rate rules; The step of determining a data storage strategy based on the access characteristics corresponding to the target business data and in combination with preset rules through the caching middleware includes: The cache middleware determines a first hit rate of the first business data based on the access characteristics. The first hit rate is used to indicate the number of times the first business data is the target business data. The first business data includes the i-th business data, where i is a positive integer. If the first hit rate corresponding to the i-th service data is lower than the first hit rate threshold, the i-th service data is removed from the cache.

3. The method of claim 2, wherein, The method further includes: If the i-th business data is not stored in the database, the i-th business data will be stored in the database.

4. The method according to any one of claims 1 to 3, characterized in that, The preset rules include database load rules; The step of determining a data storage strategy based on the access characteristics corresponding to the target business data and in combination with preset rules through the caching middleware includes: If the target business data does not belong to the first business data, the second hit rate of the second business data is determined by the caching middleware based on the access characteristics. The second hit rate is used to indicate the number of times the second business data is the target business data, wherein the second business data includes the nth business data, and n is a positive integer. If the second hit rate of the nth service data is higher than the second hit rate threshold, the nth service data will be stored in the cache.

5. The method of claim 4, wherein, The method further includes: Extend the storage duration of the nth business data in the cache.

6. The method according to any one of claims 1 to 3, characterized in that, The preset rules include node storage adjustment rules; the cache has a first cache node, which stores the m-th business data, where m is a positive integer; The step of determining a data storage strategy based on the access characteristics corresponding to the target business data and in combination with preset rules through the caching middleware includes: The cache middleware determines the third hit rate corresponding to the m-th business data based on the access characteristics, and the third hit rate is used to indicate the number of times the m-th business data is the target business data; If the memory corresponding to the first cache node is lower than a preset memory threshold, and the third hit rate is lower than a third hit rate threshold, the m-th service data is removed from the cache.

7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Acquire hotspot data within a historical time range and acquire historical business data within the historical time range. Hotspot data refers to business data whose access frequency is higher than a preset access threshold, and historical business data refers to business data that does not belong to at least one of the first business data or the second business data. The hot data is matched with the historical business data, and the historical data with a matching degree higher than a preset matching degree threshold is cached in the cache or the database, or at least one of them.

8. The method according to claim 7, characterized in that, The cache includes multiple cache nodes; The method further includes: Based on the distribution of time periods within the historical time range, the start and stop results of the multiple cache nodes are adjusted.

9. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the target business data corresponding to the business requests received within a preset time range through the caching middleware. The cache stores the first business data and the database stores the second business data. The access frequency of the first business data is lower than the access frequency of the second business data. The determination module is used to determine a data storage strategy based on the access characteristics corresponding to the target business data through the caching middleware and in combination with preset rules. The data storage strategy is used to determine at least one of the storage methods of the first business data or the storage methods of the second business data. The update module is used to update at least one of the first business data in the cache or the second business data in the database based on the data storage strategy.

10. A computer device comprising a memory, one or more processors, and a computer program stored in the memory and executable on the one or more processors, characterized in that, When the one or more processors execute the computer program, the computer device causes the computer device to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 8.

Images