Data processing method and device, equipment and storage medium

Abstract

This application discloses a data processing method, apparatus, device, and storage medium, applicable to the field of computer technology. The method includes: obtaining a first request; determining the target media data requested by the first request; in response to the presence of non-expired target media data in a data cache and the target media data being hot media data, routing the first request to a first cluster replica, and returning the target media data to the requester of the first request through the first cluster replica; wherein the first cluster replica is any cluster replica among the cluster shards in the data cache that cache the target media data, the data cache includes at least one cluster shard, different cluster shards cache different media data, each cluster shard caches media data through at least one cluster replica, and the hot media data cached by each cluster replica in each cluster shard is the same. Using this application embodiment can improve media data processing efficiency and has high applicability.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of computer technology, the dissemination of media data is becoming increasingly widespread. To achieve rapid data access, existing technologies typically rely on consistent hashing to store and access cached data. However, in scenarios with high concurrency access requests, existing mechanisms are prone to problems such as cache breakdown, cache penetration, and cache avalanche, which in turn reduces the efficiency of data access and retrieval. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and storage medium, which can improve the efficiency of media data acquisition and has high applicability.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising: Obtain the first request and determine the target media data requested in the first request. In response to the fact that the target media data is cached in the data cache and is not expired, and the target media data is hot media data, the first request is routed to the first cluster replica, and the target media data is returned to the requester of the first request through the first cluster replica. The first cluster replica is any one of the cluster shards in the data cache that caches the target media data. The data cache includes at least one cluster shard, and different cluster shards cache different media data. Each cluster shard caches media data through at least one cluster replica, and the hot media data cached by each cluster replica in each cluster shard is the same.

[0005] On the other hand, embodiments of this application provide a data processing apparatus, which includes: The request acquisition module is used to acquire a first request and determine the target media data requested by the first request. The data processing module is used to respond to the fact that there is unexpired target media data cached in the data cache and the target media data is hot media data, to route the first request to the first cluster replica, and to return the target media data to the requester of the first request through the first cluster replica; The first cluster replica is any one of the cluster shards in the data cache that caches the target media data. The data cache includes at least one cluster shard, and different cluster shards cache different media data. Each cluster shard caches media data through at least one cluster replica, and the hot media data cached by each cluster replica in each cluster shard is the same.

[0006] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory, which are interconnected; The aforementioned memory is used to store computer programs; The processor described above is used to execute the data processing method provided in the embodiments of this application when the computer program described above is invoked.

[0007] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the data processing method provided in embodiments of this application.

[0008] On the other hand, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the data processing method provided in embodiments of this application.

[0009] In this embodiment, by caching hot media data across multiple cluster replicas, requests for hot media data retrieval can be responded to quickly. Since the hot media data is already cached, the target media data can be directly returned through the cluster replicas within the cluster shards, significantly reducing data retrieval latency and avoiding frequent database accesses. This reduces database load and pressure, improving overall system availability. Furthermore, because the hot media data within each cluster shard is repeatedly stored across multiple cluster shards, random request routing effectively distributes request traffic, preventing a single cluster replica from becoming overloaded due to excessive requests. Simultaneously, the deployment of multiple cluster replicas enhances data redundancy; even if one replica fails, other cluster replicas can still provide the same hot media data, improving the fault tolerance of the caching system. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1This is a schematic diagram of the network structure provided in an embodiment of this application; Figure 2 This is a schematic diagram of data caching provided in an embodiment of this application; Figure 3 This is a schematic diagram of the functional architecture of the caching system provided in the embodiments of this application; Figure 4 This is one of the flowcharts illustrating the data processing method provided in the embodiments of this application; Figure 5 This is a second schematic flowchart of the data processing method provided in the embodiments of this application; Figure 6 This is a schematic diagram of concurrent request processing provided in an embodiment of this application; Figure 7 This is a schematic diagram of data diversion processing provided in an embodiment of this application; Figure 8 This is a schematic diagram of a data backup scenario provided in an embodiment of this application; Figure 9 This is a schematic diagram of the caching architecture provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] This application provides a data processing method that can be applied to the field of computer technology, specifically to the field of media data processing. It can effectively prevent problems such as cache penetration, cache avalanche, and cache breakdown, thereby improving data processing efficiency and enhancing the stability of media data services.

[0014] Cache penetration occurs when the requested media data is neither in the data cache nor in the database used to store media data. This is a problem that can arise in existing caching mechanisms. It happens when the requested media data is neither in the data cache nor in the backend database. Because the data cache cannot be accessed, all requests will directly query the database, potentially overwhelming the database with a large number of invalid requests and thus impacting media service performance.

[0015] Cache avalanche refers to a situation where, when using data caching, a valid expiration time is typically set for cached media data. However, if a large number of frequently accessed media data in the cache are given the same valid expiration time, this can cause a simultaneous expiration or restart of the cached data at a certain moment. Consequently, all requests related to this frequently accessed media data directly access the backend database, leading to a surge in database pressure, potentially causing a system crash, and ultimately triggering a chain reaction that results in system failure.

[0016] Cache breakdown refers to a situation where, when using data caching, a popular media data in the cache becomes invalid or is deleted, and at the same time, a large number of requests simultaneously request to retrieve that media data. In this case, the requests will directly access the backend database, causing excessive pressure on the backend system, slowing down the response, or even causing it to crash.

[0017] See Figure 1 , Figure 1 This is a schematic diagram of the network structure provided in an embodiment of this application. For example... Figure 1 As shown, any requester can initiate a first request to the caching system 30 through the terminal device 10 to request the acquisition of target media data. The target media data requested in any request is the media data that the requester expects to obtain. For the caching system 30, after receiving the first request, it needs to determine whether the data cache 20 contains any expired target media data, and whether the target media data is hot media data.

[0018] If data cache 20 has non-expired target media data and the target media data is hot media data, then the first request will be routed to any cluster replica of the first cluster shard in the first cluster shard of data cache 20 (for ease of description, it will be referred to as the first cluster replica).

[0019] The first cluster shard is the cluster shard in data cache 20 that caches the target media data.

[0020] The data cache 20 includes at least one cluster shard. Different clusters allocate different media data for caching. Each cluster shard caches media data through at least one cluster replica, and the hot media data cached by each cluster replica in each cluster shard is the same.

[0021] Based on this, the target media data can be returned to the requester of the first request through the first cluster replica.

[0022] In cluster sharding, each shard consists of several cluster replicas (nodes). Cluster sharding refers to dividing a large dataset or workload into smaller parts and then distributing these parts to different nodes or servers in the cluster for processing. This technique helps improve system performance, scalability, and fault tolerance because the workload can be distributed, thus avoiding excessive pressure on a single node or making it a single point of failure. Even if one node fails, other nodes can still continue to process the remaining shard data.

[0023] For cluster replicas, each replica consists of several nodes. Cluster replicas are a process of redundantly storing media data on one or more nodes to ensure data reliability and availability. For frequently accessed media data, each node's media data replica is a complete copy of the original media data and can independently provide service. When a node fails, replicas on other nodes can take over its service, thus ensuring system continuity and stability.

[0024] In other words, for each cluster shard, different nodes that cache the same hot media data can be called different cluster replicas.

[0025] See Figure 2 , Figure 2 This is a schematic diagram of data caching provided in an embodiment of this application. For example... Figure 2 As shown, the data cache in this embodiment can be regarded as a cluster, which includes multiple cluster shards, such as cluster shard A1, cluster shard A2, ..., cluster shard An.

[0026] Among them, the media data cached between all cluster shards from cluster shard A1 to cluster shard An is different, and the media data cached by cluster shards from cluster shard A1 to cluster shard An constitutes the full media data of the data cache (cluster).

[0027] For each cluster shard, taking cluster shard A1 as an example, cluster shard A1 also includes multiple nodes (node ​​1 to node n). Each node can be regarded as a cluster replica of cluster shard A1, and the hot media data cached by nodes 1 to n is the same. Other media data cached besides the hot media data can be the same or different.

[0028] After receiving the first request, assuming that cluster shard A2 has cached the target media data requested by the first request, and that the target media data is hot media data, the caching system 30 can route the first request to any cluster replica in cluster shard A2, such as node 1 in cluster shard A2. Node 1 in cluster shard A2 receives and processes the first request, so as to return the target media data to the requester of the first request through node 1 in cluster shard A2.

[0029] A cluster refers to a large computer service system that connects multiple nodes together through a high-speed communication network. The aggregation of these nodes can improve the stability, performance, and availability of the system.

[0030] The functional architecture of the aforementioned cache system 30 can be found in [reference needed]. Figure 3 As shown, Figure 3 This is a schematic diagram of the functional architecture of the caching system provided in an embodiment of this application. Figure 3 In this system, the caching system can realize a data platform for data source management, data aggregation, and aggregated caching. It can be used to process media data for services related to long videos, short videos, communities, live streaming, living rooms, and memberships.

[0031] The caching system directly serves read operations. When a requester initiates a request to retrieve media data, the caching system can attempt to provide the media data from the cache, reducing access to the backend database. Caching frequently accessed media data helps improve the speed of media data retrieval, reduces database load, and provides a degree of data redundancy and fault tolerance in the event of a backend database failure.

[0032] Besides primarily serving read operations, the caching system also has indirect connections with write operations. Write operations are responsible for writing data to the backend database, while the caching system is responsible for caching some frequently accessed media data from the database. Based on this, cache preheating can be implemented. For frequently accessed media data, write operations can proactively load this data into the cache to improve the efficiency of subsequent retrieval of this data.

[0033] The caching system can support plug-in extensions, allowing plug-in providers to develop and integrate new functions or optimize existing functions. This makes it easy to add new caching strategies, data analysis and other functions through plug-ins, and easily adapt to different application scenarios and needs.

[0034] Caching systems can interact with message subscribers through message queues or event notification mechanisms. When hot media data in the data cache changes (such as data being updated, deleted, or expired), the caching system can send messages to subscribers so that they can take appropriate actions, thereby ensuring that message subscribers can obtain information about changes to cached data in a timely manner.

[0035] In the embodiments of this application, the terminal device 10 may be a smartphone, tablet computer, laptop computer, desktop computer (including monitor), smartwatch, vehicle terminal, aircraft, smart home appliance (e.g., smart TV) or wearable device with display function.

[0036] In the embodiments of this application, the nodes involved in cluster sharding and cluster replicas can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0037] It should be noted that the data processing method provided in this application embodiment can be executed by the backend management platform that provides media data services, or by the caching system that runs data caching, or by other control platforms or servers that provide media data services. The specific method can be determined based on the actual application scenario requirements and is not limited here.

[0038] For example, the data processing method provided in this application embodiment can be executed by a backend management platform that provides media data services, such as a video service management platform, which is deployed with... Figure 1 The caching system 30 shown is used to cache hot video data and provide hot video data to the requester.

[0039] For example, the data processing method provided in the embodiments of this application can be executed by a caching system that runs a data cache, such as... Figure 1 The caching system 30 shown can be a caching system for short video platforms, a caching system for news clients, etc., used to cache and provide hot media data.

[0040] See Figure 4 , Figure 4 This is one of the flowcharts illustrating the data processing method provided in the embodiments of this application. For example... Figure 4 As shown, the data processing method provided in this application embodiment may include the following steps: S41, Obtain the first request and determine the target media data requested by the first request.

[0041] In some feasible implementations, after obtaining the first request, the target unique identifier of the target media data to be obtained can be determined through the first request, and then the target unique identifier can be used to determine the specific target media data requested by the first request.

[0042] Among them, the unique identifier for any media data can be the Content ID, which can be a number or a string, used to uniquely identify the media data.

[0043] Optionally, the unique identifier for any media data can be a Uniform Resource Identifier (URI) or a Uniform Resource Locator (URL). URIs and URLs typically include information such as protocol, domain name, and path, directly pointing to the storage or access location of the media file.

[0044] Optionally, the unique identifier for any media data can be a hash value calculated based on the data information of the media data, including but not limited to the title and episode number. For example, the hash value obtained by hashing the TV series name and episode number can be used to obtain the unique identifier of the media data for that episode of the TV series.

[0045] Optionally, the unique identifier for any media data can be a combination of the corresponding metadata: including elements such as title, edition number, and episode number, forming a relatively concise unique identifier.

[0046] Optionally, the unique identifier for any media data can be the data cache or the primary key value of the media data in the database.

[0047] It should be noted that the specific form of the aforementioned unique identifier can be determined based on the actual application scenario requirements, and is not restricted here. After obtaining the first request, the target unique identifier can be directly determined based on the first request, or the target unique identifier carried in the first request can be parsed to obtain the target unique identifier, and the media data corresponding to the determined target unique identifier can be identified as the target media data requested by the first request.

[0048] In this embodiment of the application, the data cache and the storage format of the media data in the backend database used to store the media data can be stored in the form of key-value pairs, where the key is a unique identifier of the media data and the value is the media data.

[0049] S42, in response to the fact that the target media data is cached in the data cache and is not expired and is hot media data, the first request is routed to the first cluster replica and the target media data is returned to the requester of the first request through the first cluster replica.

[0050] In some feasible implementations, after obtaining the first request and determining the target media data requested by the first request, it can be determined whether there is any expired target media data cached in the data cache, and whether the target media data belongs to hot media data.

[0051] Specifically, to determine whether the data cache contains valid target media data, we first need to determine if the data cache already contains such data. If it is determined that the data cache contains the target media data, we further determine the validity period of the cached target media data and compare the current time (i.e., the time of retrieval of the first request) with the validity period of the cached target media data. If the current time is within the validity period of the cached target media data, it can be determined that the data cache contains valid target media data; if the current time is outside the validity period of the cached target media data, it can be determined that the cached target media data has expired.

[0052] Specifically, a data table can be pre-stored in each cluster shard of the data cache. The data table is used to indicate the validity period of the media data in the corresponding cluster shard, and the specific storage and indication method is not limited here. For example, each data table stores the unique identifier and validity period of each media data in the corresponding cluster shard, and the unique identifier and validity period of each media data are stored in key-value format, where the key is the unique identifier and the value is the validity period.

[0053] Based on this, after determining the target media data requested in the first request, it is possible to query whether a target unique identifier for the target media data exists in the data tables of each cluster shard. If the target unique identifier for the target media data does not exist in the data tables of any cluster shard, it is determined that the target media data is not cached in the data cache. If the target unique identifier for the target media data exists in the data tables of any cluster shard, it is determined that the target media data is cached in the data cache, and it is further determined whether the current time is within the valid time corresponding to the target unique identifier in the data table. If the current time is within the valid time corresponding to the target unique identifier in the data table, it is determined that the target media data cached in the data cache is valid. If the current time is outside the valid time corresponding to the target unique identifier in the data table, it is determined that the target media data cached in the data cache is invalid.

[0054] It should be noted that for hot media data, the validity period of hot media data is the same in each cluster replica of the same cluster shard.

[0055] Specifically, when determining whether target media data belongs to popular media data, the target unique identifier of the target media data can be compared with each unique identifier in a preset list. If the target unique identifier of the target media data exists in the preset list, the target media data is determined to belong to popular media data; if the target unique identifier of the target media data does not exist in the preset list, the target media data is determined not to belong to popular media data.

[0056] The aforementioned preset list can be stored in any location, such as a backend database, or in any or all cluster shards. The specific location can be determined based on the actual application scenario requirements, and no restrictions are imposed here.

[0057] The aforementioned preset list stores unique identifiers for all hot media data cached in the data cache, and these unique identifiers change as the hot media data in the data cache changes. For example, when a hot media data is deleted from the data cache, its unique identifier is deleted from the preset list; when a hot media data is added to the data cache, its unique identifier is added to the preset list.

[0058] Alternatively, a data table can be pre-stored in each cluster shard of the data cache. This table indicates the validity period of media data within the corresponding cluster shard and whether it belongs to the "hot" media data category. The specific storage and indication methods are not limited here. For example, each data table can associate the unique identifier, validity period, and type value of each media data item. A first type value indicates that the corresponding media data belongs to the "hot" media data category, while a second type value indicates that the corresponding media data does not belong to the "hot" media data category.

[0059] The newly added trending media data can include newly released episodes of popular dramas, and there are no restrictions on this.

[0060] Based on this, after determining the target media data requested in the first request, the system can query the data tables in each cluster shard to check if a target unique identifier for the target media data exists. If the target unique identifier for the target media data does not exist in the data tables of any cluster shard, it is determined that the target media data is not cached in the data cache. If the target unique identifier for the target media data exists in the data tables of any cluster shard, it is determined that the target media data is cached in the data cache. Furthermore, based on the type value corresponding to the target media data in the data tables, it is determined whether the target media data belongs to hot media data. If it is determined that the target media data belongs to hot media data, it is further determined whether the current time is within the valid time corresponding to the target unique identifier in the data table. If the current time is within the valid time corresponding to the target unique identifier in the data table, it is determined that the target media data cached in the data cache is valid. If the current time is outside the valid time corresponding to the target unique identifier in the data table, it is determined that the target media data cached in the data cache is invalid.

[0061] In some feasible implementations, the hot media data in the embodiments of this application are media data whose browsing indicators meet the first indicator condition.

[0062] Among these, browsing metrics include, but are not limited to, number of views, browsing duration, average viewing duration, completion rate, number of comments, number of likes, number of favorites, etc. The specific metrics can be determined based on the actual application scenario requirements and are not limited here.

[0063] Optionally, the browsing metrics that meet the first metric condition may include any one or more of the following: The number of views exceeded the preset limit; The number of views ranks in the top 1% (K1%) of all media data in the database. The browsing time exceeded the preset browsing time. The browsing duration ranks in the top 2% (K2%) of browsing duration across all media data in the database; The average viewing time exceeded the preset average viewing time. The average viewing time ranks in the top 3% (K3%) of the average viewing time across all media data in the database. The completion rate exceeded the preset completion rate; The completion rate ranks in the top 4 percent among all media data in the database; The number of comments exceeded the preset number of comments; The number of comments ranks in the top 5% of the comment counts across all media data in the database; The number of likes exceeded the preset number of likes; The number of likes ranks in the top 6% of likes across all media data in the database. The number of favorites exceeds the preset number; The number of collections ranks in the top 7% among all media data in the database; The number of shares exceeded the preset number of shares; The number of shares ranks in the top 8 percent among all media data in the database.

[0064] Among them, K1-K8 are positive numbers, and the specific values ​​can be determined based on the actual application scenario requirements, without any restrictions here.

[0065] In other words, the hot media data in this application embodiment refers to media data with high attention, such as popular TV dramas and popular short videos, and there are no restrictions here.

[0066] In this embodiment of the application, every first time interval, the browsing index of each piece of media data in the database used to store media data (i.e., the backend database) can be determined, and based on the browsing index of each piece of media data in the database, it can be determined whether there is first media data in the database that has been converted into hot media data.

[0067] Specifically, when determining whether there is first media data that has turned into hot media data in the database based on the browsing metrics of each media data item in the database, if there is media data in the database whose browsing metrics meet the first metric condition, it can be determined that the media data will turn from non-hot media data into hot media data after the current first time interval, and thus it can be identified as the first media data.

[0068] Based on this, browsing metrics for each media data item in the database can be continuously evaluated, thereby identifying hot media data and adding it to the data cache. This avoids a large number of requests directly accessing the backend database when the data cache does not store newly added hot media data, which could cause system crashes. It also improves the efficiency of acquiring hot media data.

[0069] In some feasible implementations, if it is determined that the target media data is cached in the data cache and the media data is hot media data, since each cluster replica of the cluster shard that caches the hot media data also caches the hot media data, the first request can be routed to any one of the cluster replicas in the cluster shard that caches the target media data (hereinafter referred to as the first cluster replica for ease of description), and then the target media data can be returned to the request method of the first request through the first cluster replica, thereby completing the request and return of the target media data.

[0070] Based on this, when multiple requests for obtaining the same hot media data are received, each request can be randomly routed to any cluster replica within the corresponding cluster shard to return the requested hot media data to the corresponding requester. This avoids multiple requests accessing the same cluster replica at the same time, which can cause request congestion and help ensure balanced cluster traffic while improving the resource utilization of cluster replicas.

[0071] See Figure 5 , Figure 5 This is a second schematic flowchart of the data processing method provided in the embodiments of this application. Figure 5 As shown, the data processing method provided in this application embodiment may include the following steps: S51, Obtain the first request and determine the target media data requested by the first request.

[0072] For some feasible implementations, see the specific implementation of S51. Figure 4 The implementation shown in S41 will not be elaborated here.

[0073] S52, in response to the fact that the target media data is cached in the data cache and is not hot media data, the second cluster replica for processing the first request is determined from the first cluster shard according to the first request and the consistent hash algorithm, and the target media data is returned to the request of the first request through the second cluster replica.

[0074] In some feasible implementations, the specific implementation methods for determining whether the data cache contains expired target media data and whether the target media data belongs to hot media data can be found in [reference needed]. Figure 4 The implementation shown in S42 will not be elaborated here.

[0075] In some feasible implementations, non-hotspot media data can be cached in different cluster shards, and each cluster shard caches non-hotspot media data through a separate cluster replica.

[0076] In this case, after determining the target unique identifier of the target media data according to the first request, the first cluster fragment for caching the target media data can be directly determined based on the data table and data list. Then, the cluster replica in the first cluster fragment used to store non-hot media data separately is determined as the second cluster replica for processing the first request. The target media data is then returned to the requester of the first request through the second cluster replica.

[0077] Optionally, non-hot media data can be cached in different cluster shards, and each cluster shard can cache non-hot media data through multiple separate cluster replicas, with different media data cached by each cluster replica used to cache non-hot media data in each cluster shard.

[0078] In this case, after determining the target unique identifier in addition to the target media data according to the first request, the first cluster fragment containing the cached target media data can be determined based on the target unique identifier.

[0079] Furthermore, the target unique identifier can be hashed to obtain the target hash value. Based on the target hash value, the second cluster replica in the first cluster shard used to store the target media data can be determined, and then the target media data can be returned to the requester of the first request through the second cluster replica.

[0080] In other words, for each cluster shard, when caching each non-hot media data item, a hash value can be obtained by hashing the unique identifier of the media data. Based on the hash value, the cluster replica in the cluster shard used to cache the media data can be determined, and the media data can be cached.

[0081] In some feasible implementations, after obtaining the first request and determining the target media data requested by the first request, if the data cache meets the first condition, it is determined that the target media data cannot be returned to the requester of the first request through the data cache.

[0082] In this case, the first request can be routed to a database used to store media data. If the database contains the target media data, the target media data can be returned to the requester of the first request through the database.

[0083] Among them, the data cache meets the first condition, including at least one of the following: The target media data in the data cache has become invalid. The target media data is not cached in the data cache; The target media data in the data cache is being updated.

[0084] As an example, after obtaining the first request and determining the target media data requested by the first request, if the target media data is cached in the data cache, but the current time is outside the validity period of the target media data, the first request can be routed to the database used to store the media data. If the database stores the target media data, the target media data can be returned to the requester of the first request through the database.

[0085] As an example, after obtaining the first request and determining the target media data requested by the first request, if the target media data is not cached in the data cache, the first request can be routed to the database used to store the media data. If the database stores the target media data, the target media data can be returned to the requester of the first request through the database.

[0086] As an example, after obtaining the first request and determining the target media data requested by the first request, if the target media data is cached in the data cache but is being updated, it means that the data cache cannot provide data services related to the target media data. In this case, the first request can be routed to the database used to store the media data. If the database stores the target media data, the target media data can be returned to the requester of the first request through the database.

[0087] In some feasible implementations, if the data cache meets the first condition, and at least one second request for obtaining target media data is obtained simultaneously with the first request, i.e., when multiple requests simultaneously request the target media data, the requests can compete for the request lock. That is, if the first request and each of the second requests successfully compete for the target request, the target request is routed to the database used to store the media data. If the database stores the target media data, the target media data is returned to the requester of the first request through the database.

[0088] Meanwhile, after returning the target media data to the requester of the target request, since other requests have not yet obtained the target media data, the data cache can be updated based on the target media data. That is, the target media data is added to the data cache so that the first request and other requests in each of the second requests, except for the target request, can directly obtain the target media data from the data cache.

[0089] In other words, for the first request and other requests in each of the second requests besides the target request, the request can be routed to any cluster replica of the cluster shard storing the target media data, and the target media data can be returned to the corresponding requester through that cluster replica.

[0090] When adding target media data to the data cache, the hash value of the target unique identifier of the target media data can be determined based on the same consistent hashing method. The cluster shard used to cache the target media data is determined based on the hash value, and the target media data is cached in the cluster shard.

[0091] Combination Figure 6 , Figure 6 This is a schematic diagram of concurrent request processing provided in an embodiment of this application. For example... Figure 6 As shown, assume that requests 1 to m (where m is an integer greater than 1) are obtained, and all requests 1 to m are used to retrieve target media data. If the target media data is not cached in the data cache, or if the cached target media data is invalid, or if the target media data in the data cache is being updated, and if the database is determined to contain the target media data based on its unique identifier, requests 1 to m can compete for the request lock.

[0092] If request s (where s is an integer from 1 to m) successfully acquires the request lock, request s can be routed to the database, and the target media data can be returned to the requester corresponding to request s through the database. At the same time, the data cache is updated based on the target media data.

[0093] For each request other than request s, the request can be randomly routed to any cluster replica of the cluster shard that stores the target media data in the data cache, and the target media data can be returned to the corresponding requester through that cluster replica.

[0094] Based on request locking, in the case of multiple concurrent requests, updating the cluster shards can be completed with only one database request. This helps to reduce the pressure on the database, avoid cache breakdown issues in the cluster shards, and ensure the availability of media services.

[0095] In some feasible implementations, if only the first request is obtained and it is determined that the data cache meets the first condition, the first request can be routed to the database used to store media data, and if the database does not store the target media data, the first information can be returned to the requester of the first request through the database.

[0096] The first piece of information indicates that the acquisition of the target media data failed.

[0097] Subsequently, when other requests for retrieving target media data are received, the system can return the first information to the corresponding requester based on the data identifier of the target media data in the data cache, so as to indicate that the requester has failed to retrieve the target media data.

[0098] The unique identifier of the target media data is stored in the data cache. This can be done using key-value pairs, where the key is the unique identifier and the value is an empty value. This is used to indicate media data for which the corresponding unique identifier is not stored in the data cache or the database.

[0099] If other requests for retrieving target media data are received, and it is determined that a unique identifier for the target media data exists in the data cache and the value is empty, the first information can be directly returned to the corresponding requester.

[0100] Optionally, if the data cache meets the first condition, and at least one second request for obtaining target media data is obtained simultaneously with the first request (i.e., when multiple requests simultaneously request the target media data), the requests can compete for the request lock. That is, if the first request and each of the second requests successfully compete for the target request, the target request is routed to the database used to store the media data. If the database does not store the target media data, the first information is returned to the requester of the first request through the database.

[0101] The first piece of information indicates that the acquisition of the target media data failed.

[0102] At the same time, after returning the first information to the requester of the target request, since other requests have not yet obtained the target media data, the first information can be returned directly to the requesters of other requests through data caching.

[0103] Subsequently, if other requests for retrieving target media data are received, and it is determined that a unique identifier for the target media data exists in the data cache and the value is empty, the first information can be directly returned to the corresponding requester.

[0104] Based on the above implementation method, cache penetration and cache breakdown can be effectively prevented, thereby improving the stability of media services and the response efficiency of requests for obtaining media data.

[0105] In some feasible implementations, media data that has been converted from trending media data to non-trending media data is considered media data that meets the second indicator condition.

[0106] Among these, browsing metrics include, but are not limited to, number of views, browsing duration, average viewing duration, completion rate, number of comments, number of likes, number of favorites, etc. The specific metrics can be determined based on the actual application scenario requirements and are not limited here.

[0107] Optionally, the browsing metric must meet the second metric condition, which may include any one or more of the following: The increase in pageviews was less than the preset increase. The number of views ranks in the bottom 1% (H1) of the total number of views for all media data in the database. The increase in browsing time is less than the preset increase time. Browsing duration is ranked in the bottom 2% of browsing duration for all media data in the database. The average viewing time was less than the preset average viewing time. The average viewing time ranks in the bottom 3% (H3) of the average viewing time across all media data in the database. The completion rate is lower than the preset completion rate; The completion rate ranks in the bottom 4 percent among all media data completion rates in the database; The increase in the number of comments is less than the preset increase in the number of comments; The number of comments ranks in the bottom 5 percent of the comment counts for each media category in the database. The increase in likes is less than the preset increase in likes; The number of likes ranks in the bottom 6% of the likes across all media data in the database. The increase in the number of collections is less than the preset increase in the number of collections. The number of collections ranks in the bottom 7% among all media data in the database. The number of shares increased less than the preset number of shares. The number of shares ranks in the bottom 8 percent among all media data in the database.

[0108] Among them, H1-H8 are positive numbers, and the specific values ​​can be determined based on the actual application scenario requirements, without any restrictions here.

[0109] In other words, the media data that has been converted to non-hotspot media data in this application embodiment refers to media data that has received less attention and is no longer accessed frequently.

[0110] In this embodiment of the application, every second time interval, the browsing index of each media data in the data cache can be determined, and based on the browsing index of each media data in the data cache, it can be determined whether there is second media data in the data cache that has been converted to non-hot media data.

[0111] Specifically, when determining whether there is second media data that has been converted to non-hot media data in the data cache based on the browsing metrics of each media data item in the data cache, if there is media data in the data cache whose browsing metrics meet the second metric conditions, it can be determined that the media data will be converted from hot media data to non-hot media data after the current second time interval, and thus it can be identified as the second media data.

[0112] Based on this, the browsing metrics of each media data item in the data cache can be continuously evaluated, and non-hot media data can be identified and removed from the data cache to avoid occupying cache space.

[0113] In some feasible implementations, the data processing method provided in this application can also update the cached hot media data in real time. Specifically, update information can be determined in response to an update instruction for third media data in the data cache.

[0114] Among them, the third-party media data consists of trending media data that is waiting to be updated in the data cache.

[0115] The updated information for third-party media data can include version information (such as version number) and browsing privileges (such as VIP to non-VIP, non-VIP to VIP, etc.). The specific details can be determined based on the actual application scenario requirements and are not restricted here.

[0116] After obtaining the update information, the update information can be sent to each cluster replica in the cluster shard storing the third-party media data, so that each cluster replica can update the third-party media data in the cluster replica according to the update information, such as version number update, browsing rights update, etc.

[0117] Based on the above update method, it can be ensured that the cluster replicas of the same media data cached in the data cache are updated synchronously, which is conducive to maintaining the timeliness of the media data in the data cache.

[0118] See Figure 7 , Figure 7 This is a schematic diagram of data diversion processing provided in an embodiment of this application. For example... Figure 7 As shown, after obtaining multiple requests for retrieving hot media data, each request can be routed to a different cluster shard. Furthermore, for each cluster shard, the request can be routed to any cached replica to retrieve the hot media data. When an update instruction is given for a specific hot media data item, the update information can be routed to the corresponding cluster shard, and then further routed to each cluster replica of that shard, ensuring that each cluster replica can update the hot media data.

[0119] In some feasible implementations, combined with Figure 8 , Figure 8 This is a schematic diagram illustrating a data backup scenario provided in an embodiment of this application. For example... Figure 8 As shown, every third time interval, since the hot media data cached by each cluster replica in each cluster shard is the same, the hot media data cached by any cluster replica in each cluster shard can be backed up to the preset storage space and overwritten with the hot media data of the previous backup.

[0120] Based on this, when the data cache service restarts or the cache system is republished, the hot media data backed up in the preset storage space can be directly migrated to the data cache during the service interruption, thereby realizing the advance warm-up of the data cache and improving the data acquisition efficiency.

[0121] The aforementioned preset storage space includes, but is not limited to, cloud storage, object storage-based databases, blockchain, etc. The specific storage space can be determined based on the actual application scenario requirements and is not restricted here.

[0122] During the migration of backed-up hot media data, the version information of the hot media data can also be obtained, and the version information of the hot media data after the data cache migration is updated so that the hot media data in the data cache always keeps the latest version.

[0123] See Figure 9 , Figure 9 This is a schematic diagram of the caching architecture provided in an embodiment of this application. For example... Figure 9 As shown, the access layer can route requests for obtaining hot media data to the cluster replicas within the corresponding cache shards of the data cache, providing data access and retrieval services.

[0124] The caching layer can aggregate and cache different data sources to accelerate data access and retrieval, providing high-concurrency, low-latency data access services to the outside world.

[0125] The update service can send update information to the corresponding cluster replicas in the data cache when hot media data in the data cache is updated, so that the cluster replicas can update the data according to the update information (such as version number).

[0126] The central control service is responsible for regularly backing up the hot media data in the data cache, thereby enabling the data cache to be quickly warmed up when the cache service restarts.

[0127] The data cache can include multi-level caching, such as local caching based on Linux shared memory to ensure that data in each cluster shard cache is not lost, or distributed caching based on Redis to further back up the data in the data cache and prevent data loss.

[0128] In this embodiment, by caching hot media data across multiple cluster replicas, requests for hot media data retrieval can be responded to quickly. Since the hot media data is already cached, the target media data can be directly returned through the cluster replicas within the cluster shards, significantly reducing data retrieval latency and avoiding frequent database accesses. This reduces database load and pressure, improving overall system availability. Furthermore, because the hot media data within each cluster shard is repeatedly stored across multiple cluster shards, random request routing effectively distributes request traffic, preventing a single cluster replica from becoming overloaded due to excessive requests. Simultaneously, the deployment of multiple cluster replicas enhances data redundancy; even if one replica fails, other cluster replicas can still provide the same hot media data, improving the fault tolerance of the caching system.

[0129] At the same time, since hot media data is prone to updates, media data that has become hot media data can be added to the data cache periodically based on the browsing metrics of the media data. Media data that has become non-hot media data can also be removed from the data cache periodically based on the browsing metrics of the media data. This helps to prevent problems such as cache breakdown, cache penetration, and cache avalanche, and has high applicability.

[0130] See Figure 10 , Figure 10 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application. The data processing apparatus provided in the embodiments of this application includes: The request acquisition module 101 is used to acquire a first request and determine the target media data requested by the first request. Data processing module 102 is used to respond to the fact that there is unexpired target media data cached in the data cache and the target media data is hot media data, to route the first request to the first cluster replica, and to return the target media data to the requester of the first request through the first cluster replica; The first cluster replica is any one of the cluster shards in the data cache that caches the target media data. The data cache includes at least one cluster shard, and different cluster shards cache different media data. Each cluster shard caches media data through at least one cluster replica, and the hot media data cached by each cluster replica in each cluster shard is the same.

[0131] In some feasible implementations, the data processing module 102 described above is further used for: In response to the fact that the aforementioned data cache contains non-expired target media data and that the aforementioned target media data is not hot media data, a second cluster replica for processing the aforementioned first request is determined from the aforementioned first cluster shard based on the aforementioned first request and the consistent hashing algorithm. The target media data is returned to the requester of the first request via the second cluster replica.

[0132] In some feasible implementations, the data processing module 102 described above is further used for: In response to the above data cache meeting the first condition, the above first request is routed to the database used to store media data; In response to the fact that the aforementioned database stores the aforementioned target media data, the aforementioned target media data is returned to the requester of the aforementioned first request through the aforementioned database; The aforementioned data cache meets the first condition, including at least one of the following: The aforementioned target media data in the aforementioned data cache has become invalid; The aforementioned target media data is not cached in the aforementioned data cache; The target media data in the aforementioned data cache is currently being updated.

[0133] In some feasible implementations, the data processing module 102 described above is further used for: In response to the above data cache meeting the first condition, and at least one second request being obtained at the same time as the first request, the target request that successfully competed for the request lock among the first request and each of the second requests is determined; wherein each of the second requests is used to request the acquisition of the target media data; The target request is routed to the database used to store media data; in response to the database storing the target media data, the target media data is returned to the requester of the target request through the database, and the data cache is updated according to the target media data; For each request other than the target request mentioned above, the target media data is returned to the requester of that request through the data cache mentioned above.

[0134] In some feasible implementations, the data processing module 102 described above is further used for: In response to the fact that the database does not store the target media data, the unique identifier of the target media data is stored in the data cache, and first information is returned to the corresponding requester, the first information being used to indicate that the acquisition of the target media data failed. After returning the first information, for other requests to obtain the target media data, the first information is returned to the corresponding requester based on the data identifier of the target media data in the data cache.

[0135] In some feasible implementations, the data processing module 102 described above is further used for: Every first time interval, determine the browsing index of each media data item in the database used to store media data, and based on the browsing index of each media data item in the database, determine whether there is a first media data item in the database that has been converted into hot media data. In response to the existence of first media data that has been converted into hot media data in the aforementioned database, the first media data is added to each cluster replica of the cluster shard used to store the first media data.

[0136] In some feasible implementations, the data processing module 102 described above is further used for: Every second time interval, determine the browsing metrics for each media data item in the aforementioned data cache; Based on the browsing metrics of each media data item in the above data cache, determine whether there is any second media data in the above data cache that has been converted into non-hot media data; In response to the second media data that has been converted to non-hot media data in the aforementioned data cache, the aforementioned second media data is removed from the aforementioned data cache.

[0137] In some feasible implementations, the data processing module 102 described above is further used for: In response to an update instruction for the third media data in the aforementioned data cache, update information is determined; wherein the aforementioned third media data is hot media data; The aforementioned update information is sent to each cluster replica in the cluster shard storing the aforementioned third media data, so that each cluster replica in the cluster shard storing the aforementioned third media data updates the aforementioned third media data according to the aforementioned update information.

[0138] In some feasible implementations, the data processing module 102 described above is further used for: Every third time interval, the hot media data cached in any cluster replica in each cluster shard is backed up to the preset storage space, and the hot media data of the previous backup is overwritten. In response to the restart of the aforementioned data cache caching service, the hot media data backed up in the aforementioned preset storage space will be migrated to the aforementioned data cache.

[0139] In specific implementation, the aforementioned data processing device can perform the above-described actions through its built-in functional modules. Figure 4 and / or Figure 5 The implementation methods provided for each step are detailed in the above-mentioned implementation methods, and will not be repeated here.

[0140] See Figure 11 , Figure 11 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 11 As shown, the electronic device 1100 in this embodiment can be Figure 1 The device containing the caching system 30 can also be the aforementioned backend management platform for providing media data services or other caching systems based on data caching, or it can be related equipment or servers of other data platforms that provide media data services. The electronic device 1100 can be used to implement the data processing method provided in any embodiment of this application.

[0141] The electronic device 1100 in this embodiment may include a processor 1101, a network interface 1104, and a memory 1105. Furthermore, the electronic device 1100 may also include an object interface 1103 and at least one communication bus 1102. The communication bus 1102 is used to enable communication between these components. The object interface 1103 may include a display screen and a keyboard; optionally, the object interface 1103 may also include a standard wired interface or a wireless interface. The network interface 1104 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1105 may be a high-speed RAM or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory 1105 may also be at least one storage device located remotely from the processor 1101. Figure 11 As shown, the memory 1105, which is a computer-readable storage medium, may include an operating system, a network communication module, an object interface module, and a computer program.

[0142] exist Figure 11 In the illustrated electronic device 1100, the network interface 1104 provides network communication functionality; the object interface 1103 is mainly used to provide an input interface for objects; and the processor 1101 can be used to call the computer program stored in the memory 1105 to implement the media data processing method provided in any embodiment of this application, such as... Figure 4 and / or Figure 5 The implementation methods provided for each step are detailed in the above-mentioned implementation methods, and will not be repeated here.

[0143] It should be understood that in some feasible implementations, the processor 1101 described above may be a central processing unit (CPU), which may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0144] In specific implementation, the aforementioned electronic device 1100 can perform the above-described actions through its built-in functional modules. Figure 4 and / or Figure 5 The implementation methods provided for each step are detailed in the above-mentioned implementation methods, and will not be repeated here.

[0145] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement... Figure 4 and / or Figure 5 The methods provided in each step are detailed in the implementation methods provided in the above steps, and will not be repeated here.

[0146] The aforementioned computer-readable storage medium can be an internal storage unit of the data processing apparatus or electronic device provided in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. The aforementioned computer-readable storage medium can also include magnetic disks, optical disks, read-only memory (ROM), or random access memory (RAM), etc. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0147] This application provides a computer program product, which includes a computer program that is executed by a processor. Figure 4 and / or Figure 5 The methods provided for each step in the process.

[0148] The terms "first," "second," etc., used in the claims, description, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or electronic device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or electronic devices. References to "embodiment" herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The presentation of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and / or" as used in this application's description and appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0149] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

[0150] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the first request and determine the target media data requested by the first request; In response to the presence of non-expired target media data in the data cache and the target media data being hot media data, the first request is routed to the first cluster replica, and the target media data is returned to the requester of the first request through the first cluster replica; Wherein, the first cluster replica is any one of the cluster shards in the data cache that caches the target media data. The data cache includes at least one cluster shard, and different cluster shards cache different media data. Each cluster shard caches media data through at least one cluster replica, and the hot media data cached by each cluster replica in each cluster shard is the same.

2. The method according to claim 1, characterized in that, The method further includes: In response to the fact that the data cache contains unexpired target media data and the target media data is not hot media data, a second cluster replica for processing the first request is determined from the first cluster shard according to the first request and the consistent hashing algorithm. The target media data is returned to the requester of the first request via the second cluster replica.

3. The method according to claim 1, characterized in that, The method further includes: In response to the data cache meeting the first condition, the first request is routed to the database used to store media data; In response to the fact that the database stores the target media data, the target media data is returned to the requester of the first request through the database; The data cache meets the first condition, including at least one of the following: The target media data in the data cache has become invalid. The target media data is not cached in the data cache. The target media data in the data cache is in an updated state.

4. The method according to claim 3, characterized in that, The method further includes: In response to the data cache meeting the first condition, and at least one second request being obtained simultaneously with the first request, the target request that successfully competed for the request lock among the first request and each of the second requests is determined; wherein each second request is used to request the acquisition of the target media data; The target request is routed to a database for storing media data; in response to the database storing the target media data, the target media data is returned to the requester of the target request through the database, and the data cache is updated according to the target media data; For each request other than the target request, the target media data is returned to the requester of that request through the data cache.

5. The method according to claim 3 or 4, characterized in that, The method further includes: In response to the database not storing the target media data, the unique identifier of the target media data is stored in the data cache, and first information is returned to the corresponding requester, the first information being used to indicate that the acquisition of the target media data failed; After returning the first information, for other requests to obtain the target media data, the first information is returned to the corresponding requester based on the data identifier of the target media data in the data cache.

6. The method according to claim 1, characterized in that, The method further includes: Every first time interval, the browsing index of each media data item in the database used to store media data is determined, and based on the browsing index of each media data item in the database, it is determined whether there is a first media data item in the database that has been converted into hot media data. In response to the existence of first media data in the database that has been converted into hot media data, the first media data is added to each cluster replica of the cluster shard used to store the first media data.

7. The method according to claim 1, characterized in that, The method further includes: Every second time interval, the browsing metrics for each piece of media data in the data cache are determined; Based on the browsing metrics of each media data item in the data cache, determine whether there is any second media data in the data cache that has been converted to non-hotspot media data; In response to the second media data in the data cache being converted to non-hotspot media data, the second media data is removed from the data cache.

8. The method according to claim 1, characterized in that, The method further includes: In response to an update instruction for third media data in the data cache, update information is determined; wherein the third media data is hot media data; The update information is sent to each cluster replica in the cluster shard storing the third media data, so that each cluster replica in the cluster shard storing the third media data updates the third media data according to the update information.

9. The method according to claim 1, characterized in that, The method further includes: Every third time interval, the hot media data cached in any cluster replica in each cluster shard is backed up to the preset storage space, and the hot media data of the previous backup is overwritten. In response to the restart of the cache service of the data cache, the hot media data backed up in the preset storage space is migrated to the data cache.

10. A data processing apparatus, characterized in that, The device includes: The request acquisition module is used to acquire a first request and determine the target media data requested by the first request. The data processing module is used to respond to the fact that there is unexpired target media data cached in the data cache and the target media data belongs to hot media data, to route the first request to the first cluster replica, and to return the target media data to the requester of the first request through the first cluster replica; Wherein, the first cluster replica is any one of the cluster shards in the data cache that caches the target media data. The data cache includes at least one cluster shard, and different cluster shards cache different media data. Each cluster shard caches media data through at least one cluster replica, and the hot media data cached by each cluster replica in each cluster shard is the same.

11. An electronic device, characterized in that, It includes a processor and a memory, which are interconnected; The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 9 when the computer program is invoked.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images