Data processing method and device, routing device, and storage medium

Abstract

Embodiments of the present application disclose a data processing method and device, a routing device and a storage medium, wherein the method comprises: receiving an acquisition request of a data invoker for target object data, and querying target routing information of the target object data; if the target routing information is queried, sending a device identifier of a first node device in a node device set to the data invoker, so that the data invoker acquires the target object data locally from the first node device; if the target routing information is not queried, sending a device identifier of a second node device in the node device set to the data invoker, so that the data invoker acquires the target object data through the second node device; the target object data is acquired from an object storage engine by the second node device and sent to the data invoker, and the second node device caches the target object data locally after acquiring the target object data, so that the data pressure of an underlying storage device is reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, routing device, and storage medium. Background Technology

[0002] With the continuous development of computer technology, the amount of data on the Internet is increasing exponentially. Currently, most of this data is stored in the underlying storage engine. Therefore, when data needs to be retrieved, it must be retrieved from this underlying storage. However, due to the large volume of data in the underlying storage, numerous devices make access requests, leading to high loads on the underlying storage. Consequently, how to effectively retrieve the required data and reduce the load on the underlying storage has become a hot research topic. Summary of the Invention

[0003] This invention provides a data processing method, apparatus, routing device, and storage medium that can reduce the data pressure on the underlying storage device.

[0004] On one hand, embodiments of the present invention provide a data processing method, wherein the data processing method is executed by a routing device, the routing device establishing data communication with any node device in a set of node devices, and routing information generated by any node device after locally storing object data is stored in the routing device; the method includes:

[0005] Receive a request from a data caller for obtaining data of a target object, and query the target routing information of the target object data;

[0006] If the target routing information is found, the device identifier of the first node device in the node device set is sent to the data caller so that the data caller can obtain the target object data from the local machine of the first node device.

[0007] If the target routing information is not found, the device identifier of the second node device in the node device set is sent to the data caller so that the data caller can obtain the target object data through the second node device. The target object data is obtained by the second node device from the object storage engine and sent to the data caller. After obtaining the target object data, the second node device caches the target object data locally.

[0008] In another aspect, embodiments of the present invention provide a data processing apparatus, which is built into a routing device. The routing device establishes data communication with any node device in a set of node devices. Routing information generated by any node device after storing object data locally is stored in the routing device. The apparatus includes:

[0009] The receiving unit is used to receive a request from a data caller for obtaining data of a target object, and to query the target routing information of the target object data.

[0010] The processing unit is configured to, if the target routing information is found, send the device identifier of the first node device in the node device set to the data caller, so that the data caller can obtain the target object data from the local machine of the first node device;

[0011] The processing unit is further configured to send the device identifier of the second node device in the node device set to the data caller when the target routing information is not found, so that the data caller can obtain the target object data through the second node device; the target object data is obtained by the second node device from the object storage engine and sent to the data caller, and after obtaining the target object data, the second node device caches the target object data locally.

[0012] In one embodiment, when the target routing information is not found, the processing unit is further configured to obtain node devices in normal working condition from the node device set, and designate any node device in normal working condition as the second node device.

[0013] In one embodiment, the processing unit is further configured to acquire heartbeat data sent by any node device in the node device set;

[0014] The processing unit is further configured to determine that any node device is in normal working condition if the heartbeat data sent by any node device is sent according to a preset period; and to determine that any node device is in abnormal working condition if the heartbeat data is not sent according to the preset period.

[0015] In one embodiment, the processing unit is further configured to obtain the number of node devices in the node device cluster that are in normal working condition;

[0016] The processing unit is further configured to, when the number of node devices in normal working condition in the node device set is less than the target number, regard all node devices included in the node device set as node devices in normal working condition.

[0017] In one embodiment, if the data caller fails to obtain the target object data through the second node device;

[0018] The processing unit is further configured to receive a new acquisition request sent by the data caller, the new acquisition request indicating that the target object data be acquired again, and the new acquisition request including the device identifier of the second node device;

[0019] The processing unit is further configured to send the device identifier of any other node device in the node device set other than the second node device to the data caller, so that the data caller can obtain the target object data through any other node device.

[0020] In one embodiment, if the device identifier obtained by the data caller is the device identifier of the first node device, the data caller sends the data acquisition request to the first node device according to the device identifier of the first node device, so that the first node device can acquire the target object data from its local storage according to the acquisition request and send it to the data caller.

[0021] If the device identifier obtained by the data caller is the device identifier of the second node device, then after sending the acquisition request to the second node device, the data caller forwards the acquisition request to the object storage engine and obtains the target object data from the object storage engine.

[0022] In one embodiment, the first node device stores the target object data locally using a three-level cache. The acquisition request includes the source identifier of the data source to which the target object data belongs, and the index offset of the target object data. The processing unit is specifically used for:

[0023] The first node device determines the storage location of the first-level cache corresponding to the target object from the local location based on the source identifier, and obtains the second-level cache data from the storage location of the first-level cache;

[0024] The first node device determines the object range storing the data identifier from the second-level cache data based on the index offset of the target object data, and obtains the third-level storage data from the object range; the obtained third-level storage data is the target object data.

[0025] In one embodiment, the target object data includes a sequence of objects consisting of multiple object data;

[0026] The index offset of the target object data belongs to the identifier range consisting of the start data identifier corresponding to the start object data of the object sequence and the end data identifier corresponding to the end object data of the object sequence.

[0027] In one embodiment, the target object data is the data to be cached determined by the data caller when accessing the application, based on the object data already obtained by the data caller.

[0028] The source identifier of the data source to which the target object data belongs includes the program identifier of the application.

[0029] In one embodiment, the method by which the first node device stores the target object data using the three-level cache includes:

[0030] The first node device obtains the source identifier of the data source to which the target object belongs, stores the source identifier in the first-level cache, and generates an index identifier corresponding to the first-level cache based on the source identifier;

[0031] The first node device acquires the data identifier of the target object data, stores the data identifier in the second-level cache associated with the first-level cache, and generates an index identifier corresponding to the second-level cache based on the data identifier;

[0032] The first node device stores the target object data in a third-level cache associated with the second-level cache, and generates an index identifier corresponding to the third-level cache based on the data name of the target object data.

[0033] In another aspect, embodiments of the present invention provide a routing device, including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program supporting the routing device in executing the above-described method, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the following steps:

[0034] Receive a request from a data caller for obtaining data of a target object, and query the target routing information of the target object data;

[0035] If the target routing information is found, the device identifier of the first node device in the node device set is sent to the data caller so that the data caller can obtain the target object data from the local machine of the first node device.

[0036] If the target routing information is not found, the device identifier of the second node device in the node device set is sent to the data caller so that the data caller can obtain the target object data through the second node device. The target object data is obtained by the second node device from the object storage engine and sent to the data caller. After obtaining the target object data, the second node device caches the target object data locally.

[0037] In another aspect, embodiments of the present invention provide a data processing system, the system comprising: a routing device set, a node device set, and a data caller, wherein the routing device set includes one or more routing devices, and the node device set includes one or more node devices; any routing device in the routing device set establishes data communication with any node device in the node device set, routing information generated by any node device after storing object data locally is stored in the routing device, and each node device in the node device set is connected to an object storage engine;

[0038] The data caller is used to send a request to obtain target object data to any routing device in the routing device set;

[0039] Any one of the routing devices in the routing device set is used to obtain the acquisition request sent by the data caller, query the target routing information corresponding to the target object data, and when the target routing information is found, send the device identifier of the first node device in the node device set to the data caller, so that the data caller can obtain the target object data from the local of the first node device;

[0040] If any routing device in the routing device set fails to find the target routing information, it sends the device identifier of the second node device in the node device set to the data caller, so that the data caller can obtain the target object data through the second node device. The target object data is obtained by the second node device from the object storage engine and sent to the data caller.

[0041] In another aspect, embodiments of the present invention provide a computer-readable storage medium storing program instructions, which, when executed by a processor, are used to perform the data processing method as described in the first aspect.

[0042] In this embodiment, the node devices in the node device cluster store the object data obtained from the object storage engine in their local cache. They then generate corresponding routing information based on the locally cached object data and broadcast this information to each routing device in the routing cluster. Therefore, by broadcasting the routing information corresponding to the locally cached object data to each node device, each node in the routing cluster can obtain the routing information of the object data already cached locally. After a routing device obtains the routing information of the object data cached locally, if it receives a request from a data caller for retrieving target object data, it can query the target routing information corresponding to the target object data. If the target routing information is found, it indicates that the corresponding target object data has been cached locally on the corresponding node device. The routing device can then obtain the device identifier associated with the target routing information. If the obtained device identifier is the device identifier of the first node device, it sends the device identifier of the first node device to the data caller so that the data caller can retrieve the target object data locally from the first node device, thus improving the cache hit rate for cached data. If the routing device cannot find routing information for the target object data, it can randomly return a usable device identifier for the second node. This allows the data caller to retrieve the target object data from the object storage engine via the second node device. After retrieving the target object data, the second node device caches it locally and broadcasts the generated routing information. Based on the distributed routing architecture and the object data caching and broadcasting mechanism, the cache hit rate can be improved while reducing the latency of requests to retrieve the object data. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1a This is a schematic diagram of a data processing system provided in an embodiment of the present invention;

[0045] Figure 1b This is a schematic diagram of a data processing system provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic flowchart of a data processing method provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of a sequential access method provided by an embodiment of the present invention;

[0048] Figure 4 This is a schematic flowchart of a three-level storage system provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of a cache hit rate provided in an embodiment of the present invention;

[0050] Figure 6 This is a schematic block diagram of a data processing device provided in an embodiment of the present invention;

[0051] Figure 7 This is a schematic block diagram of a routing device provided in an embodiment of the present invention. Detailed Implementation

[0052] This application proposes a data processing method. After a routing device receives a data retrieval request (such as a request for target object data) sent by a data caller based on an ordered index, it can query routing information to determine whether the requested target object data has been cached in a node device of the node cluster. If the target object data is found to be cached, the routing device returns the node identifier of the node device that cached the target object data to the data caller, enabling the data caller to retrieve the target object data from the cache of the corresponding node device, thus ensuring the cache hit rate of the node device. Conversely, if the routing device determines through querying routing information that the requested target object data is not cached, it can randomly return the device identifier of an available node device to the data caller. After receiving the randomly returned device identifier, the data caller can access the node device corresponding to that device identifier to retrieve the target object data from the underlying object storage engine, thereby improving the performance of the routing device in successfully returning object data to the data caller. In one embodiment, ordered indexing refers to the data caller retrieving object data in a certain order. This order can be determined based on the index of the object data. For example, if the data caller has already retrieved data with corresponding indices a and b, then the data retrieval request currently sent by the data caller indicates that the object data with index c should be retrieved.

[0053] Because the node devices in the node cluster cache object data locally, they generate corresponding routing information based on this local cache and broadcast it. Based on this broadcast, the routing device can obtain the locally cached routing information. After obtaining this information, the routing device associates and stores it with the device identifier of the corresponding node device. Therefore, if the routing device receives a request from a data caller, it can determine the specific node device where the requested object data is cached by querying the stored routing information. In one embodiment, the routing device can be any routing device in the routing cluster. That is, by broadcasting the routing information corresponding to the locally cached object data from the node devices in the cluster, each routing device in the cluster stores this routing information. This allows other data callers to subsequently access any routing device in the cluster to obtain the required object data's routing information, thereby reducing the consistency requirements when data callers request data from routing devices.

[0054] In one embodiment, the data processing method can be applied to, for example... Figure 1a The data processing system described above exhibits high cache hit rate, timely garbage collection, and high concurrency when acquiring object data, effectively protecting the underlying system load even under extreme conditions of identical hot requests. For example... Figure 1a As shown, the data processing system includes a set of routing devices, a set of node devices, and a data caller. The set of routing devices can be, for example, as shown below. Figure 1a The set of devices marked with 10 in the middle, the routing device set includes one or more routing devices, and the node device set can be, for example, as shown in the figure. Figure 1a The set of devices marked with 11, and this set of node devices also includes one or more node devices, can be used as follows: Figure 1a Any one of the devices marked by 12 in the set of routing devices 10, wherein the data caller 12 is used to send a request to obtain target object data to any routing device in the routing device set 10; any routing device in the routing device set 10 is used to obtain the request sent by the data caller 12, query the target routing information corresponding to the target object data, and when the target routing information is found, send the device identifier of the corresponding node device in the node device set 11 to the data caller 12 so that the data caller 12 can obtain the target object data from the local device of the corresponding node device.

[0055] In one embodiment, if any routing device in the routing device set 10 has established data communication with any node device in the node device set, then after any node device in the node device set 11 stores object data locally, it can broadcast the routing information generated after storing the object data to all routing devices in the routing device set 10, so that the routing information can be stored in all routing devices of the routing device set 10. In one embodiment, the object data cached by any node device in the node device set 11 is obtained from an object storage engine connected to the node device set 11. The object storage engine can be, for example,... Figure 1a If a storage device is marked with 13, then after the node device in the node device set 11 broadcasts the routing information of the cached object data to the routing device set 11, the data caller 12 can obtain the device identifier of the node device storing the object data by accessing any routing device in the routing device set 11. For example, the data caller 12 can access the routing device 110 in the routing device set 11 and obtain the device identifier of the corresponding node device from the routing device 110. After obtaining the device identifier, the data caller 12 can access the corresponding node device through the device identifier and obtain the required object data from the local cache of the accessed node device.

[0056] In one embodiment, the node identifier corresponding to the node device can be an IP (Internet Protocol) identifier used to distinguish node devices, or it can be the physical address (MAC address) of the corresponding node device, etc. In one embodiment, if the data caller 12, after accessing any routing device in the routing device set 10, does not obtain routing information from the corresponding routing device to cache the required object data, it indicates that there is no node device in the node device set that has locally cached the object data. Therefore, if... Figure 1b As shown, the routing device can return the device identifier of any available node device in the node device set to the data caller 12. The data caller can then access the corresponding node device through the device identifier, allowing the accessed node device to retrieve the required object data from the object storage engine 13 and return it to the data caller 12. Specifically, after the node device retrieves the corresponding object data from the object storage engine 13 based on the request from the data caller 12, in addition to returning the retrieved object data to the data caller 12, it will also broadcast the routing information after storing the object data to the routing cluster.

[0057] Please see Figure 2This is a schematic flowchart of a data processing method proposed in an embodiment of this application. This data processing method can be executed by any routing device in the aforementioned routing device set. The routing device establishes data communication with any node device in the node device set. Routing information generated by any node device after locally storing object data is stored in the routing device. For example... Figure 2 As shown, the method may include:

[0058] S201: Receive the data caller's request to obtain the target object data, and query the target routing information of the target object data.

[0059] In one embodiment, a data caller can retrieve object data from an object storage engine through a list interface. The list interface supports object data retrieval, and the data caller can obtain object data by sending a list request. The retrieved object data includes one or more objects. In one embodiment, object data refers to stored data obtained after data storage using object storage methods. This object data can be stored on a cloud server, which is also called cloud object storage. The corresponding cloud server can be called a cloud object storage device. When data is stored on the corresponding object storage device using object storage methods, to improve the speed of data retrieval from the object storage device after data storage, a high QPS (Query Per Second) optimization strategy can be used. This involves distributing the corresponding object index and storing them separately. For example, the ordered object index can be divided into 32 parts before storage. QPS can be understood as the concurrent reading efficiency when reading stored data. A higher QPS means higher concurrent reading efficiency of stored data (such as object data as mentioned above), and a greater amount of data can be read and retrieved each time. In one embodiment, the cloud server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0060] In one embodiment, breaking down the object indexes of data means breaking down the data identifiers. For example, if a data point is A and its corresponding index identifier is a, and the existing data to be stored corresponds to indices 1 to 64, then when using a high QPS optimization strategy for data storage, the index identifiers 1 to 64 will be broken down for storage. For instance, the data corresponding to indices 1 and 32 can be stored as one set in one storage location, while the data corresponding to indices 2 and 33 can be stored as another set in another storage location. Thus, the high QPS optimization rule used in the object storage device will result in the object data being broken down into 32 parts for storage. In another embodiment, when the data caller retrieves object data through a list request, based on the special requirements of the list interface for data return, the list interface will retrieve the object list from the object storage device according to specific prefixes and other parameters, and return it in a specific order (such as lexicographical order, i.e., the order of the corresponding index identifiers a, b, ..., z). When the object storage device stores data, it uses high QPS optimization to scatter the object index before storing it. Therefore, when the data caller retrieves the object data through the list interface, it will retrieve all 32 data sets, merge and sort them, and then return them.

[0061] In one embodiment, the list interface has strong caching effectiveness, and because the data caller retrieves object data through the list interface in a specific order, such as... Figure 3 As shown, when retrieving object data based on a list request, the process is generally from front to back. Therefore, if the records that will be accessed soon are cached in advance, they can be returned quickly in subsequent requests. For example, if the data caller has already retrieved the object data corresponding to index 3, then the subsequent object data retrieval by the data caller will start from index 4. In addition, since high QPS optimization projects require merging and sorting 32 sets of data, the data read from the underlying storage (such as the object storage device mentioned above) will be 32 times the actual amount. Because the 32 sets of data are relatively evenly distributed, the extra records also meet the ordered requirements of the list, so they can continue to provide subsequent list requests. Therefore, after retrieving object data from the underlying storage through the list interface, it is possible to directly cache it locally, which can reduce the latency of subsequent data requests to the device and reduce the access load pressure on the underlying storage device.

[0062] In one embodiment, the `list` interface serves as an external interface for accessing the underlying storage device (such as an object storage engine). To cache object data retrieved from the underlying storage device via the `list` interface, a set of node devices (i.e., a Node set) associated with the underlying storage device can be deployed to cache the retrieved object data. When a data caller requests object data, it can request the corresponding cached data through a node device in the Node set, or it can access the underlying object storage engine through a node device to retrieve the corresponding object data. Specifically, when a node device in the Node set retrieves object data from the underlying object storage engine and caches it locally, the node device can retrieve the object data from the underlying object storage engine according to a preset period to achieve periodic updates to the cached data. Alternatively, each node device in the Node set can retrieve the target object data from the underlying object storage engine only after receiving a request from the data caller and not finding the required target object data in its local cache.

[0063] In one embodiment, each node device in the node cluster generates corresponding routing information based on locally stored object data and broadcasts the generated routing information to the routing device cluster so that each routing device in the cluster can obtain the routing information. When a data caller needs to obtain target object data, it can first send the request for obtaining the target object data to any routing device in the routing device cluster. When the routing device receives the request for obtaining the target object data from the data caller, it can query the object data to determine whether there is routing information for the target object data. If it determines that there is, it returns the device identifier corresponding to the node device that sent the routing information to the data caller, i.e., proceeds to step S202. If the routing device determines that there is no target routing information for the target object data after receiving the request for obtaining the target object data from the data caller, the routing device will randomly return an available node device to the data caller. Then, the data caller can obtain the target object data from the object storage engine by accessing the corresponding node device, i.e., proceeds to step S203. In this context, the data caller can access the routing device through the Common Gateway Interface (CGI).

[0064] S202, if the target routing information is found, the device identifier of the first node device in the node device set is sent to the data caller, so that the data caller can obtain the target object data from the local of the first node device.

[0065] S203, when no target routing information is found, the device identifier of the second node device in the node device set is sent to the data caller so that the data caller can obtain the target object data through the second node device; the target object data is obtained by the second node device from the object storage engine and sent to the data caller. After obtaining the target object data, the second node device caches the target object data locally.

[0066] In steps S202 and S203, after receiving a request from the data caller, if the routing device finds the routing information for the requested target object data, it indicates that the requested target object data has been cached in a node device within the node device set (assuming the target object data is cached in the first node device in the node device set). Since the routing device associates the broadcast routing information with the device identifier of the corresponding node device when obtaining it from the node device, after finding the routing information for the target object data, it can obtain the device identifier associated with that routing information and send it to the data caller so that the data caller can determine the cache node corresponding to the target object data. After the data caller obtains the device identifier (e.g., the Node IP of the first node device) of the node device caching the target object data from the routing device, the data caller can access the first node device based on the obtained device identifier and retrieve the required target object data from its cache. In one embodiment, after the data caller obtains the device identifier of the first node device that caches the target object data from the routing device, when the data caller accesses the first node device based on the device identifier, the request sent by the data caller for retrieving the target object data when accessing the first node device is a list request. Due to the special nature of sequential access based on the list request, the list request sent by the data caller to the node device includes the index identifier (i.e., the data identifier of the target object data) corresponding to the target object data when using sequential indexing. Then, after the first node device obtains the list request from the data caller, it retrieves the corresponding object data from the local cache based on the index identifier included in the list request.

[0067] In one embodiment, when a routing device receives a request from a data caller for data of a target object, if it finds the routing information for that target object data, the routing device can also directly forward the request to the corresponding node device (such as the first node device mentioned above) based on the device identifier associated with the routing information.

[0068] If a routing device fails to find routing information for the target object data, it indicates that none of the nodes in the node device set have cached the target object data, or that the node device that has cached the target object data (such as the first node device mentioned above) is unreachable. Therefore, when a routing device fails to find routing information for the target object data, it can randomly return the device identifier of an available node device to the data caller. For example, the routing device can return the device identifier of the second node device to the data caller. After obtaining the device identifier of the second node device, the data caller can send a list request for the target object data to the second node device. Upon receiving the list request, the second node device, unable to find the target object data in its local cache, can retrieve the target object data by accessing the object storage engine and return the retrieved target object data to the data caller. After retrieving the target object data from the object storage engine, the second node device can cache the target object data using a three-level cache storage structure implemented based on a single-machine cache design. Furthermore, after storing the target object data, the second node device will also broadcast the routing information for that target object data.

[0069] In one embodiment, when the routing device fails to find routing information (i.e., target routing information) for the target object data, it can obtain node devices in normal working condition from the node device set and designate any such node device as a second node device. In addition to broadcasting the routing information of the locally stored object data, each node device in the node device set also broadcasts heartbeat data to the routing device every n seconds. Therefore, the routing device can determine whether a corresponding node device in the node device set is in normal working condition based on the heartbeat data from each node device in the set. In its implementation, the routing device can obtain heartbeat data sent by any node device in the node device set. If any node device sends heartbeat data according to a preset period, it is determined that the node device is in normal working condition. If any node device does not send heartbeat data according to the preset period, the routing device can determine that the node device is in abnormal working condition. The preset period can be 3 seconds. In this case, the routing device can obtain heartbeat data sent by the node device once every 3 seconds. Therefore, if the routing device does not receive heartbeat data from the node device within 3 periods, it can set the corresponding node device as unhealthy (i.e., abnormal working condition). The node device set as abnormal working condition by the routing device will no longer be allocated externally. In addition, the routing device is equipped with a removal protection mechanism. If the routing device determines that the number of nodes to be removed is greater than a preset number (e.g., the proportion of removed nodes is greater than 0.5), the routing device can activate the removal protection mechanism, which means that it will no longer mark nodes in abnormal working states. In specific implementation, the routing device can obtain the number of nodes in normal working state in the node device set; and when the number of nodes in normal working state in the node device set is less than the target number, it can trigger the removal protection mechanism and regard all nodes included in the node device set as nodes in normal working state.

[0070] In one embodiment, since the routing device returns an available IP address to the data caller each time it accesses the routing device, if a downstream node device becomes momentarily unavailable, and heartbeat detection takes at least 9 seconds, a fast reporting and removal strategy can be used to immediately remove abnormal downstream node devices. That is, if the data caller fails to request the corresponding node device based on the device identifier obtained from the routing device, it can re-request the IP address from the routing device and send the IP address of the node that failed to access the routing device along with the re-request. When the routing device re-shards available IP addresses for the data caller, it will skip the faulty node IP address sent by the data caller and record the faulty IP address. When the number of reported IP addresses exceeds a threshold, removal can also be triggered to block faulty node nodes more quickly. In a specific implementation, if the data caller fails to obtain the target object data through the second node device, the routing device can receive a new acquisition request sent by the data caller. The new acquisition request indicates that the target object data should be obtained again, and the new acquisition request includes the device identifier of the second node device. The device identifier of any other node device in the node device set other than the second node device is sent to the data caller so that the data caller can obtain the target object data through any other node device.

[0071] In one embodiment, since node devices need to broadcast routing information of locally stored object data to routing devices, and considering the large amount of broadcast traffic, a smaller number of routing devices can be deployed in a single cluster. However, deploying a smaller number of routing devices requires high performance in their design. Therefore, the Router also integrates the ability to separately control the frequency of the cluster and storage to avoid node overload. In actual operation, a normal single instance can reach 100,000 / s, with a single instance of 100,000 / s capable of serving the entire campus. The Router's average processing latency is 0.013ms (in a South China campus), and it effectively eliminates multiple version release operations.

[0072] In this embodiment, the node devices in the node device cluster store the object data obtained from the object storage engine in their local cache. They then generate corresponding routing information based on the locally cached object data and broadcast this information to each routing device in the routing cluster. Therefore, by broadcasting the routing information corresponding to the locally cached object data to each node device, each node in the routing cluster can obtain the routing information of the object data already cached locally. After a routing device obtains the routing information of the object data cached locally, if it receives a request from a data caller for retrieving target object data, it can query the target routing information corresponding to the target object data. If the target routing information is found, it indicates that the corresponding target object data has been cached locally on the corresponding node device. The routing device can then obtain the device identifier associated with the target routing information. If the obtained device identifier is the device identifier of the first node device, it sends the device identifier of the first node device to the data caller so that the data caller can retrieve the target object data locally from the first node device, thus improving the cache hit rate for cached data. If the routing device cannot find routing information for the target object data, it can randomly return a usable device identifier for the second node. This allows the data caller to retrieve the target object data from the object storage engine via the second node device. After retrieving the target object data, the second node device caches it locally and broadcasts the generated routing information. Based on the distributed routing architecture and the object data caching and broadcasting mechanism, the cache hit rate can be improved while reducing the latency of requests to retrieve the object data.

[0073] In one embodiment, the target object data cached locally by the first node device is stored using a three-level cache method, such as... Figure 4As shown, when the first node device uses a three-level cache to store target object data, it first obtains the source identifier of the data source to which the target object belongs, stores the source identifier in the first-level cache, and generates an index identifier corresponding to the first-level cache based on the source identifier. The first-level cache is a hash table. The source identifier includes the user (bucket) name and specific request parameter indexes, such as the request prefix, flag (used to indicate file size), and delimiter (used to identify subfiles). Since the target object data is the data to be cached determined by the data caller based on the object data already obtained by the data caller when accessing the application, the username can be the program name (or program identifier) ​​corresponding to the accessed application. Therefore, in the first-level cache, object data from different applications can be cached separately, which helps to reduce the lock granularity between different request sessions (a type of request configuration information) to a certain extent, thereby improving the efficiency of concurrent object data retrieval. In the first-level cache, if the corresponding index identifier generated based on the source identifier is denoted as key, then key = bucket (user) + specific request parameter index. The second-level index of the first-node device is the data identifier for the target object data obtained by the first-node device. This data identifier is stored in the second-level cache associated with the first-level cache, and an index identifier corresponding to the second-level cache is generated based on the data identifier. The second-level cache, also called the second-level index, is a range index stored using an ordered graph (ordered map, such as a session map). This ordered graph includes multiple index ranges, each storing consecutive records after merging all lists. Each index range is stored using key-value pairs (KV pairs), where the key of each index range equals the starting value of the range (keymin), and the entire index range serves as the value corresponding to the index key. The third-level cache stores the actual target object data associated with the second-level cache. It stores one or more target object data using an ordered map (such as a range map). Similarly, an ordered map in the third-level cache corresponds to the key value of a cache range in the second-level cache. The third-level cache stores the actual data name of the target object in an ordered manner. The structures of the ordered graphs in the second-level and third-level caches can be the same or different.

[0074] Based on the three - level cache method adopted by the first node device, the list request sent by the data requester includes the source identifier of the data source to which the target object data belongs, and the index offset (next_offset) of the target object data. Then, when the first node device obtains the target object data from the local according to the list request, it can first determine the storage location of the first - level cache corresponding to the target object from the local according to the source identifier, and obtain the second - level cache data from the storage location of the first - level cache. Among them, when performing range positioning based on the index offset, the located range i satisfies: session[i].keymin <= next_offset < session[i].keymax. After locating the second - level cache data, the first node device can determine the object range storing the data identifier from the second - level cache data according to the index offset of the target object data, and obtain the third - level storage data from the object range. Similarly, the third - level cache data is also positioned by the next_offset of the request parameter, and the located node i satisfies range[i - 1] <= next_offset < range[i]. After locating the third - level cache, the third - level storage data obtained from the third - level cache is the target object data. In one embodiment, as Figure 4 The shown index relationship is that the source identifier of the data source to which the target object data obtained by the first node device belongs is Program 2, and the index offset is 0. Then, the target object data with the corresponding data identifier of 0 can be obtained based on the above - mentioned three - level index relationship. In one embodiment, the target object data includes an object sequence composed of multiple object data; the index offset of the target object data belongs to the identifier range composed of the starting data identifier corresponding to the starting object data of the object sequence and the ending data identifier corresponding to the ending object data of the object sequence.

[0075] In one embodiment, in terms of memory recovery after cache expiration, the memory is scanned globally by an asynchronous thread every 10 milliseconds. For example, the nodes in the ordered graph in the second - level cache expire as basic units. When the node cache time exceeds the configured time (usually 3 seconds), it is moved to the FIFO (First In First Out) queue and is uniformly cleared before the next round of global scanning by the asynchronous thread. This deferred cleaning method can effectively avoid the process coredump (that is, the process makes an error) caused by violent cleaning due to the memory being occupied by other threads. In addition, as shown in Table 1, through the cache mechanism, the List request latency is greatly reduced when the cache is hit. From the perspective of overall operation data, the cache hit rate can be reduced by 80% on average by adopting the embodiment of this application, and the latency can be reduced by 95% when the request parameter count = 1K.

[0076] Table 1

[0077] Full sort latency in milliseconds 7.5 24 43 615 Hit latency (ms) 1.7 4.5 8.1 34

[0078] Furthermore, distributed routing allows access machines to immediately detect machines with cached data, improving cache hit rate. For example, in a certain availability zone in region A, the hit rate is 90%, with an average latency of 10ms (listcount average 50). Figure 5 As shown. When caching object data, multiple copies of data are merged and sorted before being cached in a single machine's memory. Therefore, due to the spatial order of the list, the proportion of users using ordered lists is relatively high, ensuring a high cache hit rate. In addition, since the cache adopts a distributed architecture to avoid single points of failure, but the routing of ordered indexes cannot use deterministic consistent hashing algorithms, this application embodiment designs a global broadcast strategy for routes. The route management module ensures high performance, and the cluster can be deployed with minimal nodes to reduce the broadcast coverage.

[0079] Based on the description of the above data processing method embodiments, this invention also proposes a data processing apparatus. This data processing apparatus can be a computer program (including program code) running on the aforementioned routing device. The routing device establishes data communication with any node device in the node device set, and routing information generated by any node device after locally storing object data is stored in the routing device. This data processing apparatus can be used to perform operations such as... Figure 2 For the data processing method described above, please refer to [link / reference]. Figure 6 The data processing device includes a receiving unit 601 and a processing unit 602.

[0080] The receiving unit 601 is used to receive a request from a data caller for obtaining target object data, and to query the target routing information of the target object data;

[0081] The processing unit 602 is configured to, if the target routing information is found, send the device identifier of the first node device in the node device set to the data caller, so that the data caller can obtain the target object data from the local machine of the first node device;

[0082] The processing unit 602 is further configured to send the device identifier of the second node device in the node device set to the data caller when the target routing information is not found, so that the data caller can obtain the target object data through the second node device; the target object data is obtained by the second node device from the object storage engine and sent to the data caller, and after obtaining the target object data, the second node device caches the target object data locally.

[0083] In one embodiment, when the target routing information is not found, the processing unit 602 is further configured to obtain node devices in normal working condition from the node device set, and designate any node device in normal working condition as the second node device.

[0084] In one embodiment, the processing unit 602 is further configured to acquire heartbeat data sent by any node device in the node device set;

[0085] The processing unit 602 is further configured to determine that any node device is in normal working condition if the heartbeat data sent by any node device is sent according to a preset period; and to determine that any node device is in abnormal working condition if the heartbeat data is not sent according to the preset period.

[0086] In one embodiment, the processing unit 602 is further configured to obtain the number of node devices in the node device cluster that are in normal working condition;

[0087] The processing unit 602 is further configured to, when the number of node devices in normal working condition in the node device set is less than the target number, regard all node devices included in the node device set as node devices in normal working condition.

[0088] In one embodiment, if the data caller fails to obtain the target object data through the second node device;

[0089] The processing unit 602 is further configured to receive a new acquisition request sent by the data caller, the new acquisition request indicating that the target object data be acquired again, and the new acquisition request including the device identifier of the second node device;

[0090] The processing unit 602 is further configured to send the device identifier of any other node device in the node device set other than the second node device to the data caller, so that the data caller can obtain the target object data through any other node device.

[0091] In one embodiment, if the device identifier obtained by the data caller is the device identifier of the first node device, the data caller sends the data acquisition request to the first node device according to the device identifier of the first node device, so that the first node device can acquire the target object data from its local storage according to the acquisition request and send it to the data caller.

[0092] If the device identifier obtained by the data caller is the device identifier of the second node device, then after sending the acquisition request to the second node device, the data caller forwards the acquisition request to the object storage engine and obtains the target object data from the object storage engine.

[0093] In one embodiment, the first node device stores the target object data locally using a three-level cache. The acquisition request includes the source identifier of the data source to which the target object data belongs, and the index offset of the target object data. The processing unit 602 is specifically used for:

[0094] The first node device determines the storage location of the first-level cache corresponding to the target object from the local location based on the source identifier, and obtains the second-level cache data from the storage location of the first-level cache;

[0095] The first node device determines the object range storing the data identifier from the second-level cache data based on the index offset of the target object data, and obtains the third-level storage data from the object range; the obtained third-level storage data is the target object data.

[0096] In one embodiment, the target object data includes a sequence of objects consisting of multiple object data;

[0097] The index offset of the target object data belongs to the identifier range consisting of the start data identifier corresponding to the start object data of the object sequence and the end data identifier corresponding to the end object data of the object sequence.

[0098] In one embodiment, the target object data is the data to be cached determined by the data caller when accessing the application, based on the object data already obtained by the data caller.

[0099] The source identifier of the data source to which the target object data belongs includes the program identifier of the application.

[0100] In one embodiment, the method by which the first node device stores the target object data using the three-level cache includes:

[0101] The first node device obtains the source identifier of the data source to which the target object belongs, stores the source identifier in the first-level cache, and generates an index identifier corresponding to the first-level cache based on the source identifier;

[0102] The first node device acquires the data identifier of the target object data, stores the data identifier in the second-level cache associated with the first-level cache, and generates an index identifier corresponding to the second-level cache based on the data identifier;

[0103] The first node device stores the target object data in a third-level cache associated with the second-level cache, and generates an index identifier corresponding to the third-level cache based on the data name of the target object data.

[0104] In this embodiment, the node devices in the node device cluster store the object data obtained from the object storage engine into their local cache, generate corresponding routing information based on the locally cached object data, and broadcast the generated routing information. Therefore, based on the broadcast of the routing information corresponding to the locally cached object data by each node device, the receiving unit 601 can obtain the routing information of the object data that has been cached locally. After the receiving unit 601 obtains the routing information of the object data cached locally on the node device, if the receiving unit 601 receives a request from the data caller to obtain the target object data, the processing unit 602 can query the target routing information corresponding to the target object data. When the target routing information is found, it indicates that the corresponding target object data has been cached locally on the corresponding node device. Then, the processing unit 602 can obtain the device identifier associated with the target routing information. If the obtained device identifier is the device identifier of the first node device, the processing unit 602 can send the device identifier of the first node device to the data caller so that the data caller can obtain the target object data locally from the first node device, thereby improving the cache hit rate for cached data. If processing unit 602 fails to find routing information for the target object data, it can randomly return an available device identifier for the second node. This allows the data caller to retrieve the target object data from the object storage engine via the second node device. After retrieving the target object data, the second node device performs local caching and broadcasts the generated routing information. Based on the distributed routing architecture and the object data caching and broadcasting mechanism, the cache hit rate can be improved while reducing the request latency for retrieving the object data.

[0105] Please see Figure 7 This is a schematic block diagram of a routing device provided in an embodiment of the present invention. Figure 7 The routing device shown in this embodiment may include: one or more processors 701; one or more input devices 702; one or more output devices 703; and a memory 704. The processors 701, input devices 702, output devices 703, and memory 704 are connected via a bus 705. The memory 704 stores a computer program, which includes program instructions, and the processor 701 executes the program instructions stored in the memory 704.

[0106] The memory 704 may include volatile memory, such as random-access memory (RAM); the memory 704 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 704 may also include a combination of the above types of memory.

[0107] The processor 701 may be a central processing unit (CPU). The processor 701 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc. The PLD may be a field-programmable gate array (FPGA), generic array logic (GAL), etc. The processor 701 may also be a combination of the above structures.

[0108] In this embodiment of the invention, the memory 704 is used to store a computer program, the computer program including program instructions, and the processor 701 is used to execute the program instructions stored in the memory 704 to implement the above-mentioned... Figure 2 The steps of the corresponding method.

[0109] In one embodiment, the processor 701 is configured to invoke the program instructions to execute:

[0110] Receive a request from a data caller for obtaining data of a target object, and query the target routing information of the target object data;

[0111] If the target routing information is found, the device identifier of the first node device in the node device set is sent to the data caller so that the data caller can obtain the target object data from the local machine of the first node device.

[0112] If the target routing information is not found, the device identifier of the second node device in the node device set is sent to the data caller so that the data caller can obtain the target object data through the second node device. The target object data is obtained by the second node device from the object storage engine and sent to the data caller. After obtaining the target object data, the second node device caches the target object data locally.

[0113] In one embodiment, when the target routing information is not found, the processor 701 is configured to invoke the program instructions to execute:

[0114] Obtain node devices that are in normal working condition from the node device set, and designate any node device that is in normal working condition as the second node device.

[0115] In one embodiment, the processor 701 is configured to invoke the program instructions to execute:

[0116] Obtain heartbeat data sent by any node device in the node device set;

[0117] If any node device sends heartbeat data according to a preset period, it is determined that the node device is in normal working condition; if any node device does not send heartbeat data according to the preset period, it is determined that the node device is in abnormal working condition.

[0118] In one embodiment, the processor 701 is configured to invoke the program instructions to execute:

[0119] Obtain the number of node devices in the node device set that are in normal working condition;

[0120] When the number of node devices in the node device set that are in normal working condition is less than the target number, all node devices included in the node device set are considered to be node devices in normal working condition.

[0121] In one embodiment, if the data caller fails to obtain the target object data through the second node device, the processor 701 is configured to invoke the program instructions to execute:

[0122] Receive a new retrieval request sent by the data caller, the new retrieval request indicating to re-retrieve the target object data, and the new retrieval request includes the device identifier of the second node device;

[0123] The device identifier of any other node device in the node device set, excluding the second node device, is sent to the data caller so that the data caller can obtain the target object data through any other node device.

[0124] In one embodiment, if the device identifier obtained by the data caller is the device identifier of the first node device, the data caller sends the data acquisition request to the first node device according to the device identifier of the first node device, so that the first node device can acquire the target object data from its local storage according to the acquisition request and send it to the data caller.

[0125] If the device identifier obtained by the data caller is the device identifier of the second node device, then after sending the acquisition request to the second node device, the data caller forwards the acquisition request to the object storage engine and obtains the target object data from the object storage engine.

[0126] In one embodiment, the first node device stores the target object data locally using a three-level cache, and the acquisition request includes the source identifier of the data source to which the target object data belongs, and the index offset of the target object data;

[0127] In one embodiment, the processor 701 is configured to invoke the program instructions to execute:

[0128] The first node device determines the storage location of the first-level cache corresponding to the target object from the local location based on the source identifier, and obtains the second-level cache data from the storage location of the first-level cache;

[0129] The first node device determines the object range storing the data identifier from the second-level cache data based on the index offset of the target object data, and obtains the third-level storage data from the object range; the obtained third-level storage data is the target object data.

[0130] In one embodiment, the target object data includes a sequence of objects consisting of multiple object data;

[0131] The index offset of the target object data belongs to the identifier range consisting of the start data identifier corresponding to the start object data of the object sequence and the end data identifier corresponding to the end object data of the object sequence.

[0132] In one embodiment, the target object data is the data to be cached determined by the data caller when accessing the application, based on the object data already obtained by the data caller.

[0133] The source identifier of the data source to which the target object data belongs includes the program identifier of the application.

[0134] In one embodiment, the method by which the first node device stores the target object data using the three-level cache includes:

[0135] The first node device obtains the source identifier of the data source to which the target object belongs, stores the source identifier in the first-level cache, and generates an index identifier corresponding to the first-level cache based on the source identifier;

[0136] The first node device acquires the data identifier of the target object data, stores the data identifier in the second-level cache associated with the first-level cache, and generates an index identifier corresponding to the second-level cache based on the data identifier;

[0137] The first node device stores the target object data in a third-level cache associated with the second-level cache, and generates an index identifier corresponding to the third-level cache based on the data name of the target object data.

[0138] This invention provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned actions. Figure 2 The method embodiment shown. The computer-readable storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0139] The above-disclosed embodiments are merely partial examples of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments, and equivalent variations made in accordance with the claims of the present invention, still fall within the scope of the invention.

Claims

1. A data processing method, characterized in that, The data processing method is executed by any routing device in the routing device set, and the routing device establishes data communication with any node device in the node device set. The routing information generated by the node device after storing object data locally is stored in each routing device after being broadcast to each routing device in the routing device set; the method includes: Receive a request from a data caller for obtaining data of a target object, and query the target routing information of the target object data; If the target routing information is found, the device identifier of the first node device in the node device set is sent to the data caller so that the data caller can obtain the target object data from the local device of the first node device; the target routing information is associated with the device identifier of the first node device. If the target routing information is not found, the device identifier of the second node device in the node device set is sent to the data caller so that the data caller can obtain the target object data through the second node device; The target object data is obtained by the second node device from the object storage engine and sent to the data caller. After obtaining the target object data, the second node device caches the target object data locally and, after generating the target routing information of the target object data, broadcasts the target routing information to each routing device in the routing device set so as to store the target routing information in each routing device. The first node device or the second node device locally caches the target object data using a three-level cache structure. The three-level cache structure includes a first-level cache, a second-level cache, and a third-level cache. The first-level cache is used to distinguish and cache object data from different data sources. The second-level cache is used to locate the data identifier of the object data within a range based on the index offset. The third-level cache is used to read the target object data within the located index range based on the index offset.

2. The method as described in claim 1, characterized in that, When the target route information is not found, the method further includes: Obtain node devices that are in normal working condition from the node device set, and designate any node device that is in normal working condition as the second node device.

3. The method as described in claim 2, characterized in that, The method further includes: Obtain heartbeat data sent by any node device in the node device set; If any node device sends heartbeat data according to a preset period, it is determined that the node device is in normal working condition; if any node device does not send heartbeat data according to the preset period, it is determined that the node device is in abnormal working condition.

4. The method as described in claim 2, characterized in that, The method further includes: Obtain the number of node devices in the node device set that are in normal working condition; When the number of node devices in the node device set that are in normal working condition is less than the target number, all node devices included in the node device set are considered to be node devices in normal working condition.

5. The method as described in claim 1, characterized in that, If the data caller fails to obtain the target object data through the second node device, the method further includes: Receive a new retrieval request sent by the data caller, the new retrieval request indicating to re-retrieve the target object data, and the new retrieval request includes the device identifier of the second node device; The device identifier of any other node device in the node device set, excluding the second node device, is sent to the data caller so that the data caller can obtain the target object data through any other node device.

6. The method as described in claim 1, characterized in that, If the device identifier obtained by the data caller is the device identifier of the first node device, then the data caller sends the data acquisition request to the first node device according to the device identifier of the first node device, so that the first node device can acquire the target object data from its local storage according to the acquisition request and send it to the data caller; If the device identifier obtained by the data caller is the device identifier of the second node device, then after sending the acquisition request to the second node device, the data caller forwards the acquisition request to the object storage engine and obtains the target object data from the object storage engine.

7. The method as described in claim 6, characterized in that, The acquisition request includes the source identifier of the data source to which the target object data belongs, and the index offset of the target object data; The first node device obtains the target object data from its local storage according to the acquisition request in the following ways: The first node device determines the storage location of the first-level cache corresponding to the target object from the local location based on the source identifier, and obtains the second-level cache data from the storage location of the first-level cache; The first node device determines the object range for storing the data identifier from the second-level cache data based on the index offset of the target object data, and obtains the third-level storage data from the object range; the obtained third-level storage data is the target object data.

8. The method as described in claim 7, characterized in that, The target object data includes an object sequence consisting of multiple object data; The index offset of the target object data belongs to the identifier range consisting of the start data identifier corresponding to the start object data of the object sequence and the end data identifier corresponding to the end object data of the object sequence.

9. The method as described in claim 7, characterized in that, The target object data is the data to be cached determined by the data caller when accessing the application, based on the object data already obtained by the data caller. The source identifier of the data source to which the target object data belongs includes the program identifier of the application.

10. The method as described in claim 1, characterized in that, The first node device stores the target object data locally using the three-level cache method, including: The first node device obtains the source identifier of the data source to which the target object data belongs, stores the source identifier in the first-level cache, and generates an index identifier corresponding to the first-level cache based on the source identifier; The first node device acquires the data identifier of the target object data, stores the data identifier in the second-level cache associated with the first-level cache, and generates an index identifier corresponding to the second-level cache based on the data identifier; The first node device stores the target object data in a third-level cache associated with the second-level cache, and generates an index identifier corresponding to the third-level cache based on the data name of the target object data.

11. A data processing system, characterized in that, The system includes: a set of routing devices, a set of node devices, and a data caller. The set of routing devices includes one or more routing devices, and the set of node devices includes one or more node devices. Each routing device in the set of routing devices establishes data communication with each node device in the set of node devices. The routing information generated by each node device after storing object data locally is broadcast to each routing device in the set of routing devices and then stored in each routing device. Each node device in the set of node devices is connected to the object storage engine. The data caller is used to send a request to obtain target object data to any routing device in the routing device set; Any routing device in the routing device set is used to obtain the acquisition request sent by the data caller, query the target routing information corresponding to the target object data, and when the target routing information is found, send the device identifier of the first node device in the node device set to the data caller, so that the data caller can obtain the target object data from the local device of the first node device; the target routing information is associated with the device identifier of the first node device. When any routing device in the routing device set fails to find the target routing information, it sends the device identifier of the second node device in the node device set to the data caller, so that the data caller can obtain the target object data through the second node device. The target object data is obtained by the second node device from the object storage engine and sent to the data caller. After obtaining the target object data, the second node device caches the target object data locally, and after generating the target routing information of the target object data, it broadcasts the target routing information to each routing device in the routing device set so that the target routing information is stored in each routing device. The first node device or the second node device locally caches the target object data using a three-level cache structure. The three-level cache structure includes a first-level cache, a second-level cache, and a third-level cache. The first-level cache is used to distinguish and cache object data from different data sources. The second-level cache is used to locate the data identifier of the object data within a range based on the index offset. The third-level cache is used to read the target object data within the located index range based on the index offset.

12. A data processing apparatus, characterized in that, The data processing device is built into any routing device in the routing device set. This routing device establishes data communication with any node device in the node device set. The routing information generated by this node device after locally storing object data is then stored in each routing device after being broadcast to them. The device includes: The receiving unit is used to receive a request from a data caller for obtaining data of a target object, and to query the target routing information of the target object data. The processing unit is configured to, if the target routing information is found, send the device identifier of the first node device in the node device set to the data caller, so that the data caller can obtain the target object data from the local device of the first node device; the target routing information is associated with the device identifier of the first node device. The processing unit is further configured to send the device identifier of the second node device in the node device set to the data caller when the target routing information is not found, so that the data caller can obtain the target object data through the second node device; the target object data is obtained by the second node device from the object storage engine and sent to the data caller; after obtaining the target object data, the second node device caches the target object data locally, and after generating the target routing information of the target object data, broadcasts the target routing information to each routing device in the routing device set so as to store the target routing information in each routing device; The first node device or the second node device locally caches the target object data using a three-level cache structure. The three-level cache structure includes a first-level cache, a second-level cache, and a third-level cache. The first-level cache is used to distinguish and cache object data from different data sources. The second-level cache is used to locate the data identifier of the object data within a range based on the index offset. The third-level cache is used to read the target object data within the located index range based on the index offset.

13. A routing device, characterized in that, The device includes a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, wherein the memory is used to store a computer program, the routing device includes program instructions, and the processor is configured to invoke the program instructions to execute the method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 10.

15. A computer product, characterized in that, The computer product includes a computer program or computer instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 10.

Images