Cloud vehicle data caching method

Abstract

The invention provides a cloud vehicle data caching method, which is applied to a multi-layer storage architecture, and the multi-layer storage architecture comprises a memory queue layer, a distributed cache layer and a persistent storage layer. The caching method comprises the following steps: acquiring cloud vehicle data; caching the cloud vehicle data to a memory queue layer, obtaining a first cache capacity and a first cache duration of the memory queue layer, and when the first cache capacity exceeds a preset first capacity threshold or the first cache duration exceeds a preset first time interval, transferring the cloud vehicle data to a distributed cache layer; and acquiring a second cache capacity and a second cache duration of the distributed cache layer in real time, and when the second cache capacity exceeds a preset second capacity threshold or the second cache duration exceeds a preset second time interval, transferring the remaining cloud vehicle data to a persistent storage layer. According to the caching method provided by the invention, the memory resource utilization rate can be improved, the vehicle data caching efficiency is improved, and the fluency of the caching process is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle data caching, and in particular to a cloud vehicle data caching method. Background Art

[0002] Cloud vehicle data refers to vehicle-related information collected, transmitted, and stored in the cloud through various sensors, devices, networks, etc. This information can be used to improve vehicle safety, efficiency, and user experience. With the continuous progress of automotive technology, the application scope and importance of cloud vehicle data will become more prominent.

[0003] However, for the caching of cloud vehicle data, the following problems exist in the prior art:

[0004] 1) The data writing speed is slow, and high-concurrency data cannot be processed in time, which easily leads to jamming of the caching system;

[0005] 2) The access to hot data is congested, and the system is often busy when accessing hot data;

[0006] 3) The utilization rate of memory resources is low. While the problems in 1) and 2) exist, there is a large amount of free space in the memory resources;

[0007] 4) It is difficult to ensure data consistency, and the same data obtained by requests at the same time may be inconsistent. Summary of the Invention

[0008] The present invention aims to provide a cloud vehicle data caching method to solve the above technical problems, effectively improve the utilization rate of memory resources, improve the caching efficiency of vehicle data, and ensure the smoothness of the caching process.

[0009] To solve the above technical problems, the present invention provides a cloud vehicle data caching method applied to a multi-layer storage architecture. The multi-layer storage architecture includes a memory queue layer, a distributed cache layer, and a persistent storage layer; the caching method includes the following steps:

[0010] Obtain cloud vehicle data;

[0011] Cache the cloud vehicle data into the memory queue layer, and obtain the first cache capacity and the first cache duration of the memory queue layer in real time. When the first cache capacity exceeds a preset first capacity threshold or the first cache duration exceeds a preset time interval, transfer the remaining cloud vehicle data to the distributed cache layer;

[0012] During the process of transferring the remaining cloud vehicle data to the distributed cache layer, the second cache capacity and the second cache duration of the distributed cache layer are obtained in real time. When the second cache capacity exceeds the preset second capacity threshold or the second cache duration exceeds the preset second time interval, the remaining cloud vehicle data is transferred to the persistent storage layer.

[0013] It should be noted that the first time interval and the second time interval can be set according to actual requirements. For single data caching, it is equivalent to a waiting time. That is, when the first cache duration exceeds the preset first time interval, it means that there is congestion in the cache of the memory queue layer. When the second cache duration exceeds the preset second time interval, it means that there is congestion in the cache of the distributed cache layer.

[0014] In the above solution, the cloud vehicle data can be cached in the order of the memory queue layer, the distributed cache layer, and the persistent storage layer. By determining the cache capacity and cache duration of each cache layer, on the one hand, it can avoid congestion of cloud vehicle data during the caching process, and on the other hand, it can improve the utilization rate of memory resources. That is, when there is congestion in the memory queue layer, the cloud vehicle data is cached in the distributed cache layer in a timely manner. When there is congestion in the distributed cache layer, the cloud vehicle data is cached in the persistent storage layer in a timely manner. The above solution can effectively improve the utilization rate of memory resources, improve the caching efficiency of vehicle data, and ensure the smoothness of the caching process.

[0015] It should be noted that when the cloud vehicle data is directly cached to the distributed cache layer because the first cache duration of the memory queue layer exceeds the preset first time interval, it may be because there is a large amount of data to be cached at the same time. Therefore, when the first cache duration of the memory queue layer decreases, the cloud vehicle data will be cached in the order of the memory queue layer, the distributed cache layer, and the persistent storage layer. Similarly, the cache conversion relationship between the distributed cache layer and the persistent storage layer is the same. This method can effectively improve the utilization rate of memory resources.

[0016] It should be noted that the number of storage layers of the multi-layer storage architecture and the storage capacity of each storage layer can be set according to actual storage needs.

[0017] Further, the memory queue layer is a Disruptor ring buffer layer; caching the cloud vehicle data into the memory queue layer, and obtaining the first cache capacity and the first cache duration of the memory queue layer in real time. When the first cache capacity exceeds a preset first capacity threshold or the first cache duration exceeds a preset first time interval, transferring the remaining cloud vehicle data to the distributed cache layer, including: allocating a sequence number to each vehicle data in the cloud vehicle data to obtain sequence data corresponding to each vehicle data; caching all the sequence data into the ring buffer of the Disruptor ring buffer layer in a circular manner, and obtaining the first cache capacity and the first cache duration of the memory queue layer in real time; when the first cache capacity exceeds the preset first capacity threshold or the first cache duration exceeds the preset first time interval, transferring the remaining cloud vehicle data to the distributed cache layer.

[0018] In the above solution, the Disruptor ring buffer layer is a high-performance and low-latency memory queue architecture, suitable for high-concurrency data storage scenarios. Using the architecture as the first cache layer of the multi-layer storage architecture can ensure the efficient caching of cloud vehicle data, improve the data writing speed, and process data in high-concurrency data scenarios to avoid the situation of cache system jamming.

[0019] Further, the Disruptor ring buffer layer is a fixed-size buffer implemented by a circular array, and its data is stored in a circular manner, avoiding the overhead of dynamic memory allocation. When storing cloud vehicle data, it is necessary to allocate a sequence number to each vehicle data in the cloud vehicle data to identify the storage and reading order of the data. Among them, a lock-free algorithm can be used to achieve efficient data writing during data storage; concurrent processing and broadcast mode can be supported during data reading to improve the data reading efficiency.

[0020] It should be noted that in the Disruptor ring buffer layer, the cloud vehicle data is stored in the ring buffer in fixed-size slots, and each slot stores a sequence data. The Disruptor ring buffer layer is suitable for high-concurrency data writing scenarios, such as vehicle real-time data reporting, where the data retention time is short and it is mainly used for short-term buffering and fast forwarding.

[0021] In the above solution, when the ring buffer of the Disruptor ring buffer layer is not full and the first cache duration does not exceed the preset first time interval, the sequence data will be written into the corresponding slots one by one in a circular manner until the ring buffer is full or the first cache duration exceeds the preset first time interval.

[0022] Further, the caching method further includes obtaining the retention time of each sequence data in the Disruptor ring buffer layer at a preset first time interval, and transferring the sequence data corresponding to the retention time exceeding the preset first retention time threshold to the distributed cache layer in the order of the sequence numbers.

[0023] In the above solution, since the Disruptor ring buffer layer is suitable for short-term buffering and fast forwarding, in order to further improve the utilization rate of memory resources, the sequence data with the retention time exceeding the preset first retention time threshold is transferred to the distributed cache layer in the order of the sequence numbers. Among them, the first retention time threshold can be preset according to the actual parameters of the Disruptor ring buffer layer, and is used to determine the longest retention time of the data in the Disruptor ring buffer layer.

[0024] Further, the distributed cache layer is a Redis cluster cache layer; when caching the cloud vehicle data into the memory queue layer, the first cache capacity and the first cache duration of the memory queue layer are obtained in real time. When the first cache capacity exceeds the preset first capacity threshold or the first cache duration exceeds the preset first time interval, the remaining cloud vehicle data is transferred to the distributed cache layer, including: when the remaining cloud vehicle data is transferred to the distributed cache layer, calculating the storage node of each cloud vehicle data based on the consistent hashing algorithm; storing the cloud vehicle data into the corresponding storage node.

[0025] In the above solution, the Redis cluster cache layer is a distributed storage architecture with multiple storage shards. Each storage shard has a master node and multiple slave nodes. The master node is responsible for writing data, and the slave nodes are used for reading data, effectively improving the data reading performance. Among them, when the master node fails, the slave node is automatically promoted to the master node, which can ensure data availability. When the cloud vehicle data is transmitted to the distributed cache layer for caching, the storage node of each cloud vehicle data can be calculated through the consistent hashing algorithm, and then the cloud vehicle data is stored into the corresponding storage node, avoiding the single-point bottleneck problem that may exist in single-point storage and improving the utilization rate of memory resources.

[0026] It should be noted that when the cloud vehicle data is stored into the corresponding storage node, the master node stores the cloud vehicle data and synchronizes it to the slave nodes through master-slave replication.

[0027] Further, the step of calculating the storage node of each cloud vehicle data based on the consistent hashing algorithm when the remaining cloud vehicle data is transferred to the distributed cache layer includes: when the remaining cloud vehicle data is transferred to the distributed cache layer, obtaining the vehicle ID data based on the cloud vehicle data; calculating the storage node of each cloud vehicle data through the consistent hashing algorithm based on the vehicle ID data and the number of storage shards of the distributed cache layer.

[0028] In the above solution, the cloud vehicle data in the Redis cluster cache layer is stored in the form of key-value pairs, where the key is the vehicle ID data (such as VIN code), and the value is the JSON object of the cloud vehicle data.

[0029] Furthermore, the caching method further includes: obtaining the retention time of the cloud vehicle data corresponding to each storage node in the Redis cluster cache layer at a preset second time interval, and transferring the cloud vehicle data whose retention time exceeds the preset second retention time threshold to the persistent storage layer in the order of the storage nodes.

[0030] It should be noted that the Redis cluster cache layer supports setting the expiration time of data, that is, the second retention time threshold can be set to automatically clean up the expired data and release the memory.

[0031] Furthermore, the persistent storage layer is a time series data storage layer; during the process of transferring the remaining cloud vehicle data to the distributed cache layer, the second cache capacity and the second cache duration of the distributed cache layer are obtained in real time. When the second cache capacity exceeds the preset second capacity threshold or the second cache duration exceeds the preset second time interval, the remaining cloud vehicle data is transferred to the persistent storage layer, including: when the remaining cloud vehicle data is transferred to the persistent storage layer, the remaining cloud vehicle data is grouped and sorted to obtain a vehicle classification data group; the vehicle classification data group is compressed and stored to reduce the occupation of memory resources.

[0032] In the above solution, the time series data storage layer is a storage architecture designed specifically for storing and querying time series data. The cloud vehicle data stored in it is organized by timestamp, and each piece of data stored includes the timestamp, the vehicle ID data, and the data value of the cloud vehicle data. Therefore, it can efficiently query the stored data based on the time range and ID tags. This storage architecture can use an efficient compression algorithm (such as Gorilla encoding) to store time series data when storing data, greatly reducing the storage cost.

[0033] It should be noted that the time series data storage layer can adopt distributed storage, distribute the cloud vehicle data on multiple nodes, and support horizontal expansion and high availability. The time series data storage layer also supports setting a data retention policy, that is, a preset third retention time threshold can be set to automatically clean up the data whose retention time on the time series data storage layer exceeds the preset third retention time threshold.

[0034] Further, when the remaining cloud vehicle data is transferred to the persistent storage layer, the remaining cloud vehicle data is grouped and sorted to obtain a vehicle classification data group, including: when the remaining cloud vehicle data is transferred to the persistent storage layer, vehicle ID data and data timestamps are obtained based on the cloud vehicle data; the cloud vehicle data is grouped based on the vehicle ID data and sorted according to the data timestamps to obtain the vehicle classification data group.

[0035] It should be noted that the time series data storage layer is suitable for long-term storage of historical data (such as cloud vehicle data in the most recent 7 days) and low-frequency access scenarios, and is mainly used for offline analysis and historical queries.

[0036] Further, the first cache capacity and the second cache capacity are obtained through the following formula, specifically including:

[0037] C(L i )=V i ×F i ×T i

[0038] In the formula, i = 1 or 2; when i = 1, C(L1) represents the first cache capacity, V1 represents the number of vehicles in the data reporting memory queue layer, F1 represents the frequency of the data reporting memory queue layer, and T1 represents the data cache retention time of the memory queue layer; when i = 2, C(L2) represents the second cache capacity, V2 represents the number of vehicles in the data reporting distributed cache layer, F2 represents the frequency of the data reporting distributed cache layer, and T2 represents the data cache retention time of the distributed cache layer.

[0039] It should be noted that the third cache capacity of the persistent storage layer can also be calculated through the above formula.

[0040] Further, when caching the cloud vehicle data to the memory queue layer, the first cache capacity and the first cache duration of the memory queue layer are obtained in real time. When the first cache capacity exceeds the preset first capacity threshold or the first cache duration exceeds the preset first time interval, before transferring the remaining cloud vehicle data to the distributed cache layer, it further includes: assigning a version number to each vehicle data in the cloud vehicle data; when the cloud vehicle data is cached to the memory queue layer, transferred to the distributed cache layer, or transferred to the persistent storage layer, if there are two pieces of data for the same vehicle at the same timestamp, the data with the larger version number is used to overwrite the data with the smaller version number.

[0041] To solve the data consistency problem in a distributed storage environment, the above solution assigns a version number to each piece of vehicle data in the cloud vehicle data. This version number can include the timestamp and sequence number of the data. When caching data, version number comparison is used to ensure that the most recent data can overwrite the old data. That is, when multi-version data conflicts occur, the data with a larger version number overwrites the data with a smaller version number, effectively ensuring data consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 FIG. is a schematic flowchart of a method for caching cloud vehicle data according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0044] Please refer to Figure 1 , this embodiment provides a method for caching cloud vehicle data, which is applied to a multi-layer storage architecture. The multi-layer storage architecture includes a memory queue layer, a distributed cache layer, and a persistent storage layer. The caching method includes the following steps:

[0045] S1: Obtain cloud vehicle data;

[0046] S2: Cache the cloud vehicle data into the memory queue layer, and obtain the first cache capacity and the first cache duration of the memory queue layer in real time. When the first cache capacity exceeds a preset first capacity threshold or the first cache duration exceeds a preset first time interval, transfer the remaining cloud vehicle data to the distributed cache layer;

[0047] S3: During the process of transferring the remaining cloud vehicle data to the distributed cache layer, obtain the second cache capacity and the second cache duration of the distributed cache layer in real time. When the second cache capacity exceeds a preset second capacity threshold or the second cache duration exceeds a preset second time interval, transfer the remaining cloud vehicle data to the persistent storage layer.

[0048] It should be noted that the first time interval and the second time interval can be set according to actual needs. For single-data caching, it is equivalent to a waiting time. That is, when the first cache duration exceeds the preset first time interval, it means that there is congestion in the cache of the memory queue layer. When the second cache duration exceeds the preset second time interval, it means that there is congestion in the cache of the distributed cache layer.

[0049] In this embodiment, the cloud vehicle data can be cached in the order of the memory queue layer, the distributed cache layer, and the persistent storage layer. By determining the cache capacity and cache duration of each cache layer, on the one hand, it can avoid congestion of cloud vehicle data during the caching process, and on the other hand, it can improve the utilization rate of memory resources. That is, when congestion occurs in the memory queue layer, the cloud vehicle data is cached in the distributed cache layer in a timely manner. When congestion occurs in the distributed cache layer, the cloud vehicle data is cached in the persistent storage layer in a timely manner. This embodiment can effectively improve the utilization rate of memory resources, improve the caching efficiency of vehicle data, and ensure the smoothness of the caching process.

[0050] It should be noted that when the cloud vehicle data is directly cached to the distributed cache layer because the first cache duration in the memory queue layer exceeds the preset first time interval, it may be because a large amount of data needs to be cached at the same time. Therefore, when the first cache duration in the memory queue layer decreases, the cloud vehicle data will be cached in the order of the memory queue layer, the distributed cache layer, and the persistent storage layer. Similarly, the cache conversion relationship between the distributed cache layer and the persistent storage layer is the same. This method can effectively improve the utilization rate of memory resources.

[0051] It should be noted that the number of storage layers of the multi-layer storage architecture and the storage capacity of each storage layer can be set according to actual storage needs.

[0052] In one embodiment, the memory queue layer is a Disruptor ring buffer layer; when caching the cloud vehicle data to the memory queue layer, the first cache capacity and the first cache duration of the memory queue layer are obtained in real time. When the first cache capacity exceeds the preset first capacity threshold or the first cache duration exceeds the preset first time interval, the remaining cloud vehicle data is transferred to the distributed cache layer, including: allocating a sequence number to each vehicle data in the cloud vehicle data to obtain the sequence data corresponding to each vehicle data; caching all the sequence data into the ring buffer of the Disruptor ring buffer layer in a circular manner, and obtaining the first cache capacity and the first cache duration of the memory queue layer in real time; when the first cache capacity exceeds the preset first capacity threshold or the first cache duration exceeds the preset first time interval, transfer the remaining cloud vehicle data to the distributed cache layer.

[0053] In this embodiment, the Disruptor ring buffer layer is a high-performance and low-latency memory queue architecture, which is suitable for high-concurrency data storage scenarios. Using the architecture as the first cache layer of the multi-layer storage architecture can ensure the efficient caching of cloud vehicle data, improve the data writing speed, and process data in high-concurrency data to avoid the situation of cache system jamming.

[0054] In one embodiment, the Disruptor ring buffer layer is a fixed-size buffer implemented as a circular array, and its data is stored in a circular manner, avoiding the overhead of dynamic memory allocation. When storing cloud vehicle data, a sequence number needs to be assigned to each piece of vehicle data in the cloud vehicle data to identify the storage and reading order of the data. Among them, a lock-free algorithm can be used to achieve efficient writing of data during data storage; concurrent processing and broadcast mode can be supported during data reading to improve the data reading efficiency.

[0055] It should be noted that in the Disruptor ring buffer layer, cloud vehicle data is stored in the ring buffer in fixed-size slots, and each slot stores a sequence of data. The Disruptor ring buffer layer is suitable for high-concurrency data writing scenarios, such as vehicle real-time data reporting, where the data retention time is short and it is mainly used for short-term buffering and fast forwarding.

[0056] In this embodiment, when the Disruptor ring buffer layer is not full and the first cache duration does not exceed the preset first time interval, the sequence data will be written into the corresponding slots one by one in a circular manner until the ring buffer is full or the first cache duration exceeds the preset first time interval.

[0057] In one embodiment, the caching method further includes obtaining the retention time of each sequence data in the Disruptor ring buffer layer at a preset first time interval, and transferring the sequence data corresponding to the retention time exceeding the preset first retention time threshold to the distributed cache layer in the order of the sequence numbers.

[0058] In this embodiment, since the Disruptor ring buffer layer is suitable for short-term buffering and fast forwarding, in order to further improve the utilization rate of memory resources, the sequence data with a retention time exceeding the preset first retention time threshold is transferred to the distributed cache layer in the order of the sequence numbers. Among them, the first retention time threshold can be preset according to the actual parameters of the Disruptor ring buffer layer to determine the longest retention time of the data in the Disruptor ring buffer layer.

[0059] In one embodiment, the distributed cache layer is a Redis cluster cache layer; when caching cloud vehicle data into the memory queue layer, the first cache capacity and the first cache duration of the memory queue layer are obtained in real time. When the first cache capacity exceeds the preset first capacity threshold or the first cache duration exceeds the preset first time interval, the remaining cloud vehicle data is transferred to the distributed cache layer, including: when the remaining cloud vehicle data is transferred to the distributed cache layer, the storage node of each cloud vehicle data is calculated based on the consistent hashing algorithm; the cloud vehicle data is stored in the corresponding storage node.

[0060] In this embodiment, the Redis cluster cache layer is a distributed storage architecture with multiple storage shards. Each storage shard has a master node and multiple slave nodes. The master node is responsible for writing data, and the slave nodes are used for reading data, effectively improving the data reading performance. Among them, when the master node fails, the slave node automatically promotes to the master node, which can ensure data availability. When cloud vehicle data is transmitted to the distributed cache layer for caching, the storage node of each cloud vehicle data can be calculated through the consistent hashing algorithm, and then the cloud vehicle data is stored in the corresponding storage node, avoiding the single-point bottleneck problem that may exist in single-point storage and improving the utilization rate of memory resources.

[0061] It should be noted that when cloud vehicle data is stored in the corresponding storage node, the master node stores the cloud vehicle data and synchronizes it to the slave nodes through master-slave replication.

[0062] In one embodiment, when the remaining cloud vehicle data is transferred to the distributed cache layer, calculating the storage node of each cloud vehicle data based on the consistent hashing algorithm includes: when the remaining cloud vehicle data is transferred to the distributed cache layer, obtaining vehicle ID data based on the cloud vehicle data; based on the vehicle ID data and the number of storage shards of the distributed cache layer, calculating the storage node of each cloud vehicle data through the consistent hashing algorithm.

[0063] In this embodiment, the cloud vehicle data in the Redis cluster cache layer is stored in the form of key-value pairs, where the key is the vehicle ID data (such as VIN code), and the value is the JSON object of the cloud vehicle data.

[0064] In one embodiment, the consistent hashing algorithm can be expressed as:

[0065] S(vid) = hash(vid) mod N

[0066] In the formula, vid represents the vehicle ID data, S(vid) represents the storage node corresponding to the vehicle ID data; hash(·) represents the hash function; N represents the number of storage shards. It should be noted that the number of storage shards is an integer multiple of the storage nodes. Without reaching the cpu and memory limits, the more the number of storage shards, the faster the system read and write rate.

[0067] The data storage method using storage shards in this embodiment can effectively disperse the data pressure and avoid the single-point storage bottleneck.

[0068] In one embodiment, the caching method further includes: obtaining the retention time of the cloud vehicle data corresponding to each storage node in the Redis cluster cache layer at a preset second time interval, and transferring the cloud vehicle data whose retention time exceeds the preset second retention time threshold to the persistent storage layer in the order of the storage nodes.

[0069] It should be noted that the Redis cluster cache layer supports setting the expiration time of data, that is, the second retention time threshold can be set to automatically clean up expired data and release memory.

[0070] In one embodiment, the persistent storage layer is a time series data storage layer; during the process of transferring the remaining cloud vehicle data to the distributed cache layer, the second cache capacity and the second cache duration of the distributed cache layer are obtained in real time. When the second cache capacity exceeds the preset second capacity threshold or the second cache duration exceeds the preset second time interval, the remaining cloud vehicle data is transferred to the persistent storage layer, including: when the remaining cloud vehicle data is transferred to the persistent storage layer, the remaining cloud vehicle data is grouped and sorted to obtain a vehicle classification data group; the vehicle classification data group is compressed and stored to reduce the occupation of memory resources.

[0071] In this embodiment, the time series data storage layer is a storage architecture designed specifically for storing and querying time series data. The cloud vehicle data stored in it is organized by timestamp, and each piece of data stored includes a timestamp, vehicle ID data, and the data value of the cloud vehicle data. Therefore, it can efficiently query the stored data based on the time range and ID tags. This storage architecture can use an efficient compression algorithm (such as Gorilla encoding) to store time series data when storing data, greatly reducing the storage cost.

[0072] It should be noted that the time series data storage layer can adopt distributed storage, distribute the cloud vehicle data on multiple nodes, and support horizontal expansion and high availability. The time series data storage layer also supports setting a data retention policy, that is, a third retention time threshold can be preset to automatically clean up the data that has been retained on the time series data storage layer for more than the preset third retention time threshold.

[0073] In one embodiment, when the remaining cloud vehicle data is transferred to the persistent storage layer, the remaining cloud vehicle data is grouped and sorted to obtain a vehicle classification data group, including: when the remaining cloud vehicle data is transferred to the persistent storage layer, the vehicle ID data and the data timestamp are obtained based on the cloud vehicle data; the cloud vehicle data is grouped based on the vehicle ID data and sorted according to the data timestamp to obtain a vehicle classification data group.

[0074] It should be noted that the time series data storage layer is suitable for long-term storage of historical data (such as cloud vehicle data in the recent 7 days) and low-frequency access scenarios, and is mainly used for offline analysis and historical queries.

[0075] In one embodiment, the first cache capacity and the second cache capacity are obtained through the following formula, specifically including:

[0076] C(L i ) = V i ×F i ×T i

[0077] In the formula, i = 1 or 2; when i = 1, C(L1) represents the first cache capacity, V1 represents the number of vehicles in the data reporting memory queue layer, F1 represents the frequency of the data reporting memory queue layer, and T1 represents the data cache retention time of the memory queue layer; when i = 2, C(L2) represents the second cache capacity, V2 represents the number of vehicles in the data reporting distributed cache layer, F2 represents the frequency of the data reporting distributed cache layer, and T2 represents the data cache retention time of the distributed cache layer.

[0078] It should be noted that the third cache capacity of the persistent storage layer can also be calculated by the above formula. This cache capacity calculation method can ensure the reasonable allocation of cache resources and improve the utilization rate of memory resources.

[0079] In an embodiment, when caching the cloud vehicle data to the memory queue layer, the first cache capacity and the first cache duration of the memory queue layer are obtained in real time. When the first cache capacity exceeds the preset first capacity threshold or the first cache duration exceeds the preset first time interval, before transferring the remaining cloud vehicle data to the distributed cache layer, it further includes: allocating a version number to each vehicle data in the cloud vehicle data; when the cloud vehicle data is cached to the memory queue layer, transferred to the distributed cache layer, or transferred to the persistent storage layer, if there are two pieces of data for the same vehicle at the same timestamp, the data with the larger version number is used to overwrite the data with the smaller version number.

[0080] To solve the data consistency problem in the distributed storage environment, in this embodiment, a version number is allocated to each vehicle data in the cloud vehicle data. This version number can include the timestamp and sequence number of the data. When caching data, by comparing the version numbers, it is ensured that the latest data can overwrite the old data, that is, when multi-version data conflicts occur, the data with the larger version number is used to overwrite the data with the smaller version number, effectively ensuring data consistency.

[0081] Further, when data needs to be read from the multi-layer storage architecture, it is preferentially read from the memory queue layer; if the corresponding data is not found in the memory queue layer, it is read from the distributed cache layer; if the corresponding data is not found in the distributed cache layer, it is read from the persistent storage layer.

[0082] The above data reading method can optimize the data reading efficiency, ensure the fast reading of hot data, and avoid the situation of congestion in accessing hot data or the system being busy frequently when accessing hot data.

[0083] Further, to further illustrate the technical advantages of the cloud vehicle data caching method proposed by the present invention, the performance of the multi-layer storage architecture can be calculated using the following formula, specifically including:

[0084] The multi-layer storage architecture can improve data caching performance:

[0085] P = min(C(L2) / t, N×W)

[0086] In the formula, P represents the overall caching performance of the distributed caching layer; C(L2) represents the second cache capacity, indicating the maximum write capacity that L2 can handle; t represents the time interval at which write operations arrive, indicating the rate of write requests; N represents the number of storage shards; W represents the write capacity of a single storage shard, indicating the maximum write capacity that each storage shard can handle. Through the multi-level caching and sharding mechanism, the processing ability for high-concurrency writes is significantly improved.

[0087] The multi-layer storage architecture can reduce access latency:

[0088] L = ∑(p(i)×I(i))

[0089] In the formula, L represents the average access latency of the system; p(i) represents the hit probability of the i-th layer cache, indicating the probability of reading data from the i-th layer cache; I(i) represents the access latency of the i-th layer cache, indicating the time required to read data from the i-th layer cache. The higher the cache hit probability, the lower the access latency. Through the multi-layer storage architecture, hot data can be preferentially obtained from the high-performance cache layer (such as the memory queue layer or the distributed caching layer), thereby reducing the overall access latency.

[0090] The multi-layer storage architecture can optimize resource utilization:

[0091] U = ∑(C(i) / M(i))

[0092] In the formula, U represents the overall resource utilization rate of the system; C(i) represents the actual used capacity of the i-th layer cache, and M(i) represents the total capacity of the i-th layer cache. By reasonably allocating the cache capacities at all levels, the utilization rate of memory and storage resources is improved.

[0093] It should be noted that for the i-th layer cache in the above calculation process, when i = 1, it represents the memory queue layer, when i = 2, it represents the distributed caching layer, and when i = 3, it represents the persistent storage layer.

[0094] The cloud vehicle data caching method provided in this embodiment can effectively improve the utilization rate of memory resources, improve the vehicle data caching efficiency, and ensure the smoothness of the caching process during the cloud vehicle data storage process.

[0095] The above are the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A cloud vehicle data caching method, characterized in that: Applied to a multi-layer storage architecture, the multi-layer storage architecture includes a memory queue layer, a distributed cache layer and a persistent storage layer; the cache method includes the following steps: Get cloud-based vehicle data; Cache the cloud vehicle data in the memory queue layer, obtain the first cache capacity and the first cache duration of the memory queue layer in real time, and when the first cache capacity exceeds the preset first capacity threshold or the first cache duration exceeds the preset first time interval, transfer the remaining cloud vehicle data to the distributed cache layer; During the process of transferring the remaining cloud-based vehicle data to the distributed cache layer, the second cache capacity and second cache duration of the distributed cache layer are obtained in real time. When the second cache capacity exceeds the preset second capacity threshold or the second cache duration exceeds the preset second time interval, the remaining cloud-based vehicle data is transferred to the persistent storage layer.

2. A cloud vehicle data caching method according to claim 1, characterized in that: The memory queue layer is a Disruptor ring buffer layer; the cloud vehicle data is cached in the memory queue layer, a first cache capacity and a first cache duration of the memory queue layer are obtained in real time, and when the first cache capacity exceeds a preset first capacity threshold or the first cache duration exceeds a preset first time interval, the remaining cloud vehicle data is transferred to the distributed cache layer, including: Assign a serial number to each piece of vehicle data in the cloud vehicle data to obtain the serial data corresponding to each piece of vehicle data; All sequence data are cached in a circular manner in the ring buffer of the Disruptor ring buffer layer, and the first cache capacity and first cache duration of the memory queue layer are obtained in real time; When the first cache capacity exceeds a preset first capacity threshold or the first cache duration exceeds a preset first time interval, the remaining cloud vehicle data is transferred to the distributed cache layer.

3. A cloud vehicle data caching method according to claim 2, characterized in that: It also includes obtaining the retention time of each sequence data in the Disruptor ring buffer layer according to a preset first time interval, and transferring the sequence data corresponding to the retention time exceeding the preset first retention time threshold to the distributed cache layer in the order of the sequence number.

4. A cloud vehicle data caching method according to claim 1, characterized in that: The distributed cache layer is a Redis cluster cache layer; the cloud vehicle data is cached in the memory queue layer, and the first cache capacity and the first cache duration of the memory queue layer are obtained in real time. When the first cache capacity exceeds the preset first capacity threshold or the first cache duration exceeds the preset first time interval, the remaining cloud vehicle data is transferred to the distributed cache layer, including: When the remaining cloud vehicle data is transferred to the distributed cache layer, the storage node of each cloud vehicle data is calculated based on the consistent hashing algorithm; Store the cloud vehicle data in the corresponding storage node.

5. A cloud vehicle data caching method according to claim 4, characterized in that: When the remaining cloud vehicle data is transferred to the distributed cache layer, the storage node of each cloud vehicle data is calculated based on the consistent hashing algorithm, including: When the remaining cloud vehicle data is transferred to the distributed cache layer, the vehicle ID data is obtained based on the cloud vehicle data; Based on the vehicle ID data and the number of storage shards in the distributed cache layer, the storage node for each cloud vehicle data is calculated using a consistent hashing algorithm.

6. A cloud vehicle data caching method according to claim 4, characterized in that: It also includes obtaining the retention time of the cloud vehicle data corresponding to each storage node in the Redis cluster cache layer according to a preset second time interval, and transferring the cloud vehicle data corresponding to the retention time exceeding the preset second retention time threshold to the persistent storage layer in the order of the storage nodes.

7. The cloud vehicle data caching method according to claim 1, characterized in that: The persistent storage layer is a time series data storage layer; in the process of transferring the remaining cloud vehicle data to the distributed cache layer, the second cache capacity and the second cache duration of the distributed cache layer are obtained in real time, and when the second cache capacity exceeds a preset second capacity threshold or the second cache duration exceeds a preset second time interval, the remaining cloud vehicle data is transferred to the persistent storage layer, including: When the remaining cloud vehicle data is transferred to the persistent storage layer, the remaining cloud vehicle data is grouped and sorted to obtain a vehicle classification data group; The vehicle classification data group is stored after data compression.

8. A cloud vehicle data caching method according to claim 7, characterized in that: When the remaining cloud vehicle data is transferred to the persistent storage layer, the remaining cloud vehicle data is grouped and sorted to obtain a vehicle classification data group, including: When the remaining cloud vehicle data is transferred to the persistent storage layer, the vehicle ID data and data timestamp are obtained based on the cloud vehicle data; The cloud vehicle data is grouped based on the vehicle ID data and sorted according to the data timestamp to obtain a vehicle classification data group.

9. A cloud vehicle data caching method according to any one of claims 1 to 8, characterized in that: The first cache capacity and the second cache capacity are obtained by the following formulas, specifically including: C(L i )=V i ×F i ×T i Wherein, i=1 or 2; when i=1, C(L1) represents the first cache capacity, V1 represents the number of vehicles reporting data to the memory queue layer, F1 represents the frequency of data reporting to the memory queue layer, and T1 represents the data cache retention time of the memory queue layer; when i=2, C(L2) represents the second cache capacity, V2 represents the number of vehicles reporting data to the distributed cache layer, F2 represents the frequency of data reporting to the distributed cache layer, and T2 represents the data cache retention time of the distributed cache layer.

10. A cloud vehicle data caching method according to claim 2, characterized in that: In the step of caching the cloud vehicle data in the memory queue layer, obtaining a first cache capacity and a first cache duration of the memory queue layer in real time, and before transferring the remaining cloud vehicle data to the distributed cache layer when the first cache capacity exceeds a preset first capacity threshold or the first cache duration exceeds a preset first time interval, further comprising: assigning a version number to each piece of vehicle data in the cloud vehicle data; When the cloud vehicle data is cached in the memory queue layer, transferred to the distributed cache layer, or transferred to the persistent storage layer, if there are two data for the same vehicle at the same timestamp, the data with a larger version number will overwrite the data with a smaller version number.

Images