Method and system for matching big data with offline data warehouse, electronic equipment and medium

Abstract

The invention provides a method and system for matching big data with an offline data warehouse, electronic equipment and a storage medium, and aims to solve the problems of high difficulty in multi-domain data fusion and high database burden, the method comprises the following steps: carrying out custom hash algorithm processing on a user identifier in a real-time computing engine to distribute the user identifier to a plurality of channel instances; user portrait information in the external database is read in each channel instance through the same algorithm; the user portrait information meeting the first activeness is loaded to a first-level cache, and the first-level cache exists in the real-time big data platform; taking a lightweight memory bank deployed in a plurality of servers in a distributed manner as a second-level cache, and loading the user portrait information meeting the second activeness into the second-level cache; and caching the original total data of the user portrait information in a preset physical database as a third-level cache. According to the method, the efficiency of industry-level real-time data processing is effectively improved, and higher service adaptability is achieved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of big data cache, and in particular to a method for matching big data with an offline data warehouse, a system for matching big data with an offline data warehouse, an electronic device, and a computer-readable storage medium. Background Art

[0002] At present, the amount of data in various industries is growing exponentially. How to quickly extract value from these data, respond to market changes in real time, and make timely decisions is a challenge that enterprises need to face. Real-time processing technology is the key to solving this problem. In actual situations, data production and collection come from different fields and systems, which often involve the design of data hierarchy and structure. Some data are aggregated from the original layer, and some data are conditionally associated at the application layer. The first thing that real-time data processing faces is a massive data source. In the process of fast calculation, other data sources are often required to be supplemented and integrated. When multiple data sources are aggregated, it will bring exponential pressure to computing power, network IO (input / output), and disk reading and writing. Especially in the field of communication technology, the production and operation data of operators generally come from different domains and different systems. In the face of some innovative marketing scenarios in digital transformation, data across multiple domains or platforms is usually required. The integration of multi-domain data is difficult. The network domain and business domain will produce massive data, each with its own data link. Moreover, innovative scenarios during digital transformation require collision and matching of data from different domains. For example, when the real-time signaling data on the network side needs to be associated with and matched with the user portrait data on the business side, if each piece of signaling data is matched and collided with the user portrait information database in the data warehouse, a large amount of computing power and network IO will be consumed, and the peak value will exceed 10,000 Qps (Queries-per-second), causing a huge burden on the database. Currently, no database or storage medium can meet such high concurrent query access.

[0003] At present, there are still certain challenges for real-time marketing in specific scenarios. The usual practice is to use a period (for example, one hour) as a calculation batch, accumulate nearly one hour of data in advance, and then perform correlation matching of user profile related elements based on this data. This method can be called quasi-real-time computing. Its advantage is that it can process a large amount of data and analyze it, but its main disadvantage is that there is a lag in computing, which may lead to missing the best marketing opportunity. Compared with traditional big data models, the difference of quasi-real-time computing is not significant because it still relies on batch data processing and cannot achieve real-time response. However, in some specific scenarios, real-time monitoring and timely response to user behavior and rapid marketing decision-making are the key to exerting marketing effectiveness. Real-time marketing can maximize user conversion rate and brand influence by quickly identifying user behavior and instantly triggering corresponding personalized recommendations or promotion strategies. It can be seen that this method is still difficult to meet the requirements in certain situations. Summary of the invention

[0004] In order to at least solve the problems in the prior art that the fusion of multi-domain data is difficult, the collision and matching of data in different domains consumes a lot of computing power, causing a huge burden on the database, and batch data processing cannot achieve real-time response. The present disclosure provides a method for matching big data with an offline data warehouse, a system for matching big data with an offline data warehouse, an electronic device, and a computer-readable storage medium that can effectively solve the performance bottleneck in the process of associating and matching high-flow real-time data with massive data warehouse data, effectively improve the efficiency of industry-level real-time data processing, and have stronger business adaptability.

[0005] In a first aspect, the present disclosure provides a method for matching big data with an offline data warehouse, the method comprising:

[0006] The user ID is processed using a custom hash algorithm in the real-time computing engine, and the user ID is assigned to multiple channel instances based on the processing results;

[0007] In each channel instance, the user profile information in the external database is read through the same hash algorithm;

[0008] Loading user profile information that meets the first level of activity into a first-level cache that exists within the real-time big data platform;

[0009] A lightweight memory library distributed across multiple servers will be used as the secondary cache, and user profile information that meets the second level of activity will be broken up and loaded into the secondary cache;

[0010] The original full data of user portrait information is cached in the preset physical database as a third-level cache.

[0011] Furthermore, the method further comprises:

[0012] Design life cycle indicators for data in the first-level cache, including hit rate, hit count, cache miss rate, and cache expiration time;

[0013] If the user portrait information data does not meet the life cycle indicators, the cache will be downgraded or removed.

[0014] Furthermore, the method further comprises:

[0015] Set lifecycle management and cache update rules for the first-level cache, including:

[0016] Set the maximum number of cache entries. When the cache exceeds this number, the least frequently used data will be eliminated according to the preset strategy.

[0017] Set the cache expiration time. When a cache item is not created or overwritten within the specified time, the cache item will be automatically removed.

[0018] Set the cache timeout. When a cache item is not read or written within the specified time, the cache item will be removed.

[0019] Set the cache refresh time. If the cache item is not created or overwritten within the specified time, the system will force the cache to be refreshed and always return the old data until new data arrives.

[0020] Furthermore, the method further comprises:

[0021] By analyzing the historical data of the most recent preset time, the access patterns of different types of users are collected, and the activity level of each user in the most recent preset time is analyzed. The users are then divided into high-frequency active users, secondary active users, and inactive users according to their activity levels.

[0022] The high-frequency active users are set as users who meet the first level of activity, and the secondary active users are set as users who meet the second level of activity.

[0023] Furthermore, the lightweight memory library is selected from any one of the following:

[0024] Redis, Aerospike, or memcached, deploy the selected memory library as a lightweight component on the server.

[0025] Furthermore, the preset physical database is an RDS (Relational Database Service) database.

[0026] Furthermore, the method further comprises:

[0027] The user data read from the first-level cache is set to account for X proportion of the total user data, the user data read from the second-level cache is set to account for Y proportion of the total user data, and the user data read from the third-level cache is set to account for Z proportion of the total user data;

[0028] Among them, 30%≤X≤40%, 40%≤Y≤50%, 10%≤Z≤30%, and X+Y+Z=100%.

[0029] In a second aspect, the present disclosure provides a system for matching big data with an offline data warehouse, the system comprising:

[0030] An allocation module, which is configured to perform a custom hash algorithm process on the user ID in the real-time computing engine, and allocate the user ID to multiple channel instances according to the processing result;

[0031] A reading module is configured to read user profile information from an external database using the same hash algorithm in each channel instance;

[0032] A first loading module is configured to load user portrait information that meets a first activity level into a first-level cache, where the first-level cache exists inside the real-time big data platform;

[0033] The second loading module is configured to use a lightweight memory library distributedly deployed in multiple servers as a secondary cache, break up the user profile information that meets the second activity level, and load it into the secondary cache;

[0034] The cache module is configured to cache the original full data of the user portrait information in a preset physical database as a third-level cache.

[0035] In a third aspect, the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method for matching big data with an offline data warehouse as described in any one of the first aspects.

[0036] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for matching big data with an offline data warehouse as described in any one of the first aspects above is implemented.

[0037] Beneficial effects:

[0038] The present invention provides a method for matching big data with an offline data warehouse, a system for matching big data with an offline data warehouse, an electronic device, and a storage medium; a three-level cache system is constructed between the real-time signaling data on the network side and the user information data on the service side. Through the sharing, buffering, and synchronization of multiple layers of cache, the user number in the signaling data is efficiently matched and associated with other user portrait information. It can effectively solve the performance bottleneck in the process of associating and matching high-speed real-time data with massive data warehouse data. Through the combination of refined caching strategies, dynamic user portrait matching, and efficient storage and computing, it effectively improves the efficiency of industry-level real-time data processing, has stronger business adaptability and higher technical value, and thus provides strong technical support and empowerment for operators' innovative business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a method for matching big data with an offline data warehouse provided in the first embodiment of the present disclosure;

[0040] Figure 2A schematic diagram of user activity distribution statistics provided by an embodiment of the present disclosure;

[0041] Figure 3 A schematic diagram of an overall implementation scheme of a big data matching offline data warehouse provided in an embodiment of the present disclosure;

[0042] Figure 4 An architecture diagram of a system for matching big data with an offline data warehouse provided in an embodiment of the present disclosure;

[0043] Figure 5 This is an architecture diagram of an electronic device provided in Embodiment 2 of the present disclosure. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention, rather than to limit the present invention.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence; and, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.

[0046] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings.

[0047] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present disclosure, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0048] At present, the first thing that operators face in real-time data processing is a massive amount of data sources. In the process of fast calculation, other data sources are often required to supplement and integrate. When multiple data sources are aggregated, it will bring exponential pressure to computing power, network IO, and disk reading and writing. Specific problems include:

[0049] (1) Data sources are scattered and integration is difficult

[0050] Operators' production and operation data usually come from different domains and systems. In the process of digital transformation, some innovative marketing scenarios require data analysis and application across multiple domains or platforms. However, data integration across different domains often faces great challenges. The network domain and business domain each generate massive amounts of data, and these data have independent links, making cross-domain integration and collaboration complex and difficult.

[0051] (2) High concurrency problem of data matching in innovation scenarios

[0052] In the innovative marketing scenarios of digital transformation, it is often necessary to collide and match data from different domains. For example, the real-time signaling data on the network side needs to be associated with the user profile data on the business side. If each piece of signaling data needs to be matched with the user profile information in the data warehouse, it will inevitably consume a lot of computing resources and network IO. Considering that the peak query requests may exceed 10,000 per second (QPS), this high concurrent query pressure will bring a great burden to the database. At present, any database or storage medium can hardly withstand such high concurrent query access.

[0053] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned problems existing in the prior art are described in detail below with specific embodiments. It is understandable that in the embodiments of the present application, the execution subject can perform some or all of the steps in the embodiments of the present application, and these steps or operations are only examples. The embodiments of the present application can also perform other operations or deformations of various operations. In addition, each step can be performed in a different order presented in the embodiments of the present application, and it is possible not to perform all the operations in the embodiments of the present application. And, the following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0054] Figure 1 A flowchart of a method for matching big data with an offline data warehouse provided in the first embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes:

[0055] Step S101: performing a custom hash algorithm process on the user ID in the real-time computing engine, and assigning the user ID to multiple channel instances according to the processing result;

[0056] Step S102: reading user portrait information in an external database using the same hash algorithm in each channel instance;

[0057] Step S103: Loading user portrait information that meets the first activity level into a first-level cache, where the first-level cache exists inside the real-time big data platform;

[0058] Step S104: using a lightweight memory library distributedly deployed in multiple servers as a secondary cache, and breaking up the user portrait information that meets the second activity level and loading it into the secondary cache;

[0059] Step S105: Cache the original full data of the user portrait information in a preset physical database as a third-level cache.

[0060] The disclosed embodiment refers to the three-level cache structure of the CPU and draws on the mechanism of L1, L2, and L3 caches to synchronize the read and write speeds of each component through collaboration. On this basis, an efficient cache architecture that adapts to the needs of large-scale real-time data processing is constructed.

[0061] First, in the selection of big data real-time computing engine technology, Apache Flink is selected as the core computing framework. After deployment, Flink's internal partitions are partitioned using a hash algorithm to evenly distribute data to each computing node, thereby improving computing efficiency and resource utilization. The specific code structure is as follows:

[0062] public int partition(String msisdn,int numPartitions){

[0063] long partId=Tools.convPartition(msisdn,numPartitions);

[0064] return Long.valueOf(partId).intValue();

[0065] }

[0066] public static long convPartition(String msisdn,int numPartitions){

[0067] try{

[0068] CRC32 crc = new CRC32();

[0069] crc.update(msisdn.getBytes());

[0070] long hashValue=crc.getValue();

[0071] long partitionId=hashValue%numPartitions;

[0072] return partitionId;

[0073] }catch(Exception e){

[0074] e.printStackTrace();

[0075] }

[0076] return -1;

[0077] }

[0078] Among them, the input parameters: msisdn is the mobile phone number of China Unicom users and the unique identifier of the partition; numPartitions is the number of partitions, which is usually set according to the user scale and the computing resources of the platform;

[0079] Use the CRC32 hash algorithm to hash the msisdn and convert the phone number into a hash value. Perform a modulo operation on the hash value and numPartitions to obtain a partition ID in the range of [0, numPartitions) to ensure that user data can be evenly distributed to each partition.

[0080] For example, if there are 10 million users, 15 40-core 128GB servers can support 20 partitions. Based on this configuration, each partition will be allocated 500,000 users. Through the hash algorithm, the user's mobile phone number (msisdn) will be evenly distributed to the 20 partitions.

[0081] The Flink engine uses the same partitioning algorithm to read user profile information from the external database and performs multi-level caching, including:

[0082] First level cache (localCache):

[0083] After the real-time signaling data is processed by a custom hash algorithm on the user's mobile phone number, it is evenly distributed to multiple channel instances. In each channel instance, the same hash algorithm is used to load part of the user profile information into the system's heap memory for storage. This memory area is called localCache, which is located inside the real-time big data platform. The part of the user profile information stored in it is the user data with higher activity that meets the first activity level.

[0084] Second level cache (scaleCache):

[0085] When the real-time signaling data does not hit in the first-level cache (localCache), cache breakdown occurs, and the system will turn to the second-level cache. The second-level cache adopts a distributed architecture, deploys lightweight memory libraries on multiple servers, and disperses and caches the full amount of user portrait information in multiple server nodes. Compared with the first-level cache, the second-level cache can store a larger amount of data, contains more user portrait information, and has stronger data sharing capabilities and load balancing advantages. However, the disadvantage of the second-level cache is that it consumes more server memory resources. The user portrait information cached in the second-level cache is user data with an intermediate activity that meets the second activity. The first activity and the second activity can be set according to actual conditions. For example, in a preferred implementation, the value of the first activity can be set according to the data that can be stored in the first-level cache and the user activity of the overall data, and the user portrait information with a high activity ranking is cached in the first-level cache. The second activity can be set in the same way.

[0086] Level 3 cache (RDSCache):

[0087] When the user profile is not hit in both the first-level cache and the second-level cache, the system will finally access the preset physical database RDSCache. As the last backup cache, the preset physical database needs to be able to meet the needs of large-scale data access in terms of storage and performance, while ensuring the persistent storage of data. Although the access speed is slower than the first two levels of cache, it provides data reliability and integrity, and is an effective supplement to the cache system.

[0088] Through the collaborative work of the three-level cache, the system can optimize the cache hit rate and data access speed at different levels, thereby ensuring that when faced with massive amounts of data, it can still efficiently support the matching of real-time signaling data with user portraits and improve overall system performance.

[0089] The disclosed embodiment builds a three-level cache system between the real-time signaling data on the network side and the user information data on the service side. Through the sharing, buffering and synchronization of multiple layers of cache, the user number in the signaling data is efficiently matched and associated with other user profile information (such as name, age, gender, package, monthly rent, etc.). The system can effectively solve the performance bottleneck in the process of associating and matching high-speed real-time data with massive data warehouse data, thereby providing strong technical support and empowerment for operators' innovative business scenarios.

[0090] Furthermore, the method further comprises:

[0091] Design life cycle indicators for data in the first-level cache, including hit rate, hit count, cache miss rate, and cache expiration time;

[0092] If the user portrait information data does not meet the life cycle indicators, the cache will be downgraded or removed.

[0093] In order to ensure efficient access, multiple lifecycle indicators are designed for the first-level cache, such as hit rate, number of hits, cache miss rate, and cache expiration time. Hit rate refers to the ratio of the number of times data is successfully obtained from the cache to the total number of requests. Cache miss rate refers to the ratio of the number of cache misses to the total number of accesses when the cache is used. These indicators can help determine and optimize cache performance. Through lifecycle indicators, data can be updated to ensure that frequently queried user data is loaded into the cache as much as possible, thereby improving access speed and overall system performance. Prioritize, lifecycle indicators can also be designed for the second-level cache.

[0094] Since the heap memory area of ​​the computing engine is limited and valuable, it is reasonable to load about 1 / 3 of the number of users in this partition according to experience;

[0095] According to the above assumptions, if there are 500,000 users in each partition and the cache is loaded at a ratio of 1 / 3, then each partition will have about 150,000 user portraits, each of which has about 50 tags, which is about 4,500 bytes (considering character encoding, object overhead, and data storage structure). Therefore, the memory consumption of each partition is about: 150,000 × 4,500 bytes ≈ 675MB;

[0096] If there are 20 partitions, the total memory consumption is approximately: 675MB×20≈13.5GB;

[0097] In this way, the first-level cache (localCache) will be able to effectively store the data of each partition within the limit of 20GB of memory, and reasonably allocate memory resources to ensure efficient real-time computing and user profile matching.

[0098] Furthermore, the method further comprises:

[0099] Set lifecycle management and cache update rules for the first-level cache, including:

[0100] Set the maximum number of cache entries. When the cache exceeds this number, the least frequently used data will be eliminated according to the preset strategy.

[0101] Set the cache expiration time. When a cache item is not created or overwritten within the specified time, the cache item will be automatically removed.

[0102] Set the cache timeout. When a cache item is not read or written within the specified time, the cache item will be removed.

[0103] Set the cache refresh time. If the cache item is not created or overwritten within the specified time, the system will force the cache to be refreshed and always return the old data until new data arrives.

[0104] To ensure the hit rate of the first-level cache, it is necessary to design reasonable lifecycle management and cache update rules. The important indicators are as follows:

[0105] (1) maximumSize (maximum number of cache entries): the maximum number of entries allowed to be stored. When this number is exceeded, the least frequently used data will be eliminated according to certain strategies (such as LRU, LFU, etc.). Assume that a maximum of 150,000 user profiles are stored in the cache. If the cache capacity is full and more user data needs to be cached, the entries that have not been accessed for the longest time will be deleted according to the cache strategy (for example, the least used strategy) to ensure that new user data can be loaded into the cache.

[0106] (2) expireAfterWrite (cache expiration time): When a cache item is not created or overwritten within the specified time, the cache item will be automatically removed. This strategy ensures that the data stored in the cache is relatively "fresh" and avoids storing expired or invalid data. For example, if the user profile is last written to the cache for more than 1 hour without new updates or overwriting, the cache item will be automatically cleared. Assuming that the user data has not been modified or accessed within 1 hour, after the time expires, the cache system will remove the user's profile information to ensure that the memory always stores recently active data.

[0107] (3) expireAfterAccess (cache timeout): When a cache item has not been read or written within a specified time, the cache item will be removed. This mechanism helps clean up cached data that has not been accessed for a long time. Suppose a user has not performed any operations or queries in the past 3 hours, then his cache entry will be removed. In this way, if a user has not accessed or modified their data for a period of time, the cached data will expire and will not occupy cache space, thereby optimizing memory usage.

[0108] (4) refreshAfterWrite (cache refresh time): If the cache item is not created or overwritten within the specified time, the system will force a cache refresh and always return the old data until new data arrives. This mechanism can ensure that cached data will not expire in high concurrency situations, while avoiding frequent access to storage. Suppose the cache of user data needs to be forced to refresh every 24 hours. Even if the cache item has not been updated or accessed within 24 hours, the system will actively refresh the cache. On the next access, even if the new value has not yet arrived, the old value in the cache will continue to be returned to ensure that the system will not cause service interruptions due to data refresh delays during data processing.

[0109] Through lifecycle management rules, cache data is reasonably managed within a certain period of time through expiration, timeout and refresh strategies. This not only ensures efficient use of memory, but also improves cache hit rate and access performance. Through this mechanism, the efficiency of the cache can be maximized, and the system can maintain efficient response speed when facing large-scale high-concurrency requests.

[0110] Furthermore, the method further comprises:

[0111] By analyzing the historical data of the most recent preset time, the access patterns of different types of users are collected, and the activity level of each user in the most recent preset time is analyzed. The users are then divided into high-frequency active users, secondary active users, and inactive users according to their activity levels.

[0112] The high-frequency active users are set as users who meet the first level of activity, and the secondary active users are set as users who meet the second level of activity.

[0113] By analyzing historical data from the most recent preset time (such as the past three months), the access patterns of different types of users are collected. Each user's behavior in the past period of time (such as access frequency, number of visits, etc.) can be analyzed, and users can be divided into high-frequency active users, secondary active users, and inactive users according to their activity level.

[0114] In a preferred implementation, user activity is defined as:

[0115] High-frequency active users (visit at least once every 2 hours, i.e. users frequently visit within a short period of time);

[0116] Secondary active users (visit at least once every 2 to 8 hours, i.e. users are relatively active but visit less frequently);

[0117] Inactive users (no access within 8 hours, that is, the user access frequency is low or the user number is not in use).

[0118] Notes (the above statistics are limited to the daytime period from 8:00 to 18:00)

[0119] An example of user activity statistics classification is shown in Table 1:

[0120] Table 1: User activity statistics classification

[0121] User Number Number of visits (month) Average visit interval (hours) User Category 130****2045 110 2.3 Active users 131****5678 43 5.7 Active users 132****1234 18 12.4 Inactive users 155****8765 135 1.9 Highly active users 156****4321 34 6.2 Active users 185****9876 18 13.1 Inactive users 186****2345 156 1.5 Highly active users 130****5678 98 3.2 Active users 155****2341 41 7.5 Active users 132****4567 234 10.2 Inactive users 131****9876 272 0.8 Highly active users 156****6789 49 5.9 Active users 186****3456 141 2 Highly active users 130****8765 67 4.1 Active users 131****5432 27 9.3 Inactive users

[0122] Through user activity statistics, the activity of the user can be determined, thereby determining the location of the user portrait cache corresponding to the user number.

[0123] Furthermore, the lightweight memory library is selected from any one of the following:

[0124] Redis, Aerospike, or memcached, deploy the selected memory library as a lightweight component on the server.

[0125] In order to further improve the cache performance and expand the scope of the first-level cache, a lightweight memory library is deployed as a second-level cache, which can be quickly deployed on the existing server cluster. Optional memory libraries include Redis, Aerospike, Memcached, etc. The embodiment of the present disclosure preferably uses Redis as a second-level cache solution.

[0126] As an efficient memory data storage component, Redis can be easily deployed and cache data efficiently. Due to its in-memory database characteristics, Redis supports high-concurrency data access, which can significantly improve cache access speed and make up for the storage range limitation of the first-level cache (localCache). The deployment of Redis not only optimizes the performance of data access, but also provides efficient user portrait information storage for the second-level cache (scaleCache).

[0127] However, since Redis does not have a data persistence mechanism by default, there is a risk of losing cached data under abnormal circumstances. In this solution, Redis is used as a secondary cache to further supplement the storage range of the primary cache and improve the access performance of the cache. Even in the event of data loss, Redis can still provide a faster cache access speed, ensuring that the system can handle large data traffic and provide fast response.

[0128] Furthermore, the preset physical database is an RDS database.

[0129] The original full data of user profile information must be stored in the physical database, which is used as a backup for the third-level cache when the first two caches (first-level cache and second-level cache) miss. Since the third-level cache is the last link in the entire cache system, its access frequency is usually the lowest. If some user profiles frequently access the third-level cache, this may mean that the design and configuration of the first two levels of cache have not achieved the best effect and failed to effectively meet data access requirements.

[0130] The choice of the third-level cache (RDSCache) is crucial. The disclosed embodiment selects the RDS database developed internally by China Unicom, which can meet the needs of large-scale data access in terms of storage and performance, while ensuring the persistent storage of data. The RDS database has efficient data query and management capabilities, and can provide stable and reliable access support in the event of a cache miss. Alternatives to RDSCache may be other mainstream databases, such as MySQL, OceanBase, etc. These databases can also provide similar storage and query performance, and are compatible with existing systems to ensure high availability and data consistency of the system. By using the third-level cache as the last supplement, the reliability of data access is ensured, while avoiding the performance bottleneck of the cache system under high load.

[0131] Furthermore, the method further comprises:

[0132] The user data read from the first-level cache is set to account for X proportion of the total user data, the user data read from the second-level cache is set to account for Y proportion of the total user data, and the user data read from the third-level cache is set to account for Z proportion of the total user data;

[0133] Among them, 30%≤X≤40%, 40%≤Y≤50%, 10%≤Z≤30%, and X+Y+Z=100%.

[0134] X, Y, and Z can be set according to actual conditions, such as 30%, 35%, and 40% for X, 40%, 45%, and 50% for Y, and 10%, 20%, and 30% for Z. Priority can also be given to user activity statistics, and X, Y, and Z can be updated according to the overall distribution. For example, if the overall distribution of user activity is Figure 2 As shown in Figure 1, after long-term verification, the optimal cache access ratio is: about 35% of high-frequency active users read from the first-level cache, about 45% of secondary active users read from the second-level cache, and the remaining 20% ​​of inactive users read from the third-level cache. When the user allocation ratio of the cache system reaches this configuration, the role of the third-level cache will be fully utilized, and the system can ensure the balance and stability of data access while responding efficiently.

[0135] The overall implementation scheme of the present disclosure is as follows Figure 3As shown in the figure, since the signaling data needs to match the portrait tag in the business system according to the user's mobile phone number before it can be used for marketing, the big data real-time platform links the signaling data source through data, associates the portrait information through the mobile phone number, and then caches the data through the three-level cache system. The first-level cache actually uses the internal memory of the big data platform, which has the best performance and loads highly active users by partition; the second-level cache adopts a step-by-step approach, deploys lightweight memory libraries on multiple servers, and accesses secondary active users by node; the third-level cache uses China Unicom's internal RDS database, which has both performance and persistence, and mainly handles inactive users. Based on the user portraits of the multi-level cache, the big data platform performs calculations for each business scenario and conducts marketing touchpoints, such as voice calls, SMS promotion, BI analysis, etc.

[0136] The key differences between the three-level cache system of the embodiment of the present disclosure and other cache systems are mainly reflected in the following aspects:

[0137] 1. Designed for high-concurrency real-time big data

[0138] The three-level cache system disclosed in this disclosure is designed for real-time big data computing and high-throughput scenarios, especially for performance bottlenecks in industry data (such as the computing and association of massive user data in the communications industry). Traditional cache solutions are usually used for relatively static data storage and access, but the present invention solves the problem of how to effectively perform data storage, reading and association under high-frequency and large-scale data requests.

[0139] 2. Combination of cache tiering and computing allocation

[0140] This disclosure combines the partitioning strategy with the three-level cache design, and distributes data to each cache level in a refined manner through hash algorithms and user access frequency analysis. Unlike traditional cache systems that focus on the hierarchical division of data storage and access, this disclosure emphasizes improving the cache hit rate through the combination of algorithms and data, and effectively balancing the memory consumption and access performance of each cache level.

[0141] 3. Efficient collaboration between cache levels

[0142] The three-level cache system disclosed in the present invention adopts a close cooperation mechanism between L1 (first-level cache) and L3 (third-level cache), so that different cache levels can cope with the changing user request load through synchronization, sharing and dynamic adjustment. The design of this mechanism not only reduces data transmission and reading delays, but also ensures high availability and high performance of the system, especially under high concurrent data requests.

[0143] 4. Focus on dynamic cache scheduling for high-frequency, semi-active, and inactive users

[0144] Different from the static allocation of traditional cache, this paper proposes the idea of ​​dynamically adjusting the cache allocation ratio for different user groups (high-frequency, semi-active, and inactive users). Through precise cache lifecycle management and hit rate design, the system can automatically adjust the storage and reading strategies of the data cache according to the user's activity, thereby maximizing the cache hit rate and optimizing the use of memory and computing resources.

[0145] 5. Deep integration of technical level and performance optimization

[0146] In the cache design, high-performance memory databases such as Redis are combined as the second-level cache (scaleCache), and the RDS database developed internally by China Unicom is used as the third-level cache to specifically solve the persistence and query problems under high-throughput data access. Compared with the design of traditional three-level cache, this disclosure pays more attention to the efficient coordination of storage and computing, ensuring that each layer of cache performs optimally in scenarios such as real-time data processing, deep processing, coloring and mining.

[0147] 6. Dynamic data matching mechanism based on user portrait

[0148] This disclosure not only focuses on the cache technology itself, but also proposes a dynamic data matching mechanism based on user portraits and behavioral characteristics. When real-time signaling data and user portrait information need to be associated, a series of cache technology optimizations are used to ensure that the system can complete data matching and query response in a very short time, improve real-time performance and accuracy, and solve the industry's challenges in real-time data processing and user behavior analysis.

[0149] Compared with traditional cache systems, the biggest difference of this disclosure is that it is tailored for real-time big data, high concurrency and massive data processing environments, solving the problems of high throughput, low latency and multi-level data matching. Through the combination of refined cache strategies, dynamic user portrait matching and efficient storage and computing, the efficiency of industry-level real-time data processing is effectively improved, with stronger business adaptability and higher technical value.

[0150] The second embodiment of the present disclosure provides a system for matching big data with an offline data warehouse, such as Figure 4 As shown, the system comprises:

[0151] An allocation module 11 is configured to perform a custom hash algorithm process on the user identification in the real-time computing engine, and allocate the user identification to multiple channel instances according to the processing result;

[0152] A reading module 12, which is configured to read user portrait information in an external database through the same hash algorithm in each channel instance;

[0153] A first loading module 13, which is configured to load user portrait information that meets a first activity level into a first-level cache, where the first-level cache exists inside the real-time big data platform;

[0154] The second loading module 14 is configured to use a lightweight memory library distributedly deployed in multiple servers as a secondary cache, and to break up the user portrait information that meets the second activity level and load it into the secondary cache;

[0155] The cache module 15 is configured to cache the original full data of the user portrait information in a preset physical database as a third-level cache.

[0156] Furthermore, the first loading module 13 is also configured to:

[0157] Design life cycle indicators for data in the first-level cache, including hit rate, hit count, cache miss rate, and cache expiration time;

[0158] If the user portrait information data does not meet the life cycle indicators, the cache will be downgraded or removed.

[0159] Furthermore, the first loading module 13 is also configured to:

[0160] Set lifecycle management and cache update rules for the first-level cache, including:

[0161] Set the maximum number of cache entries. When the cache exceeds this number, the least frequently used data will be eliminated according to the preset strategy.

[0162] Set the cache expiration time. When a cache item is not created or overwritten within the specified time, the cache item will be automatically removed.

[0163] Set the cache timeout. When a cache item is not read or written within the specified time, the cache item will be removed.

[0164] Set the cache refresh time. If the cache item is not created or overwritten within the specified time, the system will force the cache to be refreshed and always return the old data until new data arrives.

[0165] Furthermore, the system also includes a division module 16;

[0166] The classification module 16 is configured to collect access patterns of different types of users by analyzing historical data of the most recent preset time, and analyze the activity level of each user in the most recent preset time, and classify the users into high-frequency active users, secondary active users, and inactive users according to their activity levels; and,

[0167] The high-frequency active users are set as users who meet the first level of activity, and the secondary active users are set as users who meet the second level of activity.

[0168] Furthermore, the lightweight memory library is selected from any one of the following:

[0169] Redis, Aerospike or memcached,

[0170] The second loading module 14 also configures me to deploy the selected memory library as a lightweight component on the server.

[0171] Furthermore, the preset physical database is an RDS database.

[0172] Furthermore, the system also includes a setting module 17;

[0173] The setting module 17 is configured to set the user data read from the first-level cache to account for an X proportion of the total user data, the user data read from the second-level cache to account for a Y proportion of the total user data, and the user data read from the third-level cache to account for a Z proportion of the total user data;

[0174] Among them, 30%≤X≤40%, 40%≤Y≤50%, 10%≤Z≤30%, and X+Y+Z=100%.

[0175] The system for matching big data with offline data warehouses in the disclosed embodiment is used to implement the method for matching big data with offline data warehouses in method embodiment 1, so the description is relatively simple. For details, please refer to the relevant description in the previous method embodiment, which will not be repeated here.

[0176] In addition, if Figure 5 As shown, the third embodiment of the present disclosure further provides an electronic device, including a memory 100 and a processor 200, wherein the memory 100 stores a computer program, and when the processor 200 runs the computer program stored in the memory 100, the processor 200 executes the above-mentioned various possible methods.

[0177] The memory 100 is connected to the processor 200. The memory 100 may be a flash memory, a read-only memory or other memory, and the processor 200 may be a central processing unit or a single-chip microcomputer.

[0178] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the above-mentioned various possible methods.

[0179] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), Digital Video Disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0180] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A method for matching big data with an offline data warehouse, characterized in that: The method comprises: The user ID is processed using a custom hash algorithm in the real-time computing engine, and the user ID is assigned to multiple channel instances based on the processing results; In each channel instance, the user profile information in the external database is read through the same hash algorithm; Loading user profile information that meets the first level of activity into a first-level cache that exists within the real-time big data platform; A lightweight memory library distributed across multiple servers will be used as the secondary cache, and user profile information that meets the second level of activity will be broken up and loaded into the secondary cache; The original full data of user portrait information is cached in the preset physical database as a third-level cache.

2. The method according to claim 1, characterized in that The method further comprises: Design life cycle indicators for data in the first-level cache, including hit rate, hit count, cache miss rate, and cache expiration time; If the user portrait information data does not meet the life cycle indicators, the cache will be downgraded or removed.

3. The method according to claim 1, characterized in that The method further comprises: Set lifecycle management and cache update rules for the first-level cache, including: Set the maximum number of cache entries. When the cache exceeds this number, the least frequently used data will be eliminated according to the preset strategy. Set the cache expiration time. When a cache item is not created or overwritten within the specified time, the cache item will be automatically removed. Set the cache timeout. When a cache item is not read or written within the specified time, the cache item will be removed. Set the cache refresh time. If the cache item is not created or overwritten within the specified time, the system will force the cache to be refreshed and always return the old data until new data arrives.

4. The method according to claim 1, characterized in that The method further comprises: By analyzing the historical data of the most recent preset time, the access patterns of different types of users are collected, and the activity level of each user in the most recent preset time is analyzed. The users are then divided into high-frequency active users, secondary active users, and inactive users according to their activity levels. The high-frequency active users are set as users who meet the first level of activity, and the secondary active users are set as users who meet the second level of activity.

5. The method according to claim 1, characterized in that The lightweight memory library is selected from any of the following: Redis, Aerospike, or memcached, deploy the selected memory library as a lightweight component on the server.

6. The method according to claim 1, characterized in that The preset physical database is a relational database service RDS database.

7. The method according to claim 1, characterized in that The method further comprises: The user data read from the first-level cache is set to account for X proportion of the total user data, the user data read from the second-level cache is set to account for Y proportion of the total user data, and the user data read from the third-level cache is set to account for Z proportion of the total user data; Among them, 30%≤X≤40%, 40%≤Y≤50%, 10%≤Z≤30%, and X+Y+Z=100%.

8. A system for matching big data with an offline data warehouse, characterized in that: The system comprises: An allocation module, which is configured to process the user ID using a custom hash algorithm in the real-time computing engine, and allocate the user ID to multiple channel instances according to the processing result; A reading module is configured to read user profile information from an external database using the same hash algorithm in each channel instance; A first loading module is configured to load user portrait information that meets a first activity level into a first-level cache, where the first-level cache exists inside the real-time big data platform; The second loading module is configured to use a lightweight memory library distributedly deployed in multiple servers as a secondary cache, and to break up the user portrait information that meets the second activity level and load it into the secondary cache; The cache module is configured to cache the original full data of the user portrait information in a preset physical database as a third-level cache.

9. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method for matching big data with an offline data warehouse as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for matching big data with an offline data warehouse according to any one of claims 1 to 7 is implemented.

Images