Network advertisement data high-concurrency access service method

Abstract

The invention relates to the technical field of advertisement data high-concurrency access service methods, in particular to a network advertisement data high-concurrency access service method which specifically comprises the following steps: step 1, data feature analysis and classification modeling: comprehensively analyzing features of network advertisement data, and establishing a corresponding classification model; 2, designing a multi-level cache architecture, and designing the multi-level cache architecture based on a classification result in the step 1; 3, constructing a dynamic resource scheduling model: constructing the dynamic resource scheduling model according to the access requirements and priorities of different types of data; 4, the data are stored on a plurality of nodes in a scattered mode; the method solves the problem of insufficient consideration of data features in a traditional method.

Description

Technical Field

[0001] This invention relates to the technical field of methods for high-concurrency access to advertising data services, and in particular to a method for high-concurrency access to online advertising data services. Background Technology

[0002] Online advertising has become a crucial means for businesses to promote their products and services, with its market size continuously expanding and advertising formats becoming increasingly diversified, such as banner ads, video ads, and native ads. With the rapid growth in the number of internet users and the demand for more precise advertising, online advertising data is experiencing explosive growth. This data encompasses multiple dimensions, including ad impressions, clicks, conversion rates, and user profile data, playing a vital role in helping advertisers optimize their advertising strategies and improve advertising effectiveness.

[0003] However, online advertising data access scenarios are characterized by high concurrency, low latency, large data volumes, and high real-time requirements. When a large number of users access advertising data simultaneously, or when advertising platforms perform real-time data analysis and report generation, existing data access service methods face numerous challenges.

[0004] Traditional centralized database architectures often encounter problems such as database connection exhaustion, slow query response, or even system crashes when faced with high concurrency access. Summary of the Invention

[0005] To address the technical problems that traditional centralized database architectures often encounter when facing high concurrency access, such as database connection exhaustion, slow query response, or even system crashes, this invention provides a method for providing high-concurrency access services for online advertising data.

[0006] The technical solution adopted in this invention is: a method for high-concurrency access to network advertising data, specifically including the following steps:

[0007] Step 1: Data Feature Analysis and Classification Modeling. Conduct a comprehensive analysis of the characteristics of online advertising data and establish a corresponding classification model.

[0008] Step 2: Multi-level caching architecture design. Based on the classification results of Step 1, design a multi-level caching architecture.

[0009] Step 3: Build a dynamic resource scheduling model. Based on the access needs and priorities of different types of data, build a dynamic resource scheduling model.

[0010] Step 4: Distribute the data across multiple nodes;

[0011] Step 5: In high-concurrency access scenarios, establish concurrency control and data consistency guarantee mechanisms;

[0012] Step Six: Build a performance monitoring system to monitor various performance indicators of the high-concurrency access service for online advertising data in real time;

[0013] Step 7: Establish a multi-level disaster recovery system, including local disaster recovery, off-site disaster recovery, and cloud disaster recovery.

[0014] In one embodiment, step one involves data feature analysis and classification modeling, which comprehensively analyzes the characteristics of online advertising data and establishes a corresponding classification model, mainly including data access frequency, data timeliness, data size, data value, and the correlation of data access.

[0015] Data access frequency refers to the number of times data is accessed per unit of time, denoted by f, and its calculation formula is:

[0016]

[0017] Where N represents the total number of times the data is accessed within time T;

[0018] Data timeliness refers to the effective time range of data. Beyond this time range, the value of the data decreases significantly or even becomes worthless. (t) alid This refers to the time interval between the generation and expiration of data.

[0019] Data size refers to the storage space occupied by the data, denoted by s, and the unit is bytes or kilobytes;

[0020] Data value is represented by 'v'. Data value is derived from the data's contribution to improving ad conversion rates and the scarcity of the data. The calculation formula is as follows:

[0021] v = w1×c + w2×r + w3×s r ;

[0022] Where c represents the contribution of data to the improvement of ad conversion rate, r represents the scarcity of data, and s r To represent the stability of the data, w1, w2, and w3 are the weights of each factor, and w1 + w2 + w3 = 1;

[0023] The correlation of data access refers to the probability that different pieces of data can be accessed simultaneously, denoted by r. rel This means that the formula for calculating the correlation between data i and data j is:

[0024]

[0025] Where Ni,j represents the number of times data i and data j are accessed simultaneously, and N i N represents the total number of times data i was accessed. j This represents the total number of times data j was accessed.

[0026] In one embodiment, in step two: multi-level cache architecture design, based on the classification results of step one, the specific method for designing a multi-level cache architecture is as follows:

[0027] A multi-level caching architecture includes three layers: local cache, distributed cache, and disk cache.

[0028] The local cache is deployed locally on the application server to store frequently accessed, small-volume, hot data. The local cache capacity is set to C. local The cache content needs to be dynamically adjusted based on data characteristics. The distributed cache adopts a cluster deployment method, specifically a Redis cluster, to store high-frequency, high-value data, strongly correlated datasets, and data with medium access frequency. The distributed cache capacity is set to C. dist It can support multiple application servers to share cached data;

[0029] Disk cache is used to store data that is accessed infrequently but still has some value, as well as some historical data backups; the disk cache capacity is set to C. disk ;

[0030] When the data in the local cache reaches its capacity limit, the cache priority P of the data is calculated based on the data's access frequency and value. cache The calculation formula is as follows:

[0031]

[0032] Among them, t elapsed This represents the time interval from data generation to the present, where a, b, and d are the weights of each factor, and a+b+d=1.

[0033] In one embodiment, step three involves constructing a dynamic resource scheduling model. This model is designed to address the access needs and priorities of different data categories. The specific method is as follows:

[0034] First, define the priority P of the access request. req The calculation formula is as follows:

[0035] P req =e×v+f×u+g×t urgent ;

[0036] Where v is the value of the requested data, and u is the user level t. urgent To represent the urgency of the request, e, f, and g are the weights of each factor, and e+f+g=1;

[0037] Determine the amount of resources R required for each request based on the priority and data type of the access request. reqThis includes CPU resources, memory resources, and network bandwidth resources, for a data size of s and a processing complexity of c. p The request requires CPU resources R cpu The calculation formula is:

[0038] R cpu =h×s+i×c p ;

[0039] Where h and i are proportionality coefficients, and memory resources R mem The calculation formula is:

[0040] R mem =j×s+k×n conn ;

[0041] Where, n conn The current number of database connections requested, j and k are proportional coefficients; network bandwidth resource R bandwidth The calculation formula is:

[0042] R bandwidth =l×s / t resp ;

[0043] Among them, t resp Let l be the desired response time, and l be the scaling factor.

[0044] The dynamic resource scheduling model adopts a priority-based weighted fair queue scheduling algorithm, which puts requests of different priorities into different queues, and each queue obtains a corresponding resource weight according to its priority.

[0045] Let the total system resources be R. total The priority of queue i is P i Then the resource share R obtained by queue i i for:

[0046]

[0047] Where m is the number of queues;

[0048] When the system load exceeds the threshold L thr When this happens, low-priority requests will be rate-limited, with a rate-limiting ratio r. limit The calculation formula is:

[0049]

[0050] In one embodiment, in step four: the data is distributed and stored across multiple nodes, and the data sharding uses a consistent hashing algorithm, which maps both the storage nodes and the data to a range of [0, 2]. 32On the hash ring of [-1], for data d, its hash value hash(d) is calculated by the hash function;

[0051] Then, the data d is assigned to the storage node n on the hash ring that is closest to hash(d), that is, the smallest hash(n) that satisfies hash(n)≥hash(d). If no such node exists, it is assigned to the first node on the hash ring.

[0052] The granularity of data sharding needs to be determined based on the data characteristics. For small data with high access frequency, a smaller granularity is used; for large data with low access frequency, a larger granularity is used. Let the data sharding granularity be G, and for data of size s, the number of shards be N. shard The calculation formula is:

[0053] N shard =ceil(s / G);

[0054] Here, ceil() is the floor function.

[0055] In one embodiment, step five involves establishing a concurrency control and data consistency guarantee mechanism in high-concurrency access scenarios. The specific method is as follows:

[0056] An optimistic concurrency control mechanism is adopted, which assumes that concurrency conflicts will not occur in most cases, allowing multiple transactions to operate on the data simultaneously. Conflict detection is performed when a transaction commits. Each data record contains a version number. When a transaction reads data, it also reads this version number. When a transaction modifies data and commits, it checks whether the version number of the current data is consistent with the version number read at that time. If they are consistent, the transaction is committed and the version number is incremented by 1. If they are inconsistent, it indicates that a concurrency conflict has occurred, and the transaction needs to be rolled back and re-executed.

[0057] The update rules for the version number are as follows:

[0058] version new =version old +1;

[0059] When concurrent conflicts occur, the number of retries N for a transaction retry Limits need to be imposed to avoid wasting system resources due to infinite retries. retry ≤N max , where N m ax represents the maximum number of retries.

[0060] In one embodiment, step six involves constructing a performance monitoring system to monitor various performance indicators of the high-concurrency access service for online advertising data in real time, and performing adaptive optimization based on the monitoring results to ensure that the system is always in the best operating state. The specific method is as follows:

[0061] Performance monitoring metrics mainly include response time t resp Throughput (QPS), cache hit rate (H) cache Error rate E error ;

[0062] Response time t resp This refers to the time interval from when a request is sent to when a response is received, and its calculation formula is:

[0063] t resp =t receive -t send ;

[0064] Among them, t send t represents the time the request was sent. receive For response reception time;

[0065] The formula for calculating throughput QPS is:

[0066]

[0067] Where, N req The total number of requests processed within time T;

[0068] Cache hit rate H cache The calculation formula is:

[0069]

[0070] Where, N cache_hit N represents the number of cache hits. cache_miss To cache the number of cache misses;

[0071] Error rate E error The calculation formula is:

[0072]

[0073] Where, N error N represents the number of requests that resulted in errors. total This represents the total number of requests.

[0074] In one embodiment, step seven: establishing a multi-level disaster recovery system, including local disaster recovery, off-site disaster recovery, and cloud disaster recovery, is specifically implemented as follows:

[0075] Local disaster recovery refers to establishing a backup system within the same data center to maintain real-time data synchronization with the main system. The synchronization method adopts incremental synchronization, which means that only the changed parts of the data are transmitted to reduce the amount of data transmission and synchronization time.

[0076] The beneficial effects of this invention are as follows: Compared with the prior art, this invention solves the problem of insufficient consideration of data characteristics in traditional methods; the multi-level caching architecture, combined with dynamic migration strategy and consistency guarantee mechanism, significantly improves the cache hit rate and reduces the database load. Compared with the existing simple cache eviction strategy, the cache hit rate is improved, effectively reducing data access latency and solving problems such as database connection exhaustion, slow query response, and even system crash. Attached Figure Description

[0077] Figure 1 This is a flowchart illustrating the present invention; Detailed Implementation

[0078] In the description of this invention, it should be noted that the terms "front", "up", "down", "left", "right", "vertical", "horizontal", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0079] To address the problems existing in the background technology, this application proposes the following technical solution: a method for high-concurrency access to network advertising data, specifically including the following steps:

[0080] Step 1: Data Feature Analysis and Classification Modeling. Conduct a comprehensive analysis of the characteristics of online advertising data and establish a corresponding classification model.

[0081] In step one: data feature analysis and classification modeling, a comprehensive analysis of the characteristics of online advertising data is conducted, and a corresponding classification model is established, mainly including data access frequency, data timeliness, data size, data value, and the correlation of data access.

[0082] Data access frequency refers to the number of times data is accessed per unit of time, denoted by f, and its calculation formula is:

[0083]

[0084] Where N represents the total number of times the data is accessed within time T;

[0085] Data timeliness refers to the effective time range of data. Beyond this time range, the value of the data decreases significantly or even becomes worthless. (t) valid This refers to the time interval between the generation and expiration of data.

[0086] Data size refers to the storage space occupied by the data, represented by s, and the unit is either bytes (B) or kilobytes (KB);

[0087] Data value is a comprehensive indicator that reflects the importance of data to advertisers' decision-making and advertising effectiveness optimization, denoted by 'v'. Data value is derived from the data's contribution to improving advertising conversion rates and the scarcity of the data. Its calculation formula is as follows:

[0088] V = w1 × C + w2 × r + w3 × s r ;

[0089] Where c represents the contribution of data to the improvement of ad conversion rate, r represents the scarcity of data, and s r The stability of the data (i.e., the length of time the data remains unchanged) is represented by w1, w2, and w3, which are the weights of each factor, and w1 + w2 + w3 = 1; the correlation of data access refers to the probability that different data are accessed simultaneously, denoted by r. rel This means that the formula for calculating the correlation between data i and data j is:

[0090]

[0091] Where, N i，j N represents the number of times data i and data j are accessed simultaneously. i N represents the total number of times data i was accessed. j This represents the total number of times data j was accessed;

[0092] Based on the above characteristics, clustering algorithms from machine learning are used to classify and model online advertising data. Here, the K-means clustering algorithm is chosen to divide the data into different categories, such as high-frequency high-value, high-frequency low-value, low-frequency high-value, low-frequency low-value, and strongly correlated categories. During the clustering process, the data's access frequency (f), data value (v), and correlation (r) are used as the basis for classification. rel As a clustering feature, classification modeling can provide a clear basis for subsequent caching strategy formulation and resource scheduling. For example, high-frequency, high-value data should be given priority in caching and allocated more resources.

[0093] Step 2: Multi-level caching architecture design. Based on the classification results of Step 1, design a multi-level caching architecture.

[0094] In step two: multi-level caching architecture design, based on the classification results of step one, the specific method for designing a multi-level caching architecture is as follows: to improve cache hit rate and reduce database access pressure;

[0095] A multi-level caching architecture includes three layers: local cache, distributed cache, and disk cache.

[0096] Local caching is deployed locally on the application server to store frequently accessed but small-volume hot data, such as real-time statistics on ad impressions and clicks. The advantage of local caching is its fast access speed, eliminating network transmission overhead. The local cache size is set to C. local The cache content needs to be dynamically adjusted based on data characteristics; the distributed cache adopts a cluster deployment method, specifically a Redis cluster, to store high-frequency, high-value data, strongly correlated datasets, and data with medium access frequency; the distributed cache capacity is set to C. dist It can support multiple application servers to share cached data, thereby improving cache utilization.

[0097] Disk cache is used to store data that is accessed infrequently but still has some value, as well as some historical data backups; the disk cache capacity is set to C. disk , ;

[0098] When the data in the local cache reaches its capacity limit, the cache priority P of the data is calculated based on the data's access frequency and value. cache The calculation formula is as follows:

[0099]

[0100] Among them, t elapsed This represents the time interval from data generation to the present, where a, b, and d are the weights of each factor, and a + b + d = 1. Lower-priority data will be migrated to the distributed cache. When the distributed cache reaches its capacity limit, lower-priority data will also be migrated to the disk cache based on cache priority. To ensure cache data consistency, a cache update mechanism is used. When data in the database is updated, the cache system is notified to update the data using a publish-subscribe pattern. For local cache, an immediate update method is used; for distributed cache, an asynchronous update method is used to reduce the impact on the main business process. Its update latency t... update Certain requirements need to be met, namely t update ≤t tol , where t tol This represents the maximum allowable time for data inconsistency in the system.

[0101] Step 3: Build a dynamic resource scheduling model. Based on the access needs and priorities of different types of data, build a dynamic resource scheduling model.

[0102] In step three: dynamic resource scheduling model construction. Based on the access needs and priorities of different types of data, a dynamic resource scheduling model is constructed. The specific method is as follows, so as to achieve reasonable allocation of resources and improve the system's response speed and service quality.

[0103] First, define the priority P of the access request. req The calculation formula is as follows:

[0104] P req =e×v+f×u+g×t urgent ;

[0105] Where v is the value of the requested data, u is the user level of the request (e.g., paid users are more valuable than free users), and t is the user level of the requester. urgent The urgency of the request is indicated by a value ranging from 0 to 1, with larger values ​​indicating greater urgency. e, f, and g represent the weights of each factor, and e + f + g = 1.

[0106] Determine the amount of resources R required for each request based on the priority and data type of the access request. req This includes CPU resources, memory resources, and network bandwidth resources; the calculation of resource quantity is related to factors such as data size and processing complexity. For data of size s and processing complexity c... p The request requires CPU resources R cpu The calculation formula is:

[0107] R cpu =h×s+i×c p ;

[0108] Where h and i are proportionality coefficients; memory resources R mem The calculation formula is:

[0109] R mem =j×s+k×n conn ;

[0110] Where, n conn The current number of database connections requested, j and k are proportional coefficients; network bandwidth resource R bandwidth The calculation formula is:

[0111] R bandwidth =l×s / t resp ;

[0112] Among them, t resp Let l be the desired response time, and l be the scaling factor.

[0113] The dynamic resource scheduling model adopts a priority-based weighted fair queue (WFQ) scheduling algorithm, which places requests of different priorities into different queues, and each queue obtains a corresponding resource weight according to its priority.

[0114] Let the total system resources be R. total The priority of queue i is P i Then the resource share R obtained by queue i i for:

[0115]

[0116] Where m is the number of queues;

[0117] At the same time, the model also needs to consider the system load. When the system load exceeds the threshold L... thr System load L can be calculated by comprehensively considering indicators such as CPU utilization, memory utilization, and network bandwidth utilization. L = m1 × L cpu +m2×L mem +m3×L bandwidth L c pu, L meem L bandwidth These represent the utilization rates of CPU, memory, and network bandwidth, respectively (m1, m2, and m3 are weights, and m1 + m2 + m3 = 1). Low-priority requests will be rate-limited, with a rate-limiting ratio r. limit The calculation formula is:

[0118]

[0119] The dynamic resource scheduling model ensures that high-priority access requests receive sufficient resource support, thereby improving the overall service quality of the system.

[0120] Step 4: Distribute the data across multiple nodes;

[0121] In step four: data is distributed across multiple nodes to improve parallel access capabilities and system scalability; data sharding uses a consistent hashing algorithm, which can evenly distribute data across different storage nodes, and when a node joins or leaves the cluster, only a small amount of data needs to be remapped, reducing data migration overhead; the consistent hashing algorithm maps both storage nodes and data to a range [0, 2]. 32 On the hash ring of [-1], for data d, its hash value hash(d) is calculated by a hash function, such as the MD5 hash function;

[0122] Then, the data d is assigned to the storage node n on the hash ring that is closest to hash(d), that is, the smallest hash(n) that satisfies hash(n)≥hash(d). If no such node exists, it is assigned to the first node on the hash ring.

[0123] The granularity of data sharding needs to be determined based on the data characteristics. For small data with high access frequency, a smaller sharding granularity can be used to improve parallel access capabilities; for large data with low access frequency, a larger sharding granularity can be used to reduce the overhead of shard management. Let the data sharding granularity be G, and for data of size s, the number of shards is N. shard The calculation formula is:

[0124] N shard =ceil(s / G)

[0125] Here, ceil() is the floor function; in the distributed storage process, a data replication mechanism is adopted to ensure data reliability and availability; each data shard stores multiple replicas on different storage nodes, let the number of replicas be k, usually k=3; the replicas are placed using a cross-rack deployment strategy to avoid data loss due to the failure of a single rack; at the same time, the data consistency between multiple replicas is guaranteed through a replica consistency protocol (such as the Paxos protocol); when the node where the primary replica resides fails, the system can automatically switch from the replica to the primary replica to ensure continuous data access.

[0126] Step 5: In high-concurrency access scenarios, establish concurrency control and data consistency guarantee mechanisms;

[0127] In step five: In high-concurrency access scenarios, establish a concurrency control and data consistency guarantee mechanism to prevent data errors or inconsistencies; adopt an optimistic concurrency control mechanism, which assumes that concurrency conflicts will not occur in most cases, allowing multiple transactions to operate on the data simultaneously, and performing conflict detection when the transaction is committed; each data record contains a version number, which is read when a transaction reads data; when a transaction modifies data and commits, it checks whether the version number of the current data is consistent with the version number read; if they are consistent, the transaction is committed and the version number is incremented by 1; if they are inconsistent, it indicates that a concurrency conflict has occurred, and the transaction needs to be rolled back and re-executed.

[0128] The update rules for the version number are as follows:

[0129] version new =version old +1

[0130] When concurrent conflicts occur, the number of retries N for a transaction retryLimits need to be imposed to avoid wasting system resources due to infinite retries. retry ≤N max , where N m `ax` represents the maximum number of retries. For updates to critical data, such as deductions from advertising budgets, a pessimistic concurrency control mechanism is used, employing locking to ensure that only one transaction can modify the data at a time. The granularity of locks can be adjusted based on data characteristics, including row-level locks and table-level locks. Row-level locks are suitable for high-concurrency, small-scale data modifications, while table-level locks are suitable for large-scale data modifications or schema changes. To ensure data consistency in a distributed environment, a distributed transaction management mechanism is adopted, based on the two-phase commit (2PC) protocol. The two-phase commit protocol includes a preparation phase and a commit phase. In the preparation phase, the coordinator sends a preparation request to all participating nodes. Each node performs the transaction operation and writes undo and redo information to the log, then returns a success or failure response to the coordinator. If all nodes return success, the coordinator sends a commit request to all nodes in the commit phase. Each node performs the commit operation and releases resources. If any node returns failure, the coordinator sends a rollback request, and each node performs the rollback operation.

[0131] Step Six: Build a performance monitoring system to monitor various performance indicators of the high-concurrency access service for online advertising data in real time;

[0132] In step six: Build a performance monitoring system to monitor various performance indicators of the high-concurrency access service for network advertising data in real time, and perform adaptive optimization based on the monitoring results to ensure that the system is always in the best operating state;

[0133] Performance monitoring metrics mainly include response time t resp Throughput (QPS, queries per second) and cache hit rate (H) cache Error rate E error wait;

[0134] Response time t resp This refers to the time interval from when a request is sent to when a response is received, and its calculation formula is:

[0135] t resp =t receive -t send

[0136] Among them, t send t represents the time the request was sent. receive For response reception time;

[0137] The formula for calculating throughput QPS is:

[0138]

[0139] Where, N req The total number of requests processed within time T;

[0140] Cache hit rate H cache The calculation formula is:

[0141]

[0142] Where, N cache_hit N represents the number of cache hits. cache_miss To cache the number of cache misses;

[0143] Error rate E error The calculation formula is:

[0144]

[0145] Where, N error N represents the number of requests that resulted in errors. total Total number of requests;

[0146] Step 7: Establish a multi-level disaster recovery system, including local disaster recovery, off-site disaster recovery, and cloud disaster recovery.

[0147] Step seven involves establishing a multi-level disaster recovery system, including local disaster recovery, off-site disaster recovery, and cloud disaster recovery. Local disaster recovery refers to establishing a backup system within the same data center, maintaining real-time data synchronization with the primary system. The synchronization method uses incremental synchronization, meaning only the changed parts of the data are transmitted to reduce data transmission volume and synchronization time. When the primary system fails, the local disaster recovery system can take over services quickly, with a switchover time typically controlled within seconds. Off-site disaster recovery involves establishing a disaster recovery center in a geographically distant location from the primary data center, connected by a dedicated communication line. Data synchronization in off-site disaster recovery uses a combination of scheduled and incremental synchronization. Cloud disaster recovery utilizes… Use cloud service provider infrastructure for data backup and disaster recovery; back up important advertising data to cloud storage, such as object storage services, and achieve rapid service recovery through the elastic computing resources of the cloud platform; cloud disaster recovery has high scalability and flexibility, and can dynamically adjust storage resources according to the growth of data volume. Its data transmission adopts encrypted transmission method to ensure data security during transmission. By reasonably setting the cycle of full backup and incremental backup, both data security can be guaranteed and the impact of backup on system performance can be reduced; backup data needs to be verified regularly to ensure the integrity and recoverability of backup data, and the verification cycle is set to once a week.

[0148] Establish a fault drill mechanism and conduct regular fault simulation drills to verify the effectiveness of fault tolerance and disaster recovery mechanisms. Fault drills include various scenarios such as single-machine failure, node cluster failure, and data center failure. During the drills, record indicators such as system recovery time and data consistency, and optimize fault handling procedures and disaster recovery strategies based on the drill results. The fault drill cycle is set to once per quarter.

[0149] In addition, a robust fault alarm mechanism needs to be established to promptly send alarm information to operations and maintenance personnel when system failures or abnormal situations occur. Alarm levels are categorized into three levels based on the severity of the fault: urgent, important, and general. An urgent alarm indicates a serious system failure, such as a main data center outage, requiring immediate attention. An important alarm indicates a significant system anomaly, such as a sharp drop in cache hit rate, requiring prompt action. A general alarm indicates a minor system anomaly, such as high load on individual nodes, which can be addressed at an appropriate time. Alarm information is sent through multiple channels, such as SMS, email, and WeChat Work, to ensure that operations and maintenance personnel can receive and respond promptly.

[0150] In summary, this invention addresses the problem of insufficient consideration of data characteristics in traditional methods; the multi-level caching architecture, combined with dynamic migration strategies and consistency guarantee mechanisms, significantly improves cache hit rate and reduces database load. Compared to existing simple cache eviction strategies, it increases cache hit rate and effectively reduces data access latency; the dynamic resource scheduling model allocates resources rationally based on request priority and resource requirements, ensuring fast response times for high-priority requests and reducing response time under high system load; data sharding and distributed storage strategies improve the system's parallel access capability and scalability, supporting the storage and access of massive amounts of data and increasing data processing throughput; the concurrency control and data consistency guarantee mechanism combine optimistic and pessimistic control. This combined approach improves concurrent processing efficiency and reduces data error rates while ensuring data consistency. Performance monitoring and adaptive optimization strategies enable the system to adjust operating parameters in real time, maintaining optimal performance and significantly enhancing system stability and reliability. A robust fault tolerance and disaster recovery mechanism ensures rapid system recovery in the event of a failure, reducing business interruption time to minutes (RT0) and seconds (RP0), greatly guaranteeing the continuity and data security of online advertising services. It comprehensively improves the performance, reliability, and availability of high-concurrency access services for online advertising data, meeting advertisers' needs for real-time data analysis and precise ad targeting, demonstrating significant practical application value and economic benefits.

[0151] Although embodiments of the invention have been shown and described, the scope of the invention will be defined by the appended claims and their equivalents by those skilled in the art.

Claims

1. A method for high-concurrency access to online advertising data, characterized in that, Specifically, the following steps are included: Step 1: Data Feature Analysis and Classification Modeling. Conduct a comprehensive analysis of the characteristics of online advertising data and establish a corresponding classification model. Step 2: Multi-level caching architecture design. Based on the classification results of Step 1, design a multi-level caching architecture. Step 3: Build a dynamic resource scheduling model. Based on the access needs and priorities of different types of data, build a dynamic resource scheduling model. Step 4: Distribute the data across multiple nodes; Step 5: In high-concurrency access scenarios, establish concurrency control and data consistency guarantee mechanisms; Step Six: Build a performance monitoring system to monitor various performance indicators of the high-concurrency access service for online advertising data in real time; Step 7: Establish a multi-level disaster recovery system, including local disaster recovery, off-site disaster recovery, and cloud disaster recovery.

2. The method for high-concurrency access to network advertising data according to claim 1, characterized in that, In step one: data feature analysis and classification modeling, a comprehensive analysis of the characteristics of online advertising data is conducted, and a corresponding classification model is established, mainly including data access frequency, data timeliness, data size, data value, and the correlation of data access. Data access frequency refers to the number of times data is accessed per unit of time, denoted by f, and its calculation formula is: Where N represents the total number of times the data is accessed within time T; Data timeliness refers to the effective time range of data. Beyond this time range, the value of the data decreases significantly or even becomes worthless. (t) valid This refers to the time interval between the generation and expiration of data. Data size refers to the storage space occupied by the data, denoted by s, and the unit is bytes or kilobytes; Data value is represented by 'v'. Data value is derived from the data's contribution to improving ad conversion rates and the scarcity of the data. The calculation formula is as follows: V=w1×c+w2×r+w3×s r ; Where c represents the contribution of data to the improvement of ad conversion rate, r represents the scarcity of data, and s r To represent the stability of the data, w1, w2, and w3 are the weights of each factor, and w1 + w2 + w3 = 1; The correlation of data access refers to the probability that different pieces of data can be accessed simultaneously, denoted by r. rel This means that the formula for calculating the correlation between data i and data j is: Where, N i，j N represents the number of times data i and data j are accessed simultaneously. i N represents the total number of times data i was accessed. j This represents the total number of times data j was accessed.

3. The method for high-concurrency access to network advertising data according to claim 2, characterized in that, Step Two: Multi-level caching architecture design. Based on the classification results of Step One, the specific method for designing a multi-level caching architecture is as follows: A multi-level caching architecture includes three layers: local cache, distributed cache, and disk cache. The local cache is deployed locally on the application server to store frequently accessed, small-volume, hot data. The local cache capacity is set to C. local The cache content needs to be dynamically adjusted based on data characteristics. The distributed cache adopts a cluster deployment method, specifically a Redis cluster, to store high-frequency, high-value data, strongly correlated datasets, and data with medium access frequency. The distributed cache capacity is set to C. dist It can support multiple application servers to share cached data; Disk cache is used to store data that is accessed infrequently but still has some value, as well as some historical data backups; the disk cache capacity is set to C. disk ; When the data in the local cache reaches its capacity limit, the cache priority P of the data is calculated based on the data's access frequency and value. cache The calculation formula is as follows: Among them, t elapsed This represents the time interval from data generation to the present, where a, b, and d are the weights of each factor, and a+b+d=1.

4. The method for high-concurrency access to network advertising data according to claim 3, characterized in that, Step 3: Dynamic resource scheduling model construction. A dynamic resource scheduling model is constructed based on the access needs and priorities of different types of data. The specific method is as follows: First, define the priority P of the access request. req The calculation formula is as follows: P req =e×v+f×u+g×t urgent ; Where v is the value of the requested data, and u is the user level t. urgent To represent the urgency of the request, e, f, and g are the weights of each factor, and e+f+g=1; Determine the amount of resources R required for each request based on the priority and data type of the access request. req This includes CPU resources, memory resources, and network bandwidth resources, for a data size of s and a processing complexity of c. p The request requires CPU resources R cpu The calculation formula is: R cpu =h×s+i×c p ; Where h and i are proportionality coefficients, and memory resources R mem The calculation formula is: R mem =j×s+k×n conn ; Where, n conn The current number of database connections requested, j and k are proportional coefficients; network bandwidth resource R bandwidth The calculation formula is: R bandwidt h=l×s / t resp ; Among them, t resp Let l be the desired response time, and l be the scaling factor. The dynamic resource scheduling model adopts a priority-based weighted fair queue scheduling algorithm, which puts requests of different priorities into different queues, and each queue obtains a corresponding resource weight according to its priority. Let the total system resources be R. total The priority of queue i is P i Then the resource share R obtained by queue i i for: Where m is the number of queues; When the system load exceeds the threshold L thr When this happens, low-priority requests will be rate-limited, with a rate-limiting ratio r. limit The calculation formula is:

5. The method for high-concurrency access to network advertising data according to claim 4, characterized in that, In step four: the data is distributed and stored across multiple nodes. Data sharding uses a consistent hashing algorithm, which maps both the storage nodes and the data to a range [0, 2]. 32 On the hash ring of [-1], for data d, its hash value hash(d) is calculated by the hash function; Then, the data d is assigned to the storage node n on the hash ring that is closest to hash(d), that is, the smallest hash(n) that satisfies hash(n)≥hash(d). If no such node exists, it is assigned to the first node on the hash ring. The granularity of data sharding needs to be determined based on the data characteristics. For small data with high access frequency, a smaller granularity is used; for large data with low access frequency, a larger granularity is used. Let the data sharding granularity be G, and for data of size s, the number of shards be N. shard The calculation formula is: N shard =ceil(s / G); Here, ceil() is the floor function.

6. The method for high-concurrency access to network advertising data according to claim 5, characterized in that, Step 5: In high-concurrency access scenarios, establish a concurrency control and data consistency guarantee mechanism. The specific methods are as follows: An optimistic concurrency control mechanism is adopted, which assumes that no concurrency conflicts will occur in most cases, allowing multiple transactions to operate on the data at the same time. Conflict detection is performed when a transaction is committed. Each data record contains a version number. When a transaction reads data, it also reads this version number. When a transaction modifies data and commits, it checks whether the version number of the current data is consistent with the version number read at the time. If they match, commit the transaction and increment the version number by 1; If there is a discrepancy, it indicates a concurrent conflict, and the transaction needs to be rolled back and re-executed. The update rules for the version number are as follows: version new =version old +1; When concurrent conflicts occur, the number of retries N for a transaction retry Limits need to be imposed to avoid wasting system resources due to infinite retries. retry ≤N max , where N m ax represents the maximum number of retries.

7. The method for providing high-concurrency access to network advertising data according to claim 6, characterized in that, Step six involves building a performance monitoring system to monitor various performance metrics of the high-concurrency access service for online advertising data in real time, and to adaptively optimize the system based on the monitoring results to ensure that the system is always in the best operating state. The specific method is as follows: Performance monitoring metrics mainly include response time t resp Throughput (QPS), cache hit rate (H) cache Error rate E error ; Response time t resp This refers to the time interval from when a request is sent to when a response is received, and its calculation formula is: t resp =t receive -t send ; Among them, t send t represents the time the request was sent. receive For the response reception time; The formula for calculating throughput QPS is: Where, N req The total number of requests processed within time T; Cache hit rate H cache The calculation formula is: Where, N cache_hit N represents the number of cache hits. cache_miss To cache the number of cache misses; Error rate E error The calculation formula is: Where, N error N represents the number of requests that resulted in errors. total This represents the total number of requests.

8. A method for high-concurrency access to network advertising data according to claim 7, characterized in that, Step 7: Establish a multi-level disaster recovery system, including local disaster recovery, off-site disaster recovery, and cloud disaster recovery. The specific methods are as follows: Local disaster recovery refers to establishing a backup system within the same data center to maintain real-time data synchronization with the main system. The synchronization method adopts incremental synchronization, which means that only the changed parts of the data are transmitted to reduce the amount of data transmission and synchronization time.

Images