Cross-platform live broadcast visitor data fusion and routing load balancing method and system

Abstract

The present invention provides a cross-platform live broadcast visitor data fusion and routing load balancing method and system, which relates to the technical field of load balancing, including filtering duplicate visitor identifications through a Bloom filter array, calculating the number of independent visitors using a logarithmic cardinality counting algorithm, and dividing visitor priorities based on data flow entropy values; using a consistent hashing algorithm to build a ring routing structure, mapping visitors to virtual nodes, and performing shard routing based on priorities; recording virtual node response times in real time, identifying abnormal nodes, migrating associated visitors to spare nodes, and realizing dynamic load balancing; the present invention effectively improves the visitor data processing efficiency and system stability of a live broadcast platform, reduces server load pressure, and ensures the smoothness of live broadcast services.

Description

Technical Field

[0001] The present invention relates to the technical field of load balancing, and in particular to a method and system for cross-platform live broadcast visitor data fusion and routing load balancing. Background Art

[0002] With the rapid development of the live broadcast industry, cross-platform live broadcast has become a mainstream trend, which requires the live broadcast system to simultaneously access the visitor data streams of multiple live broadcast platforms and perform unified data processing and load distribution. Existing live broadcast systems usually adopt a distributed architecture to support large-scale live broadcast access needs through the collaborative work of multiple server nodes. In order to achieve efficient processing and reasonable distribution of visitor data, the system needs to deduplicate, count and route visitor data from different platforms.

[0003] There are still several problems in the existing technology. There is a problem of duplicate identification in cross-platform visitor data, and the traditional deduplication method takes up a lot of memory when processing large-scale data; the existing load balancing mechanism often adopts a static hash mapping scheme, which cannot dynamically adjust the routing strategy according to the real-time performance status of the server node; when the node performance fluctuates, the visitor migration process lacks effective status monitoring and progress management, which can easily lead to reduced service quality and data inconsistency.

[0004] In summary, a cross-platform live visitor data fusion and routing load balancing method is urgently needed, which realizes efficient visitor deduplication through filter array, accurately calculates the number of independent visitors by using logarithmic cardinality counting algorithm, builds an extensible routing structure by combining consistent hashing and virtual node technology, and introduces a dynamic load balancing mechanism based on performance monitoring to ensure the stability and reliability of the system. The present invention can solve the problems in the prior art. Summary of the invention

[0005] The embodiment of the present invention provides a cross-platform live broadcast visitor data fusion and routing load balancing method and system, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] A method for cross-platform live broadcast visitor data fusion and routing load balancing is provided, including:

[0008] Receive visitor data streams from multiple live broadcast platforms, filter duplicate visitor identifiers through a filter array composed of Bloom filters, and obtain a valid visitor set; calculate the number of independent visitors of the valid visitor set using a logarithmic cardinality counting algorithm, calculate a data stream entropy value based on the number of independent visitors, prioritize the valid visitor set according to the data stream entropy value, and generate a priority diversion table;

[0009] Read the visitor set and priority information in the priority diversion table, map the visitor set to a hash value through a consistent hashing algorithm, build a ring routing structure according to the hash value, and set a plurality of virtual nodes corresponding to the server nodes in the ring routing structure; establish a jump index tree in the ring routing structure, and map the visitor set to the corresponding virtual node; determine the fragmentation window size based on the priority information, group the visitor set mapped to the same virtual node according to the fragmentation window size, generate a fragmentation routing record, and write it into the routing allocation table;

[0010] Based on the binary heap structure, the response time of all virtual nodes in the ring routing structure is recorded, and the response time is used as the node weight for heap sorting; a fixed-size counting window is set for all virtual nodes in the ring routing structure, and the virtual nodes whose abnormal response times exceed a preset threshold in the counting window are identified, and the virtual nodes are marked as target nodes; the visitor set associated with the target node in the routing allocation table is read, and a backup node is selected from the ring routing structure; a migration progress bitmap is established, and the visitor set is migrated to the backup node in batches, and the migration progress is marked in the migration progress bitmap; the routing allocation table is updated according to the migration result, and the target node is removed from the ring routing structure to complete dynamic load balancing.

[0011] In an optional embodiment,

[0012] The visitor data streams of multiple live broadcast platforms are received, and repeated visitor identifiers are filtered through a filter array composed of Bloom filters to obtain a valid visitor set; the number of independent visitors is calculated by a logarithmic cardinality counting algorithm for the valid visitor set, and a data stream entropy value is calculated based on the number of independent visitors, and the valid visitor set is prioritized according to the data stream entropy value, and a priority diversion table is generated, including:

[0013] Based on a preset size, a bitmap structure is set, a Bloom filter array is constructed based on the bitmap structure, and a plurality of independent hash functions are configured for the Bloom filter array, wherein coefficients and offsets of the independent hash functions are random prime numbers;

[0014] Receive visitor data streams of multiple live broadcast platforms, extract visitor identifiers from the visitor data streams, input the visitor identifiers into the independent hash function for mapping, and obtain a location mapping set; determine whether the visitor identifier is a duplicate visitor identifier according to the binary bit state of the location mapping set in the bitmap structure; write the non-duplicate visitor identifier into the valid visitor set;

[0015] Mapping valid visitor identifiers in the valid visitor set to generate hash values, obtaining location records based on the hash values, performing bucket statistics and mean calculation based on the location records, and obtaining the number of independent visitors in the valid visitor set;

[0016] In a preset time window, based on the number of independent visitors, the proportion of visitor behavior types in the effective visitor set is counted, a data flow entropy value in the preset time window is calculated according to the proportion of visitor behavior types, the data flow entropy value is normalized, and the effective visitor set is divided into a plurality of priority visitor subsets according to the normalized data flow entropy value;

[0017] A priority diversion table is generated based on the visitor identifiers in the priority visitor subset, the timestamp of the preset time window, and the corresponding priority information.

[0018] In an optional embodiment,

[0019] The valid visitor identifiers in the valid visitor set are mapped to generate hash values, and based on the hash values, location records are obtained. Bucket statistics and mean calculations are performed based on the location records to obtain the number of independent visitors in the valid visitor set, including:

[0020] The valid visitor identifiers in the valid visitor set are mapped by a hash function to generate hash values, and based on the binary expansion of the hash values, the positions of the leftmost 1s in the binary expansion corresponding to each hash value are counted to determine as position records;

[0021] Using the high-order bits of the hash value as bucket indexes, bucketing and storing the location records to obtain multiple record buckets;

[0022] Recording the maximum value of each position record in the record bucket, and at the same time counting the variance of the position records in each record bucket, and calculating the information entropy of each record bucket based on the variance;

[0023] Classify the record buckets according to the information entropy, when the information entropy is less than a first preset threshold, mark it as a stable bucket, when the information entropy is greater than a second preset threshold, mark it as a fluctuation bucket, and when the information entropy is between the first preset threshold and the second preset threshold, mark it as a transition bucket;

[0024] For the stable bucket, the maximum value of the position record is used as the stable bucket statistic; for the fluctuation bucket, the median of the position record in the fluctuation bucket is used as the fluctuation bucket statistic; for the transition bucket, the maximum value and the median are weighted averaged according to the difference ratio between the information entropy and the first threshold value to determine the transition bucket statistic;

[0025] According to the preset weight coefficients, the stable bucket statistics, the fluctuation bucket statistics and the transition bucket statistics are respectively assigned corresponding weights, and the harmonic mean is calculated to obtain the number of independent visitors in the effective visitor set.

[0026] In an optional embodiment,

[0027] Read the visitor set and priority information in the priority diversion table, map the visitor set to a hash value through a consistent hashing algorithm, build a ring routing structure according to the hash value, and set multiple virtual nodes corresponding to server nodes in the ring routing structure; establish a jump index tree in the ring routing structure, map the visitor set to the corresponding virtual node; determine the fragmentation window size based on the priority information, group the visitor set mapped to the same virtual node according to the fragmentation window size, generate a fragmentation routing record, and write it into the routing allocation table, including:

[0028] Reading a visitor set and priority information in a priority distribution table, wherein the visitor set includes a visitor identifier and an access timestamp, and the priority information includes a priority level and a scheduling weight;

[0029] Calculating the processing performance and current load status of each server node to obtain a load factor, and determining the number of virtual nodes corresponding to each server node based on the load factor;

[0030] The node identifier of each server node is concatenated with the load coefficient to generate a node composite identifier, the node composite identifier is mapped to a virtual node hash value through a consistent hashing algorithm, a plurality of virtual node hash values ​​are generated according to the number of virtual nodes, and the virtual node hash values ​​are distributed within a preset range to form a ring routing structure;

[0031] Concatenate each visitor ID in the visitor set with the corresponding access timestamp to generate a visitor composite ID, map the visitor composite ID to a visitor hash value through the consistent hashing algorithm, and map the visitor hash value to the ring routing structure;

[0032] Constructing a jump index tree in the ring routing structure, the jump index tree comprising a plurality of index nodes, each of the index nodes recording the position information of the virtual node in the corresponding hash value interval and the load coefficient;

[0033] Searching in the jump index tree according to the visitor hash value, selecting a target index node with the smallest load factor on the search path, and selecting a virtual node with the closest hash value from the virtual nodes recorded in the target index node as the target virtual node; grouping a visitor set with the same target virtual node to obtain a target visitor set;

[0034] Calculate a priority base based on the priority level and the scheduling weight, and multiply the priority base by the load factor of the target virtual node to obtain a shard window size; calculate the time distribution density of the target visitor set at the access timestamp, determine a shard interval according to the time distribution density and the shard window size, and divide the target visitor set into multiple visitor subsets according to the shard interval;

[0035] Based on the node composite identifier corresponding to the target virtual node, the shard interval, the visitor subset, the priority information and the time distribution density, a shard routing record is generated; and the shard routing record is written into a routing allocation table.

[0036] In an optional embodiment,

[0037] Constructing a jump index tree in the ring routing structure includes:

[0038] Counting the distribution density of virtual nodes in the ring routing structure, and dividing the ring routing structure into a plurality of hash segments based on the distribution density;

[0039] Using each of the hash segments as an index node, determining a first-level index node;

[0040] Recursively divide the hash value interval range of each layer of index nodes by binary search to generate the next layer of index nodes, until the number of virtual nodes in the hash value interval range of the next layer of index nodes is less than the preset node number threshold, forming a multi-layer index node structure, recording the hash value interval range of each index node, the corresponding virtual node hash value set and the node composite identifier set;

[0041] Establish a horizontal pointer connection between index nodes in the same layer, the horizontal pointer points to the next index node adjacent to the hash value interval range; establish a vertical pointer connection between index nodes in adjacent layers, the vertical pointer points to the index node corresponding to the hash value interval range of the next layer;

[0042] Calculate the average load coefficient of the virtual nodes within the hash value interval of each index node, and use the average load coefficient as the load indicator of the index node;

[0043] Establishing a jump pointer table in each of the index nodes, wherein the jump pointer table contains remote index node positions with distances that are powers of 2;

[0044] A jump index tree is constructed based on the hash value interval range of the index node, the connection of the horizontal pointer, the connection of the vertical pointer, the load index and the jump pointer table.

[0045] In an optional embodiment,

[0046] Based on the binary heap structure, the response time of all virtual nodes in the ring routing structure is recorded, and the response time is used as the node weight for heap sorting; a fixed-size counting window is set for all virtual nodes in the ring routing structure, and the virtual node whose abnormal response times exceed a preset threshold in the counting window is identified, and the virtual node is marked as a target node; the visitor set associated with the target node in the routing allocation table is read, and a spare node is selected from the ring routing structure; a migration progress bitmap is established, and the visitor set is migrated to the spare node in batches, and the migration progress is marked in the migration progress bitmap; the routing allocation table is updated according to the migration result, and the target node is removed from the ring routing structure. The dynamic load balancing includes:

[0047] A binary heap structure is used to record the response time of a virtual node in a ring routing structure, wherein the response time includes a timestamp, a request identifier, and a processing delay; an average response time of the virtual node and a response time variance are calculated, and a performance weight of the virtual node is calculated based on the average response time and the response time variance; the virtual nodes are heap-sorted according to the performance weight to generate a performance sorting sequence;

[0048] According to a preset window size, a sliding count window is set for the virtual node, and a request-response state sequence is recorded in the sliding count window;

[0049] Calculate a response time threshold according to the average response time and the response time variance, the response time threshold being the sum of the average response time and twice the response time variance;

[0050] Counting the number of abnormal responses in the request response state sequence whose response time exceeds the response time threshold; when the number of abnormal responses exceeds a preset abnormal threshold, marking the virtual node as a target node;

[0051] Extracting a set of visitors of the target node from a routing allocation table, and selecting a set of standby nodes based on the performance ranking sequence;

[0052] Building a migration progress bitmap according to the number of the visitor set;

[0053] Compare one tenth of the number of the visitor set with a preset benchmark value, determine the total number of migration batches based on the minimum value, and divide the visitor set into multiple migration batches according to the total number of migration batches; perform parallel state migration on each migration batch, update the node mapping relationship of the migrated visitor to the routing allocation table, and mark the migration completion status of the corresponding visitor in the migration progress bitmap;

[0054] The migration integrity of the visitor set is verified through the migration progress bitmap, and all mapping relationships of the target node are deleted from the routing allocation table; the target node is marked as a pending cleanup state, and the existing request processing of the target node is waited for to be completed, and the target node is removed from the ring routing structure to complete dynamic load balancing.

[0055] In an optional embodiment,

[0056] Using a binary heap structure to record the response time of a virtual node in a ring routing structure, the response time including a timestamp, a request identifier, and a processing delay; calculating an average response time and a response time variance of the virtual node, calculating a performance weight of the virtual node based on the average response time and the response time variance, performing heap sorting on the virtual nodes according to the performance weight, and generating a performance sorting sequence includes:

[0057] Constructing a complete binary heap structure, each heap node in the complete binary heap structure stores a response time record of a virtual node; when a new response time record arrives, inserting the response time record into the last position of the complete binary heap structure;

[0058] Calculating an average response time of the virtual node based on the processing delay;

[0059] When calculating the response time variance, traverse all response time records of the virtual node, calculate the sum of squares of the difference between each processing delay and the average response time, and divide it by the number of response time records; perform weighted calculation on the average response time and the response time variance according to a preset weight ratio to obtain the performance weight of the virtual node;

[0060] Using the performance weight as a sorting key value, performing a heapization operation on the complete binary heap structure includes:

[0061] Compare the performance weight of the current node with that of the corresponding parent node. When the performance weight of the current node is greater than that of the parent node, swap the positions of the current node and the parent node, and repeat until the top of the heap is reached.

[0062] According to a preset heapization cycle, a heapization operation is performed on all nodes in the complete binary heap structure; starting from the top of the complete binary heap structure, virtual nodes corresponding to the maximum performance weights are extracted in sequence to generate a performance sorting sequence.

[0063] According to a second aspect of the embodiments of the present invention,

[0064] A cross-platform live broadcast visitor data fusion and routing load balancing system is provided, including:

[0065] The first unit is used to receive visitor data streams of multiple live broadcast platforms, filter repeated visitor identifiers through a filter array composed of Bloom filters, and obtain a valid visitor set; calculate the number of independent visitors of the valid visitor set by using a logarithmic cardinality counting algorithm, calculate a data stream entropy value based on the number of independent visitors, prioritize the valid visitor set according to the data stream entropy value, and generate a priority diversion table;

[0066] The second unit is used to read the visitor set and priority information in the priority diversion table, map the visitor set to a hash value through a consistent hashing algorithm, construct a ring routing structure according to the hash value, and set a plurality of virtual nodes corresponding to the server nodes in the ring routing structure; establish a jump index tree in the ring routing structure, map the visitor set to the corresponding virtual node; determine the fragmentation window size based on the priority information, group the visitor set mapped to the same virtual node according to the fragmentation window size, generate a fragmentation routing record, and write it into the routing allocation table;

[0067] The third unit is used to record the response time of all virtual nodes in the ring routing structure based on a binary heap structure, and use the response time as the node weight for heap sorting; set a fixed-size counting window for all virtual nodes in the ring routing structure, identify virtual nodes whose abnormal response times within the counting window exceed a preset threshold, and mark the virtual nodes as target nodes; read the visitor set associated with the target node in the routing allocation table, and select a backup node from the ring routing structure; establish a migration progress bitmap, migrate the visitor set to the backup node in batches, and mark the migration progress in the migration progress bitmap; update the routing allocation table according to the migration result, and remove the target node from the ring routing structure to complete dynamic load balancing.

[0068] According to a third aspect of the embodiments of the present invention,

[0069] An electronic device is provided, comprising:

[0070] processor;

[0071] a memory for storing processor-executable instructions;

[0072] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0073] According to a fourth aspect of the embodiments of the present invention,

[0074] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0075] In an embodiment of the present invention, a Bloom filter array is used to quickly filter duplicate visitors, and a logarithmic cardinality counting algorithm is used to accurately count the number of independent visitors, thereby reducing data redundancy and computing overhead and improving data processing efficiency. The data flow entropy value is used to divide visitor priorities, achieve differentiated services, and optimize resource utilization; a consistent hashing algorithm and virtual nodes are used to build a ring routing structure to effectively disperse visitor traffic and avoid single-point overload. The application of a jump index tree accelerates visitor routing lookup speed and improves routing efficiency. The dynamic load balancing mechanism can monitor node response time in real time, identify abnormal nodes according to preset thresholds, and migrate visitors to backup nodes to ensure stable system operation; the priority diversion table is combined with a sharding window mechanism to achieve refined diversion based on visitor priority and improve the service quality of important visitors. The use of a migration progress bitmap can accurately track the visitor migration process, ensure smooth and complete migration, and facilitate system management and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of a flow chart of a method for cross-platform live broadcast visitor data fusion and routing load balancing according to an embodiment of the present invention;

[0077] Figure 2 It is a structural diagram of the cross-platform live broadcast visitor data fusion and routing load balancing system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0080] Figure 1 FIG. 1 is a flow chart of a method for cross-platform live broadcast visitor data fusion and routing load balancing according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0081] S101. Receive visitor data streams from multiple live broadcast platforms, filter duplicate visitor identifiers through a filter array composed of Bloom filters, and obtain a valid visitor set; calculate the number of independent visitors of the valid visitor set using a logarithmic cardinality counting algorithm, calculate a data stream entropy value based on the number of independent visitors, prioritize the valid visitor set according to the data stream entropy value, and generate a priority diversion table;

[0082] S102. Read the visitor set and priority information in the priority diversion table, map the visitor set to a hash value through a consistent hashing algorithm, build a ring routing structure according to the hash value, and set a plurality of virtual nodes corresponding to the server nodes in the ring routing structure; establish a jump index tree in the ring routing structure, map the visitor set to the corresponding virtual node; determine the fragmentation window size based on the priority information, group the visitor set mapped to the same virtual node according to the fragmentation window size, generate a fragmentation routing record, and write it into the routing allocation table;

[0083] S103. Based on the binary heap structure, the response time of all virtual nodes in the ring routing structure is recorded, and the response time is used as the node weight for heap sorting; a fixed-size counting window is set for all virtual nodes in the ring routing structure, and the virtual nodes whose abnormal response times in the counting window exceed a preset threshold are identified, and the virtual nodes are marked as target nodes; the visitor set associated with the target node in the routing allocation table is read, and a backup node is selected from the ring routing structure; a migration progress bitmap is established, and the visitor set is migrated to the backup node in batches, and the migration progress is marked in the migration progress bitmap; the routing allocation table is updated according to the migration result, and the target node is removed from the ring routing structure to complete dynamic load balancing.

[0084] In an optional implementation, visitor data streams of multiple live broadcast platforms are received, and repeated visitor identifiers are filtered through a filter array composed of Bloom filters to obtain a valid visitor set; the number of independent visitors is calculated for the valid visitor set using a logarithmic cardinality counting algorithm, a data stream entropy value is calculated based on the number of independent visitors, and the valid visitor set is prioritized according to the data stream entropy value, and generating a priority diversion table includes:

[0085] Based on a preset size, a bitmap structure is set, a Bloom filter array is constructed based on the bitmap structure, and a plurality of independent hash functions are configured for the Bloom filter array, wherein coefficients and offsets of the independent hash functions are random prime numbers;

[0086] Receive visitor data streams of multiple live broadcast platforms, extract visitor identifiers from the visitor data streams, input the visitor identifiers into the independent hash function for mapping, and obtain a location mapping set; determine whether the visitor identifier is a duplicate visitor identifier according to the binary bit state of the location mapping set in the bitmap structure; write the non-duplicate visitor identifier into the valid visitor set;

[0087] Mapping valid visitor identifiers in the valid visitor set to generate hash values, obtaining location records based on the hash values, performing bucket statistics and mean calculation based on the location records, and obtaining the number of independent visitors in the valid visitor set;

[0088] In a preset time window, based on the number of independent visitors, the proportion of visitor behavior types in the effective visitor set is counted, a data flow entropy value in the preset time window is calculated according to the proportion of visitor behavior types, the data flow entropy value is normalized, and the effective visitor set is divided into a plurality of priority visitor subsets according to the normalized data flow entropy value;

[0089] A priority diversion table is generated based on the visitor identifiers in the priority visitor subset, the timestamp of the preset time window, and the corresponding priority information.

[0090] In a specific implementation, first, a bitmap structure is set and a Bloom filter array is constructed based on the bitmap structure. The size of the bitmap structure is determined according to a preset value, for example, the bitmap size is set to 1024 bits. Multiple independent hash functions are configured for the Bloom filter array, for example, 3 independent hash functions are configured. The coefficients and offsets of these hash functions use random prime numbers, for example, prime numbers 7, 11 and 17 are selected as coefficients, and prime numbers 2, 3 and 5 are selected as offsets.

[0091] Next, receive visitor data streams from multiple live broadcast platforms. Assume that the received visitor identification data stream is: user1, user2, user3, user1, user4, user2, user5. Extract the visitor identification in the data stream, and input each visitor identification into the configured independent hash function for mapping. For example, after "user1" is mapped by three hash functions, the location mapping set {123, 321, 543} is obtained. Determine whether the visitor identification is a duplicate visitor identification according to the binary bit state of the location mapping set in the bitmap structure. In the initial state, all bits of the bitmap are 0. Set the bits corresponding to the three location mapping values ​​of "user1" to 1. When "user1" appears for the second time, the same location mapping set {123, 321, 543} is obtained through hash function mapping. Since the bits of these three positions are all 1, "user1" is judged to be a duplicate visitor identification. Write the non-duplicate visitor identification into the valid visitor set. In this example, the valid visitor set is {user1, user2, user3, user4, user5}.

[0092] Then, the valid visitor IDs in the valid visitor set are mapped to generate hash values. For example, a hash function is used to hash the valid visitor IDs. Based on the hash value, a location record is obtained. For example, visitors whose hash values ​​fall within a certain range are assigned to the same bucket. Bucket statistics and mean calculations are performed based on the location records to obtain the number of independent visitors in the valid visitor set. Assuming that after bucket statistics, 5 buckets are obtained, and the number of visitors in each bucket is 1, 1, 1, 1, 1, respectively, then the number of independent visitors in the valid visitor set is 5.

[0093] Within the preset time window, for example, the time window is set to 10 minutes. Count the proportion of visitor behavior types in the valid visitor set. Assume that the visitor behavior types are divided into: watching live broadcasts, sending comments, and giving gifts. The statistical results show that watching live broadcasts accounts for 80%, sending comments accounts for 15%, and giving gifts accounts for 5%. Calculate the data flow entropy value within the preset time window based on the proportion of visitor behavior types. Normalize the data flow entropy value. Divide the valid visitor set into multiple priority visitor subsets based on the normalized data flow entropy value. For example, divide visitors into three priorities: high, medium, and low. Assume that according to the calculation results, user1 and user2 are high priority, user3 and user4 are medium priority, and user5 is low priority.

[0094] Finally, based on the visitor IDs in the priority visitor subset, the timestamps of the preset time window, and the corresponding priority information, a priority distribution table is generated. For example, the following priority distribution table is generated: timestamp: 2024-10-27 10:00:00, user1, high; timestamp: 2024-10-27 10:00:00, user2, high; timestamp: 2024-10-27 10:00:00, user3, medium; timestamp: 2024-10-27 10:00:00, user4, medium; timestamp: 2024-10-27 10:00:00, user5, low.

[0095] In this embodiment, through priority diversion, limited resources can be allocated preferentially to high-priority visitors, thereby improving resource utilization efficiency and avoiding resource waste; high-priority visitors can obtain better services, such as lower latency, clearer images, etc., thereby improving user experience and satisfaction; refined operations are performed based on visitor priority, such as pushing different content or activities to visitors of different priorities, thereby improving operational efficiency and conversion rate.

[0096] In an optional implementation, mapping valid visitor identifiers in the valid visitor set to generate hash values, obtaining location records based on the hash values, performing bucket statistics and mean calculation based on the location records, and obtaining the number of independent visitors in the valid visitor set includes:

[0097] The valid visitor identifiers in the valid visitor set are mapped by a hash function to generate hash values, and based on the binary expansion of the hash values, the positions of the leftmost 1s in the binary expansion corresponding to each hash value are counted to determine as position records;

[0098] Using the high-order bits of the hash value as bucket indexes, bucketing and storing the location records to obtain multiple record buckets;

[0099] Recording the maximum value of each position record in the record bucket, and at the same time counting the variance of the position records in each record bucket, and calculating the information entropy of each record bucket based on the variance;

[0100] Classify the record buckets according to the information entropy, when the information entropy is less than a first preset threshold, mark it as a stable bucket, when the information entropy is greater than a second preset threshold, mark it as a fluctuation bucket, and when the information entropy is between the first preset threshold and the second preset threshold, mark it as a transition bucket;

[0101] For the stable bucket, the maximum value of the position record is used as the stable bucket statistic; for the fluctuation bucket, the median of the position record in the fluctuation bucket is used as the fluctuation bucket statistic; for the transition bucket, the maximum value and the median are weighted averaged according to the difference ratio between the information entropy and the first threshold value to determine the transition bucket statistic;

[0102] According to the preset weight coefficients, the stable bucket statistics, the fluctuation bucket statistics and the transition bucket statistics are respectively assigned corresponding weights, and the harmonic mean is calculated to obtain the number of independent visitors in the effective visitor set.

[0103] In a specific implementation, first, valid visitor identifiers within a period of time are collected. The valid visitor identifier can be a user's cookie ID, device ID, or other identifier that can uniquely identify the user. For example, the collected valid visitor identifier set is {A123, B456, C789, A123, D012, E345, B456, F678, G901, A123}.

[0104] Then, a hash operation is performed on each valid visitor ID to map it into a hash value. A common hash function, such as MD5 or SHA-256, can be used. Assume that a certain hash function is used and the corresponding hash value set is {12, 45, 78, 12, 1, 34, 45, 67, 90, 12}.

[0105] Next, perform binary expansion on each hash value, find the position of the leftmost 1, and record the position as the position record. For example, the binary representation of the decimal number 12 is 1100, and the leftmost 1 is in the fourth position (counting from right to left), so the position record is 4. Similarly, the set of position records corresponding to all hash values ​​can be obtained as {4, 6, 7, 4, 1, 6, 6, 7, 7, 4}.

[0106] Use the high-order bits of the hash value as the bucket index to store the location records in buckets. For example, use the first two bits of the hash value as the bucket index to divide the location records into different buckets. Assuming that 12, 45, and 78 are all in bucket 1, 1 and 34 are in bucket 0, and 67 and 90 are in bucket 2, three record buckets are obtained: bucket 0 contains {1, 6}, bucket 1 contains {4, 6, 7, 4, 6, 4}, and bucket 2 contains {7, 7}.

[0107] In each record bucket, record the maximum value of the position record. For example, the maximum value of bucket 0 is 6, the maximum value of bucket 1 is 7, and the maximum value of bucket 2 is 7. At the same time, calculate the variance of the position record in each bucket, and calculate the information entropy based on the variance. Assume that the information entropy of bucket 0 is calculated to be 0.5, the information entropy of bucket 1 is 0.8, and the information entropy of bucket 2 is 0.

[0108] The record buckets are classified according to the preset thresholds. Assuming that the first preset threshold is 0.6 and the second preset threshold is 0.9, bucket 0 is marked as a stable bucket, bucket 1 is marked as a transition bucket, and bucket 2 is marked as a stable bucket.

[0109] For stable buckets, the maximum value of the position record is used as the stable bucket statistic. For fluctuating buckets, the median of the position record is used as the fluctuating bucket statistic. For transition buckets, the maximum value and the median are weighted averaged according to the difference ratio between the information entropy and the first threshold to determine the transition bucket statistic. For example, the stable bucket statistic of bucket 0 is 6, and the stable bucket statistic of bucket 2 is 7. The median of bucket 1 is 6, the maximum value is 7, the information entropy is 0.8, and the first threshold is 0.6. Then the transition bucket statistic is 6×(0.9-0.8) / (0.9-0.6)+7×(0.8-0.6) / (0.9-0.6)≈6.67.

[0110] Finally, according to the preset weight coefficient, the stable bucket statistics, fluctuation bucket statistics and transition bucket statistics are assigned corresponding weights, and the harmonic mean is calculated to obtain the final number of effective visitors. Assuming that the weight of the stable bucket is 0.5, the weight of the transition bucket is 0.3, and the weight of the fluctuation bucket is 0.2 (there is no fluctuation bucket in the example), the final number of effective visitors is 1 / (0.5 / 6+0.3 / 6.67+0.5 / 7)≈6.54.

[0111] In this embodiment, through the introduction of bucketing strategy and information entropy, this method can effectively identify and process data fluctuations, thereby improving the accuracy of independent visitor statistics; compared with traditional precise deduplication methods, this method is based on hashing and bucketing strategies, which greatly reduces the computational complexity and is suitable for real-time processing of massive data; the preset thresholds and weight coefficients can be adjusted according to actual conditions, and can be flexibly adapted to different data distributions and business scenarios.

[0112] In an optional implementation, the visitor set and priority information in the priority diversion table are read, the visitor set is mapped to a hash value through a consistent hashing algorithm, a ring routing structure is constructed according to the hash value, and a plurality of virtual nodes corresponding to server nodes are set in the ring routing structure; a jump index tree is established in the ring routing structure, and the visitor set is mapped to the corresponding virtual node; a sharding window size is determined based on the priority information, and the visitor set mapped to the same virtual node is grouped according to the sharding window size, a sharding routing record is generated, and written into the routing allocation table, including:

[0113] Reading a visitor set and priority information in a priority distribution table, wherein the visitor set includes a visitor identifier and an access timestamp, and the priority information includes a priority level and a scheduling weight;

[0114] Calculating the processing performance and current load status of each server node to obtain a load factor, and determining the number of virtual nodes corresponding to each server node based on the load factor;

[0115] The node identifier of each server node is concatenated with the load coefficient to generate a node composite identifier, the node composite identifier is mapped to a virtual node hash value through a consistent hashing algorithm, a plurality of virtual node hash values ​​are generated according to the number of virtual nodes, and the virtual node hash values ​​are distributed within a preset range to form a ring routing structure;

[0116] Concatenate each visitor ID in the visitor set with the corresponding access timestamp to generate a visitor composite ID, map the visitor composite ID to a visitor hash value through the consistent hashing algorithm, and map the visitor hash value to the ring routing structure;

[0117] Constructing a jump index tree in the ring routing structure, the jump index tree comprising a plurality of index nodes, each of the index nodes recording the position information of the virtual node in the corresponding hash value interval and the load coefficient;

[0118] Searching in the jump index tree according to the visitor hash value, selecting a target index node with the smallest load factor on the search path, and selecting a virtual node with the closest hash value from the virtual nodes recorded in the target index node as the target virtual node; grouping a visitor set with the same target virtual node to obtain a target visitor set;

[0119] Calculate a priority base based on the priority level and the scheduling weight, and multiply the priority base by the load factor of the target virtual node to obtain a shard window size; calculate the time distribution density of the target visitor set at the access timestamp, determine a shard interval according to the time distribution density and the shard window size, and divide the target visitor set into multiple visitor subsets according to the shard interval;

[0120] Based on the node composite identifier corresponding to the target virtual node, the shard interval, the visitor subset, the priority information and the time distribution density, a shard routing record is generated; and the shard routing record is written into a routing allocation table.

[0121] In a specific implementation, first, the visitor set and priority information are read from the priority diversion table. The visitor set contains the unique identifier and access timestamp of each visitor. The priority information contains the priority level and the scheduling weight. For example, the priority level is divided into three levels: high, medium, and low, corresponding to the scheduling weights of 10, 5, and 1, respectively. Assume that the visitor set read is as follows: Visitor A, timestamp 1678886400; Visitor B, timestamp 1678886400; Visitor C, timestamp 1678886460; Visitor D, timestamp 1678886520; Visitor E, timestamp 1678886520. The priority information is: Visitors A, B, and C are high priority, and Visitors D and E are medium priority.

[0122] Next, calculate the processing performance and current load status of each server node. For example, by monitoring the server's CPU usage, memory usage, and network traffic, comprehensively evaluate the server's processing capacity and load. Suppose there are two server nodes, the load factor of node 1 is 0.6, and the load factor of node 2 is 0.4.

[0123] Then, the number of virtual nodes is determined according to the load factor of the server node. The server node with a lower load factor is allocated with more virtual nodes. For example, node 1 is allocated with 3 virtual nodes and node 2 is allocated with 5 virtual nodes. The server node identifier and the load factor are concatenated to generate a node composite identifier, for example, "node 1-0.6" and "node 2-0.4". The node composite identifier is mapped to a virtual node hash value using a consistent hashing algorithm. Assuming that the preset hash value range is 0 to 100, the hash values ​​of the three virtual nodes of node 1 are 20, 50, and 80, respectively, and the hash values ​​of the five virtual nodes of node 2 are 10, 30, 40, 60, and 90, respectively, forming a ring routing structure.

[0124] Next, the visitor ID and the access timestamp are concatenated to generate a visitor composite ID, for example, "Visitor A-1678886400". The visitor composite ID is mapped to the visitor hash value using the same consistent hashing algorithm. For example, the hash values ​​of visitors A, B, and C are 15, 35, and 55, respectively, and the hash values ​​of visitors D and E are 45 and 75, respectively. The visitor hash values ​​are mapped to the ring routing structure.

[0125] A jump index tree is constructed in a ring routing structure. Each index node of the jump index tree records the location information and load factor of the virtual node in the corresponding hash value interval. For example, an index node records a hash value interval of 0 to 50, including virtual nodes 10, 20, 30, 40, 50, and their corresponding load factors. Retrieve according to the visitor hash value in the jump index tree. For example, the hash value of visitor A is 15, and the index nodes with a hash value interval of 0 to 50 are retrieved, and the virtual node 10 with the smallest load factor is selected as the target virtual node.

[0126] The visitor sets with the same target virtual node are grouped. For example, visitors A and B are both mapped to virtual node 10, forming a target visitor set.

[0127] Calculate the priority cardinality based on the priority level and scheduling weight. For example, the cardinality of high priority is 10, and the cardinality of medium priority is 5. Multiply the priority cardinality by the load factor of the target virtual node to get the shard window size. For example, the priority cardinality of visitors A and B is 10, and the load factor of the target virtual node 10 is 0.4, then the shard window size is 4. Calculate the time distribution density of the target visitor set on the access timestamp. For example, if the access timestamps of visitors A and B are the same, the time distribution density is higher. Determine the shard interval based on the time distribution density and the shard window size. For example, divide the time range near the timestamp 1678886400 into a shard interval. Divide the target visitor set into multiple visitor subsets according to the shard interval.

[0128] Finally, based on the node composite identifier, shard interval, visitor subset, priority information and time distribution density corresponding to the target virtual node, a shard routing record is generated and written into the routing allocation table. For example, a shard routing record contains: target server node 1, shard interval 1678886390 to 1678886410, visitor subset {visitor A, visitor B}, high priority, and high time distribution density.

[0129] In this embodiment, through priority and time distribution density control, high-priority visitors and requests during peak access periods are given priority to ensure the stable operation of key businesses; dynamic sharding based on consistent hashing and load factor evenly distributes visitor traffic to different server nodes, avoiding server overload or idleness and improving resource utilization; fast retrieval of the jump index tree and dynamic adjustment of the sharding window can effectively cope with sudden traffic and load changes, enhancing the stability and robustness of the system.

[0130] In an optional implementation, constructing a jump index tree in the ring routing structure includes:

[0131] Counting the distribution density of virtual nodes in the ring routing structure, and dividing the ring routing structure into a plurality of hash segments based on the distribution density;

[0132] Using each of the hash segments as an index node, determining a first-level index node;

[0133] Recursively divide the hash value interval range of each layer of index nodes by binary search to generate the next layer of index nodes, until the number of virtual nodes in the hash value interval range of the next layer of index nodes is less than the preset node number threshold, forming a multi-layer index node structure, recording the hash value interval range of each index node, the corresponding virtual node hash value set and the node composite identifier set;

[0134] Establish a horizontal pointer connection between index nodes in the same layer, the horizontal pointer points to the next index node adjacent to the hash value interval range; establish a vertical pointer connection between index nodes in adjacent layers, the vertical pointer points to the index node corresponding to the hash value interval range of the next layer;

[0135] Calculate the average load coefficient of the virtual nodes within the hash value interval of each index node, and use the average load coefficient as the load indicator of the index node;

[0136] Establishing a jump pointer table in each of the index nodes, wherein the jump pointer table contains remote index node positions with distances that are powers of 2;

[0137] A jump index tree is constructed based on the hash value interval range of the index node, the connection of the horizontal pointer, the connection of the vertical pointer, the load index and the jump pointer table.

[0138] In a specific implementation, first, the distribution density of virtual nodes in the ring routing structure is counted. The ring routing structure can be regarded as a ring-shaped numerical range, such as 0 to 1023. Virtual nodes are distributed on this ring according to their hash values. The distribution density of virtual nodes is counted, that is, the number of virtual nodes in different numerical intervals on the ring is counted. For example, the ring is divided into 16 segments, and the number of virtual nodes in each segment is counted. Assume that the statistical results are: 10 in the first segment, 5 in the second segment, 20 in the third segment, and so on.

[0139] Next, the ring routing structure is divided into multiple hash segments based on the distribution density of virtual nodes. The division method can be selected according to the actual situation. For example, the area with high virtual node density can be divided into smaller segments and the area with low density can be divided into larger segments according to the number of virtual nodes to ensure that the number of virtual nodes in each segment is relatively balanced. Assume that according to the above statistical results, the ring is divided into 8 hash segments, the first two segments are merged into one segment, containing 15 virtual nodes; the third segment is a separate segment, containing 20 virtual nodes; and so on.

[0140] Then, each hash segment is used as an index node to determine the first-level index node. Each index node records its corresponding hash value interval range, the hash value set of all virtual nodes in the interval, and the node composite identifier set (such as IP address and port number). Taking the above 8 hash segments as an example, they constitute the first-level index node.

[0141] Next, the hash value interval range of each layer of index nodes is recursively divided by binary division to generate the next layer of index nodes. For example, the hash value interval range of an index node in the first layer is divided into two, and two new index nodes are generated, corresponding to two sub-intervals respectively. Continue the recursive division until the number of virtual nodes in the hash value interval range of the next layer of index nodes is less than the preset node number threshold, such as 10. Assuming that an index node in the first layer contains 20 virtual nodes and its hash value interval is 200-300, it is divided into two sub-intervals 200-250 and 250-300, and the corresponding index nodes each contain a part of virtual nodes. If the number of virtual nodes in a sub-interval is still greater than 10, continue the division until the number of virtual nodes in all intervals is less than or equal to 10.

[0142] At the same time, pointer connections are established between index nodes. Horizontal pointer connections are established between index nodes on the same layer, pointing to the next index node with an adjacent hash value interval range. Vertical pointer connections are established between index nodes on adjacent layers, pointing to the index node corresponding to the hash value interval range on the next layer. For example, the hash value interval of the first-layer index node A is 0-100, and that of B is 100-200, then the horizontal pointer of A points to B. A's child nodes A1 and A2 correspond to the intervals 0-50 and 50-100, respectively, then the vertical pointer of A points to A1 and A2, respectively.

[0143] After that, the average load factor of each index node is calculated as the load indicator of the index node. The load factor can be defined according to the actual application scenario, for example, it can be the number of requests processed by the virtual node. The average load factor of all virtual nodes within the hash value range of each index node is calculated.

[0144] A jump pointer table is created in each index node. The jump pointer table contains the remote index node positions with distances that are powers of 2. For example, the jump pointer table contains the index node position information with distances of 1, 2, 4, 8, etc. This is similar to the structure of a jump linked list and can speed up node search.

[0145] Finally, a jump index tree is constructed based on the hash value interval range of the index node, the connection of the horizontal pointer, the connection of the vertical pointer, the load indicator and the jump pointer table.

[0146] In this embodiment, by jumping the index tree, the target virtual node can be quickly located to avoid traversing the entire ring routing structure, thereby improving query efficiency; utilizing the structure of the index tree, range queries can be easily performed to quickly find all virtual nodes within a specific hash value interval; by recording the load index of each index node, requests can be directed to virtual nodes with lower loads to achieve load balancing and improve system stability and performance.

[0147] In an optional implementation, based on a binary heap structure, the response time of all virtual nodes in the ring routing structure is recorded, and the response time is used as the node weight for heap sorting; a fixed-size counting window is set for all virtual nodes in the ring routing structure, and the virtual node whose abnormal response times in the counting window exceed a preset threshold is identified, and the virtual node is marked as a target node; the visitor set associated with the target node in the routing allocation table is read, and a backup node is selected from the ring routing structure; a migration progress bitmap is established, and the visitor set is migrated to the backup node in batches, and the migration progress is marked in the migration progress bitmap; the routing allocation table is updated according to the migration result, and the target node is removed from the ring routing structure. The completion of dynamic load balancing includes:

[0148] A binary heap structure is used to record the response time of a virtual node in a ring routing structure, wherein the response time includes a timestamp, a request identifier, and a processing delay; an average response time of the virtual node and a response time variance are calculated, and a performance weight of the virtual node is calculated based on the average response time and the response time variance; the virtual nodes are heap-sorted according to the performance weight to generate a performance sorting sequence;

[0149] According to a preset window size, a sliding count window is set for the virtual node, and a request response state sequence is recorded in the sliding count window;

[0150] Calculate a response time threshold according to the average response time and the response time variance, the response time threshold being the sum of the average response time and twice the response time variance;

[0151] Counting the number of abnormal responses in the request response state sequence whose response time exceeds the response time threshold; when the number of abnormal responses exceeds a preset abnormal threshold, marking the virtual node as a target node;

[0152] Extracting a set of visitors of the target node from a routing allocation table, and selecting a set of standby nodes based on the performance ranking sequence;

[0153] Building a migration progress bitmap according to the number of the visitor set;

[0154] Compare one tenth of the number of the visitor set with a preset benchmark value, determine the total number of migration batches based on the minimum value, and divide the visitor set into multiple migration batches according to the total number of migration batches; perform parallel state migration on each migration batch, update the node mapping relationship of the migrated visitor to the routing allocation table, and mark the migration completion status of the corresponding visitor in the migration progress bitmap;

[0155] The migration integrity of the visitor set is verified through the migration progress bitmap, and all mapping relationships of the target node are deleted from the routing allocation table; the target node is marked as a pending cleanup state, and the existing request processing of the target node is waited for to be completed, and the target node is removed from the ring routing structure to complete dynamic load balancing.

[0156] In a specific implementation, firstly, virtual nodes in a ring routing structure are defined. Each virtual node is responsible for processing a portion of visitor requests and has a unique identifier. The visitor requests are distributed to different virtual nodes according to preset routing rules.

[0157] Record the virtual node response time. Use a binary heap structure to store the response time information of all virtual nodes in the ring routing structure. Each record contains a timestamp, request identifier, and processing delay. For example, if virtual node A processes request ID 123 at 10:00:00 on January 1, 2024 with a delay of 50 milliseconds, the record is {2024-01-01 10:00:00, 123, 50}.

[0158] Calculate the performance weight of the virtual node. Periodically calculate the average response time and response time variance of each virtual node. For example, virtual node A has processed 100 requests in the past minute, with a total delay of 5000 milliseconds, and the average response time is 50 milliseconds. The response time variance is calculated as the average of the sum of the squares of the difference between the delay of each request and the average response time. Assume that the calculated variance is 25. The performance weight is jointly determined by the average response time and the response time variance. A smaller average response time and variance corresponds to a higher performance weight. In this embodiment, the performance weight calculation method is the sum of the average response time and the response time variance. For example, the performance weight of virtual node A is 50+25=75. Heap sort all virtual nodes according to the performance weight to generate a performance sort sequence.

[0159] Identify the target node. Set a fixed-size sliding count window for each virtual node, for example, the window size is 100. Record the request response state sequence in the sliding count window, including the response time of each request. Calculate the response time threshold. The response time threshold is calculated as the sum of the average response time and twice the variance of the response time. For example, the response time threshold of virtual node A is 50+2×25=100 milliseconds. Count the number of abnormal responses in the sliding count window whose response time exceeds the threshold. For example, if the response time of 15 requests of virtual node A in the last 100 requests exceeds 100 milliseconds, the number of abnormal responses is 15. When the number of abnormal responses exceeds the preset abnormal threshold (for example, 10), mark the virtual node as a target node.

[0160] Select a backup node. Extract the visitor set of the target node from the routing allocation table. For example, the visitor set of the target node A is {user1, user2, user3, ..., user100}. Select a backup node set based on the performance sorting sequence. Select several nodes with high performance weights as backup nodes, for example, select the top three nodes with the best performance.

[0161] Perform visitor migration. Construct a migration progress bitmap based on the number of visitor sets of the target node. Each bit of the bitmap corresponds to a visitor, and the initial state is 0. For example, if the target node A has 100 visitors, a 100-bit bitmap is constructed. Compare one-tenth of the number of visitor sets with a preset benchmark value (for example, 5), and take the smaller value as the total number of migration batches. For example, if the target node A has 100 visitors, one-tenth is 10, and compared with the benchmark value 5, 5 is taken as the total number of migration batches. Divide the visitor set into 5 migration batches evenly. Perform parallel state migration for each migration batch to migrate the visitors to the backup node. For example, migrate the first batch of 20 visitors to the backup node B. After the migration is completed, update the mapping relationship between nodes and visitors in the routing allocation table. For example, update the mapping relationship from user1 to user20 in the routing allocation table from node A to node B. And mark the corresponding visitor's bit as 1 in the migration progress bitmap.

[0162] Remove the target node. Verify that all guests have been migrated through the migration progress bitmap. When all bits in the bitmap are 1, the migration is complete. Delete all mappings of the target node from the routing allocation table. Mark the target node as pending cleanup and wait for its existing requests to be processed. Finally, remove the target node from the ring routing structure.

[0163] In this embodiment, by dynamically migrating visitors from overloaded nodes, single point failures are avoided and the overall stability and reliability of the system are improved; by migrating visitors from overloaded nodes to nodes with lower loads, dynamic allocation and balanced utilization of resources are achieved, avoiding waste of resources; by migrating visitors to nodes with faster response, the processing delay of user requests is reduced and the user experience is improved.

[0164] In an optional implementation, a binary heap structure is used to record the response time of a virtual node in a ring routing structure, wherein the response time includes a timestamp, a request identifier, and a processing delay; an average response time of the virtual node and a response time variance are calculated, and a performance weight of the virtual node is calculated based on the average response time and the response time variance; the virtual nodes are heap-sorted according to the performance weight, and generating a performance sorting sequence includes:

[0165] Constructing a complete binary heap structure, each heap node in the complete binary heap structure stores a response time record of a virtual node; when a new response time record arrives, inserting the response time record into the last position of the complete binary heap structure;

[0166] Calculating an average response time of the virtual node based on the processing delay;

[0167] When calculating the response time variance, traverse all response time records of the virtual node, calculate the sum of squares of the difference between each processing delay and the average response time, and divide it by the number of response time records; perform weighted calculation on the average response time and the response time variance according to a preset weight ratio to obtain the performance weight of the virtual node;

[0168] Using the performance weight as a sorting key value, performing a heapization operation on the complete binary heap structure includes:

[0169] Compare the performance weight of the current node with that of the corresponding parent node. When the performance weight of the current node is greater than that of the parent node, swap the positions of the current node and the parent node, and repeat until the top of the heap is reached.

[0170] According to a preset heapization cycle, a heapization operation is performed on all nodes in the complete binary heap structure; starting from the top of the complete binary heap structure, virtual nodes corresponding to the maximum performance weights are extracted in sequence to generate a performance sorting sequence.

[0171] The virtual node performance ranking method and system in the ring routing structure can track the response performance of the virtual nodes in real time, and dynamically adjust the routing strategy accordingly to improve the overall system efficiency.

[0172] First, the system initializes an empty complete binary heap to store the response time records of the virtual nodes. Each record contains three fields: timestamp, request identifier, and processing delay. For example, a record can be {timestamp: 2024-07-27 10:00:00, request identifier: Request123, processing delay: 50ms}.

[0173] Next, when a new response time record arrives, the system inserts it into the last position of the complete binary heap. Assuming the new record is {timestamp: 2024-07-27 10:00:05, request ID: Request124, processing delay: 60ms}, it is added to the end of the heap.

[0174] Then, the system calculates the average response time of the virtual node. For example, if the processing delays of the response time records of a virtual node are 50ms, 60ms, and 70ms respectively, the average response time of the virtual node is (50+60+70) / 3=60ms.

[0175] Then, the system calculates the variance of the response time of the virtual node. Continuing with the above virtual node as an example, the square of the difference between each processing delay and the average response time (60ms) is calculated, and then these square values ​​are added and divided by the number of records. That is, ((50-60) 2 +(60-60)2 +(70-60) 2 ) / 3=66.67.

[0176] Next, the system weights the average response time and the response time variance according to the preset weight ratio to obtain the performance weight of the virtual node. Assuming that the weight of the average response time is 0.7 and the weight of the response time variance is 0.3, the performance weight of the virtual node is 0.7×60+0.3×66.67=62.001.

[0177] After obtaining the performance weight, the system performs a heapization operation on the complete binary heap. The performance weight of the newly inserted node is compared with that of its parent node. If the performance weight of the newly inserted node is greater than that of its parent node, the positions of the two nodes are swapped. This process is repeated until the newly inserted node reaches the correct position or reaches the top of the heap.

[0178] The system performs a heapization operation on all nodes in the complete binary heap according to a preset heapization cycle, for example, every 10 seconds, to maintain the properties of the heap.

[0179] Finally, the system starts from the top of the complete binary heap and extracts the virtual nodes corresponding to the maximum performance weights in turn to generate a performance sorting sequence. For example, assuming that the three nodes at the top of the heap correspond to virtual nodes A, B, and C, and their performance weights are 80, 70, and 60, respectively, then the generated performance sorting sequence is A, B, and C.

[0180] In this embodiment, a binary heap structure is used to efficiently update the performance weights of virtual nodes and generate performance ranking sequences in real time, thereby realizing dynamic adjustment of the ring routing structure; the average response time and the response time variance are comprehensively considered, so that the performance of virtual nodes can be evaluated more accurately and the deviation caused by a single indicator can be avoided; the heap sorting algorithm is used, which has a low computational complexity and can quickly sort virtual nodes, saving system resources.

[0181] Figure 2 FIG. 1 is a schematic diagram of the structure of a cross-platform live visitor data fusion and routing load balancing system according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0182] The first unit is used to receive visitor data streams of multiple live broadcast platforms, filter repeated visitor identifiers through a filter array composed of Bloom filters, and obtain a valid visitor set; calculate the number of independent visitors of the valid visitor set by using a logarithmic cardinality counting algorithm, calculate a data stream entropy value based on the number of independent visitors, prioritize the valid visitor set according to the data stream entropy value, and generate a priority diversion table;

[0183] The second unit is used to read the visitor set and priority information in the priority diversion table, map the visitor set to a hash value through a consistent hashing algorithm, construct a ring routing structure according to the hash value, and set a plurality of virtual nodes corresponding to the server nodes in the ring routing structure; establish a jump index tree in the ring routing structure, map the visitor set to the corresponding virtual node; determine the fragmentation window size based on the priority information, group the visitor set mapped to the same virtual node according to the fragmentation window size, generate a fragmentation routing record, and write it into the routing allocation table;

[0184] The third unit is used to record the response time of all virtual nodes in the ring routing structure based on a binary heap structure, and use the response time as the node weight for heap sorting; set a fixed-size counting window for all virtual nodes in the ring routing structure, identify virtual nodes whose abnormal response times within the counting window exceed a preset threshold, and mark the virtual nodes as target nodes; read the visitor set associated with the target node in the routing allocation table, and select a backup node from the ring routing structure; establish a migration progress bitmap, migrate the visitor set to the backup node in batches, and mark the migration progress in the migration progress bitmap; update the routing allocation table according to the migration result, and remove the target node from the ring routing structure to complete dynamic load balancing.

[0185] According to a third aspect of the embodiments of the present invention,

[0186] An electronic device is provided, comprising:

[0187] processor;

[0188] a memory for storing processor-executable instructions;

[0189] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0190] According to a fourth aspect of the embodiments of the present invention,

[0191] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0192] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-platform live broadcast visitor data fusion and routing load balancing method, characterized in that: include: Receive visitor data streams from multiple live broadcast platforms, filter duplicate visitor identifiers through a filter array composed of Bloom filters, and obtain a valid visitor set; Calculating the number of independent visitors by using a logarithmic cardinality counting algorithm for the valid visitor set, calculating a data flow entropy value based on the number of independent visitors, prioritizing the valid visitor set according to the data flow entropy value, and generating a priority diversion table; Reading the visitor set and priority information in the priority distribution table, mapping the visitor set to a hash value through a consistent hashing algorithm, constructing a ring routing structure according to the hash value, and setting a plurality of virtual nodes corresponding to the server nodes in the ring routing structure; Establishing a jump index tree in the ring routing structure, mapping the visitor set to the corresponding virtual node; determining the fragmentation window size based on the priority information, grouping the visitor set mapped to the same virtual node according to the fragmentation window size, generating fragmentation routing records, and writing them into the routing allocation table; Based on the binary heap structure, the response time of all virtual nodes in the ring routing structure is recorded, and the response time is used as the node weight for heap sorting; a fixed-size counting window is set for all virtual nodes in the ring routing structure, and the virtual nodes whose abnormal response times exceed a preset threshold in the counting window are identified, and the virtual nodes are marked as target nodes; the visitor set associated with the target node in the routing allocation table is read, and a backup node is selected from the ring routing structure; a migration progress bitmap is established, and the visitor set is migrated to the backup node in batches, and the migration progress is marked in the migration progress bitmap; the routing allocation table is updated according to the migration result, and the target node is removed from the ring routing structure to complete dynamic load balancing.

2. The method according to claim 1, characterized in that The visitor data streams of multiple live broadcast platforms are received, and repeated visitor identifiers are filtered through a filter array composed of Bloom filters to obtain a valid visitor set; the number of independent visitors is calculated by a logarithmic cardinality counting algorithm for the valid visitor set, and a data stream entropy value is calculated based on the number of independent visitors, and the valid visitor set is prioritized according to the data stream entropy value, and a priority diversion table is generated, including: Based on a preset size, a bitmap structure is set, a Bloom filter array is constructed based on the bitmap structure, and a plurality of independent hash functions are configured for the Bloom filter array, wherein coefficients and offsets of the independent hash functions are random prime numbers; Receive visitor data streams of multiple live broadcast platforms, extract visitor identifiers from the visitor data streams, input the visitor identifiers into the independent hash function for mapping, and obtain a location mapping set; determine whether the visitor identifier is a duplicate visitor identifier according to the binary bit state of the location mapping set in the bitmap structure; write the non-duplicate visitor identifier into the valid visitor set; Mapping valid visitor identifiers in the valid visitor set to generate hash values, obtaining location records based on the hash values, performing bucket statistics and mean calculation based on the location records, and obtaining the number of independent visitors in the valid visitor set; In a preset time window, based on the number of independent visitors, the proportion of visitor behavior types in the effective visitor set is counted, a data flow entropy value in the preset time window is calculated according to the proportion of visitor behavior types, the data flow entropy value is normalized, and the effective visitor set is divided into a plurality of priority visitor subsets according to the normalized data flow entropy value; A priority diversion table is generated based on the visitor identifiers in the priority visitor subset, the timestamp of the preset time window, and the corresponding priority information.

3. The method according to claim 2, characterized in that Mapping valid visitor identifiers in the valid visitor set to generate hash values, obtaining location records based on the hash values, performing bucket statistics and mean calculation based on the location records, and obtaining the number of independent visitors in the valid visitor set includes: The valid visitor identifiers in the valid visitor set are mapped by a hash function to generate hash values, and based on the binary expansion of the hash values, the positions of the leftmost 1s in the binary expansion corresponding to each hash value are counted to determine as position records; Using the high-order bits of the hash value as bucket indexes, bucketing and storing the location records to obtain multiple record buckets; Recording the maximum value of each position record in the record bucket, and at the same time counting the variance of the position records in each record bucket, and calculating the information entropy of each record bucket based on the variance; Classify the record buckets according to the information entropy, when the information entropy is less than a first preset threshold, mark it as a stable bucket, when the information entropy is greater than a second preset threshold, mark it as a fluctuation bucket, and when the information entropy is between the first preset threshold and the second preset threshold, mark it as a transition bucket; For the stable bucket, the maximum value of the position record is used as the stable bucket statistic; for the fluctuation bucket, the median of the position record in the fluctuation bucket is used as the fluctuation bucket statistic; for the transition bucket, the maximum value and the median are weighted averaged according to the difference ratio between the information entropy and the first threshold value to determine the transition bucket statistic; According to the preset weight coefficients, the stable bucket statistics, the fluctuation bucket statistics and the transition bucket statistics are respectively assigned corresponding weights, and the harmonic mean is calculated to obtain the number of independent visitors in the effective visitor set.

4. The method according to claim 1, characterized in that: Reading the visitor set and priority information in the priority distribution table, mapping the visitor set to a hash value through a consistent hashing algorithm, constructing a ring routing structure according to the hash value, and setting a plurality of virtual nodes corresponding to the server nodes in the ring routing structure; Establishing a jump index tree in the ring routing structure, mapping the visitor set to the corresponding virtual node; determining the fragmentation window size based on the priority information, grouping the visitor set mapped to the same virtual node according to the fragmentation window size, generating fragmentation routing records, and writing them into the routing allocation table includes: Reading a visitor set and priority information in a priority distribution table, wherein the visitor set includes a visitor identifier and an access timestamp, and the priority information includes a priority level and a scheduling weight; Calculating the processing performance and current load status of each server node to obtain a load factor, and determining the number of virtual nodes corresponding to each server node based on the load factor; The node identifier of each server node is concatenated with the load coefficient to generate a node composite identifier, the node composite identifier is mapped to a virtual node hash value through a consistent hashing algorithm, a plurality of virtual node hash values ​​are generated according to the number of virtual nodes, and the virtual node hash values ​​are distributed within a preset range to form a ring routing structure; Concatenate each visitor ID in the visitor set with the corresponding access timestamp to generate a visitor composite ID, map the visitor composite ID to a visitor hash value through the consistent hashing algorithm, and map the visitor hash value to the ring routing structure; Constructing a jump index tree in the ring routing structure, the jump index tree comprising a plurality of index nodes, each of the index nodes recording the position information of the virtual node in the corresponding hash value interval and the load coefficient; Searching in the jump index tree according to the visitor hash value, selecting a target index node with the smallest load factor on the search path, and selecting a virtual node with the closest hash value from the virtual nodes recorded in the target index node as the target virtual node; grouping a visitor set with the same target virtual node to obtain a target visitor set; Calculate a priority base based on the priority level and the scheduling weight, and multiply the priority base by the load factor of the target virtual node to obtain a shard window size; calculate the time distribution density of the target visitor set at the access timestamp, determine a shard interval according to the time distribution density and the shard window size, and divide the target visitor set into multiple visitor subsets according to the shard interval; Based on the node composite identifier corresponding to the target virtual node, the shard interval, the visitor subset, the priority information and the time distribution density, a shard routing record is generated; and the shard routing record is written into a routing allocation table.

5. The method according to claim 4, characterized in that Constructing a jump index tree in the ring routing structure includes: Counting the distribution density of virtual nodes in the ring routing structure, and dividing the ring routing structure into a plurality of hash segments based on the distribution density; Using each of the hash segments as an index node, determining a first-level index node; Recursively divide the hash value interval range of each layer of index nodes by binary search to generate the next layer of index nodes, until the number of virtual nodes in the hash value interval range of the next layer of index nodes is less than the preset node number threshold, forming a multi-layer index node structure, recording the hash value interval range of each index node, the corresponding virtual node hash value set and the node composite identifier set; Establish a horizontal pointer connection between index nodes in the same layer, the horizontal pointer points to the next index node adjacent to the hash value interval range; establish a vertical pointer connection between index nodes in adjacent layers, the vertical pointer points to the index node corresponding to the hash value interval range of the next layer; Calculate the average load coefficient of the virtual nodes within the hash value interval of each index node, and use the average load coefficient as the load indicator of the index node; Establishing a jump pointer table in each of the index nodes, wherein the jump pointer table contains remote index node positions with distances that are powers of 2; A jump index tree is constructed based on the hash value interval range of the index node, the connection of the horizontal pointer, the connection of the vertical pointer, the load index and the jump pointer table.

6. The method according to claim 1, characterized in that Based on the binary heap structure, the response time of all virtual nodes in the ring routing structure is recorded, and the response time is used as the node weight for heap sorting; a fixed-size counting window is set for all virtual nodes in the ring routing structure, and the virtual nodes whose abnormal response times exceed a preset threshold in the counting window are identified, and the virtual nodes are marked as target nodes; the visitor set associated with the target node in the routing allocation table is read, and a backup node is selected from the ring routing structure; a migration progress bitmap is established, and the visitor set is migrated to the backup node in batches, and the migration progress is marked in the migration progress bitmap; The routing allocation table is updated according to the migration result, and the target node is removed from the ring routing structure to complete the dynamic load balancing, including: A binary heap structure is used to record the response time of a virtual node in a ring routing structure, wherein the response time includes a timestamp, a request identifier, and a processing delay; an average response time of the virtual node and a response time variance are calculated, and a performance weight of the virtual node is calculated based on the average response time and the response time variance; the virtual nodes are heap-sorted according to the performance weight to generate a performance sorting sequence; According to a preset window size, a sliding count window is set for the virtual node, and a request-response state sequence is recorded in the sliding count window; Calculate a response time threshold according to the average response time and the response time variance, the response time threshold being the sum of the average response time and twice the response time variance; Counting the number of abnormal responses in the request response state sequence whose response time exceeds the response time threshold; when the number of abnormal responses exceeds a preset abnormal threshold, marking the virtual node as a target node; Extracting a set of visitors of the target node from a routing allocation table, and selecting a set of standby nodes based on the performance ranking sequence; Building a migration progress bitmap according to the number of the visitor set; Compare one tenth of the number of the visitor set with a preset benchmark value, determine the total number of migration batches based on the minimum value, and divide the visitor set into multiple migration batches according to the total number of migration batches; perform parallel state migration on each migration batch, update the node mapping relationship of the migrated visitor to the routing allocation table, and mark the migration completion status of the corresponding visitor in the migration progress bitmap; The migration integrity of the visitor set is verified through the migration progress bitmap, and all mapping relationships of the target node are deleted from the routing allocation table; the target node is marked as a pending cleanup state, and the existing request processing of the target node is waited for to be completed, and the target node is removed from the ring routing structure to complete dynamic load balancing.

7. The method according to claim 6, characterized in that Using a binary heap structure to record the response time of a virtual node in a ring routing structure, the response time including a timestamp, a request identifier, and a processing delay; calculating an average response time and a response time variance of the virtual node, calculating a performance weight of the virtual node based on the average response time and the response time variance, performing heap sorting on the virtual nodes according to the performance weight, and generating a performance sorting sequence includes: Constructing a complete binary heap structure, each heap node in the complete binary heap structure stores a response time record of a virtual node; when a new response time record arrives, inserting the response time record into the last position of the complete binary heap structure; Calculating an average response time of the virtual node based on the processing delay; When calculating the response time variance, traverse all response time records of the virtual node, calculate the sum of squares of the difference between each processing delay and the average response time, and divide it by the number of response time records; perform weighted calculation on the average response time and the response time variance according to a preset weight ratio to obtain the performance weight of the virtual node; Using the performance weight as a sorting key value, performing a heapization operation on the complete binary heap structure includes: Compare the performance weight of the current node with that of the corresponding parent node. When the performance weight of the current node is greater than that of the parent node, swap the positions of the current node and the parent node, and repeat until the top of the heap is reached. According to a preset heapization cycle, a heapization operation is performed on all nodes in the complete binary heap structure; starting from the top of the complete binary heap structure, virtual nodes corresponding to the maximum performance weights are extracted in sequence to generate a performance sorting sequence.

8. A cross-platform live broadcast visitor data fusion and routing load balancing system, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to receive visitor data streams of multiple live broadcast platforms, filter repeated visitor identifiers through a filter array composed of Bloom filters, and obtain a valid visitor set; calculate the number of independent visitors of the valid visitor set by using a logarithmic cardinality counting algorithm, calculate a data stream entropy value based on the number of independent visitors, prioritize the valid visitor set according to the data stream entropy value, and generate a priority diversion table; The second unit is used to read the visitor set and priority information in the priority diversion table, map the visitor set to a hash value through a consistent hashing algorithm, build a ring routing structure according to the hash value, and set a plurality of virtual nodes corresponding to the server nodes in the ring routing structure; Establishing a jump index tree in the ring routing structure, mapping the visitor set to the corresponding virtual node; determining the fragmentation window size based on the priority information, grouping the visitor set mapped to the same virtual node according to the fragmentation window size, generating fragmentation routing records, and writing them into the routing allocation table; The third unit is used to record the response time of all virtual nodes in the ring routing structure based on a binary heap structure, and use the response time as the node weight for heap sorting; set a fixed-size counting window for all virtual nodes in the ring routing structure, identify virtual nodes whose abnormal response times within the counting window exceed a preset threshold, and mark the virtual nodes as target nodes; read the visitor set associated with the target node in the routing allocation table, and select a backup node from the ring routing structure; establish a migration progress bitmap, migrate the visitor set to the backup node in batches, and mark the migration progress in the migration progress bitmap; update the routing allocation table according to the migration result, and remove the target node from the ring routing structure to complete dynamic load balancing.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images