Hotspot data detection method and device, equipment and storage medium

Abstract

The invention discloses a hotspot data detection method and device, equipment and a storage medium, and the method comprises the steps: intercepting a user access request received by a service instance through a client SDK integrated in the service instance, and extracting to-be-detected data; distributing the to-be-detected data to each computing node of a computing component through the client SDK; performing hotspot judgment on the to-be-detected data through each computing node, and marking the to-be-detected data meeting a preset hotspot condition as hotspot data; and pushing the hotspot data to the corresponding service instance from each computing node. According to the invention, the centralized calculation of the calculation component is matched with the distributed acquisition of the client SDK, so that the accuracy and real-time performance of hotspot data detection are improved.

Description

Technical Field

[0001] The present application relates to the technical field of distributed computing, and particularly to a method, device, equipment and storage medium for detecting hot data. Background Art

[0002] In a distributed service architecture, the real-time detection of hot data is a key technical challenge to ensure the stability and performance of the system. Usually, a centralized storage-based solution is adopted to identify hot spots by counting the data access frequency per unit time. However, such methods rely on external storage, introducing additional network communication overhead, resulting in increased detection latency and difficulty in meeting the millisecond-level real-time requirement; secondly, in a distributed multi-instance deployment environment, access requests for the same data may be scattered across different service instances, and centralized statistics are difficult to quickly and accurately aggregate global access data, affecting the timeliness and accuracy of hot spot determination. In addition, the introduction of third-party storage not only increases the complexity of the system architecture but may also further reduce the overall stability of the system due to single-point failures or performance bottlenecks of storage components. With the growth of business concurrency, existing technologies often face problems such as response latency and increased resource competition in high-load scenarios, and cannot effectively balance the requirements of real-time detection and system lightweighting. Summary of the Invention

[0003] The main purpose of the present application is to provide a method, device, equipment and storage medium for detecting hot data, aiming to solve the technical problem of how to detect hot data in real time and efficiently in a distributed service architecture.

[0004] To achieve the above object, the present application provides a method for detecting hot data, which includes the following steps:

[0005] Intercept user access requests received by the service instance through the client SDK integrated in the service instance, and extract the data to be detected;

[0006] Distribute the data to be detected to each computing node of the computing component through the client SDK;

[0007] Perform hot spot determination on the data to be detected by each computing node, and mark the data to be detected that meets the preset hot spot condition as hot data;

[0008] Push the hot data from each computing node to the corresponding service instance.

[0009] In an embodiment, before the step of distributing the data to be detected to each computing node of the computing component through the client SDK, it includes:

[0010] Generate a mapping relationship table between the service name and the connection channel based on the connection channels established between the client SDK and each computing node of the computing component, where the service name is the unique identifier of the cluster to which the service instance belongs.

[0011] In one embodiment, after the step of generating a mapping relationship table between the service name and the connection channel based on the connection channels established between the client SDK and each computing node of the computing component, the following steps are included:

[0012] Update the routing change data stored in the configuration center based on the self-routing information reported by each computing node of the computing component;

[0013] Monitor the routing change data of the configuration center through the client SDK, and adjust the mapping relationship table between the service name and the connection channel according to the routing change data.

[0014] In one embodiment, before the step of distributing the data to be detected to each computing node of the computing component through the client SDK, the following steps are further included:

[0015] In response to the client SDK monitoring a rule configuration change event of the configuration center, pull the latest rule configuration associated with the event from the configuration center;

[0016] If the data to be detected hits the latest rule configuration and is not marked as hot data, then execute the step: distribute the data to be detected to each computing node of the computing component through the client SDK.

[0017] In one embodiment, the step of distributing the data to be detected to each computing node of the computing component through the client SDK includes:

[0018] Perform a hash calculation on the key value of the data to be detected to generate a hash value;

[0019] Perform a modulo operation on the hash value and the total number of computing nodes to obtain a target node index;

[0020] According to the target node index, match the corresponding target connection channel from the mapping relationship table, and send the data to be detected to the target computing node through the target connection channel.

[0021] In one embodiment, the step of performing a hot spot determination on the data to be detected by each computing node and marking the data to be detected that meets the preset hot spot conditions as hot data includes:

[0022] Obtain an algorithm object corresponding to the data to be detected, where the algorithm object includes a time window length, a preset access threshold, a window start time, and an access counter;

[0023] Calculate the time difference between the current system time and the start time of the window;

[0024] When the time difference is not greater than the length of the time window, update the access counter and determine whether the updated access counter exceeds the preset access threshold;

[0025] If it exceeds the preset access threshold, mark the data to be detected as hot data.

[0026] In one embodiment, before the step of obtaining the algorithm object corresponding to the data to be detected, it includes:

[0027] If it is determined that the data to be detected is distributed for the first time, create a corresponding algorithm object according to the key value of the data to be detected, and initialize the time window length and the preset access threshold of the algorithm object based on the rule configuration obtained from the configuration center.

[0028] In addition, to achieve the above object, the present application further provides a hot data detection device, and the hot data detection device includes:

[0029] An extraction module, configured to intercept a user access request received by the service instance through a client SDK integrated in the service instance, and extract data to be detected;

[0030] A distribution module, configured to distribute the data to be detected to each computing node of the computing component through the client SDK; [

[0031] A determination module, configured to perform hot determination on the data to be detected through each computing node, and mark the data to be detected that meets the preset hot conditions as hot data;

[0032] A push module, configured to push the hot data from each computing node to the corresponding service instance.

[0033] In addition, to achieve the above object, the present application further provides a terminal device, and the terminal device includes a memory, a processor, and a hot data detection program stored on the memory and executable on the processor. When the hot data detection program is executed by the processor, the steps of the hot data detection method described above are implemented.

[0034] In addition, to achieve the above object, the present application further provides a computer-readable storage medium, and a hot data detection program is stored on the computer-readable storage medium. When the hot data detection program is executed by a processor, the steps of the hot data detection method described above are implemented.

[0035] One or more technical solutions proposed by the present application have at least the following technical effects:

[0036] This application adopts a combination of distributed collection and centralized computing to achieve real-time and efficient detection of hot data under a distributed service architecture.

[0037] Specifically, by integrating the client SDK in the service instance to directly intercept user access requests and extract the data to be detected, there is no need to obtain data using external storage, reducing additional communication overhead and lowering data transmission latency. Secondly, the data to be detected is distributed to each computing node of the computing component through the client SDK, and the hotness determination is performed on the data to be detected by each computing node. The data to be detected that meets the preset hot conditions is marked as hot data. A cluster of computing components is used to centrally process the data to be detected reported in a distributed manner. Through the parallel computing of multiple computing nodes, the accuracy and timeliness of hotness determination are ensured. In summary, this application combines the centralized computing of the computing component with the distributed collection of the client SDK to improve the accuracy and real-time performance of hot data detection. In addition, since all processing is completed within the service cluster and does not rely on external storage components, not only is the system operation and maintenance complexity reduced, but the reliability and scalability of the overall solution are also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic flowchart of the first exemplary embodiment of the hot data detection method of this application;

[0039] Figure 2 It is a schematic diagram of the mapping relationship between the service name and the connection channel of this application;

[0040] Figure 3 It is an exemplary system architecture diagram of the hot data detection of this application;

[0041] Figure 4 It is a schematic flowchart of the second exemplary embodiment of the hot data detection method of this application;

[0042] Figure 5 It is a schematic flowchart of the third exemplary embodiment of the hot data detection method of this application;

[0043] Figure 6 It is a schematic flowchart of the fourth exemplary embodiment of the hot data detection method of this application;

[0044] Figure 7 It is a schematic diagram of the module structure of the hot data detection device in the embodiment of this application;

[0045] Figure 8 It is a schematic diagram of the device structure of the hardware operating environment involved in the hot data detection method in the embodiment of this application.

[0046] The realization, functional features, and advantages of the present application will be further described in conjunction with embodiments and with reference to the accompanying drawings. Specific Embodiments

[0047] It should be understood that the specific embodiments described herein are merely used to explain the present application and are not used to limit the present application.

[0048] The main technical solution of the present application is as follows: intercept the user access requests received by the service instance through the client SDK integrated in the service instance, and extract the data to be detected; distribute the data to be detected to each computing node of the computing component through the client SDK; perform hotspot determination on the data to be detected by each computing node, and mark the data to be detected that meets the preset hotspot conditions as hotspot data; push the hotspot data from each computing node to the corresponding service instance.

[0049] The present application actually considers that under a distributed service architecture, hotspot data detection faces multiple technical challenges. First, common solutions rely on external storage for access statistics, resulting in real-time and system coupling problems - cross-instance data aggregation requires frequent communication with external components, introducing not only millisecond-level delays but also increasing system vulnerability due to strong dependence on third-party services. Second, there is a problem of balanced load in the face of a large amount of data to be detected. Traditional centralized computing nodes are prone to becoming performance bottlenecks due to sudden traffic increases, and simple polling distribution strategies are difficult to ensure the efficient use of computing resources, further exacerbating response delays. In addition, although offline analysis based on all historical data can improve accuracy, it is difficult to meet the real-time processing requirements in high-concurrency scenarios. The interweaving of these problems makes it difficult for existing technologies to achieve lightweight, low-latency, and high-accuracy hotspot detection in a distributed architecture.

[0050] Based on this, the embodiments of the present application propose a solution: intercept the user access requests received by the service instance through the client SDK integrated in the service instance, and extract the data to be detected; establish a connection channel between the client SDK and each computing node of the computing component, and establish a mapping relationship table between the service name and the connection channel; distribute the data to be detected to each computing node of the computing component through the client SDK; listen to the rule configuration change data of the configuration center through the client SDK, and dynamically load and update the rule configuration; if the data to be detected hits the rule configuration and is not marked as hotspot data, then distribute the data to be detected to the computing component; perform hotspot determination on the data to be detected by each computing node, and mark the data to be detected that meets the preset hotspot conditions as hotspot data; push the hotspot data from each computing node to the corresponding service instance.

[0051] Specifically, the following are the detailed steps of the first exemplary embodiment of the hotspot data detection method of the present application:

[0052] Reference Figure 1 , Figure 1 is a flowchart of the first exemplary embodiment of the hot data detection method of this application. In this embodiment, the hot data detection method includes steps S10 to S40:

[0053] Step S10, intercept the user access request received by the service instance through the client SDK integrated in the service instance, and extract the data to be detected;

[0054] Specifically, the client SDK needs to be initialized when the service instance starts. During the initialization process, the SDK will obtain the necessary configuration information from the configuration center, including but not limited to rule configuration, TCP routing of computing components, etc. These configuration information will serve as the basis for the subsequent work of the SDK to ensure that it can correctly intercept and process user access requests.

[0055] When the service instance receives a user access request, the client SDK can intercept the request processing flow of the service instance through AOP (Aspect-Oriented Programming) technology, or directly call the relevant methods of the client SDK at the request processing entry of the service instance. Specifically, the client SDK implants monitoring logic at the entry of the HTTP request processing link to capture the metadata of the request in real time, including but not limited to the request path, parameters, timestamp, and associated business data identifier.

[0056] In the data extraction stage, the client SDK can screen out the data to be detected from the original request according to the predefined rule configuration. For example, in an e-commerce scenario, if it is necessary to detect hot products, the product ID field in the request is extracted; if it is necessary to detect high-frequency operations, the operation type and user identifier are extracted. The extracted data to be detected is encapsulated into a message body in a unified format, which can include data key (Key), timestamp, source instance identifier, and additional context information.

[0057] In addition, to ensure data integrity, the SDK attaches a CRC check code to the message body and records a data snapshot in the local log for fault tolerance and recovery.

[0058] Step S20, distribute the data to be detected to each computing node of the computing component through the client SDK;

[0059] Specifically, after the client SDK extracts the data to be detected, it determines the addresses of each computing node in the computing component according to the TCP routing information of the computing component provided by the global configuration center. These address information have been obtained and cached from the configuration center during the SDK initialization, ensuring that the SDK can quickly and accurately find each computing node of the computing component.

[0060] Furthermore, to achieve efficient data distribution, the client SDK adopts a load balancing strategy based on the hash algorithm. Specifically, the SDK performs a modulo operation based on the hash value of the data to be detected and the number of computing nodes in the computing component, so as to determine which computing node the data to be detected should be sent to. This hash-based load balancing strategy can not only ensure that the data to be detected is evenly distributed to each computing node, avoiding the problem of overloading a single computing node, but also improve the efficiency and stability of data distribution.

[0061] After determining the target computing node of the data to be detected, the client SDK can distribute the data to be detected to each computing node of the computing component through a pre-established connection channel.

[0062] It should be noted that, to further improve the reliability and stability of data distribution, the client SDK performs a series of exception handling and retry mechanisms when sending the data to be detected. For example, if a network exception or an unavailable computing node is detected during the data sending process, the client SDK will automatically attempt to re-establish the connection and re-send the data to be detected; if the data still cannot be successfully sent after multiple retries, the client SDK will cache the data to be detected and send it again after the network returns to normal or the computing node becomes available, ensuring that the data to be detected will not be lost due to network problems or computing node failures.

[0063] In a feasible implementation manner, before step S20, step S01 may be included:

[0064] Step S01: Generate a mapping relationship table between the service name and the connection channel based on the connection channels established between the client SDK and each computing node of the computing component, where the service name is the unique identifier of the cluster to which the service instance belongs.

[0065] In the service instance startup phase, the client SDK actively establishes connection channels with each computing node of the computing component as follows: The client SDK obtains a list of all available nodes of the computing component through the service discovery mechanism. For each computing node, the client SDK creates a two-way TCP long connection based on the Netty framework and encrypts the communication channel through the SSL / TLS protocol to ensure the security of data transmission. After the connection is established, the client SDK sends an initialization handshake message packet to the computing node, and the message packet carries the unique identifier (service name) of the cluster to which the service instance belongs and the instance ID for the computing node to perform service grouping and status tracking.

[0066] Furthermore, the successfully established connection channels are grouped and stored by service name to form a Map<service name,List <channel>>Mapping relationship table of the structure. Among them, the service name is used to uniquely identify the service cluster and its running environment.

[0067] In a feasible implementation manner, step S20 may include steps S21 to S23:

[0068] Step S21, perform a hash calculation on the key value of the data to be detected to generate a hash value;

[0069] During the data distribution process, the client SDK first performs a hash calculation on the key identification field, that is, the key value, of the data to be detected to determine its target route in the computing node cluster. The purpose of the hash calculation is to convert the key value into a hash value with a fixed length, so that the subsequent modulo operation can evenly distribute the data to be detected to each computing node.

[0070] In this embodiment, various hash algorithms can be used to achieve this purpose, such as MD5, SHA-1, or MurmurHash, etc.

[0071] It should be noted that during the process of generating the hash value, the client SDK will ensure the uniqueness of the key value of the data to be detected. If the key value of the data to be detected is itself a composite key, the client SDK will first splice or serialize these fields to ensure that the generated key value can accurately reflect the uniqueness of the data. In addition, the client SDK will also verify the generated hash value to ensure that it meets the processing requirements of the subsequent computing nodes. If the generated hash value does not meet the requirements, the client SDK will recalculate or perform necessary conversions to ensure the correctness and availability of the hash value.

[0072] Step S22, perform a modulo operation on the hash value and the total number of nodes of the computing node to obtain a target node index;

[0073] Furthermore, after generating the hash value, the client SDK can periodically obtain the total number of nodes of the computing component from the configuration center. The total number of nodes is the number of all computing nodes in the computing component and is used to determine the target node of the data to be detected. The client SDK will obtain the latest total number of nodes from the configuration center.

[0074] The client SDK performs a modulo operation on the generated hash value and the total number of nodes of the computing node to obtain a target node index. In this way, the client SDK can evenly map the hash value to each node of the computing component to ensure the load balance of each node.

[0075] For example, if there are 10 nodes in the computing component and the hash value is 12345, the target node index is 5, indicating that the data to be detected will be sent to the 5th computing node.

[0076] It should be noted that to ensure the accuracy of the modulo operation, the client SDK will verify the total number of nodes in the computing nodes. If the total number of nodes is 0 or not correctly obtained, the client SDK will trigger an exception handling mechanism, pause data distribution, and re-obtain the total number of nodes. After re-obtaining the total number of nodes, the client SDK will perform the modulo operation again to ensure the correctness of the target node index.

[0077] Step S23: According to the target node index, match the corresponding target connection channel from the mapping relation table, and send the data to be detected to the target computing node through the target connection channel.

[0078] Specifically, the client SDK obtains the list of all valid connection channels corresponding to the service from the pre-established mapping relation table based on the service name. According to the target node index value, locate the specific target connection channel from the channel list. For example, if the index value is 2 and the channel list is [connection channel_0, connection channel_1, connection channel_2], then select connection channel_2 as the target channel. To ensure load balancing, when multiple computing nodes are associated with the same service name, the client SDK adopts a polling strategy to select channels from the list in index order to avoid overloading a single node.

[0079] Please refer to Figure 2 , Figure 2 which is a schematic diagram of the mapping relationship between the service name and the connection channel in the embodiment of the present application, where Figure 2 it includes an instance module, a connection channel module, and a computing node module.

[0080] The instance module represents a cluster of service instances deployed in a distributed environment. For example, "Service A Instance 1" to "Service A Instance N" belong to the same service cluster. These instances may be distributed in different physical servers or containerized platforms and achieve cross-node communication through a virtual network. When each instance starts, it establishes an initial connection with the computing component by integrating the client SDK and continuously listens to the dynamic routing information of the configuration center. The design of the instance supports elastic scaling: when the service load surges, the system can quickly create new instances, and the new instances automatically join the cluster and complete connection initialization, realizing the ability of horizontal expansion.

[0081] Multiple "online" connection channels are mounted under each instance to form a multiplexing communication link between the service instance and the computing node.

[0082] In a feasible implementation, the instance establishes a TCP long connection with the computing node through the Netty framework, and each connection channel corresponds to an independent network session. To ensure high availability, a single Pod defaults to creating multiple connection channels, monitors the channel activity in real time, and automatically switches to the standby connection channel in case of a failure.

[0083] The computing node module consists of multiple computing nodes, and each node independently processes the data received from different connection channels. Specifically, after each computing node receives the data to be detected from multiple instances, it performs hash bucketing according to the Key value, aggregates the data with the same Key into the same memory window, and provides it for the sliding window algorithm to count the access frequency. For example, for a request with a product ID of "123", all relevant data is grouped into the same computing thread to ensure the atomicity of the statistical result. Further, the computing node regularly reports its own status to the configuration center, and the client SDK dynamically adjusts the weight of the connection channel according to the feedback information. For example, when the queue depth of a certain node exceeds the threshold, the client SDK automatically reduces the priority of its associated connection channel to avoid data accumulation.

[0084] When a computing node determines that a piece of data is a hot spot, it pushes the result back to the source service instance through the reverse connection channel. When pushing, the mapping relationship table between the service name and the connection channel is used to locate all associated instances to achieve cluster-level data consistency.

[0085] Step S30: Each of the computing nodes determines whether the data to be detected is a hot spot, and marks the data to be detected that meets the preset hot spot condition as hot spot data.

[0086] In a feasible implementation manner, step S30 may include steps S31 to S34:

[0087] Step S31: Obtain the algorithm object corresponding to the data to be detected, where the algorithm object includes a time window length, a preset access threshold, a window start time, and an access counter.

[0088] Specifically, when a computing node receives the data to be detected, it obtains the algorithm object corresponding to the data to be detected. The algorithm object can be stored in a hash table, with the Key value of the data to be detected as the key and the corresponding algorithm object instance as the value. To reduce lock contention, segment locks or CAS (Compare-and-Swap) operations can be used to achieve concurrent security. For example, when multiple threads access the same Key simultaneously, the counter is updated through AtomicInteger to avoid data competition. In addition, to cope with the risk of memory overflow, the computing node regularly clears the algorithm objects that have not been accessed for a long time and persists their status to the local disk for reconstruction during fault recovery.

[0089] The data structure of the algorithm object includes core fields such as a time window length, a preset access threshold, a window start time, and an access counter. The time window length can be obtained from the rule configuration loaded from the configuration center; the preset access threshold can be defined by business rules; the window start time records the start moment of the current time window, and the initial value is the system timestamp when the data first arrives; the access counter is used to count the number of accesses to this Key within the window.

[0090] Step S32: Calculate the time difference between the current system time and the start time of the window.

[0091] Specifically, after obtaining the algorithm object, the computing node can obtain the current timestamp through the system clock and calculate the difference from the start time of the window stored in the algorithm object.

[0092] To ensure time accuracy, the system clock needs to be synchronized with the NTP server, with the error controlled within 1 millisecond. For example, if the current time is 1620000000000 milliseconds and the start time of the window is 1619999999500 milliseconds, the time difference is 500 milliseconds. This difference is used to determine whether the current access is within the valid time window or whether the window parameters need to be reset.

[0093] For clusters deployed across time zones, all computing nodes adopt a unified UTC time benchmark to avoid incorrect window calculations due to time zone differences. The calculation process of the time difference is encapsulated as a stateless function to ensure the consistency of results in a distributed environment.

[0094] Step S33: When the time difference is not greater than the length of the time window, update the access counter and determine whether the updated access counter exceeds the preset access threshold.

[0095] Specifically, if the time difference is not greater than the length of the time window, it is determined that the current access is within the valid window. Update the operation access counter and compare the value of the updated access counter with the preset threshold. If the counter value exceeds the threshold, only record the current state; if it exceeds, trigger the hotspot marking process.

[0096] If the time difference is not greater than the length of the time window, it indicates that the current window has expired, and the window reset needs to be executed. Update the start time of the window to the current system time and reset the counter to zero.

[0097] Step S34: If it exceeds the preset access threshold, mark the data to be detected as hotspot data.

[0098] Specifically, when the access counter exceeds the preset threshold, the computing node can perform the following operations: Set the isHotspot flag in the algorithm object to true and add the key to the hotspot data queue.

[0099] Step S40: Push the hotspot data from each computing node to the corresponding service instance.

[0100] After the computing component completes the determination and marking of hotspot data, the hotspot data needs to be synchronized to all instances of the corresponding service to ensure data consistency within the cluster. The specific implementation process is as follows:

[0101] The computing node extracts the labeled keys and their associated parameters from the hot data queue and encapsulates them into a push message in a unified format. The message header contains the service name, which is carried by the client SDK of the instance where the data to be detected comes from during initial reporting. The service name serves as a globally unique identifier for locating the target connection channel in the mapping table. For example, if the hot data comes from the order service cluster, the service name is order-service.v1, and it needs to be pushed to all Pod instances under this cluster.

[0102] The computing component accesses the mapping table maintained in memory and retrieves the corresponding list of connection channels through the service name. For example, the service name order-service.v1 may be associated with three valid channels [connection channel_1, connection channel_2, connection channel_3], which are respectively connected to three instances of this service. To cope with dynamic scaling, the query process of the mapping table needs to incorporate the latest routing information to ensure that only the currently active channels are included in the list.

[0103] The computing node can use the asynchronous non-blocking interface of the Netty framework to send the hot data push message to all target channels in parallel.

[0104] After receiving the hot data, the service instance loads the hot data key and associated parameters into the local memory cache to prevent memory overflow.

[0105] Further, please refer to Figure 3 , Figure 3 which is an exemplary system architecture diagram for hot data detection in this application. The core components include a computing component, a configuration center, and a client SDK, and their interaction logic is as follows:

[0106] When starting up, the computing node connects to the configuration center and reports its TCP routing information. The configuration center stores this routing information and notifies the client SDK when the routing information changes. The client SDK connects to the configuration center when the service instance starts up to obtain the rule configuration and the TCP routing information of the computing node. The client SDK listens to the rule change and routing change data of the configuration center in real time to ensure that its processing logic is always based on the latest configuration. The client SDK establishes a long connection with the computing node. When the client SDK extracts the data to be detected, it will determine whether the data needs to be reported according to the rules. If reporting is required, the client SDK will determine the target computing node through the hash algorithm and modulo operation, and send the data to the target node through the long connection. The computing node receives the data to be detected from the client SDK and performs hot determination. When the computing node detects hot data, it will push the hot data to all relevant service instances through the corresponding connection channels according to the mapping table in the configuration center.

[0107] In summary, the above system realizes millisecond-level hot data detection through a design pattern of distributed data uploading, centralized data computing, and distributed data distribution. The computing nodes are responsible for data processing, the configuration center is responsible for storing and distributing configuration information, and the client SDK is responsible for data reporting and receiving hot data. This not only improves the performance and stability of the system but also avoids the complexity and overhead of introducing third-party storage.

[0108] Further, based on Figure 4 , Figure 4 FIG. is a flowchart of a second exemplary embodiment of the hot data detection method of the present application. In the second exemplary embodiment of the present application, the hot data detection method includes steps S50 to S60:

[0109] Step S50, updating the routing change data stored in the configuration center based on the self-routing information reported by each computing node of the computing component;

[0110] Step S60, listening to the routing change data of the configuration center through the client SDK, and adjusting the mapping relationship table between the service name and the connection channel according to the routing change data.

[0111] In a feasible implementation manner, after the computing component is started, each computing node reports its own routing information to the configuration center through a timing task. After receiving the reported data, the configuration center updates the routing table in the memory and persistent storage and marks the change version number. For abnormal nodes, the configuration center sets their status to "unavailable" and triggers a routing change event. The reported data is transmitted through the HTTP / RPC protocol after serialization to ensure cross-platform compatibility.

[0112] The client SDK listens to the routing change event of the configuration center through the long polling or Watch mechanism. When detecting a version number update, it pulls the latest routing table and compares the differences with the locally cached mapping relationship table: for newly added nodes, it establishes a Netty connection and adds them to the mapping table; for invalid nodes, it closes the corresponding connection channel and removes the record. The adjustment process adopts a double-buffer mechanism, first updating the copy and then atomically switching to avoid concurrent conflicts when distributing data.

[0113] Further, based on Figure 5 , Figure 5 FIG. is a flowchart of a third exemplary embodiment of the hot data detection method of the present application. In the third exemplary embodiment of the present application, steps S70 to S80 may be included before step S20:

[0114] Step S70, in response to the client SDK listening to the rule configuration change event of the configuration center, pulling the latest rule configuration associated with the event from the configuration center;

[0115] The client SDK listens for rule configuration change events in the configuration center in real time through long polling or event listening mechanisms. When the configuration center detects a rule update, it pushes a change notification to all subscribed client SDKs. After receiving the notification, the client SDK immediately initiates an HTTP / RPC request to the configuration center to pull the latest rule configuration file associated with the event. During the pulling process, the SDK verifies the integrity of the configuration file to ensure that the file has not been tampered with. When loading, a double-buffering mechanism is adopted: the new rules are parsed and temporarily stored in a standby memory area. After the parsing and syntax checking are completed, the currently effective rule cache is replaced through atomic operations to avoid logical conflicts during rule switching. At the same time, the SDK backs up the old version of the rules to the local disk and records the version number and update timestamp to support quick rollback or fault recovery.

[0116] Step S80, if the data to be detected hits the latest rule configuration and is not marked as hot data, then execute the steps of distributing the data to be detected to each computing node of the computing component through the client SDK.

[0117] After the client SDK extracts the data to be detected, it performs multi-level rule matching based on the latest rule configuration. If the data hits any rule, it further queries the local memory cache or the distributed lock service to confirm that the data is not marked as hot. If the conditions are met, the SDK encapsulates the data into a unified protocol packet and selects a target computing node according to the preset load balancing strategy.

[0118] Further, based on Figure 6 , Figure 6 FIG. is a schematic flowchart of the fourth exemplary embodiment of the hot data detection method of the present application. In the fourth exemplary embodiment of the present application, before step S31, step S90 may be included:

[0119] Step S90, if it is determined that the data to be detected is distributed for the first time, then create a corresponding algorithm object according to the key value of the data to be detected, and initialize the time window length and the preset access threshold of the algorithm object based on the rule configuration obtained from the configuration center.

[0120] Specifically, after the computing node receives the data to be detected, if it is determined that the Key value of the data has not been processed, i.e., it is the first distribution, a unique identifier is generated based on the Key value of the data. The algorithm object cache in the memory is queried, and after confirming that there is no existing object, the algorithm object creation process is triggered. The parameters matching the Key are extracted from the pre-loaded rule configuration to initialize the fields of the algorithm object: the window start time is set to the current system timestamp, the access counter is reset to zero, and the preset access threshold and time window length are dynamically loaded from the rule configuration. To ensure thread safety, a double-checked locking mechanism can be adopted to avoid concurrent creation of duplicate objects by multiple threads. After initialization, the algorithm object is stored in the cache, and a creation log is recorded in the local persistent storage for reconstruction during fault recovery. This process ensures that the first processing of each Key is based on the latest rule configuration, and the data structure is seamlessly connected to the subsequent sliding window calculation.

[0121] In addition, the present application also proposes a hot data detection device, which includes:

[0122] An extraction module 10, configured to intercept the user access requests received by the service instance through the client SDK integrated in the service instance, and extract the data to be detected;

[0123] A distribution module 20, configured to distribute the data to be detected to each computing node of the computing component through the client SDK;

[0124] A determination module 30, configured to perform hot determination on the data to be detected by each computing node, and mark the data to be detected that meets the preset hot conditions as hot data;

[0125] A push module 40, configured to push the hot data from each computing node to the corresponding service instance.

[0126] The hot data detection device provided by the present application adopts the hot data detection method in the above embodiment, aiming to solve the technical problem of how to detect hot data in real time and efficiently under a distributed service architecture. Compared with the prior art, the beneficial effects of the hot data detection device provided by the present application are the same as those of the hot data detection method provided by the above embodiment, and other technical features in the hot data detection device are the same as those disclosed in the method of the above embodiment, and will not be elaborated here.

[0127] The present application provides a hot data detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the hot data detection method in the first embodiment above.

[0128] The hot data detection device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description: tablet computers), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The illustrated hot data detection device is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0129] As Figure 8 shown, the hot data detection device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1002 or the program loaded from the storage device 1003 into the random access memory 1004. In the random access memory 1004, various programs and data required for the operation of the hot data detection device are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems may be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the hot data detection device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a hot data detection device with various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems may be implemented or had alternatively.

[0130] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0131] The hot data detection device provided in the present application adopts the hot data detection method in the above-mentioned embodiments, aiming to solve the technical problem of how to detect hot data in real time and efficiently under a distributed service architecture. Compared with the prior art, the beneficial effects of the hot data detection device provided in the present application are the same as those of the hot data detection method provided in the above-mentioned embodiments, and other technical features in the hot data detection device are the same as the features disclosed in the method of the previous embodiment, which will not be elaborated here.

[0132] It should be understood that each part disclosed in the present application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0133] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0134] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the hot data detection method in the above-mentioned embodiments.

[0135] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0136] The above computer-readable storage medium can be included in the hotspot data detection device; or it can exist separately and not be assembled into the hotspot data detection device.

[0137] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above programming languages include object-oriented programming languages - such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0139] The modules described in the embodiments of the present application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0140] The readable storage medium provided by the present application is a computer-readable storage medium, and the computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned hot data detection method, aiming to solve the technical problem of how to detect hot data in real time and efficiently under a distributed service architecture. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the hot data detection method provided by the above embodiments, and will not be elaborated here.

[0141] The present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the hot data detection method as described above.

[0142] The computer program product provided by the present application aims to solve the technical problem of how to detect hot data in real time and efficiently under a distributed service architecture. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the hot data detection method provided by the above embodiments, and will not be elaborated here.

[0143] Compared with the prior art, the hot data detection method, device, equipment, medium and computer product proposed in the embodiments of the present application extract the service feature information of the target service, perform data standardization processing on the service feature information to obtain standard feature data, perform hashing processing on the standard feature data to obtain unique feature data, perform numerical processing and splicing processing on the unique feature data to obtain the first service feature value, accumulate the first service feature values of the target service to obtain the target service feature value, and finally compare the target service feature value with the feature value set to obtain the hot data detection result. It is more efficient, flexible and reliable than the traditional method of generating a unique key value or a continuous serial number for each service to identify duplicate services. Based on the solution of the present application, by transforming the services in complex scenarios through a series of simple transformations, it is finally transformed into a comparison of two numbers, making the comparison process very intuitive and efficient. The system only needs to simply compare whether these two values are equal to quickly determine whether two services are exactly the same.

[0144] It should be noted that in this article, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or system including that element.

[0145] The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages and disadvantages of the embodiments.

[0146] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of the present application.

[0147] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.< / channel>

Claims

1. A method for detecting hot data, characterized in that, The described hot data detection method includes: Intercepting the user access requests received by the service instance through the Client Software Development Kit (SDK) integrated in the service instance, and extracting the data to be detected; Distributing the data to be detected to each computing node of the computing component through the client SDK; Performing hot determination on the data to be detected by each computing node, and marking the data to be detected that meets the preset hot conditions as hot data; Pushing the hot data from each computing node to the corresponding service instance.

2. The hot data detection method according to claim 1, wherein Before the step of distributing the data to be detected to each computing node of the computing component through the client SDK, it includes: Generating a mapping relationship table between the service name and the connection channel based on the connection channels established between the client SDK and each computing node of the computing component, where the service name is the unique identifier of the cluster to which the service instance belongs.

3. The hot data detection method according to claim 2, characterized in that After the step of generating a mapping relationship table between the service name and the connection channel based on the connection channels established between the client SDK and each computing node of the computing component, it includes: Updating the routing change data stored in the configuration center based on the self-routing information reported by each computing node of the computing component; Listening to the routing change data of the configuration center through the client SDK, and adjusting the mapping relationship table between the service name and the connection channel according to the routing change data.

4. The hot data detection method according to claim 1, wherein Before the step of distributing the data to be detected to each computing node of the computing component through the client SDK, it also includes: In response to the client SDK listening to the rule configuration change event of the configuration center, pulling the latest rule configuration associated with the event from the configuration center; If the data to be detected hits the latest rule configuration and is not marked as hot data, execute the step: distributing the data to be detected to each computing node of the computing component through the client SDK.

5. The hot data detection method according to claim 1, characterized in that The step of distributing the data to be detected to each computing node of the computing component through the client SDK includes: Performing a hash calculation on the key value of the data to be detected to generate a hash value; Performing a modulo operation on the hash value and the total number of computing nodes to obtain the target node index; According to the target node index, matching the corresponding target connection channel from the mapping relationship table, and sending the data to be detected to the target computing node through the target connection channel.

6. The hot data detection method according to claim 1, characterized in that The step of performing hot determination on the data to be detected by each computing node and marking the data to be detected that meets the preset hot conditions as hot data includes: Obtaining an algorithm object corresponding to the data to be detected, where the algorithm object includes the time window length, the preset access threshold, the window start time, and the access counter; Calculating the time difference between the current system time and the window start time; When the time difference is not greater than the time window length, updating the access counter and determining whether the updated access counter exceeds the preset access threshold; If it exceeds the preset access threshold, marking the data to be detected as hot data.

7. The hot data detection method according to claim 6, wherein Before the step of obtaining the algorithm object corresponding to the data to be detected, the following steps are included: If it is determined that the data to be detected is distributed for the first time, an algorithm object corresponding to the data to be detected is created according to the key value of the data to be detected, and the time window length and preset access threshold of the algorithm object are initialized based on the rule configuration obtained from the configuration center.

8. A hot data detection device, characterized in that, The device includes: An extraction module, configured to intercept a user access request received by the service instance through a client SDK integrated in the service instance, and extract data to be detected; A distribution module, configured to distribute the data to be detected to each computing node of the computing component through the client SDK; A determination module, configured to perform hot spot determination on the data to be detected by each of the computing nodes, and mark the data to be detected that meets the preset hot spot condition as hot spot data; A push module, configured to push the hot spot data from each of the computing nodes to the corresponding service instance.

9. A hot data detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the computer program is configured to implement the steps of the hot spot data detection method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the hot spot data detection method according to any one of claims 1 to 7 are implemented.

Images