Cluster processing method and device, nonvolatile storage medium and computer equipment

Abstract

The invention discloses a cluster processing method and device, a nonvolatile storage medium and computer equipment. The method comprises the following steps: setting a plurality of clusters; based on a preset service scene demand, the plurality of clusters are divided into a plurality of cluster groups corresponding to service scenes, and the service scenes comprise consumer-oriented service scenes and enterprise-oriented service scenes; based on a plurality of preset data fragments, the plurality of cluster groups corresponding to the service scenes are divided into cluster subgroups corresponding to the plurality of data fragments respectively, and the cluster subgroups are used for caching data of the corresponding data fragments in the corresponding service scenes; and setting respective corresponding identifiers for the cluster subgroups corresponding to the plurality of data fragments, and determining a plurality of target cluster subgroups. According to the method and the device, the technical problems that the current local cache or central cache cluster is difficult to support the service requirement and the resource utilization is unbalanced due to unreasonable cache allocation are solved.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a cluster processing method, apparatus, non-volatile storage medium, and computer equipment. Background Art

[0002] Currently, there are two types of caches: local cache and central cache (cluster / sentinel).

[0003] Local caching: When a request is made to the application, the database is queried for the requested value, and the request keyword and result are stored as key-value pairs in a map. Subsequent requests will first check whether the requested keyword exists in the map. If so, the value in the map is directly restored. This solution has very low scalability and availability, wastes application memory, and the cache disappears after restart. There is no way to set an expiration time for the cache.

[0004] Central Cache (Cluster / Sentinel): When a request comes into the application, the cache is checked for the requested keyword. If it exists, the result is returned directly. If not, the database is queried, the result is stored in the cache, and the query result is returned. Disaster recovery is provided through the Sentinel / Cluster mechanism. However, if the CPU of the cluster / MASTER node is too high, there is still a risk of downtime, causing the entire business to crash and potentially triggering various production failures.

[0005] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0006] Embodiments of the present invention provide a cluster processing method, apparatus, non-volatile storage medium, and computer device to at least address the technical issues that current local cache or central cache clusters are unable to support business needs and that unreasonable cache allocation leads to unbalanced resource utilization.

[0007] According to one aspect of an embodiment of the present invention, a cluster processing method is provided, including: setting up multiple clusters; based on preset business scenario requirements, dividing the multiple clusters into multiple cluster groups corresponding to the business scenarios, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios; based on preset multiple data fragments, dividing the multiple cluster groups corresponding to the business scenarios into cluster sub-groups corresponding to multiple data fragments, wherein the cluster sub-groups are used to cache data of corresponding data fragments in corresponding business scenarios; setting respective corresponding identifiers for the cluster sub-groups corresponding to the multiple data fragments, and determining multiple target cluster sub-groups.

[0008] Optionally, new data is received; a business scenario and data segment corresponding to the new data are determined; based on the business scenario and data segment corresponding to the new data, a first target cluster subgroup corresponding to the new data is determined; based on an identifier corresponding to the first target cluster subgroup, the new data is cached to the first target cluster subgroup.

[0009] Optionally, it is detected whether there is a faulty cluster in the target cluster subgroup among the multiple target cluster subgroups; if there is a faulty cluster in the target cluster subgroup, the faulty cluster is removed from the corresponding target cluster subgroup.

[0010] Optionally, in the case that all clusters included in the second target cluster subgroup are faulty clusters, the second target cluster subgroup is deleted, and data segments corresponding to other cluster subgroups in the cluster group corresponding to the second target cluster subgroup are modified.

[0011] Optionally, detecting whether there is a faulty cluster in the target cluster subgroups among the multiple target cluster subgroups includes: sending detection information to the clusters in the multiple target cluster subgroups; detecting the time it takes for the clusters in the multiple target cluster subgroups to return a response message after receiving the detection information; and detecting whether there is a faulty cluster in the cluster subgroups among the multiple target cluster subgroups based on the relationship between the time it takes to return the response message and a preset threshold.

[0012] Optionally, before dividing multiple cluster groups corresponding to business scenarios into cluster subgroups corresponding to multiple data segments based on preset multiple data segments, it includes: determining the data range based on a preset hash function; and dividing the data range into multiple data segments according to a preset segmentation accuracy.

[0013] According to another aspect of an embodiment of the present invention, a cluster processing device is also provided, including: a setting module for setting multiple clusters; a grouping module for dividing the multiple clusters into multiple cluster groups corresponding to the business scenarios based on preset business scenario requirements, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios; a partitioning module for dividing the multiple cluster groups corresponding to the business scenarios into cluster subgroups corresponding to the multiple data fragments based on preset multiple data fragments, wherein the cluster subgroups are used to cache data of the corresponding data fragments in the corresponding business scenarios; a determination module for setting corresponding identifiers for the cluster subgroups corresponding to the multiple data fragments, and determining multiple target cluster subgroups.

[0014] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above cluster processing methods.

[0015] According to yet another aspect of an embodiment of the present invention, a computer device is provided. The computer device includes a processor, and the processor is configured to run a program. When the program is run, any one of the above cluster processing methods is executed.

[0016] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, any one of the above cluster processing methods is implemented.

[0017] In an embodiment of the present invention, a cluster processing method is adopted, by setting up multiple clusters; based on preset business scenario requirements, the multiple clusters are divided into multiple cluster groups corresponding to the business scenarios, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios; based on preset multiple data fragments, the multiple cluster groups corresponding to the business scenarios are divided into cluster sub-groups corresponding to multiple data fragments, wherein the cluster sub-groups are used to cache data of corresponding data fragments in corresponding business scenarios; corresponding identifiers are set for the cluster sub-groups corresponding to the multiple data fragments, and multiple target cluster sub-groups are determined, thereby achieving the purpose of allocating clusters according to business needs, thereby realizing the technical effect of improving the utilization rate of cache resources, and further solving the technical problem that the current local cache or central cache cluster is difficult to support business needs and the unreasonable cache allocation leads to uneven resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 A hardware structure block diagram of a computer terminal for implementing a cluster processing method is shown;

[0020] Figure 2 is a flow chart of a cluster processing method provided according to an embodiment of the present invention;

[0021] Figure 3 is a flowchart of a cluster processing method provided according to an optional embodiment of the present invention;

[0022] Figure 4 is a structural block diagram of a cluster processing device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to an embodiment of the present invention, a method embodiment of a cluster processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a cluster processing method. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1Different configurations shown.

[0027] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0028] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the cluster processing method in the embodiment of the present invention. The processor executes the software programs and modules stored in the memory 104 to execute various functional applications and data processing, that is, to implement the cluster processing method of the above-mentioned application. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0030] Figure 2 FIG. 1 is a flow chart of a cluster processing method according to an embodiment of the present invention. Figure 2 As shown, the method includes the following steps:

[0031] Step S202: Set up multiple clusters.

[0032] In this step, you set up multiple cache clusters (such as Redis clusters). You can first analyze your business needs and identify which applications or services require high concurrent processing capabilities and which require high data security and disaster recovery capabilities. This will help you decide the size, configuration, and grouping of the clusters. Create multiple clusters and select appropriate hardware resources, such as high-memory servers, SSD storage, etc., based on the performance requirements of the cluster to ensure the responsiveness and stability of the cluster. Configure cluster parameters such as the maximum number of connections, memory limits, data persistence strategies, etc. Enable master-slave replication to improve read performance and disaster recovery capabilities, and consider using Sentinel or high-availability cluster mode (such as Redis Cluster) to achieve automatic failover.

[0033] Step S204 : Based on preset business scenario requirements, the multiple clusters are divided into multiple cluster groups corresponding to the business scenarios, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios.

[0034] In this step, based on the preset business scenario requirements, multiple clusters are divided into multiple cluster groups corresponding to the business scenarios. The business scenarios include consumer-oriented business scenarios (TOC, Target Oriented Consumer) and enterprise-oriented business scenarios (TOB, Target Oriented Business):

[0035] TOC cluster group (consumer-oriented): First, identify consumer-oriented business scenarios, such as online shopping, flash sales, ticket grabbing, etc. These scenarios are usually characterized by high concurrency and large instantaneous traffic. The design of the TOC cluster group should focus on providing fast read and write performance, efficient data processing capabilities, and fast fault recovery mechanisms. Use high-performance hardware, optimize data structures and algorithms to ensure response speed and stability under high load. TOB cluster group (enterprise-oriented): Enterprise-oriented business scenarios may involve more complex data processing, data analysis, data security, and data persistence requirements. When designing the TOB cluster group, data integrity, transaction processing capabilities, and long-term stability should be considered, and hardware configurations suitable for long-term operation and data protection should be used. The number of nodes, hardware specifications, and software parameters of the TOC and TOB cluster groups can be configured according to business needs. Ensure that each cluster group can operate independently and provide stable services.

[0036] The above steps can better adapt to the needs of different business scenarios, provide customized caching services, and ensure the high availability and disaster recovery capabilities of the system, while reducing operation and maintenance costs and failure risks.

[0037] Step S206 : Based on the preset multiple data segments, the multiple cluster groups corresponding to the business scenarios are divided into multiple cluster subgroups corresponding to the data segments, wherein the cluster subgroups are used to cache data of the corresponding data segments in the corresponding business scenarios.

[0038] In this step, multiple cluster groups corresponding to business scenarios are subdivided into cluster subgroups corresponding to multiple data segments. This ensures even data distribution and enables more flexible response to the needs of different business scenarios. The data access patterns, traffic characteristics, and data storage requirements of each business scenario can be analyzed in detail. Determine which data segments (subsets of the dataset) are frequently accessed by specific business scenarios. Evaluate the size, access frequency, and access patterns of each data segment to determine their caching needs. Based on the business scenario and data access characteristics, the dataset is divided into multiple data segments. Each segment can be defined based on a specific key value range, data type, or business logic. Create one or more cluster subgroups for each data segment. These subgroups will be dedicated to caching data related to that data segment. Configure the hardware resources of the cluster subgroups, such as the number of servers, memory size, CPU power, etc., to meet the storage and access requirements of the data segments. Implement hash segmentation of the data segments to ensure that each data segment is assigned to the corresponding cluster subgroup.

[0039] Through the above steps, multiple cluster groups corresponding to business scenarios can be refined into multiple cluster subgroups based on pre-defined data fragments. Each subgroup focuses on caching and processing data fragments corresponding to specific business scenarios. This strategy can significantly improve the distributed performance of the cache system, strengthen disaster recovery mechanisms, and ensure system stability and responsiveness through intelligent scheduling and flow control strategies.

[0040] Step S208 : setting respective identifiers for the cluster subgroups corresponding to the plurality of data segments, and determining a plurality of target cluster subgroups.

[0041] In this step, identifiers are assigned to the cluster subgroups corresponding to each of the multiple data segments, and multiple target cluster subgroups are identified. This ensures efficient data storage and access, while also providing improved disaster recovery and resource utilization. Data segment identifiers can be defined. Each data segment has its own specific business logic and access patterns, so an identifier is required to distinguish different data segments. This identifier can be a data prefix, a portion of a hash value, or a unique ID generated by the business logic. The identifier should be designed to be easy to use in code and system configuration and reflect the characteristics of the data segment, such as data type and business scenario. Cluster subgroups are created. Based on business requirements and data access characteristics, multiple cluster subgroups can be created, each dedicated to storing and processing one or more data segments. Cluster subgroup configuration should fully consider data size, access frequency, and disaster recovery requirements, and appropriately allocate the number of nodes, memory capacity, and CPU resources to meet the storage requirements of the data segments. Next, identifiers are mapped to cluster subgroups. A mapping table or database can be established to associate data segment identifiers with corresponding cluster subgroups. This mapping table serves as the basis for the addressing mechanism, helping the system quickly locate the correct cluster subgroup based on the identifier. For example, a mapping rule can be designed so that the identifier "User" corresponds to the 1-20 segment in the TOC cluster group, and the identifier "Inventory" corresponds to the 81-100 segment in the TOB cluster group. Then, the addressing algorithm can be developed or optimized to ensure that requests can be correctly routed to specific cluster subgroups based on the data segment identifier. This usually involves implementing logic at the client or middleware layer, which may use consistent hashing, range queries, or other custom addressing strategies. The addressing algorithm should take into account the dynamic adjustment of cluster subgroups (such as expansion, reduction, reallocation, etc.) to ensure that the addressing remains accurate even when the cluster changes.

[0042] Through the above steps, identifiers are set for the cluster subgroups corresponding to multiple data fragments, and the target cluster subgroup is determined, achieving efficient data storage and access while improving the system's disaster recovery capabilities and resource utilization efficiency. This achieves the goal of allocating clusters based on business needs, thereby achieving the technical effect of improving cache resource utilization. This further solves the technical problem that current local cache or central cache clusters cannot meet business needs and that improper cache allocation leads to uneven resource utilization.

[0043] As an optional embodiment, new data is received; the business scenario and data fragment corresponding to the new data are determined; based on the business scenario and data fragment corresponding to the new data, the first target cluster subgroup corresponding to the new data is determined; based on the identifier corresponding to the first target cluster subgroup, the new data is cached to the first target cluster subgroup.

[0044] Optionally, receiving new data and efficiently caching it in the appropriate cluster subgroup is a key operation to ensure the caching system can support complex business scenarios and high-concurrency requests. When the system receives a new data request, whether from a user operation, a backend service call, or an external API interface, it must first parse the data in the request to understand its content and structure. This may involve parsing the HTTP request body, messages in a message queue, or data from database query results. Based on the parsed data content, the business scenario to which it belongs is determined. Each business scenario may correspond to one or more data segments. For example, user information may be one data segment, while product inventory may be another. This step typically requires support from business logic and may involve examining data fields, data types, or specific data identifiers. Based on the business scenario and data segment to which the new data belongs, a mapping table or database is queried to determine the mapping between identifiers and cluster subgroups to determine the cluster subgroup to which the new data should be cached. This cluster subgroup becomes the "first target cluster subgroup" for the new data. After determining the first target cluster subgroup, caching operations are performed based on its identifier. For example, a key containing the cluster subgroup identifier can be generated based on the business scenario and data segment of the new data. The key generation may be based on the ID, name, hash value, or other business-related attributes of the data segment. The generated key and addressing algorithm (such as consistent hashing) are then used to determine the specific target cluster node (i.e., one or more nodes in the first target cluster subgroup). The new data is stored in the determined target cluster node. This usually includes setting the cache key and corresponding value, and may also require defining the cache expiration time, read and write policy, etc.

[0045] As an optional embodiment, it is detected whether there is a faulty cluster in the target cluster subgroup among the multiple target cluster subgroups; if there is a faulty cluster in the target cluster subgroup, the faulty cluster is removed from the corresponding target cluster subgroup.

[0046] Optionally, a health check and performance monitoring mechanism can be integrated into the cache system to continuously monitor the operational status of each target cluster subgroup, including but not limited to key performance indicators such as node response time, CPU usage, memory consumption, and network latency. This can be achieved using off-the-shelf monitoring tools such as Prometheus and Grafana, or custom monitoring scripts. Based on this monitoring data, abnormal cluster node conditions can be identified. For example, if a node's response time is significantly higher than average, or if its CPU usage exceeds a threshold for a prolonged period, it can be preliminarily determined to be a node failure. Fault detection should be real-time, enabling rapid response to performance anomalies. After initially identifying a faulty node, more detailed diagnosis can be performed to confirm whether the node is truly faulty. This may involve sending heartbeat probes, performing simple read and write tests, or calling the node's health check endpoint. The fault confirmation mechanism should be able to accurately distinguish between temporary node performance fluctuations and true fault conditions. Once a faulty node is identified in the cluster, it should be immediately removed from the corresponding target cluster subgroup. Specifically, the cluster configuration can be modified to remove the faulty node from the cluster subgroup's node list. Routing and addressing policies can then be updated to prevent new requests from being routed to the faulty node. After removing the failed cluster, adjust and optimize resources as needed. For example, expand the capacity of a cluster subgroup by adding new nodes to compensate for the reduced capacity after removing the failed node. Adjust the data sharding strategy to redistribute data fragments to healthy nodes to ensure even data distribution. Optimize the configuration of the cluster subgroup, such as increasing memory and adjusting CPU limits, to accommodate the new load requirements.

[0047] Through continuous monitoring, real-time detection, fault confirmation, elimination and recovery mechanisms, the node status in the target cluster subgroup can be effectively managed to ensure that the system can maintain stable operation in high-concurrency scenarios. Even in the event of failure of individual nodes, measures can be taken quickly to maintain business continuity and service quality.

[0048] As an optional embodiment, when all clusters included in the second target cluster subgroup are faulty clusters, the second target cluster subgroup is deleted, and data segments corresponding to other cluster subgroups in the cluster group corresponding to the second target cluster subgroup are modified.

[0049] Optionally, if all cluster nodes in the second target cluster subgroup are detected as faulty, measures are taken to delete the entire second target cluster subgroup and reallocate the data segments it was originally responsible for to other healthy cluster subgroups within the cluster group. Specifically, the health status of all cluster nodes in the second target cluster subgroup is continuously monitored. Once all nodes are detected to be faulty or experiencing performance anomalies, and after failure confirmation, it is determined that these cluster nodes cannot be restored to a healthy state in a short period of time, the entire second target cluster subgroup is confirmed to have failed. The second target cluster subgroup can be formally deleted from the cluster group. Specifically, the cluster configuration can be updated to remove information about all nodes in the second target cluster subgroup, and the system architecture diagram can be modified to ensure that other components and systems no longer send requests to the deleted second target cluster subgroup. Data segments are then reallocated. Data segments originally processed by the second target cluster subgroup need to be reallocated. The access patterns and storage requirements of the data segments can be analyzed to determine which cluster subgroups are most suitable for taking over these data segments. The mapping table or database between data segment identifiers and cluster subgroups is then updated to redirect the affected data segments to other healthy cluster subgroups within the cluster group. Data migration can also be performed. The data in the second target cluster subgroup is copied or migrated to the new target cluster subgroup to ensure data integrity and consistency. This effectively handles the failure of an entire cluster subgroup, maintaining high system availability and continuous data access, while optimizing resource allocation and ensuring system performance and stability.

[0050] As an optional embodiment, detecting whether there is a faulty cluster in a target cluster subgroup among multiple target cluster subgroups includes: sending detection information to clusters in the multiple target cluster subgroups; detecting the time it takes for the clusters in the multiple target cluster subgroups to return response information after receiving the detection information; and detecting whether there is a faulty cluster in the cluster subgroups among the multiple target cluster subgroups based on the relationship between the time it takes to return the response information and a preset threshold.

[0051] Optionally, to detect faulty clusters in multiple target cluster subgroups, you can send a test message and determine cluster health based on response time. You can define a test message, for example, by creating a packet to detect cluster health. This packet might contain a simple command or query, such as a ping, or a short read or write operation. Ensure that the test message design minimizes the impact on cluster performance and avoids introducing additional load during the test process. Then, send the test message to clusters in multiple target cluster subgroups, ensuring even distribution of the test message and preventing single cluster nodes from being overloaded by concentrated test message transmission. Record the time between sending the test message and receiving the cluster response. Define a reasonable response time threshold to use as a criterion for determining cluster node health. The threshold should be based on historical data and business requirements. Typically, the threshold should be within the normal response time range, but slightly above the average, to avoid misjudging cluster failures due to brief performance fluctuations. Compare the response time of each cluster node against a preset threshold. If the response time exceeds the threshold, preliminarily determine that the cluster node may be faulty. If the response time of all nodes is normal, the entire cluster subgroup is considered healthy. If the response time of some nodes exceeds the threshold, further analysis is required, potentially indicating a partial failure or network issue. For nodes initially identified as faulty, further diagnosis is performed to confirm the cause. Once a node is confirmed to be faulty, it is immediately isolated from the cluster subgroup to prevent it from receiving further requests, and the operations team is notified to address the problem.

[0052] If multiple nodes in a cluster subgroup fail, impacting the availability of the entire subgroup, the disaster recovery plan must be immediately activated to reroute data requests to other healthy cluster subgroups within the cluster group. Simultaneously, consider whether to adjust resources or reconfigure the failed subgroup's services to restore its capabilities. The results of each health check can be recorded, including information such as the time the check information was sent, response time, and the ID of the failed node. Regularly analyzing this data can identify potential performance bottlenecks or failure modes, providing a basis for system optimization and failure prediction.

[0053] As an optional embodiment, before dividing multiple cluster groups corresponding to business scenarios into cluster subgroups corresponding to multiple data fragments based on preset multiple data fragments, it includes: determining the data range based on a preset hash function; and dividing the data range into multiple data fragments according to a preset segmentation accuracy.

[0054] Optionally, before dividing multiple cluster groups corresponding to business scenarios into cluster subgroups corresponding to multiple data segments, a preset hash function is used to determine the data range, and the data range is further divided into multiple data segments based on a preset segmentation precision. A hash function can be selected or designed to map data identifiers (such as keys, IDs, or names) to a fixed range of values. The selection of a hash function should take into account its uniform distribution, collision rate, and performance. For example, SHA-256, MD5, or a consistent hashing algorithm can be selected as a preset hash function. The range of the entire data space is then determined. This may involve analyzing the characteristics and scale of the business data to determine a reasonable numerical range. The segmentation precision, that is, the size of the data range that each data segment should cover, is then set based on business requirements and cluster scale. The segmentation precision should balance data uniformity and load balancing, while also considering the processing power and disaster recovery requirements of the cluster nodes. The data identifier is mapped to the data range using a preset hash function, and the data range is then divided into multiple contiguous data segments based on the set segmentation precision. For example, the TOC cluster group uses numbers 1-80 for segmentation, and the TOB cluster group uses 81-100 for segmentation. The segmentation can be divided into more detailed sections based on the actual business, such as: in the 1-80 segment, the 1-20, 21-40, 41-60, and 61-80 segments store different clusters. A mapping relationship can also be established between data fragments and cluster subgroups to ensure that each data fragment can be correctly routed to the corresponding cluster subgroup. Based on the preset hash function and segmentation accuracy, multiple cluster groups corresponding to business scenarios are divided into cluster subgroups corresponding to multiple data fragments, which can achieve refined data management and load balancing in high-concurrency scenarios.

[0055] Here is a specific example: Figure 3 is a flow chart of a cluster processing method according to an optional embodiment of the present invention, such as Figure 3 As shown:

[0056] Step 1: Set up multiple clusters and allocate nodes to the clusters based on demand.

[0057] Step 2: Cluster grouping can be divided into high-performance group and management group. The high-performance group handles TOC traffic and the management group handles TOB traffic. It can be divided more finely according to the actual needs of the enterprise. The purpose of cluster grouping is to use resources reasonably. For example, if the daily TPS of application A, B, and C interfaces is less than 100 or even single digits, then it is necessary to let it use the TOB traffic group. If application D and E handle high-traffic interfaces such as user-side flash sales, lotteries, and browsing for a long time, use the TOC cluster group.

[0058] Step 3: Constrain the total number of segments for cluster groups. For example, the TOC cluster group uses numbers 1-80 for segmentation, and the TOB cluster group uses numbers 81-100 for segmentation. Segments can be divided more finely based on actual business. For example, in the 1-80 segment, the 1-20, 21-40, 41-60, and 61-80 segments each store different clusters.

[0059] Step 4: Sort out the more detailed internal enterprise application traffic distribution and the actual business object-oriented situation, and also divide it into TOC and TOB groups. TOC applications use the REDIS cluster group of the TOC cluster, and TOB uses the TOB cluster. Each resource uses the resources of its own cluster as much as possible to decouple the business model.

[0060] Step 5: Establish an AOP mechanism as a section and use cache in the form of annotations. In AOP, the machine IP to be stored in the cache needs to be processed. The user uses (@Cache(Key=ID)User user) in the method parameter to indicate that the cache needs to be performed by ID. At this time, AOP obtains the value of the ID attribute in user through reflection, and performs a HASH 100-degree shard calculation on it to obtain the shard value through calculation.

[0061] Step 6: Establish the REDIS addressing mechanism. Use the sharding results calculated based on the current application requirements to find the cluster machines within the cluster group belonging to this segment and store the values ​​in the cluster group.

[0062] Step 7: Use the REDIS heartbeat mechanism for link liveness detection. If a cluster within a cluster group fails to detect a heartbeat or is clearly down, quickly remove the faulty cluster from the current cluster group. If an entire cluster group segment fails, such as segments 1-20, immediately remove segments 1-20 from the 1-80 range of the TOC cluster group. Temporarily change the 21-40 range to 1-40. This way, even after a cluster failure, segments 21-40 of the original cluster group will temporarily receive write requests from segments 1-20, preventing the entire business from collapsing.

[0063] Step 8: Build a direct bridge between TOC and TOB, and establish a communication channel between TOC application and TOB application from the perspective of technical middle platform. Bridge testing needs to implement multi-cache data source and do a good job of flow control. In order to prevent the bridge from being unable to withstand a large number of requests and causing the bridge to collapse, a one-way bridge should be set up. The traffic of TOC->TOB should be independent of the channel of TOB->TOC to share the traffic. Application testing can determine whether to connect to the bridge based on actual needs. The purpose of establishing a communication bridge is to prevent some data with fuzzy boundaries in the BC connection scenario from interacting on both ends of BC, such as inventory, sales, orders, etc., which are required by both BC business types. A one-way bridge can avoid the BC communication traffic from being too large to overwhelm the bridge. Bridge testing can also design traffic transfer according to the actual business type, such as: Application A calls the bridge to pass ORDER, and the bridge can send traffic to the exclusive ORDER channel.

[0064] Step 9: Establish a good monitoring system. Detect whether the heartbeat mechanism and CPU are functioning properly and issue alerts to prevent a cluster group in the production environment from crashing. Prepare the contingency plan for point 7 in advance through alerts.

[0065] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0066] Through the description of the above embodiments, those skilled in the art can clearly understand that the cluster processing method according to the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0067] According to an embodiment of the present invention, a cluster processing device for implementing the above cluster processing method is also provided. Figure 4 is a structural block diagram of a cluster processing device provided according to an embodiment of the present invention. Figure 4 As shown, the cluster processing device includes: a setting module 42, a grouping module 44, a division module 46 and a determination module 48. The cluster processing device is described below.

[0068] The setting module 42 is used to set up multiple clusters.

[0069] The grouping module 44 is connected to the setting module 42 and is used to divide the multiple clusters into multiple cluster groups corresponding to the business scenarios based on preset business scenario requirements, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios.

[0070] The partitioning module 46 is connected to the grouping module 44 and is used to divide the multiple cluster groups corresponding to the business scenarios into cluster subgroups corresponding to the multiple data segments based on the preset multiple data segments, wherein the cluster subgroups are used to cache the data of the corresponding data segments in the corresponding business scenarios.

[0071] The determination module 48 is connected to the division module 46 and is configured to set corresponding identifiers for the cluster subgroups corresponding to the plurality of data segments, and determine a plurality of target cluster subgroups.

[0072] It should be noted that the setting module 42, grouping module 44, partitioning module 46, and determination module 48 described above correspond to steps S202 to S208 in the embodiment. The examples and application scenarios implemented by these modules and corresponding steps are the same, but are not limited to those disclosed in the embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0073] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.

[0074] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the cluster processing method and device in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the above-mentioned cluster processing method. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0075] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: setting multiple clusters; based on preset business scenario requirements, dividing the multiple clusters into multiple cluster groups corresponding to the business scenarios, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios; based on preset multiple data fragments, dividing the multiple cluster groups corresponding to the business scenarios into cluster sub-groups corresponding to multiple data fragments, wherein the cluster sub-groups are used to cache data of the corresponding data fragments in the corresponding business scenarios; setting corresponding identifiers for the cluster sub-groups corresponding to the multiple data fragments, and determining multiple target cluster sub-groups.

[0076] Optionally, the above-mentioned processor can also execute the program code of the following steps: receiving new data; determining the business scenario and data fragment corresponding to the new data; determining the first target cluster subgroup corresponding to the new data based on the business scenario and data fragment corresponding to the new data; and caching the new data to the first target cluster subgroup based on the identifier corresponding to the first target cluster subgroup.

[0077] Optionally, the processor may further execute program code of the following steps: detecting whether there is a faulty cluster in the target cluster subgroups; and if there is a faulty cluster in the target cluster subgroup, removing the faulty cluster from the corresponding target cluster subgroup.

[0078] Optionally, the processor may further execute program code of the following steps: if all clusters included in the second target cluster subgroup are faulty clusters, delete the second target cluster subgroup and modify data segments corresponding to other cluster subgroups in the cluster group corresponding to the second target cluster subgroup.

[0079] Optionally, the above-mentioned processor can also execute the program code of the following steps: detecting whether there is a faulty cluster in the target cluster subgroups among multiple target cluster subgroups, including: sending detection information to the clusters in the multiple target cluster subgroups; detecting the time it takes for the clusters in the multiple target cluster subgroups to return a response information after receiving the detection information; and detecting whether there is a faulty cluster in the cluster subgroups among the multiple target cluster subgroups based on the relationship between the time it takes to return the response information and a preset threshold.

[0080] Optionally, the processor may also execute program code for the following steps: before dividing multiple cluster groups corresponding to business scenarios into cluster subgroups corresponding to multiple data segments based on a preset plurality of data segments, the steps include: determining a data range based on a preset hash function; and dividing the data range into multiple data segments according to a preset segmentation accuracy.

[0081] An embodiment of the present invention provides a cluster processing method, which is achieved by setting up multiple clusters; based on preset business scenario requirements, dividing the multiple clusters into multiple cluster groups corresponding to the business scenarios, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios; based on preset multiple data fragments, dividing the multiple cluster groups corresponding to the business scenarios into cluster sub-groups corresponding to multiple data fragments, wherein the cluster sub-groups are used to cache data of corresponding data fragments in corresponding business scenarios; setting respective corresponding identifiers for the cluster sub-groups corresponding to the multiple data fragments, and determining multiple target cluster sub-groups, thereby achieving the purpose of allocating clusters according to business needs, thereby realizing the technical effect of improving the utilization rate of cache resources, and further solving the technical problem that the current local cache or central cache cluster is difficult to support business needs and the unreasonable cache allocation leads to uneven resource utilization.

[0082] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0083] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the cluster processing method provided in the embodiment.

[0084] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0085] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: setting multiple clusters; based on preset business scenario requirements, dividing the multiple clusters into multiple cluster groups corresponding to the business scenarios, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios; based on preset multiple data fragments, dividing the multiple cluster groups corresponding to the business scenarios into cluster sub-groups corresponding to multiple data fragments, wherein the cluster sub-groups are used to cache data of the corresponding data fragments in the corresponding business scenarios; setting corresponding identifiers for the cluster sub-groups corresponding to the multiple data fragments, and determining multiple target cluster sub-groups.

[0086] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: receiving new data; determining the business scenario and data fragment corresponding to the new data; determining the first target cluster subgroup corresponding to the new data based on the business scenario and data fragment corresponding to the new data; and caching the new data to the first target cluster subgroup based on the identifier corresponding to the first target cluster subgroup.

[0087] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: detecting whether there is a faulty cluster in a target cluster subgroup among multiple target cluster subgroups; and if there is a faulty cluster in the target cluster subgroup, removing the faulty cluster from the corresponding target cluster subgroup.

[0088] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: in a case where all clusters included in the second target cluster subgroup are faulty clusters, deleting the second target cluster subgroup and modifying data segments corresponding to other cluster subgroups in the cluster group corresponding to the second target cluster subgroup.

[0089] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: detecting whether there is a faulty cluster in a target cluster subgroup among multiple target cluster subgroups, including: sending detection information to clusters in the multiple target cluster subgroups; detecting the time it takes for the clusters in the multiple target cluster subgroups to return a response information after receiving the detection information; and detecting whether there is a faulty cluster in the cluster subgroups among the multiple target cluster subgroups based on the relationship between the time it takes to return the response information and a preset threshold.

[0090] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: before dividing multiple cluster groups corresponding to business scenarios into cluster subgroups corresponding to multiple data fragments based on preset multiple data fragments, including: determining the data range based on a preset hash function; dividing the data range into multiple data fragments according to a preset segmentation accuracy.

[0091] An embodiment of the present invention also provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can achieve: setting multiple clusters; based on preset business scenario requirements, dividing the multiple clusters into multiple cluster groups corresponding to the business scenarios, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios; based on preset multiple data fragments, dividing the multiple cluster groups corresponding to the business scenarios into cluster sub-groups corresponding to multiple data fragments, wherein the cluster sub-groups are used to cache data of corresponding data fragments in corresponding business scenarios; setting corresponding identifiers for the cluster sub-groups corresponding to the multiple data fragments, and determining multiple target cluster sub-groups.

[0092] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0093] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0095] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, and other media that can store program code.

[0098] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A cluster processing method, characterized in that: include: Set up multiple clusters; Based on preset business scenario requirements, the multiple clusters are divided into multiple cluster groups corresponding to the business scenarios, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios; Based on a plurality of preset data fragments, the plurality of cluster groups corresponding to the business scenarios are respectively divided into cluster subgroups corresponding to the plurality of data fragments, wherein the cluster subgroups are used to cache data of the corresponding data fragments in the corresponding business scenarios; Corresponding identifiers are set for the cluster subgroups corresponding to the multiple data fragments, and multiple target cluster subgroups are determined.

2. The method according to claim 1, characterized in that Also includes: Receive new data; Determine the business scenario and data segment corresponding to the new data; Determine a first target cluster subgroup corresponding to the new data based on the business scenario and the data segment corresponding to the new data; Based on the identifier corresponding to the first target cluster subgroup, the new data is cached in the first target cluster subgroup.

3. The method according to claim 1, characterized in that Also includes: Detecting whether there is a faulty cluster in the target cluster subgroup among the multiple target cluster subgroups; In the case that there is a faulty cluster in the target cluster subgroup, the faulty cluster is removed from the corresponding target cluster subgroup.

4. The method according to claim 3, characterized in that Also includes: In the case that all clusters included in the second target cluster subgroup are faulty clusters, the second target cluster subgroup is deleted, and data segments corresponding to other cluster subgroups in the cluster group corresponding to the second target cluster subgroup are modified.

5. The method according to claim 3, characterized in that: The detecting whether there is a faulty cluster in the target cluster subgroups includes: sending detection information to clusters in the plurality of target cluster subgroups; Detecting the time length for clusters in the plurality of target cluster subgroups to return response information after receiving the detection information; According to the relationship between the time length of the returned response information and a preset threshold, it is detected whether there is a faulty cluster in the cluster subgroups among the multiple target cluster subgroups.

6. The method according to any one of claims 1 to 5, characterized in that: Before the plurality of cluster groups corresponding to the business scenarios are divided into cluster subgroups corresponding to the plurality of data segments based on the preset plurality of data segments, the method further comprises: Determine the data range based on the preset hash function; According to a preset segmentation accuracy, the data range is divided into the multiple data segments.

7. A cluster processing device, characterized in that: include: The setup module is used to set up multiple clusters; A grouping module, configured to group the plurality of clusters into a plurality of cluster groups corresponding to business scenarios based on preset business scenario requirements, wherein the business scenarios include consumer-oriented business scenarios and enterprise-oriented business scenarios; A division module, configured to divide the plurality of cluster groups corresponding to the business scenarios into cluster subgroups corresponding to the plurality of data segments, based on the plurality of preset data segments, wherein the cluster subgroups are used to cache data of the corresponding data segments in the corresponding business scenarios; The determination module is used to set corresponding identifiers for the cluster subgroups corresponding to the multiple data fragments, and determine multiple target cluster subgroups.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the cluster processing method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the cluster processing method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the cluster processing method according to any one of claims 1 to 6 is implemented.

Images