Data processing method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a nonvolatile storage medium and electronic equipment. The method comprises the following steps that: gateway equipment accesses an equipment cluster through a preset protocol; the equipment cluster forwards the service messages sent by the gateway equipment to a load balancing cluster in a shared subscription mode, and the load balancing cluster is used for sending the service messages with the same service characteristics to a service processing cluster in the same message queue; and service processors in one-to-one correspondence with the message queues are adopted in the service processing cluster to receive and process the service messages sent by the load balancing cluster. According to the method and the device, the technical problem that the processing delay of the service message is relatively high because the large-scale service message generated by the gateway equipment cannot be forwarded in a balanced manner in the related technology is solved.

Description

Technical Field

[0001] This application relates to the technical field of IT and software development. Specifically, it relates to a data processing method, apparatus, non-volatile storage medium, and electronic device. Background Art

[0002] With the rapid development of the Internet of Things and smart cities, the number of gateway devices has increased sharply, posing higher requirements for the processing capabilities of management platforms. In the scenario of managing tens of millions of gateway devices, long connections need to be established between the gateways and the management platform to maintain real-time communication, resulting in a sharp increase in the message volume, which may reach the million TPS level.

[0003] Generally, gateway devices use the MQTT protocol to connect to the platform and use the subscription / publishing method of MQTT messages for information interaction. The platform side processes business logic according to a customized business protocol. As the number of gateway devices increases, the pressure on a single service instance increases, and the ratio of hardware resource expansion is uneven. In this case, if multiple service instances are expanded, since each service subscribes to the same topic, all MQTT messages will be forwarded to each service instance through the message middleware. This will cause duplicate consumption of messages, increase the complexity of business processing, and also result in uneven service pressure. At the same time, the message middleware needs to forward multiple copies of messages, causing unnecessary bandwidth load.

[0004] Existing load balancing solutions lack adaptability and stickiness by diverting devices to different MQTT access clusters or pre-configuring routing rules for message forwarding. In the case of large-scale dynamic growth of devices, the processing targets of business processing nodes will shift, resulting in increased processing pressure on the nodes, data inconsistency, and increased complexity of business processing.

[0005] In response to the above problems, no effective solution has been proposed yet. Summary of the Invention

[0006] This application provides a data processing method, apparatus, non-volatile storage medium, and electronic device to at least solve the technical problem of high processing latency for business messages caused by the inability of related technologies to evenly forward large-scale business messages generated by gateway devices.

[0007] According to one aspect of this application, a data processing method is provided, including: a gateway device accesses a device cluster through a preset protocol; the device cluster forwards the business messages sent by the gateway device to a load balancing cluster in a shared subscription manner, where the load balancing cluster is used to send business messages with the same business characteristics to a business processing cluster in the same message queue; in the business processing cluster, a business processor corresponding to the message queue one by one receives and processes the business messages sent by the load balancing cluster.

[0008] Optionally, in the service processing cluster, service processors corresponding one-to-one to message queues are adopted to receive and process service messages sent by the load balancing cluster, including: collecting processing metrics generated during the process of a service processor corresponding one-to-one to a message queue processing the received service messages, where the processing of the received service messages by the service processing cluster includes: storing the service messages in a local cache module; obtaining the service message identifier of the service messages from the local cache module, comparing the service message identifier of the received service messages with the service message identifiers of the already stored service messages, if a service message with the same service message identifier is found, determining the service message as a duplicate service message, and marking or discarding the duplicate service message; counting the number of service messages received per unit time, and in the case where the number of service messages exceeds a preset threshold, starting a preset processing mechanism in the frequency reducer, where the preset processing mechanism includes: discarding service messages according to a preset time and / or a preset ratio; sequentially inputting the service messages processed by the frequency reducer into a serial processing queue, where in the serial processing queue, the service messages are processed in the order of first in first out; determining a control policy for the service messages according to the processing metrics, sending the control policy to the load balancing cluster, and the load balancing cluster determining a forwarding path according to the control policy and sending service messages with the same service characteristics to the service processing cluster through the same message queue based on the forwarding path.

[0009] Optionally, determining a control policy for service messages according to the processing metrics includes: performing feature extraction processing on the processing metrics corresponding to the service messages to obtain target features, where the target features include: data volume, service type, source, and processing time; comparing the target features with the conditional rules in the preset control policy one by one, if the target features meet all the conditional rules in the preset control policy, determining that the service messages corresponding to the target features match the control policy; placing the service messages in the corresponding calculation queue according to the processing method determined by the matched control policy, where for service messages involving data conversion calculations, processing is performed according to preset conversion rules, and for service messages involving logical judgments, judgments are made according to preset logical conditions; using a leaky bucket algorithm to control the traffic of service messages in the calculation queue, the leaky bucket processing service messages at a fixed rate, in the case where the speed of service messages entering the leaky bucket exceeds the processing speed of the leaky bucket, the excess messages enter the waiting queue, if the waiting queue exceeds the preset length, determining the priority of the service messages according to the service type and source of the service messages, and discarding or delaying the service messages exceeding the preset length in the waiting queue according to the priority of the service messages.

[0010] Optionally, the data processing method further includes: the device cluster sorts the media access control addresses of the gateway devices, performs equal-number segmentation or equal-load segmentation on the sorting result, and records the starting media access control address of each segment; after the device cluster receives a request message sent by a gateway device, it determines the segment to which the media access control address of the gateway device belongs, and forwards the request message to the processing service corresponding to the segment.

[0011] Optionally, after forwarding the request message to the processing service corresponding to the segment, the method further includes: under the condition of meeting a preset condition, re-sorting the media access control addresses of the gateway devices, performing equal-number segmentation or equal-load segmentation on the new sorting result, recording the starting media access control address of each segment, and after receiving a request message sent by a gateway device, re-determining the segment to which the media access control address of the gateway device belongs, and forwarding the request message to the processing service corresponding to the segment, where the preset condition includes: the change amount of the number of network management devices managed by the device cluster within the first preset time period is greater than the first preset threshold, the change amount of the number of network management devices managed by the device cluster within the first preset time period is less than the second preset threshold, where the second preset threshold is less than the first preset threshold, the change rate of the computing resources of the device cluster within the second preset time period is greater than the third preset threshold, the change rate of the memory resources of the device cluster within the second preset time period is greater than the fourth preset threshold, the change rate of the network resources of the device cluster within the second preset time period is greater than the fifth preset threshold, and the absolute value of the difference between the processing time of the device cluster for executing a task and the historical processing time of executing a historical task is greater than the sixth preset threshold.

[0012] Optionally, performing equal-number segmentation on the sorting result includes: obtaining the first number of segments N to be divided, calculating the number M of media access control addresses included in each segment, where both N and M are positive integers greater than 1, and M is equal to the total number of media access control addresses of the gateway devices divided by N; starting from the starting position of the sorting result, dividing every M media access control addresses into one segment.

[0013] Optionally, performing equal-load segmentation on the sorting result includes: obtaining the load information of each gateway device, where the load information includes: CPU usage rate, memory occupancy rate, network bandwidth usage rate; determining the total load of all gateway devices according to the load information of each gateway device; obtaining the second number of segments to be divided; determining the average load of each segment according to the total load of all gateway devices and the second number of segments; starting from the starting position of the sorting result, adding gateway devices to each segment in turn so that the load of each segment approaches the average load until all gateway devices are assigned to the corresponding segments.

[0014] Optionally, the gateway device accesses the device cluster through a preset protocol, including: the gateway device accesses the device cluster based on the Message Queuing Telemetry Transport (MQTT) protocol and in the form of a star topology structure and a mesh topology structure. Among them, the device cluster only supports the message forwarding capability, and the message forwarding capability includes: polling message forwarding capability, hash message forwarding capability, and random message forwarding capability.

[0015] According to another aspect of the present application, there is also provided a data processing system, including: a gateway device, a device cluster, a load balancing cluster, and a service processing cluster. The gateway device is communicatively connected to the device cluster, the device cluster is communicatively connected to the load balancing cluster, and the load balancing cluster is communicatively connected to the service processing cluster. Among them, the gateway device accesses the device cluster through a preset protocol; the device cluster forwards the service messages sent by the gateway device to the load balancing cluster in a shared subscription manner, where the load balancing cluster is used to send the service messages with the same service characteristics to the service processing cluster in the same message queue; in the service processing cluster, a service processor corresponding one-to-one to the message queue receives and processes the service messages sent by the load balancing cluster.

[0016] According to another aspect of the present application, there is also provided a non-volatile storage medium. The storage medium includes a stored program. Among them, when the program runs, it controls the device where the storage medium is located to execute the above data processing method.

[0017] According to another aspect of the present application, there is also provided an electronic device, including: a memory and a processor. The processor is used to run the program stored in the memory. Among them, when the program runs, it executes the above data processing method.

[0018] According to another aspect of the present application, there is also provided a computer program. Among them, when the computer program is executed by a processor, it implements the above data processing method.

[0019] According to another aspect of the present application, there is also provided a computer program product. The computer program product includes a non-volatile computer-readable storage medium. Among them, the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above data processing method.

[0020] In this application, a gateway device accesses a device cluster through a preset protocol; the device cluster forwards the service messages sent by the gateway device to a load balancing cluster in a shared subscription manner, where the load balancing cluster is used to send the service messages with the same service characteristics to a service processing cluster in the same message queue; in the service processing cluster, a service processor corresponding to the message queue one by one is used to receive and process the service messages sent by the load balancing cluster, achieving the purpose of balanced forwarding of a large number of service messages generated by the gateway device, thus realizing the technical effect of timely processing of service messages, and further solving the technical problem of high processing delay of service messages caused by the inability of related technologies to perform balanced forwarding of a large number of service messages generated by the gateway device. Description of the Drawings

[0021] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0022] Figure 1 is a flowchart of a data processing method according to an embodiment of the present application;

[0023] Figure 2 is a load balancing system for large-scale gateway management according to an embodiment of the present application;

[0024] Figure 3 is an interaction schematic diagram of an intelligent feedback control device, a load engine, and a service processor in a service cluster according to an embodiment of the present application;

[0025] Figure 4 is a structural diagram of a data processing system according to an embodiment of the present application;

[0026] Figure 5 is a hardware structure block diagram of a computer terminal for a data processing method according to an embodiment of the present application. Detailed Embodiments

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the description, claims and the above-mentioned drawings of this application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0029] According to an embodiment of the present application, a method embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from that here.

[0030] Figure 1 is a flowchart of a data processing method according to an embodiment of the present application. As Figure 1 shown, the method includes the following steps:

[0031] Step S102, the gateway device accesses the device cluster through a preset protocol.

[0032] In step S102, the gateway device uses the MQTT protocol to establish a long connection with the device cluster. Specifically, the device cluster sets up an MQTT access point, and the gateway device communicates with it through the publish / subscribe mode. During the access process, the device cluster will verify the identity of the gateway to ensure that only authorized devices can access. In addition, the device cluster will also record the MAC address of each gateway for subsequent sticky processing.

[0033] Step S104, the device cluster forwards the service messages sent by the gateway device to the load balancing cluster in a shared subscription manner, where the load balancing cluster is used to send the service messages with the same service characteristics to the service processing cluster in the same message queue.

[0034] Specifically, the device cluster shares and subscribes the service messages of all gateway devices under a single topic, which are then received by the load balancing cluster. The load balancing cluster is responsible for parsing these messages and extracting features based on the MAC addresses of the devices. Based on these features, the load engine uses a preset balancing strategy (such as sorting and segmenting the MAC addresses) to forward the messages to specific message queues. These queues are consumed by the processing nodes of the service processing cluster, ensuring that messages with the same service characteristics are consumed by the same processing node, thus achieving sticky processing of the messages.

[0035] Step S106: In the service processing cluster, a service processor corresponding one-to-one to the message queue receives and processes the service messages sent by the load balancing cluster.

[0036] In step S106, when the messages arrive at the queue, the service processors in the queue receive these messages and submit them to the local cache for quick access and to reduce duplicate checks. Specifically, it includes checking whether the messages have been processed to avoid duplicate work; reducing the burden on the service processors in high-concurrency scenarios by restricting the message processing frequency; and ensuring that the messages of the same device are processed in sequence to maintain data integrity.

[0037] According to the above steps, by having the gateway devices access the device cluster through a preset protocol; the device cluster uses the shared subscription method to forward the service messages sent by the gateway devices to the load balancing cluster, where the load balancing cluster is used to send service messages with the same service characteristics to the service processing cluster in the same message queue; and in the service processing cluster, a service processor corresponding one-to-one to the message queue receives and processes the service messages sent by the load balancing cluster, the purpose of evenly forwarding the large-scale service messages generated by the gateway devices is achieved, thus realizing the technical effect of timely processing of the service messages.

[0038] The following provides Figure 1 an exemplary illustration and explanation of the

[0039] According to some alternative embodiments of the present application, in a service processing cluster, a service processor corresponding one-to-one to a message queue is adopted to receive and process service messages sent by a load balancing cluster, which can be achieved through the following steps: Collect processing metrics generated during the process of a service processor corresponding one-to-one to a message queue processing the received service messages. Among them, the processing of the received service messages by the service processing cluster includes: storing the service messages in a local cache module; obtaining the service message identifier of the service messages from the local cache module, comparing the service message identifier of the received service messages with the service message identifiers of the already stored service messages. If a service message with the same service message identifier is found, determine the service message as a duplicate service message, and mark or discard the duplicate service message; count the number of service messages received per unit time. When the number of service messages exceeds a preset threshold, start a preset processing mechanism in the frequency reducer. Among them, the preset processing mechanism includes: discarding service messages according to a preset time and / or a preset ratio; inputting the service messages processed by the frequency reducer into a serial processing queue in sequence. Among them, in the serial processing queue, the service messages are processed in the order of first in first out; determine a control strategy for the service messages according to the processing metrics, and send the control strategy to the load balancing cluster. The load balancing cluster determines a forwarding path according to the control strategy, and based on the forwarding path, sends service messages with the same service characteristics to the service processing cluster through the same message queue.

[0040] In the above embodiment, after receiving the service messages, the service processor first stores them in the local cache module. The purpose of the local cache is to enable quick access and reduce requests to the backend database, thereby improving the processing speed. Specifically, it includes obtaining the service message identifier (such as MAC address or unique ID) of the service messages from the local cache, comparing the received service message identifier with the service message identifiers stored in the cache. If a service message with the same service message identifier is detected, mark it as a duplicate service message and discard it according to a preset strategy to avoid unnecessary repeated processing. Then count the number of service messages received per unit time (such as per second or per minute). When the number of service messages exceeds a preset threshold, the frequency reducer is activated. The preset processing mechanism may include discarding some service messages according to a preset time (for example, every 10 seconds) and / or a preset ratio (for example, discard 1 message for every 100 messages) to prevent the processor from being overloaded. The service messages processed by the frequency reducer are input into the serial processing queue in sequence. In this queue, the service messages are processed following the principle of first in first out, ensuring that the service messages of the same device can be processed in the order of reception.

[0041] Furthermore, based on the collected processing metrics, the service processor determines control policies, which include adjusting the allocation of message queues, optimizing the parameters of the frequency scaling mechanism, enhancing the policies of the local cache, etc., to improve processing efficiency and balance the load. The service processor sends the determined control policies to the load balancing cluster. After receiving the policies, the load balancing cluster adjusts its forwarding rules according to the policy content and determines new forwarding paths. Based on the updated policies, the load balancing cluster recalculates the specific message queues to which service messages with the same service characteristics (such as the same MAC address) should be forwarded. The updated policies ensure that service messages with the same service characteristics are always sent to the service processing cluster through the same message queue, thus achieving sticky forwarding, enhancing the continuity and consistency of processing, and at the same time avoiding resource waste and processing bottlenecks.

[0042] The above specific implementation steps realize the dynamic optimization of message processing and load balancing in the scenario of large-scale gateway management through real-time feedback control policies, not only improving the processing efficiency, but also providing an optimization solution for the scalability and resource utilization of the service processing cluster, which is of great significance for building an efficient and reliable network communication service.

[0043] According to some other optional embodiments of the present application, determining the control policy for service messages according to processing metrics can be achieved through the following steps: performing feature extraction processing on the processing metrics corresponding to the service messages to obtain target features, where the target features include: data volume, service type, source, and processing time; comparing the target features with the conditional rules in the preset control policies one by one. If the target features meet all the conditional rules in the preset control policies, it is determined that the service message corresponding to the target features matches the control policy; according to the processing method determined by the matched control policy, place the service message in the corresponding calculation queue, where, for service messages involving data conversion calculations, process them according to the preset conversion rules, and for service messages involving logical judgments, make judgments according to the preset logical conditions; use the leaky bucket algorithm to control the traffic of service messages in the calculation queue. The leaky bucket processes service messages at a fixed rate. When the speed at which service messages enter the leaky bucket exceeds the processing speed of the leaky bucket, the excess messages enter the waiting queue. If the waiting queue exceeds the preset length, determine the priority of the service messages according to the service type and source of the service messages, and discard or delay the service messages exceeding the preset length in the waiting queue according to the priority of the service messages.

[0044] In the above embodiments, feature extraction processing is performed on the processing metrics corresponding to the service messages to obtain target features. The original data size and the processed data size of the service messages are counted to determine the increase or decrease of data during the message processing and the storage pressure on the processor. Then, the content of the service messages is analyzed to identify the service types to which the messages belong, such as data collection, command issuance, status reporting, etc. And the sender information of the service messages is recorded, that is, the MAC address or other unique identifier of the gateway device. It should be noted that the source information is the basis for implementing sticky processing and dynamic load balancing. Finally, the total time from the reception to the completion of the processing of the service messages is recorded, including the local cache access time, duplicate detection time, downconverter processing time, and the waiting time in the serial control queue. The length of the processing time reflects the working efficiency and load of the processor.

[0045] Furthermore, the extracted target features are compared item by item with the conditional rules in the preset control policy. The preset control policy includes the processing priorities of different types of service messages, the processing frequencies of messages from different sources, the load upper limit of the processor, etc. If the target features meet all the preset conditional rules, such as the service type is data collection and the source MAC address belongs to a specific segment, it is determined that the service message matches the control policy, and the processing method defined in the policy will be executed. The processing methods in the control policy include data conversion calculation and logical judgment. For the service messages that require data conversion, they are processed according to the preset conversion rules (for example, data format conversion, data field filtering, etc.); for the service messages involving logical judgment, they are judged according to the preset logical conditions (for example, status detection, threshold judgment, etc.) to ensure the correctness and efficiency of message processing.

[0046] Next, according to the matched control policy, the service messages are placed in a specific calculation queue. Among them, each calculation queue corresponds to a specific processing method and a set of preset conditions. The leaky bucket algorithm is used to control the traffic of service messages in the calculation queue. The basic principle of the leaky bucket algorithm is to allow message traffic not exceeding a certain fixed rate to pass through, and the excess traffic is temporarily stored in the waiting queue. If the speed at which the service messages enter the leaky bucket exceeds the preset leaky bucket processing speed, the additional service messages will be placed in the waiting queue. The waiting queue has a preset length limit, and measures need to be taken when the queue length exceeds the preset value.

[0047] Finally, determine the priority of the service message according to its service type and source. For example, a status report has a higher priority than data collection, or messages from a specific MAC address segment are processed first. Based on the priority of the service message, make a processing decision on the service messages in the waiting queue that exceed the preset length, including discarding and delaying processing. For service messages with lower priority or those that can be retransmitted, they can be directly discarded when the queue is overloaded to reduce the processing burden; for service messages that cannot be discarded, they can be placed in a low-priority queue and processed when there is idle time.

[0048] Through the above implementation manners, it is possible to intelligently control the message processing flow according to the characteristics of service messages and the real-time load situation, and achieve the effective utilization of resources. When dealing with large-scale network communications and high-concurrency service messages, this mechanism can significantly improve the processing capacity and the ability to handle burst traffic.

[0049] In some alternative embodiments of the present application, the above data processing method further includes the following steps: the device cluster sorts the media access control addresses of the gateway devices, performs equal-number segmentation or equal-load segmentation on the sorting result, and records the starting media access control address of each segment; after the device cluster receives the request message sent by the gateway device, it determines the segment to which the media access control address of the gateway device belongs, and forwards the request message to the processing service corresponding to that segment.

[0050] In the above embodiment, the device cluster first collects the MAC addresses of all the connected gateway devices. The MAC address is the unique identifier of the device and is used to distinguish different devices in the network. Then sort the collected MAC addresses, and a quick sort, merge sort or any other efficient sorting algorithm can be used to ensure that all MAC addresses are arranged in a certain order, such as in numerical size or lexicographical order. According to actual needs, the device cluster can adopt an equal-number segmentation or equal-load segmentation strategy. Specifically, equal-number segmentation means dividing the sorted MAC address list into multiple segments on average, and each segment contains the same number of MAC addresses. For example, if there are a total of 10,000 MAC addresses, they can be divided into 10 segments, and each segment contains 1,000 MAC addresses. Equal-load segmentation takes into account that gateway devices with different MAC addresses will generate different message loads. The device cluster can perform dynamic segmentation on the MAC addresses based on historical data or real-time monitoring to ensure that the processing services of each segment bear relatively balanced loads within a certain period of time. Then record the starting MAC address of each segment. The starting MAC address marks the beginning of a segment, and this information is the key to load balancing and sticky forwarding. The subsequent processing service can quickly determine which segment the received request message belongs to according to the MAC address in the request message.

[0051] When the device cluster receives a request message sent by the gateway device, it first parses the MAC address information in the request message. Specifically, according to the MAC address, it looks up the segment to which it belongs. A lookup table or index structure can be used, which records the starting MAC address of the segment and the corresponding processing service. After determining the segment to which the MAC address belongs, the device cluster forwards the request message to the processing service corresponding to that segment. The processing service is located in the load balancing cluster and is responsible for further message processing and distribution. To ensure sticky processing of messages, that is, request messages with the same MAC address are always forwarded to the same processing service, the device cluster needs to keep the correspondence between the segment and the processing service unchanged when forwarding messages.

[0052] Through the above implementation manner, the device cluster can effectively manage and allocate the request messages of a large number of gateway devices, ensure that the load borne by each processing service is relatively balanced, and at the same time, through the sticky forwarding mechanism, ensure the continuity and consistency of the message processing of the same device, which is crucial for handling scenarios with high concurrency and large-scale device access.

[0053] As some alternative embodiments of the present application, after forwarding the request message to the processing service corresponding to the segment, the data processing method further includes the following steps: under the condition of meeting a preset condition, re-sorting the media access control addresses of the gateway devices, performing equal-number segmentation or equal-load segmentation on the new sorting result, recording the starting media access control address of each segment, and after receiving the request message sent by the gateway device, re-determining the segment to which the media access control address of the gateway device belongs, and forwarding the request message to the processing service corresponding to that segment, where the preset conditions include: the change amount of the number of network management devices managed by the device cluster within the first preset time period is greater than the first preset threshold, the change amount of the number of network management devices managed by the device cluster within the first preset time period is less than the second preset threshold, where the second preset threshold is less than the first preset threshold, the change rate of the computing resources of the device cluster within the second preset time period is greater than the third preset threshold, the change rate of the memory resources of the device cluster within the second preset time period is greater than the fourth preset threshold, the change rate of the network resources of the device cluster within the second preset time period is greater than the fifth preset threshold, and the absolute value of the difference between the processing time of the device cluster for executing tasks and the historical processing time of executing historical tasks is greater than the sixth preset threshold.

[0054] In the above embodiments, the device cluster continuously monitors the number of managed gateway devices. Within the first preset time period (e.g., every hour), if the change amount of the number of gateway devices exceeds the first preset threshold (e.g., an increase or decrease of 10% of the devices), then MAC address reordering is triggered. Within the first preset time period, if the change amount of the number of gateway devices is lower than the second preset threshold (e.g., the change amount is less than 5%), it indicates that the number of devices is relatively stable, and at this time, MAC address reordering is not required. It should be noted that the second preset threshold is set lower than the first preset threshold to avoid frequent reordering operations.

[0055] Meanwhile, the device cluster monitors the change rate of its computing resources (such as CPU usage rate). Within the second preset time period (e.g., every minute), if the change rate of the computing resources exceeds the third preset threshold (e.g., the change rate is greater than 20%), then MAC address reordering is triggered to reallocate processing tasks to achieve load balancing. Also within the second preset time period, the change rate of memory resources (such as RAM usage) is monitored. If the change rate of the memory resources exceeds the fourth preset threshold (e.g., the change rate is greater than 15%), it indicates that the memory pressure has changed significantly, and load adjustment is required through MAC address reordering.

[0056] Meanwhile, within the second preset time period, the change rates of resources such as network bandwidth and latency are monitored. If the change rate exceeds the fifth preset threshold (e.g., the change rate is greater than 10%), it indicates that the network resources are facing unbalanced pressure, and the message forwarding path needs to be optimized through reordering. The device cluster records the processing time of the current task and the processing time of historical tasks. If the absolute value of the difference between the current processing time and the historical processing time is greater than the sixth preset threshold (e.g., the processing time difference is greater than 2 seconds), it indicates that the task processing efficiency has fluctuated significantly, and the processor load needs to be adjusted through MAC address reordering to ensure the stability and efficiency of the processing time.

[0057] When it is detected that any of the above preset conditions is met, the device cluster will initiate the MAC address reordering process, which involves reordering all the MAC addresses of the gateway devices. Then, the new sorting result is segmented either by equal quantity or by equal load. When segmenting by equal quantity, all the gateway devices are evenly divided into multiple segments, and each segment contains the same number of MAC addresses; when segmenting by equal load, according to the performance monitoring data, the segments are dynamically adjusted to dynamically match the high-load processing services with the low-load processing services to achieve overall load balancing. And the starting MAC address of each new segment is recorded for the fast segmentation and forwarding of subsequent request messages.

[0058] Through the above dynamic monitoring and response mechanism, the device cluster can automatically adjust the sorting and segmentation of MAC addresses according to the actual operating status and resource load conditions, realizing intelligent traffic management and load balancing in the scenario of large-scale gateway management, thereby improving the overall performance and stability.

[0059] In some alternative embodiments of the present application, to perform equal-number segmentation on the sorting result, the following steps can be implemented: Obtain the first number of segments N to be divided, and calculate the number of media access control addresses included in each segment M, where both N and M are positive integers greater than 1, and M is equal to the total number of media access control addresses of the gateway device divided by N; Starting from the starting position of the sorting result, divide it into segments according to every M media access control addresses as a segment.

[0060] In the above embodiment, first, the device cluster needs to determine the first number of segments N to be divided, which is a parameter dynamically adjusted according to design requirements, processing capabilities, and the number of gateway devices. For example, if the device cluster hopes to divide all gateway devices into 10 processing segments, then N = 10. And calculate the number of MAC addresses included in each segment M, which involves the statistics of the total number of MAC addresses of the entire gateway device. Assume the total number of MAC addresses of the gateway device is T, then M = T / N. To ensure that M is an integer, processing such as rounding down or rounding up the result can be performed.

[0061] Then, starting from the starting position of the sorting result, that is, the first MAC address, the device cluster will divide it into segments according to every M MAC addresses as a segment. For example, if M = 1000, then the first segment will include the first 1000 MAC addresses, the second segment will include the next 1000 MAC addresses, and so on. And record the starting MAC address of each segment. By recording the starting MAC address, it is possible to quickly determine which segment's processing service a newly received request message should be forwarded to. The record of segment information is stored in a data structure, such as an array or a hash table, for convenient and quick lookup. Each segment information includes segment ID, starting MAC address, ending MAC address, and the corresponding processing service identifier.

[0062] When the device cluster receives a request message sent by a gateway device, first parse the MAC address of the sender from the message. Using the recorded segment information, the device cluster determines the segment to which the MAC address belongs, which can be achieved by finding the relationship between the starting MAC address and the received MAC address. Then the device cluster forwards the request message to the processing service of the segment to which the MAC address belongs. The processing service further processes the message based on the load balancing policy, such as data conversion, logical judgment, etc.

[0063] As some other alternative embodiments of the present application, equal-load segmentation of the sorting result can be achieved through the following steps: Obtain the load information of each gateway device, where the load information includes: central processing unit usage rate, memory occupancy rate, network bandwidth usage rate; Determine the total load of all gateway devices according to the load information of each gateway device; Obtain the number of segments to be divided in the second stage; Determine the average load of each segment according to the total load of all gateway devices and the number of segments in the second stage; Starting from the starting position of the sorting result, sequentially add gateway devices to each segment so that the load of each segment approaches the average load until all gateway devices are assigned to the corresponding segments.

[0064] In the above embodiment, the device cluster establishes connections with each gateway device and collects their real-time load information. The load information includes: Central processing unit usage rate: Obtain the average usage percentage of the CPU through the monitoring interface of the gateway device; Memory occupancy rate: Collect the proportion of the memory currently used by the gateway device in the total memory; Network bandwidth usage rate: Monitor the speed of data uploaded and downloaded by the gateway device and calculate its proportion in the total network bandwidth. At the same time, to ensure the accuracy and real-time nature of the load information, the device cluster can set a fixed collection frequency (such as once every 10 seconds) and use an appropriate data structure (such as a sliding window) to smooth short-term fluctuations and obtain more stable data.

[0065] The device cluster performs weighted aggregation on the load information of each gateway device collected to obtain the total load of all gateway devices. The weighting method can be based on the processing capacity or network location of the device to ensure that the calculation result reflects the actual load situation of the entire cluster. The calculation formula for the total load can be: Total load = ∑(CPU usage rate of gateway device i * weight of gateway device i + memory occupancy rate of gateway device i * weight of gateway device i + network bandwidth usage rate of gateway device i * weight of gateway device i).

[0066] The device cluster obtains the number of segments to be divided in the second stage, and calculates the average load of each segment according to the total load of all gateway devices and the number of segments in the second stage: Average load = Total load / Number of segments in the second stage. And perform segmentation according to the average load. Specifically, the device cluster starts from the starting position of the sorting result, that is, the first gateway device, and sequentially assigns gateway devices to each segment. When assigning gateway devices, the device cluster compares the load information of the device with the cumulative load of the current segment to ensure that the total load of each segment is close to the calculated average load. Some algorithms, such as the greedy algorithm, can be used in the segmentation assignment process to dynamically adjust the assignment strategy to ensure that while meeting the average load target, the relevance and processing efficiency between gateway devices are considered. When the cumulative load of a segment reaches or approaches the average load, the device cluster will start assigning gateway devices to the next segment until all gateway devices are assigned to the corresponding segments.

[0067] Through the above embodiments, the device cluster can dynamically divide multiple load-balanced segments according to the real-time load information of the gateway device, making the average load of each segment consistent, thereby achieving efficient resource allocation and balanced utilization of processing capabilities.

[0068] In some alternative embodiments, the gateway device can access the device cluster through a preset protocol, which can be achieved through the following steps: The gateway device accesses the device cluster based on the Message Queuing Telemetry Transport (MQTT) protocol and in the form of a star topology and a mesh topology. Among them, the device cluster only supports the message forwarding ability, and the message forwarding ability includes: polling message forwarding ability, hash message forwarding ability, and random message forwarding ability.

[0069] In this embodiment, the gateway device first directly connects to the MQTT access cluster through the star topology. Subsequently, the devices in the MQTT cluster will form a mesh connection to improve the efficiency and reliability of message delivery. The gateway device uses the MQTT protocol to establish a connection with the device cluster. MQTT is an efficient publish / subscribe messaging protocol suitable for machine-to-machine (M2M) / Internet of Things (IoT) communications, especially in environments with limited network conditions. The gateway device and the device cluster establish a long connection through the MQTT protocol for continuous data exchange.

[0070] The polling message forwarding ability of the device cluster means that when the gateway device sends a request or message, the cluster will forward the message to different processing services according to a pre-set order or polling mechanism. For example, the cluster can maintain a list of processing services. Each time a message arrives, starting from the beginning of the list, the message is sequentially forwarded to the next service in the list until all services have processed the message, and then polling starts from the beginning. The hash message forwarding ability is that the device cluster generates a hash value based on the message attributes (such as the MAC address of the gateway device or the topic of the message), and then forwards the message to a specific processing service according to the hash value. This method can ensure that messages with the same attributes are always forwarded to the same processing service, which is beneficial for sticky message processing and load balancing. The random message forwarding ability is that when the device cluster receives a message, it randomly selects a processing service to forward the message. This strategy can quickly disperse messages and avoid overloading a single service when the loads of the processing services are roughly balanced. However, it is not suitable for scenarios that require sticky processing, such as the situation where messages from the same device need to be processed in the same service.

[0071] Through the above embodiments, the gateway device can access the device cluster efficiently and securely, and through the various message forwarding capabilities of the cluster, achieve efficient management and load balancing of a large number of gateway devices, ensuring stable operation and high processing capabilities even in high-concurrency and large-scale device access scenarios.

[0072] In summary, in the context of the exponential growth of the number of gateway devices, this application aims to solve the problem of how to effectively manage and handle the high-concurrency communication between these devices and the management platform. Through innovative load balancing strategies, it ensures that the system can stably and quickly respond to the requests of each device, and can maintain efficient operation even in extreme cases of tens of millions of device accesses.

[0073] Traditional load balancing solutions often fall short when faced with large-scale data due to issues such as uneven resource allocation and processing bottlenecks. This application realizes refined management and dynamic adjustment of traffic by introducing MAC address sorting, dynamic segmentation, segmentation-based balancing, and sticky load strategies, significantly enhancing the system's processing capacity and scalability. This enables the system to easily handle larger-scale device accesses and higher concurrency requirements that may occur in the future. In the case of limited resources, how to maximize the utilization of existing resources is a problem that every system designer needs to face. This application reduces resource waste and unnecessary switching overhead through reasonable segmentation strategies and sticky session mechanisms, improving resource utilization efficiency. At the same time, the automated monitoring and update mechanism reduces operation and maintenance costs, enabling the system to run stably for a long time without excessive manual intervention.

[0074] In the field of network communication, the stability and reliability of the system are crucial. This application ensures that the system can quickly recover and continue to provide services in the face of abnormal situations such as sudden traffic and device failures by introducing dynamic update strategies and redundant designs. This not only improves the user experience but also provides strong guarantees for the continuous operation of enterprises.

[0075] Figure 2 A load balancing system for large-scale gateway management according to an embodiment of this application, as Figure 2 shown, the system includes: an access cluster, a load engine, a service cluster, and an intelligent feedback control device. Among them, first, the gateway device is connected upstream through a star + mesh MQTT access cluster. It should be noted that the access cluster only takes care of the upstream connection of the device and only supports basic message forwarding capabilities such as polling, Hash, and random.

[0076] In the traditional Internet of Things communication architecture, gateway devices usually establish connections with the management platform through shared subscriptions and transmit a large amount of data to the platform. In this mode, the conventional load balancing mechanism on the platform side only forwards messages based on the request volume of the messages, resulting in the messages received by the downstream service processors being both random and disordered, making it difficult to ensure the continuity and consistency of message processing. Especially when dealing with the massive data of large-scale gateway devices, this problem is particularly prominent.

[0077] To address the above problems, this application proposes an innovative load balancing engine design, the core of which is the ability to extract business characteristics. More specifically, the engine can identify and extract key features in business messages, such as the MAC address of a device, and then stably forward all messages with the same features to the same downstream message queue. This strategy lays the foundation for sticky processing of business messages and also ensures sequential processing of messages by using queues, that is, messages from the same device can be processed serially in the order they arrive, thus ensuring the accuracy and integrity of data processing.

[0078] At the business processing cluster level, each message queue is configured with exactly one consumer, which means that messages with the same characteristics (such as messages from the same device) will be assigned to only one business processor. In this way, a specific processor can continuously and stably process data from the same batch of devices, achieving sticky processing of business messages. This method can not only effectively handle the access of tens of millions of gateway devices, but also greatly improve the message processing efficiency and the overall performance of the system by using high-speed local caches.

[0079] However, with the increasing number of device accesses, sticky processing may cause some business processors to be overloaded, that is, the phenomenon of "overheating" occurs, thus affecting their processing capabilities. To solve this problem, this application introduces a feedback update mechanism, allowing the system to dynamically adjust the load balancing strategy according to the processor load data monitored in real time. Through intelligent algorithms, the system can automatically identify and reallocate some tasks of overloaded processors to other processors, achieving an even distribution of pressure, ensuring the stable operation of all business processors, avoiding the emergence of processing bottlenecks, and improving the stability and processing efficiency of the entire system.

[0080] Some modules of the intelligent feedback control device are embedded in the load balancing module and the business processor module and are connected through the intelligent feedback control module, as Figure 3 shown. In the business processor, business messages will go through general processing processes such as local caching, duplicate checking, frequency reducer processing, and serial control. The data collection module collects key metrics such as the data volume and processing time of each processing process, and these metrics can accurately reflect the pressure on the processor. The metric data is sent to the control module periodically. The intelligent feedback control device is responsible for collecting the pressure data metrics of all business processors, calculating them overall, and sending the updated strategy to the load engine module through the feedback module. The load balancing module uses the real-time updated strategy to calculate the forwarding path. The control strategy acts on multiple load calculation processes such as feature extraction, policy matching, queue calculation, and traffic control, and finally completes the entire adjustment process.

[0081] Preferably, the embodiment of the present application further provides a solution to increase stickiness, which is achieved through the following steps: Introduce a message queue: Allocate a dedicated message queue for each gateway MAC address to ensure that all messages of the same gateway are sent to the same queue. Single processing node: Ensure that each queue is processed by only one downstream processing node, thereby maintaining the continuity and consistency of messages for the same device. High-speed memory cache: Introduce a high-speed memory cache in the processing module to cache frequently accessed data, reduce the number of calls to the database or remote services, and improve the processing speed. Simplify middleware and reduce network interactions: By reducing unnecessary middleware and simplifying the data processing flow, reduce the complexity and overhead of network interactions and remove bottlenecks. Stateless expansion: Through adjusting the forwarding allocation algorithm and the number of queues, achieve stateless expansion of the message processor, and improve the system capacity and flexibility.

[0082] Preferably, the embodiment of the present application further provides a solution to solve the message forwarding balance, and the specific steps are as follows: Gateway MAC sorting and segmentation: Sort the MAC addresses of the gateway devices included in the management, and perform average segmentation according to a preset algorithm (such as equal quantity, equal load, etc.), and record the starting (connecting) MAC address of each segment. Dynamically allocate processing services: After receiving a gateway request, calculate the segment to which the gateway MAC address belongs, and forward the request to the corresponding processing service within the segment. The segment information is not static, but is dynamically adjusted according to the system operation status. The factors triggering the update include the increase or decrease in the number of managed gateways, changes in the amount of processing resources, extension or shortening of the processing time, etc. When it is detected that the above factors reach the preset threshold, the system automatically re-performs MAC address sorting, segmentation, and updates the segment information to ensure the effectiveness and real-time nature of the load balancing strategy.

[0083] In summary, the load balancing device and method of the present application, through innovative technologies such as intelligent traffic perception and prediction, multi-level balancing and sticky load allocation, dynamic capacity expansion and contraction, and high availability guarantee, effectively solve the traffic pressure problem in the large-scale gateway management scenario, improve the overall performance, stability, and scalability of the system, and provide strong support for building an efficient and reliable network service. Through the sticky solution and the message forwarding balance strategy, effectively solve the traffic pressure problem in the large-scale gateway device management, improve the system processing capacity and response speed. Reduce the risk of resource bottlenecks and single points of failure, and improve the overall stability and reliability of the system. Through reasonable resource allocation and caching strategies, improve resource utilization efficiency and reduce operating costs. Support a stateless expansion mechanism, enabling the system to easily handle larger-scale device access and higher concurrency requirements that may occur in the future.

[0084] Figure 4 is a structural diagram of a data processing system according to an embodiment of the present application, as Figure 4As shown in the figure, the system includes: a gateway device 41, a device cluster 42, a load balancing cluster 43, and a service processing cluster 44. The gateway device 41 is communicatively connected to the device cluster 42, the device cluster 42 is communicatively connected to the load balancing cluster 43, and the load balancing cluster 43 is communicatively connected to the service processing cluster 44. Among them,

[0085] The gateway device 41 accesses the device cluster 42 through a preset protocol; the device cluster 42 forwards the service messages sent by the gateway device 41 to the load balancing cluster 43 in a shared subscription manner, where the load balancing cluster 43 is used to send the service messages with the same service characteristics to the service processing cluster 44 in the same message queue; in the service processing cluster 44, a service processor corresponding one-to-one to the message queue is used to receive and process the service messages sent by the load balancing cluster 43.

[0086] Specifically, the gateway device 41 accesses the device cluster 42 through a preset communication protocol (such as MQTT). The preset protocol ensures efficient and secure data transmission between the gateway device and the platform. The device cluster 42 is responsible for communicating with the gateway device 41 and receiving the service messages sent by the device. In this system, the device cluster forwards the service messages sent by the gateway device 41 to the load balancing cluster 43 in a shared subscription manner. The shared subscription mode ensures wide message reception, but may lead to disordered and repeated message processing. The function of the load balancing cluster 43 is to receive the service messages forwarded from the device cluster 42 and, through an intelligent algorithm, stably forward the messages with the same service characteristics (such as the MAC address of the device) to the same message queue. This strategy ensures message processing stickiness, that is, all messages of the same device are sequentially processed by a specific service processor, avoiding data processing chaos and repetition. The load balancing cluster 43 ensures that the messages in each message queue meet the requirements of sticky processing through feature extraction, policy matching, and queue calculation. At the same time, through a traffic control mechanism, it realizes balanced message distribution, avoiding resource waste and processing bottlenecks. The service processing cluster 44 consists of multiple service processors, and each processor corresponds one-to-one to the message queue in the load balancing cluster 43. The purpose of this design is to ensure that each service message with specific service characteristics is only processed by one service processor, thus realizing sticky processing. The service processor is responsible for receiving and processing the messages in the message queue and executing specific service logics, such as data parsing, storage, and further analysis. Through the sticky mechanism, the service processor can continuously process the messages from a specific device, improving processing efficiency and data consistency.

[0087] Figure 4The data processing system shown forms an efficient channel for message reception, forwarding, and processing by organically connecting gateway devices, device clusters, load balancing clusters, and business processing clusters through a hierarchical architecture design. The load balancing strategy in this application ensures the continuity and consistency of message processing. Through the queue and sticky mechanisms, each business processor focuses on processing messages of specific devices, avoiding resource waste and processing chaos. At the same time, the intelligent dynamic load adjustment mechanism ensures that even when the number of device accesses is increasing continuously, the system can maintain a stable and efficient operating state, providing strong technical support for processing large-scale gateway device data.

[0088] It should be noted that Figure 4 For the preferred implementation manners of the illustrated embodiments, reference may be made to Figure 1 the relevant descriptions of the illustrated embodiments, which will not be elaborated herein.

[0089] Figure 5 The hardware structure block diagram of a computer terminal for implementing a data processing method is shown. As Figure 5 shown, the computer terminal 50 may include one or more processors 502 (illustrated as 502a, 502b,..., 502n in the figure) (the processor 502 may include, but is not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA), a memory 504 for storing data, and a transmission module 506 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 5 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 50 may further include more or fewer components than those Figure 5 shown, or have a different configuration from that Figure 5 shown.

[0090] It should be noted that the above one or more processors 502 and / or other data processing circuits are generally referred to as "data processing circuits" herein. The data processing circuit may be embodied in whole or in part as software, hardware, firmware, or any combination thereof. In addition, the data processing circuit may be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 50. As involved in the embodiments of the present application, the data processing circuit is a kind of processor control (such as the selection of a variable resistor terminal path connected to an interface).

[0091] The memory 504 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method in the embodiments of the present application. The processor 502 executes various functional applications and data processing by running the software programs and modules stored in the memory 504, that is, implements the above data processing method. The memory 504 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 504 may further include a memory remotely disposed relative to the processor 502, and these remote memories can be connected to the computer terminal 50 through a network. Examples of the above network include but are not limited to the Internet, enterprise intranets, local area networks, mobile communication networks, and combinations thereof.

[0092] The transmission module 506 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the computer terminal 50. In one instance, the transmission module 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one instance, the transmission module 506 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0093] The display can be, for example, a touch-screen liquid crystal display (LCD), and the liquid crystal display enables a user to interact with the user interface of the computer terminal 50.

[0094] It should be noted here that in some alternative embodiments, the above Figure 5 shown computer terminal may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. It should be pointed out that Figure 5 is only an example of a specific specific instance, and is intended to show the types of components that may exist in the above computer terminal.

[0095] It should be noted that Figure 5 the shown computer terminal is used to execute Figure 1 the shown data processing method, so the relevant explanations in the execution method of the above commands also apply to this electronic device, and will not be elaborated here.

[0096] The embodiments of the present application also provide a non-volatile storage medium, and the non-volatile storage medium includes a stored program, wherein when the program runs, it controls the device where the storage medium is located to execute the above data processing method.

[0097] A program for a non-volatile storage medium to perform the following functions: A gateway device accesses a device cluster through a preset protocol; the device cluster forwards the service messages sent by the gateway device to a load balancing cluster in a shared subscription manner, where the load balancing cluster is used to send service messages with the same service characteristics to a service processing cluster in the same message queue; in the service processing cluster, a service processor corresponding to the message queue one by one receives and processes the service messages sent by the load balancing cluster.

[0098] An embodiment of the present application also provides an electronic device, including: a memory and a processor, where the processor is used to run a program stored in the memory, and when the program runs, it executes the above data processing method.

[0099] The processor is used to run a program that performs the following functions: A gateway device accesses a device cluster through a preset protocol; the device cluster forwards the service messages sent by the gateway device to a load balancing cluster in a shared subscription manner, where the load balancing cluster is used to send service messages with the same service characteristics to a service processing cluster in the same message queue; in the service processing cluster, a service processor corresponding to the message queue one by one receives and processes the service messages sent by the load balancing cluster.

[0100] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0101] In the above embodiments of the present application, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0102] In the above embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the processing of relevant data such as collection, storage, use, processing, transmission, provision, disclosure, and application complies with relevant laws, regulations, and standards, takes necessary protection measures, does not violate public order and good customs, and provides a corresponding operation entry for the user to choose to authorize or refuse.

[0103] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0104] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0105] In addition, each functional unit in various embodiments of the present application may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0106] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the related technology, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs and other various media that can store program codes.

[0107] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A data processing method, characterized in that: include: The gateway device accesses the device cluster through a preset protocol; The device cluster forwards the service message sent by the gateway device to the load balancing cluster in a shared subscription manner, wherein the load balancing cluster is used to send service messages with the same service characteristics to the service processing cluster in the same message queue; In the service processing cluster, a service processor corresponding to a message queue one by one is used to receive and process the service message sent by the load balancing cluster.

2. The method according to claim 1, characterized in that In the service processing cluster, a service processor corresponding to a message queue is used to receive and process the service message sent by the load balancing cluster, including: Collect processing indicators generated in the process of processing received business messages by business processors corresponding to message queues one by one, wherein the processing of received business messages by the business processing cluster includes: storing business messages in a local cache module; obtaining the business message identifier of the business message from the local cache module, comparing the business message identifier of the received business message with the business message identifier of the stored business message, and if a business message with the same business message identifier is found, determining the business message as a duplicate business message, marking or discarding the duplicate business message; counting the number of business messages received within a unit time, and when the number of business messages exceeds a preset threshold, starting a preset processing mechanism in a downconverter, wherein the preset processing mechanism includes: discarding business messages according to a preset time and / or a preset ratio; and sequentially inputting the business messages processed by the downconverter into a serial processing queue, wherein in the serial processing queue, business messages are processed in a first-in-first-out order; Based on the processing indicators, a control strategy for business messages is determined, and the control strategy is sent to the load balancing cluster. The load balancing cluster determines a forwarding path based on the control strategy, and based on the forwarding path, business messages with the same business characteristics are sent to the business processing cluster in the same message queue.

3. The method according to claim 2, characterized in that According to the processing indicators, a control strategy for the business message is determined, including: Performing feature extraction processing on the processing indicators corresponding to the business message to obtain target features, wherein the target features include: data volume, business type, source, and processing time; Compare the target feature with the conditional rules in the preset control policy one by one, and if the target feature satisfies all the conditional rules in the preset control policy, determine that the service message corresponding to the target feature matches the control policy; According to the processing method determined by the matched control strategy, the business message is placed in the corresponding calculation queue, wherein the business message involving data conversion calculation is processed according to the preset conversion rules, and the business message involving logical judgment is judged according to the preset logical conditions; A leaky bucket algorithm is used to control the flow of business messages in the computing queue. The leaky bucket processes business messages at a fixed rate. When the speed at which business messages enter the leaky bucket exceeds the processing speed of the leaky bucket, the excess messages enter the waiting queue. If the waiting queue exceeds the preset length, the priority of the business message is determined according to the business type and source of the business message. Based on the priority of the business message, the business messages in the waiting queue that exceed the preset length are discarded or delayed.

4. The method according to claim 1, characterized in that: The method further comprises: The device cluster sorts the media access control addresses of the gateway devices, divides the sorting results into equal number segments or equal load segments, and records the starting media access control address of each segment; After receiving the request message sent by the gateway device, the device cluster determines the segment to which the media access control address of the gateway device belongs, and forwards the request message to the processing service corresponding to the segment.

5. The method according to claim 4, characterized in that After forwarding the request message to the processing service corresponding to the segment, the method further includes: Under the condition that the preset conditions are met, the media access control addresses of the gateway devices are re-sorted, the new sorting results are segmented into equal numbers or equal loads, the starting media access control address of each segment is recorded, and after receiving the request message sent by the gateway device, the segment to which the media access control address of the gateway device belongs is re-determined, and the request message is forwarded to the processing service corresponding to the segment, wherein the preset conditions include: the change in the number of network management devices managed by the device cluster within a first preset time period is greater than a first preset threshold, the change in the number of network management devices managed by the device cluster within the first preset time period is less than a second preset threshold, wherein the second preset threshold is less than the first preset threshold, the change rate of the computing resources of the device cluster within the second preset time period is greater than a third preset threshold, the change rate of the memory resources of the device cluster within the second preset time period is greater than a fourth preset threshold, the change rate of the network resources of the device cluster within the second preset time period is greater than a fifth preset threshold, and the absolute value of the difference between the processing time of the device cluster executing the task and the historical processing time of executing the historical task is greater than a sixth preset threshold.

6. The method according to claim 4, characterized in that The sorting result is divided into equal number of segments, including: obtaining the first segment number N to be divided, calculating the number M of media access control addresses contained in each segment, wherein N and M are both positive integers greater than 1, and M is equal to the total number of media access control addresses of the gateway device divided by N; starting from the starting position of the sorting result, dividing the segment according to each M media access control addresses as a segment; The sorting results are divided into equal load segments, including: Obtain load information for each gateway device, wherein the load information includes: CPU usage, memory occupancy, and network bandwidth usage; determine the total load of all gateway devices based on the load information of each gateway device; obtain the number of second segments to be divided; determine the average load of each segment based on the total load of all gateway devices and the number of second segments; starting from the starting position of the sorting result, add the gateway devices to each segment in turn, so that the load of each segment approaches the average load, until all the gateway devices are allocated to the corresponding segments.

7. The method according to claim 1, characterized in that The gateway device accesses the device cluster through a preset protocol, including: the gateway device accesses the device cluster based on the message queue telemetry transmission protocol and adopts a star topology and a mesh topology, wherein the device cluster only supports message forwarding capabilities, and the message forwarding capabilities include: polling message forwarding capabilities, hash message forwarding capabilities, and random message forwarding capabilities.

8. A data processing system, characterized in that: include: A gateway device, a device cluster, a load balancing cluster and a business processing cluster, wherein the gateway device is communicatively connected to the device cluster, the device cluster is communicatively connected to the load balancing cluster, and the load balancing cluster is communicatively connected to the business processing cluster, wherein the gateway device accesses the device cluster through a preset protocol; The device cluster forwards the service message sent by the gateway device to the load balancing cluster in a shared subscription manner, wherein the load balancing cluster is used to send service messages with the same service characteristics to the service processing cluster in the same message queue; In the service processing cluster, a service processor corresponding to a message queue one by one is used to receive and process the service message sent by the load balancing cluster.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the data processing method according to any one of claims 1 to 7 when running.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images