Message transmission optimization system through scalable data processing

Abstract

The invention discloses a message transmission optimization system through scalable data processing, and the system comprises a data collection module which is used for collecting equipment data based on edge nodes and carrying out the data preprocessing; the data access module is used for performing data access based on a distributed architecture and completing data synchronization within the range of all cluster nodes; the data processing module is used for automatically adjusting the number of cluster nodes according to the current load based on the micro-service architecture; and the cluster management module is used for carrying out heartbeat detection on each node in the cluster and carrying out failover on the failed node. According to the invention, rapid access and distributed transmission are carried out on the acquired equipment data, processing nodes are reasonably configured and the current data are processed based on the data volume flexibly and based on the data processing module, and automatic fault transfer is carried out on the cluster, so that the rationality of processing resource configuration, the system operation efficiency and the system stability are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of device data processing, and in particular to a message transmission optimization system, method, computing device and storage medium through scalable data processing. Background Art

[0002] With the rapid development of technologies such as big data, the Internet of Things, deep learning, and artificial intelligence, the value of data itself is increasingly recognized, especially since the practical application, technological progress, and development of the above-mentioned technologies all require a large amount of data. Therefore, how to quickly obtain data from various devices and transmit and process it has become a top priority.

[0003] Currently, the acquired data from various devices is usually transmitted based on a fixed transmission path, and the corresponding processing end processes the data after receiving it. However, in this way, when faced with situations where the data volume is significantly different and generated by different types of devices, the transmission resources and processing resources occupied are still the same.

[0004] In this way, when the types of device data are diverse and complex, and the data volume varies, this processing method will show rigid, inefficient and inflexible problems. When the data volume is small, it will lead to excessive idle occupancy rate of transmission resources and idle utilization rate of processing resources, resulting in a large amount of resource occupation and waste, and extremely low utilization efficiency; when the data volume is large, it will lead to insufficient allocation of transmission resources and processing resources, greatly extending the time of data transmission and data processing, and also reducing the overall efficiency of the system. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a message transmission optimization system through scalable data processing and a corresponding message transmission optimization method through scalable data processing, a computing device and a computer storage medium.

[0006] According to one aspect of the present invention, a message transmission optimization system through scalable data processing is provided, comprising: a data acquisition module, a data access module, a data processing module and a cluster management module; wherein,

[0007] The data acquisition module is used to collect device data based on edge nodes and perform data preprocessing;

[0008] The data access module is used to access data based on a distributed architecture and complete data synchronization within all cluster nodes;

[0009] The data processing module is used to automatically adjust the number of cluster nodes according to the current load based on the microservice architecture;

[0010] The cluster management module is used to perform heartbeat detection on each node in the cluster and perform failover on nodes with faults.

[0011] In the above solution, the edge node is arranged on the device side, and the data acquisition module is further used for:

[0012] Determine the device corresponding to each edge node, and each edge node obtains the device data of its corresponding device;

[0013] Performing data preprocessing on the acquired device data to obtain preprocessed data; wherein the data preprocessing at least includes: data cleaning and / or normalization;

[0014] Based on the lossless compression algorithm, the preprocessed data is losslessly compressed.

[0015] In the above solution, the data access module is further used to:

[0016] A pre-set recording interface between nodes in the cluster;

[0017] Using the recording interface between each node, the device data acquired and processed by the edge node is transmitted;

[0018] The device data obtained and processed by the edge nodes is backed up and stored in each node.

[0019] In the above solution, the data processing module is further used to:

[0020] Determine target indicators for capacity expansion or reduction based on device data;

[0021] According to the target indicators, use the load balancer in the microservice architecture and the preset load balancing technology to determine the cluster node adjustment method;

[0022] Expand or reduce the cluster capacity based on the cluster node adjustment method.

[0023] In the above solution, the cluster management module is further used to:

[0024] Monitor each node in the cluster and obtain the heartbeat information sent by each node according to the preset period;

[0025] Determine whether a node is in an abnormal state based on the heartbeat information of each node;

[0026] For nodes with abnormalities, failover is performed based on a preset failover strategy; wherein the failover strategy is a master-slave replication strategy, a multi-master replication strategy or a data sharding strategy.

[0027] In the above solution, the data processing module is further used to:

[0028] According to the load status of nodes in the cluster, a node load adjustment model is constructed;

[0029] Collecting historical load information of past data processing in the cluster, using the historical load information to train a node load adjustment model, and obtaining a trained node load adjustment model;

[0030] Based on the trained node load adjustment model, the node load status in the cluster is determined and adjusted.

[0031] In the above solution, the data processing module is further used to:

[0032] The node load adjustment model determines the node load state based on the node data processing performance parameter, and the node data processing performance parameter is

[0033]

[0034] Among them, P N is the node data processing performance parameter; x is the total number of tasks processed by the node; M i The amount of data when synchronizing data for the i-th task; No ei is the node CPU idle rate parameter corresponding to the i-th task; α is the coefficient of the CPU idle rate parameter; Sa ei The idle rate parameter is stored for the node corresponding to the i-th task; β is the coefficient of the stored idle rate parameter.

[0035] According to another aspect of the present invention, there is provided a method for optimizing message transmission through scalable data processing, comprising:

[0036] Collect device data based on edge nodes and perform data preprocessing;

[0037] Data access is performed based on a distributed architecture, and data synchronization is completed across all cluster nodes;

[0038] Based on the microservice architecture, the number of cluster nodes is automatically adjusted according to the current load;

[0039] Perform heartbeat detection on each node in the cluster and perform failover on failed nodes.

[0040] According to another aspect of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0041] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operations implemented by the message transmission optimization system through scalable data processing as described above.

[0042] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to execute operations implemented by the message transmission optimization system through scalable data processing as described above.

[0043] According to the technical solution provided by the present invention, a message transmission optimization system through scalable data processing includes: a data acquisition module, a data access module, a data processing module and a cluster management module; wherein the data acquisition module is used to collect device data based on edge nodes and perform data preprocessing; the data access module is used to access data based on a distributed architecture and complete data synchronization within all cluster nodes; the data processing module is used to automatically adjust the number of cluster nodes according to the current load based on a microservice architecture; the cluster management module is used to perform heartbeat detection on each node in the cluster and perform failover on nodes with faults. By setting up edge nodes on the device side, the device data close to each edge node is quickly collected, which improves the data collection efficiency. By preprocessing and lossless compression at the corresponding edge nodes, the corresponding data transmission volume is reduced, and the data transmission efficiency is improved while ensuring data integrity. Based on the distributed architecture, the collected data is accessed, and the transmission and storage of device data in the remaining nodes in the distributed architecture are quickly completed through the interfaces between each node, thereby completing data backup and improving data security. Subsequently, the microservice architecture is used for data processing. Based on the load balancer in the microservice architecture, the cluster is expanded or reduced according to the current amount of data to be processed using load balancing mediation technology, so that the number of nodes in the cluster can better match the current amount of data to be processed, reducing the possible high or low node utilization, that is, reducing the waste of resources or the reduction of processing speed, and greatly improving the overall operation efficiency of the system. By monitoring the heartbeat information of each node in the cluster, automatic failover is performed when a fault occurs, which improves the robustness of the system when a fault occurs and ensures the stable operation of the system. In addition, by building a load adjustment model, using load history for training, and adjusting the cluster based on the node data processing performance parameters determined based on the characteristics of the current data, the intelligence of the system is further improved, and it can more accurately analyze the current load requirements for processing nodes, further improving the accuracy of expanding or shrinking the cluster and improving the operating efficiency of the system.

[0044] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the corresponding drawings of the description.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0047] Figure 1 A structural block diagram of a message transmission optimization system through scalable data processing according to an embodiment of the present invention is shown;

[0048] Figure 2 A schematic flow chart of an automatic expansion and contraction method based on a scalable data processing layer according to an embodiment of the present invention is shown;

[0049] Figure 3 A schematic diagram of a process for cluster node operation and fault management according to an embodiment of the present invention is shown;

[0050] Figure 4 A schematic flow chart of a method for optimizing message transmission through scalable data processing according to an embodiment of the present invention is shown;

[0051] Figure 5 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0052] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0053] Figure 1 The structure block diagram of a message transmission optimization system through scalable data processing according to an embodiment of the present invention is shown, the system comprises: a data acquisition module 101, a data access module 102, a data processing module 103 and a cluster management module 104; wherein,

[0054] The data collection module 101 is used to collect device data based on edge nodes and perform data preprocessing.

[0055] Preferably, the edge node is arranged on the device side;

[0056] Specifically, the device corresponding to each edge node is determined, and each edge node obtains the device data of its corresponding device;

[0057] Perform data preprocessing on the acquired device data to obtain preprocessed data;

[0058] Based on the lossless compression algorithm, the preprocessed data is losslessly compressed.

[0059] Preferably, data preprocessing includes at least: data cleaning and / or normalization; wherein,

[0060] Data cleaning uses inconsistency detection to filter out abnormal data and perform noise identification, and then completes data cleaning through data filtering and correction;

[0061] In addition, data reduction or data conversion may also be included depending on the data type and processing requirements.

[0062] The data access module 102 is used to access data based on a distributed architecture and complete data synchronization within all cluster nodes.

[0063] Specifically, a recording interface pre-set between nodes in the cluster;

[0064] Using the recording interface between each node, the device data acquired and processed by the edge node is transmitted;

[0065] The device data obtained and processed by the edge nodes is backed up and stored in each node.

[0066] Preferably, the Flume architecture can be used to collect data through edge nodes, collect data from the target device, and then pull the collected device data from the destination (other nodes used for data synchronization), and cache the data before the device data is transmitted. After the data transmission process is completed, Flume deletes the cached data.

[0067] The data processing module 103 is used to automatically adjust the number of cluster nodes according to the current load based on the microservice architecture.

[0068] The cluster management module 104 is used to perform heartbeat detection on each node in the cluster and perform failover on a node with a fault.

[0069] According to the present embodiment, a message transmission optimization system through scalable data processing is provided, including: a data acquisition module, a data access module, a data processing module and a cluster management module; wherein the data acquisition module is used to collect device data based on edge nodes and perform data preprocessing; the data access module is used to perform data access based on a distributed architecture and complete data synchronization within all cluster nodes; the data processing module is used to automatically adjust the number of cluster nodes according to the current load based on a microservice architecture; the cluster management module is used to perform heartbeat detection on each node in the cluster and perform failover on faulty nodes. A message transmission optimization system through scalable data processing provided by this embodiment, by setting edge nodes on the device side, quickly collecting device data close to each edge node, improving data collection efficiency, and by preprocessing and lossless compression at the corresponding edge nodes, reducing the corresponding data transmission volume, while ensuring data integrity, improving data transmission efficiency; data access is performed on the collected data based on a distributed architecture, and the transmission and storage of device data in other nodes in the distributed architecture are quickly completed through the interface between each node, thereby completing data backup and improving data security; subsequently, a microservice architecture is used for data processing, and load balancing mediation technology is used to expand or shrink the cluster according to the current amount of data to be processed, so that the number of nodes in the cluster can better match the current amount of data to be processed, reducing the possible situation of excessive or low node utilization, and greatly improving the overall operation efficiency of the system; by monitoring the heartbeat information of each node in the cluster, automatic failover is performed when a fault occurs, improving the robustness of the system when a fault occurs, and ensuring the stable operation of the system.

[0070] Figure 2 FIG. 4 shows a flow chart of an automatic expansion and contraction method based on a scalable data processing layer according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0071] Step S201: Determine a target indicator for capacity expansion or capacity reduction based on device data.

[0072] Specifically, the capacity expansion or contraction indicator may be the total amount of device data.

[0073] Preferably, the target indicator for capacity expansion or contraction may also be a relevant indicator in the data processing process, such as the CPU utilization rate, memory utilization rate or QPS (Queries Per Second) of the node.

[0074] Step S202, according to the target indicator, using the load balancer in the microservice architecture, based on the preset load balancing technology, determines the cluster node adjustment method.

[0075] Specifically, a node load adjustment model is constructed according to the load status of nodes in the cluster;

[0076] Collecting historical load information of past data processing in the cluster, using the historical load information to train a node load adjustment model, and obtaining a trained node load adjustment model;

[0077] Based on the trained node load adjustment model, the node load status in the cluster is determined and adjusted. Preferably, the node load adjustment model determines the node load status based on the node data processing performance parameter, and the node data processing performance parameter is

[0078]

[0079] Among them, P N is the node data processing performance parameter; x is the total number of tasks processed by the node; M i The amount of data when synchronizing data for the i-th task; No ei is the node CPU idle rate parameter corresponding to the i-th task; α is the coefficient of the CPU idle rate parameter; Sa ei The idle rate parameter is stored for the node corresponding to the i-th task; β is the coefficient of the stored idle rate parameter.

[0080] Step S203: Expand or reduce the capacity of the cluster based on the cluster node adjustment method.

[0081] According to the above method, a microservice architecture is used for data processing. Based on the load balancer in the microservice architecture, the cluster is expanded or reduced in capacity using load balancing mediation technology according to the current amount of data that needs to be processed, so that the number of nodes in the cluster can better match the current amount of data to be processed, reducing the possible situation of excessive or low node utilization, that is, reducing the waste of resources or the reduction of processing speed, and greatly improving the overall operation efficiency of the system. In addition, by constructing a load adjustment model, using load history for training, and adjusting the cluster based on the node data processing performance parameters determined based on the characteristics of the current data, the intelligence of the system is further improved, and the current load requirements for processing nodes can be analyzed more accurately, further improving the accuracy of expanding or reducing the cluster, and improving the operation efficiency of the system.

[0082] Figure 3 FIG. 1 is a flow chart showing a method for cluster node operation and fault management according to an embodiment of the present invention. Figure 3 As shown, the method comprises the following steps:

[0083] Step S301: monitor each node in the cluster and obtain heartbeat information sent by each node according to a preset period.

[0084] Step S302: determine whether the node is in an abnormal state based on the heartbeat information of each node.

[0085] Step S303: For the abnormal node, failover is performed based on a preset failover strategy.

[0086] Preferably, the failover strategy is a master-slave replication strategy, a multi-master replication strategy or a data sharding strategy.

[0087] Preferably, during the automatic transfer process, the selection of a new node for processing the task can be based on the neighboring nodes of the node where the failure occurs, and the node with higher transmission stability is selected as the new processing node; wherein,

[0088] The calculation method of the stability prediction value of each adjacent node of the fault node is:

[0089] pre=var{A1(n j ),A2(n j ),……,A m-1 (n j )}

[0090] Where pre is the stability prediction value; var is the variance; m is the number of sampling times of the fault node to the jth adjacent node; {A1(n j ),A2(n j ),……,A m-1 (n j )} is the vector set corresponding to the m-1 sampling of the fault node and the jth adjacent node; n j represents the jth adjacent node.

[0091] Furthermore, the vectors in the vector set are

[0092]

[0093] Among them, d m (star→n j ) is the Euclidean distance from the starting node (faulty node) to the jth adjacent node after m samplings.

[0094] Based on this, the stability prediction values ​​of each adjacent node of the faulty node are calculated, and the obtained stability prediction values ​​are sorted according to the numerical size. The adjacent node corresponding to the stability prediction value with the highest sorting is used as the target node for fault transfer and data processing of the current faulty node.

[0095] According to the above method, by monitoring the heartbeat information of each node in the cluster, automatic failover is performed when a fault occurs, thereby improving the robustness of the system when a fault occurs and ensuring the stable operation of the system. At the same time, by selecting the neighboring nodes of the faulty node and selecting the target node for failover based on transmission stability, the security and stability of processing when a fault occurs are better guaranteed, further improving the overall stability of the system.

[0096] Figure 4 A schematic flow chart of a method for optimizing message transmission through scalable data processing according to an embodiment of the present invention is shown;

[0097] like Figure 4 As shown, the method comprises the following steps:

[0098] Step S401: collect device data based on edge nodes and perform data preprocessing.

[0099] Preferably, the edge node is arranged on the device side.

[0100] Specifically, the collecting of device data and preprocessing of data based on edge nodes further includes:

[0101] Determine the device corresponding to each edge node, and each edge node obtains the device data of its corresponding device;

[0102] Performing data preprocessing on the acquired device data to obtain preprocessed data; wherein the data preprocessing at least includes: data cleaning and / or normalization;

[0103] Based on the lossless compression algorithm, the preprocessed data is losslessly compressed.

[0104] Step S402: data access is performed based on the distributed architecture, and data synchronization is completed within the scope of all cluster nodes.

[0105] Specifically, the data access based on the distributed architecture and completing data synchronization within all cluster nodes further includes:

[0106] A pre-set recording interface between nodes in the cluster;

[0107] Using the recording interface between each node, the device data acquired and processed by the edge node is transmitted;

[0108] The device data obtained and processed by the edge nodes is backed up and stored in each node.

[0109] Step S403: Based on the microservice architecture, the number of cluster nodes is automatically adjusted according to the current load.

[0110] Specifically, the microservice architecture is based on which the number of cluster nodes is automatically adjusted according to the current load, further comprising:

[0111] Determine target indicators for capacity expansion or reduction based on device data;

[0112] According to the target indicators, use the load balancer in the microservice architecture and the preset load balancing technology to determine the cluster node adjustment method;

[0113] Expand or reduce the cluster capacity based on the cluster node adjustment method.

[0114] Preferably, a node load adjustment model is constructed according to the load status of the nodes in the cluster;

[0115] Collecting historical load information of past data processing in the cluster, using the historical load information to train a node load adjustment model, and obtaining a trained node load adjustment model;

[0116] Based on the trained node load adjustment model, the node load status in the cluster is determined and adjusted.

[0117] Preferably, the node load adjustment model determines the load state of the node based on the node data processing performance parameter, and the node data processing performance parameter is

[0118]

[0119] Among them, P N is the node data processing performance parameter; x is the total number of tasks processed by the node; M i The amount of data when synchronizing data for the i-th task; No ei is the node CPU idle rate parameter corresponding to the i-th task; α is the coefficient of the CPU idle rate parameter; Sa ei The idle rate parameter is stored for the node corresponding to the i-th task; β is the coefficient of the stored idle rate parameter.

[0120] Step S404: perform heartbeat detection on each node in the cluster and perform failover on the node with a fault.

[0121] Specifically, the heartbeat detection is performed on each node in the cluster, and the failover is performed on the node with a fault, further comprising:

[0122] Monitor each node in the cluster and obtain the heartbeat information sent by each node according to the preset period;

[0123] Determine whether a node is in an abnormal state based on the heartbeat information of each node;

[0124] For nodes with abnormalities, failover is performed based on a preset failover strategy; wherein the failover strategy is a master-slave replication strategy, a multi-master replication strategy or a data sharding strategy.

[0125] According to a message transmission optimization method through scalable data processing provided by this embodiment, device data is collected and pre-processed based on edge nodes; data access is performed based on a distributed architecture, and data synchronization is completed within all cluster nodes; based on a microservice architecture, the number of cluster nodes is automatically adjusted according to the current load; heartbeat detection is performed on each node in the cluster, and failover is performed on nodes with faults. According to a message transmission optimization method through scalable data processing provided by this embodiment, by setting edge nodes on the device side, the device data close to each edge node is quickly collected, the data collection efficiency is improved, and by preprocessing and lossless compression at the corresponding edge nodes, the corresponding data transmission volume is reduced, and the data transmission efficiency is improved while ensuring data integrity; based on the distributed architecture, the collected data is accessed, and the transmission and storage of the device data in the remaining nodes in the distributed architecture are quickly completed through the interface between each node, thereby completing data backup and improving data security; then, a microservice architecture is used for data processing, and based on the load balancer in the microservice architecture, the cluster is expanded or reduced according to the current amount of data to be processed by using load balancing mediation technology, so that the number of nodes in the cluster can better match the current amount of data to be processed, reducing the possible situation of excessive or low node utilization, that is, reducing the waste of resources or the reduction of processing speed, and greatly improving the overall operation efficiency of the system; by monitoring the heartbeat information of each node in the cluster, automatic failover is performed when a fault occurs, the robustness of the system when a fault occurs is improved, and the stable operation of the system is ensured. In addition, by building a load adjustment model, using load history for training, and adjusting the cluster based on the node data processing performance parameters determined based on the characteristics of the current data, the intelligence of the system is further improved, and it can more accurately analyze the current load requirements for processing nodes, further improving the accuracy of expanding or shrinking the cluster and improving the operating efficiency of the system.

[0126] The present invention also provides a non-volatile computer storage medium storing at least one executable instruction, which can execute an operation that can be achieved by a message transmission optimization system with scalable data processing in any of the above system embodiments.

[0127] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0128] like Figure 5 As shown, the computing device may include: a processor (processor) 502 , a communications interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0129] in:

[0130] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0131] The communication interface 504 is used to communicate with other devices such as clients or other servers.

[0132] The processor 502 is used to execute the program 510, and specifically can execute the operations that can be realized by the message transmission optimization system through scalable data processing in the above-mentioned embodiment of the message transmission optimization system through scalable data processing.

[0133] Specifically, the program 510 may include program codes, and the program codes include computer operation instructions.

[0134] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0135] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0136] The program 510 can be specifically used to enable the processor 502 to perform an operation that can be implemented by a message transmission optimization system through scalable data processing in any of the above-mentioned system embodiments. The specific implementation of each step in the program 510 can refer to the corresponding descriptions in the corresponding steps and units in the above-mentioned message transmission optimization system embodiment through scalable data processing, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the aforementioned method embodiment, which will not be repeated here.

[0137] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0138] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0139] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the individual embodiments previously disclosed. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0140] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0141] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0142] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0143] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A message transmission optimization system through scalable data processing, comprising: Data collection module, data access module, data processing module and cluster management module; among them, The data acquisition module is used to collect device data based on edge nodes and perform data preprocessing; The data access module is used to access data based on a distributed architecture and complete data synchronization within all cluster nodes; The data processing module is used to automatically adjust the number of cluster nodes according to the current load based on the microservice architecture; The cluster management module is used to perform heartbeat detection on each node in the cluster and perform failover on nodes with faults.

2. A message transmission optimization system through scalable data processing according to claim 1, characterized in that: The edge node is arranged on the device side, and the data acquisition module is further used for: Determine the device corresponding to each edge node, and each edge node obtains the device data of its corresponding device; Performing data preprocessing on the acquired device data to obtain preprocessed data; wherein the data preprocessing at least includes: data cleaning and / or normalization; Based on the lossless compression algorithm, the preprocessed data is losslessly compressed.

3. A message transmission optimization system through scalable data processing according to claim 1, characterized in that: The data access module is further used for: A pre-set recording interface between nodes in the cluster; Using the recording interface between each node, the device data acquired and processed by the edge node is transmitted; The device data obtained and processed by the edge nodes is backed up and stored in each node.

4. A message transmission optimization system through scalable data processing according to claim 1, characterized in that: The data processing module is further used for: Determine target indicators for capacity expansion or reduction based on device data; According to the target indicators, use the load balancer in the microservice architecture and the preset load balancing technology to determine the cluster node adjustment method; Expand or reduce the cluster capacity based on the cluster node adjustment method.

5. A message transmission optimization system through scalable data processing according to claim 1, characterized in that: The cluster management module is further used to: Monitor each node in the cluster and obtain the heartbeat information sent by each node according to the preset period; Determine whether a node is in an abnormal state based on the heartbeat information of each node; For nodes with abnormalities, failover is performed based on a preset failover strategy; wherein the failover strategy is a master-slave replication strategy, a multi-master replication strategy or a data sharding strategy.

6. A message transmission optimization system through scalable data processing according to claim 1, characterized in that: The data processing module is further used for: According to the load status of nodes in the cluster, a node load adjustment model is constructed; Collecting historical load information of past data processing in the cluster, using the historical load information to train a node load adjustment model, and obtaining a trained node load adjustment model; Based on the trained node load adjustment model, the node load status in the cluster is determined and adjusted.

7. A message transmission optimization system through scalable data processing according to claim 6, characterized in that: The data processing module is further used for: The node load adjustment model determines the node load state based on the node data processing performance parameter, and the node data processing performance parameter is Among them, P N is the node data processing performance parameter; x is the total number of tasks processed by the node; M i The amount of data when synchronizing data for the i-th task; No ei is the node CPU idle rate parameter corresponding to the i-th task; α is the coefficient of the CPU idle rate parameter; Sa ei The idle rate parameter is stored for the node corresponding to the i-th task; β is the coefficient of the stored idle rate parameter.

8. A method for optimizing message transmission through scalable data processing, comprising: Collect device data based on edge nodes and perform data preprocessing; Data access is performed based on a distributed architecture, and data synchronization is completed across all cluster nodes; Based on the microservice architecture, the number of cluster nodes is automatically adjusted according to the current load; Perform heartbeat detection on each node in the cluster and perform failover on failed nodes.

9. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation implemented by the message transmission optimization system through scalable data processing as described in any one of claims 1-7.

10. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, wherein the executable instruction enables a processor to execute the operation implemented by the message transmission optimization system through scalable data processing as described in any one of claims 1 to 7.

Images