Data processing method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a nonvolatile storage medium and electronic equipment. The method comprises the steps that under the condition that gateway equipment is detected to report messages, the first message capacity of each gateway management platform deployed in a cluster is determined, the first message capacity is used for indicating the message processing capacity of the gateway management platform, and the message processing capacity of the gateway management platform is in positive correlation with the message processing capacity; and a target server is determined in the cluster according to the first message capacity, the target server is used for receiving messages reported by the gateway equipment, and the target server carries out sequential processing and hierarchical caching on the messages reported by the gateway equipment. According to the method and the device, the technical problems of long message processing delay time and message blocking when a gateway equipment message processing scheme of a related technology is applied to a high-concurrency message processing scene are solved.

Description

Technical Field

[0001] The present application relates to the field of intelligent gateway technology, and more specifically, to a data processing method and device, a non-volatile storage medium, and an electronic device. Background Art

[0002] With the development of Internet technology, traditional network access methods can no longer meet the growing demand for network quality and speed. Fiber To The Room (FTTR) equipment, which extends directly to users' homes or businesses through optical fibers, significantly improves the stability and transmission speed of network signals, and has become the preferred networking solution for improving network quality, meeting high bandwidth requirements, and promoting industrial upgrading. A large number of FTTR devices (i.e., gateway devices) are deployed in the centralized information and communication technology (ICT) management platform, and the gateway devices will generate a large amount of reporting information. For example, if the gateway device reports status information every 5 seconds (s), the system will generate a throughput of more than 200,000, and the processing volume may be higher under abnormal circumstances. In the related technology, for the above-mentioned high-concurrency message processing scenario, there are problems of message processing delay and blocking.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and device, a non-volatile storage medium, and an electronic device to at least solve the technical problems of long message processing delay and message blocking when the gateway device message processing solution of the related technology is applied in a high-concurrency message processing scenario.

[0005] According to one aspect of an embodiment of the present application, a data processing method is provided, including: in the case of detecting a message reported by a gateway device, determining a first message capacity of each gateway management platform deployed in a cluster, wherein the first message capacity is used to indicate the message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the message processing capability; determining a target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time-series processing and hierarchical caching on the message reported by the gateway device.

[0006] Optionally, determining the target server in the cluster based on the first message capacity includes: comparing the first message capacities of all gateway management platforms deployed in the cluster to obtain a comparison result; when the comparison result indicates that the first message capacities of all gateway management platforms are the same, determining the target server in the cluster based on the first message forwarding strategy; when the comparison result indicates that the first message capacities of all gateway management platforms are not completely the same, determining the target server in the cluster based on the second message forwarding strategy, wherein the methods for determining the reception probability in the first message forwarding strategy and the second message forwarding strategy are different, and the reception probability is the probability that the server receives the message reported by the gateway device.

[0007] Optionally, determining a target server in a cluster according to a first message forwarding strategy includes: for each server, determining the number of clients of the server, wherein the number of clients is the number of gateway management platforms connected to the server; determining the product of the number of clients and the first message capacity, and determining a first ratio of the product to the total number, wherein the total number is the number of all gateway management platforms included in the cluster, and the first ratio is used to indicate the reception probability of the server, and the first ratio is positively correlated with the reception probability; determining the target server according to multiple first ratios corresponding to multiple servers.

[0008] Optionally, the target server is determined based on multiple first ratios corresponding to multiple servers, including: obtaining a random number, and accumulating the multiple first ratios in order from large to small, wherein each time the accumulation is performed, the accumulated result and the random number are compared until the accumulated result is greater than or equal to the random number, and the accumulation is stopped; and the server corresponding to the last first ratio accumulated is determined as the target server.

[0009] Optionally, a target server is determined in the cluster according to a second message forwarding strategy, including: for each server, determining a second message capacity of the server according to the number of gateway management platforms connected to the server and a first message capacity of each gateway management platform, wherein the second message capacity is used to indicate the message processing capability of the server, and the message processing capability of the server is positively correlated with the second message capacity; determining a third message capacity of the cluster according to all gateway management platforms deployed in the cluster and the first message capacity of each gateway management platform, wherein the third message capacity is used to indicate the message processing capability of the cluster, and the message processing capability of the cluster is positively correlated with the third message capacity; determining a second ratio of the second message capacity to the third message capacity, wherein the second ratio is used to indicate the reception probability of the server, and the second ratio is positively correlated with the reception probability; and determining the target server according to multiple second ratios corresponding to multiple servers.

[0010] Optionally, the target server is determined based on multiple second ratios corresponding to multiple servers, including: obtaining a random number, and accumulating the multiple second ratios in order from large to small, wherein each time the accumulation is performed, the accumulated result and the random number are compared until the accumulated result is greater than or equal to the random number, and the accumulation is stopped; and the server corresponding to the last accumulated second ratio is determined as the target server.

[0011] Optionally, the messages reported by the gateway device are processed in a time-series manner, including: for each message, obtaining the address mapping value carried by the message, and determining the reporting time of the message, wherein the address mapping value is the mapping value of the media access control MAC address of the gateway device that reports the message; and storing messages containing the same address mapping value in order of reporting time from earliest to latest as a message sequence.

[0012] Optionally, the cluster also includes: multiple middlewares, wherein the middlewares include at least: a first type of middleware for temporarily storing messages reported by gateway devices, and a second type of middleware for permanently storing messages reported by gateway devices; performing hierarchical caching on messages reported by gateway devices, including: after receiving the messages reported by the gateway devices, storing all the messages in the first type of middleware; and dumping the messages to the second type of middleware when the time length for which the messages are stored in the first type of middleware is greater than a preset time length.

[0013] According to another aspect of an embodiment of the present application, a data processing device is also provided, including: a detection module, used to determine a first message capacity of each gateway management platform deployed in the cluster when a message reported by a gateway device is detected, wherein the first message capacity is used to indicate the message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the message processing capability; a determination module, used to determine a target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time-series processing and hierarchical caching on the message reported by the gateway device.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a computer program is stored, wherein the above-mentioned data processing method is executed by running the computer program on a device where the non-volatile storage medium is located.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned data processing method through the computer program.

[0016] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0017] In an embodiment of the present application, when a message reported by a gateway device is detected, a first message capacity of each gateway management platform deployed in the cluster is determined, wherein the first message capacity is used to indicate the message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the message processing capability; a target server is determined in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time-series processing and hierarchical caching on the message reported by the gateway device, and when forwarding the message reported by the gateway device, a server for processing the message reported by the gateway device is determined in the cluster by a resource hierarchical forwarding method, wherein the resource hierarchical The forwarding algorithm takes into account the ability of the gateway management platform connected to the server to process messages, and achieves the purpose of considering the message processing ability of the gateway management platform when forwarding messages reported by the gateway. Therefore, the messages reported by the gateway device can be forwarded to the server with stronger message processing capabilities for processing, thereby achieving the technical effect of improving the processing speed of messages reported by the gateway device; in addition, when processing messages reported by the gateway device, a hierarchical caching strategy is adopted, which can effectively avoid message blocking and achieve the technical effect of improving message transmission efficiency, thereby solving the technical problems of long message processing delay and message blocking when the gateway device message processing solution of the related technology is applied to high-concurrency message processing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a data processing method according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of a method for processing data according to an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of an information interaction process between a gateway device and a gateway management platform according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of the architecture of a cluster according to an embodiment of the present application;

[0023] Figure 5It is a schematic diagram of a flow chart of a server processing a message reported by a gateway device according to an embodiment of the present application;

[0024] Figure 6 It is a structural diagram of a data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0028] Message Queuing Telemetry Transport (MQTT): is a client-server based message publish / subscribe transport protocol.

[0029] In the related art, the message reported by the gateway device is only forwarded to the message broker server (Broker) directly connected to the gateway device, and then the server forwards the message to the gateway management platform connected to it. Therefore, in the scenario of high concurrent message processing, there are problems such as message processing delay and blocking and low message transmission efficiency. In order to solve this problem, the embodiments of the present application provide relevant solutions, which are described in detail below.

[0030] According to an embodiment of the present application, a method embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a data processing method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0032] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0036] The present application embodiment provides a data processing method that can be run in the above-mentioned operating environment. Figure 2 is a flowchart of the steps of the data processing method provided in the embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0037] Step S202, when detecting that the gateway device reports a message, determine the first message capacity of each gateway management platform deployed in the cluster, wherein the first message capacity is used to indicate the message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the message processing capability.

[0038] Figure 3 Schematic diagram of the information interaction process between the gateway device and the gateway management platform. The method provided in the embodiment of the present application can be applied to the scenario of high concurrent message processing, for example, Figure 3 The information and communication technology function management platform for centrally managing multiple FTTR devices is shown in FIG. Figure 3As shown, in an information and communication technology function management platform for centralized management of multiple FTTR devices, multiple FTTR devices and multiple gateway management platforms (Data management platform, DMP), as well as multiple message proxy servers (MQTT Broker) that follow the message queue telemetry transmission protocol are deployed; the DMP interacts with the FTTR device through the message proxy server (MQTT Broker) to process the status and performance-related information of the gateway reported by the FTTR device, and to issue configuration commands to the FTTR device. Since the method provided in the embodiment of the present application comprehensively considers the data processing capability of the gateway platform to determine how to forward the message reported by the gateway device, in step S202, after the server (MQTT Broker) detects the message reported by the FTTR device, it first detects the message capacity (i.e., the first message capacity) of each gateway management platform included in the cluster at the current moment, wherein the (first) message capacity of the gateway management platform represents the data processing capability of the gateway management platform, for example, the number of messages processed per second by the gateway management platform, and the larger the value of the (first) message capacity of the gateway management platform, the stronger the data processing capability of the gateway management platform. Figure 4 is a schematic diagram of the cluster architecture. The cluster includes multiple FTTR devices, multiple gateway management platforms and multiple message broker servers (MQTT Broker). Figure 4 As shown in FIG. 1 , node B representing the message broker server (MQTT Broker) acts as both a cluster head (used as a local controller) and a cluster gateway (used as a bridge between two clusters). Node C representing the gateway management platform is connected to the message broker server (MQTT Broker) to receive messages reported by FTTR devices and to generate configuration commands for FTTR by processing the messages reported by FTTR devices. Figure 4 There are no loose nodes in the cluster shown, that is, the message reported by FTTR can be forwarded to different message proxy servers, and then forwarded to any gateway management platform connected to the message proxy server. In addition, the number of messages that each gateway management platform can receive is proportional to its capacity (i.e., the first message capacity). After receiving the message reported by FTTR, the gateway management platform will simultaneously reduce its remaining capacity. After processing the message, the remaining capacity increases.

[0039] Step S204: determining a target server in the cluster according to the first message capacity, wherein the target server is used to receive messages reported by the gateway device, and the target server performs time-sequential processing and hierarchical caching on the messages reported by the gateway device.

[0040] In step S204, after determining the message capacity (i.e., the first message capacity) of each gateway management platform in the cluster, the server (i.e., the target server) used to receive the message reported by the FTTR device is determined in the cluster according to the connection relationship between the gateway management platform and the server (MQTT Broker), wherein the selected server (i.e., the target server) will follow the timing and process the messages reported by the FTTR device in sequence from early to late in the order of the time when the messages are reported by the FTTR, based on hierarchical caching according to the attributes of the messages, wherein the attributes of the messages include: temporary storage of messages and long-term storage of messages.

[0041] Optionally, determining the target server in the cluster based on the first message capacity includes: comparing the first message capacities of all gateway management platforms deployed in the cluster to obtain a comparison result; when the comparison result indicates that the first message capacities of all gateway management platforms are the same, determining the target server in the cluster based on the first message forwarding strategy; when the comparison result indicates that the first message capacities of all gateway management platforms are not completely the same, determining the target server in the cluster based on the second message forwarding strategy, wherein the methods for determining the reception probability in the first message forwarding strategy and the second message forwarding strategy are different, and the reception probability is the probability that the server receives the message reported by the gateway device.

[0042] The method provided in the embodiment of the present application adopts a resource-level forwarding strategy to select a server in the cluster for receiving messages reported by the gateway device. The resource-level forwarding strategy mainly determines the server (i.e., the target server) used to receive the message reported by the FTTR device in the cluster through the connection relationship between the gateway management platform and the server (MQTTBroker), mainly considering the message capacity of the gateway management platform connected to each server. Specifically, in this embodiment, two different message forwarding strategies (i.e., the first message forwarding strategy) and (the second message forwarding strategy) are set according to whether the message capacity of all gateway management platforms included in the cluster is the same. The difference between these two message forwarding strategies is that the algorithms used to determine the probability of each server as a server (i.e., the target server) receiving the message reported by the gateway device are different, wherein the probability of each server being the target server is related to the message capacity of the gateway management platform connected to each server. When the message capacity of all gateway device management platforms in the cluster is the same, the corresponding first message forwarding strategy is adopted to determine the target server in the cluster; when the message capacity of all gateway device management platforms in the cluster is not exactly the same, the corresponding second message forwarding strategy is adopted to determine the target server in the cluster.

[0043] According to an optional embodiment of the present application, a target server is determined in a cluster according to a first message forwarding strategy, including: for each server, determining the number of clients of the server, wherein the number of clients is the number of gateway management platforms connected to the server; determining the product of the number of clients and the first message capacity, and determining a first ratio of the product to the total number, wherein the total number is the number of all gateway management platforms included in the cluster, and the first ratio is used to indicate the reception probability of the server, and the first ratio is positively correlated with the reception probability; determining the target server according to multiple first ratios corresponding to multiple servers.

[0044] As mentioned in the above embodiment, when the message capacities of all the gateway device management platforms in the cluster are the same, the probability of each server in the cluster being selected as the target server is determined in the following manner to further select the target server in the cluster. In this embodiment, when the message capacities of all the gateway device management platforms in the cluster are the same, if the number of gateway management platforms connected to the server is the same, the message processing capabilities of the servers are also exactly the same; according to the formula Determine the probability BM of each server being selected as the target server k , where k represents the kth server, m represents the number of gateway management platforms connected to the kth server (i.e., the number of clients), t represents the message capacity of the gateway device management platform (i.e., the first message capacity), and n represents the total number of gateway device management platforms contained in the cluster; in this case, the number of gateway management platforms connected to the server (Broker) is proportional to the number of messages that the server can process. After determining the probability of each server in the cluster being selected as the target server through the above method, the target server for receiving the message reported by the gateway device is further determined according to the probability corresponding to each server.

[0045] Optionally, the target server is determined based on multiple first ratios corresponding to multiple servers, including: obtaining a random number, and accumulating the multiple first ratios in order from large to small, wherein each time the accumulation is performed, the accumulated result and the random number are compared until the accumulated result is greater than or equal to the random number, and the accumulation is stopped; and the server corresponding to the last first ratio accumulated is determined as the target server.

[0046] When further determining the target server for receiving the message reported by the gateway device according to the probability corresponding to each server (i.e., the first ratio), the target server can be determined by a probability polling method or a probability accumulation method. When the probability polling method is adopted, multiple probability values ​​are arranged into a sequence in the order of probability value (i.e., the first ratio) from small to large or from large to small. When selecting for the first time, a probability value is randomly selected from the sequence generated by the probability value arrangement, and a server corresponding to it is used as the target server. When selecting for the next time, the server corresponding to the probability value on the left or right side of the probability value selected last time is used as the target server. This process ends when all probability values ​​are selected once. In each round, each server serves as the target server only once. When the probability accumulation method is used, a random number with a value in the interval [0,1] is generated, and the probability values ​​are accumulated in order starting from the first probability value in the sequence generated by arranging the probability values ​​(i.e., the first ratio) from large to small. After accumulating a probability value each time, the size relationship between the random number and the accumulated result is compared until the accumulated result is greater than or equal to the random number, and the accumulation is stopped immediately. At this time, the server corresponding to the probability value accumulated for the last time (i.e., the first ratio) is selected as the (target) server that receives the information reported by the gateway device.

[0047] According to another optional embodiment of the present application, a target server is determined in a cluster based on a second message forwarding strategy, including: for each server, determining the second message capacity of the server based on the number of gateway management platforms connected to the server and the first message capacity of each gateway management platform, wherein the second message capacity is used to indicate the message processing capability of the server, and the message processing capability of the server is positively correlated with the second message capacity; determining the third message capacity of the cluster based on all gateway management platforms deployed in the cluster and the first message capacity of each gateway management platform, wherein the third message capacity is used to indicate the message processing capability of the cluster, and the message processing capability of the cluster is positively correlated with the third message capacity; determining a second ratio of the second message capacity to the third message capacity, wherein the second ratio is used to indicate the reception probability of the server, and the second ratio is positively correlated with the reception probability; determining the target server based on multiple second ratios corresponding to multiple servers.

[0048] In this embodiment, when the message capacities of all gateway device management platforms in the cluster are not exactly the same, it is necessary to determine the message processing capacity of each server separately when selecting the target server. Specifically, according to the formula Determine the message capacity of each server (ie, the second message capacity) BR k , where k represents the kth server, t jrepresents the message capacity (i.e., the first message capacity) of each gateway management platform j, and m is the total number of gateway management platforms connected to server k. In the resource hierarchical forwarding strategy adopted in the embodiment of the present application, the larger the message capacity (i.e., the second message capacity) of the server, the more messages it can process per second, which also indicates that the server's message processing capability is stronger. The message processing capability of the server is related to the gateway management platform to which it is connected. Since the message capacities of the gateway device management platforms in this embodiment are not exactly the same, the message processing capabilities of the servers are also not exactly the same. In this embodiment, according to the formula Determine the probability of each server being selected as the target server (i.e., the second ratio), where, represents the total message processing capacity of all gateway management platforms deployed in the cluster (i.e., the third message capacity), i represents the i-th gateway management platform, n represents the total number of gateway management platforms deployed in the cluster, and t i Represents the message capacity of the ith gateway management platform. After determining the probability of each server in the cluster being selected as the target server through the above method, the target server for receiving the message reported by the gateway device is further determined according to the probability corresponding to each server.

[0049] Optionally, the target server is determined based on multiple second ratios corresponding to multiple servers, including: obtaining a random number, and accumulating the multiple second ratios in order from large to small, wherein each time the accumulation is performed, the accumulated result and the random number are compared until the accumulated result is greater than or equal to the random number, and the accumulation is stopped; and the server corresponding to the last accumulated second ratio is determined as the target server.

[0050] In the case where the message processing capabilities of the servers are different, when the target server for receiving the message reported by the gateway device is further determined according to the probability corresponding to each server (i.e., the second ratio), a probability accumulation method is adopted. Specifically, in this embodiment, a random number with a value in the interval [0,1] is obtained, and the probability values ​​are accumulated in sequence starting from the first probability value in a sequence generated by arranging multiple probability values ​​(i.e., the second ratio) from large to small. After accumulating a probability value each time, the size relationship between the random number and the accumulated result is compared until the accumulated result is greater than or equal to the random number, and the accumulation is stopped immediately. At this time, the server corresponding to the probability value accumulated for the last time (i.e., the second ratio) is selected as the (target) server for receiving the information reported by the gateway device.

[0051] According to some optional embodiments of the present application, messages reported by a gateway device are processed sequentially, including: for each message, obtaining an address mapping value carried by the message, and determining a reporting time of the message, wherein the address mapping value is a mapping value of a media access control MAC address of the gateway device that reports the message; and storing messages containing the same address mapping value in order of reporting time from earliest to latest as a message sequence.

[0052] The method provided in the embodiment of the present application also optimizes the message processing process. Specifically, a data buffer channel is added to the cluster architecture to execute a load balancing strategy for messages reported by gateway devices; the load balancing strategy refers to storing messages reported by the same gateway device as a message sequence; in this embodiment, after receiving the message reported by the gateway device, the target server identifies the address mapping value carried in the message, wherein the address mapping value carried by each message is generated by mapping the media access control (Media Access Control, MAC) address of the gateway device that reports the message, for example, the address mapping value can be a hash value generated by hash mapping the MAC address, and the same MAC address mapping generates the same hash value; then the target server can determine which messages come from the same gateway device by identifying the address mapping value carried by the message. Furthermore, after identifying the messages from the same gateway device, the information is stored in the order of reporting time from early to late as a message sequence, thereby completing the time-series processing of the messages reported by the gateway device. For example, if the target server recognizes that the hash code carried in the message is h, the identifier i of the message sequence to which the message is forwarded can be determined according to the formula i=MOD(h,n), where n represents the total number of queues contained in the middleware storing the message, and MOD represents "modulus operation". Through the method of this embodiment, all messages reported by the gateway device with the same MAC address are forwarded to the same queue and arranged according to the reporting time, thereby ensuring the timing of message processing.

[0053] According to some other optional embodiments of the present application, the cluster also includes: multiple middlewares, wherein the middlewares include at least: a first type of middleware for temporarily storing messages reported by gateway devices, and a second type of middleware for permanently storing messages reported by gateway devices; performing hierarchical caching on messages reported by gateway devices, including: after receiving the messages reported by the gateway devices, storing all the messages in the first type of middleware; and dumping the messages to the second type of middleware when the time length for which the messages are stored in the first type of middleware is greater than a preset time length.

[0054] Figure 5 This is a flow chart of the server processing messages reported by the gateway device, such as Figure 5As shown, in addition to adding a data buffer channel (msg-sink) to the server, a memory database (Remote Dictionary Server, Redis) with fast storage characteristics and a relational database (MySQL) that supports permanent data storage are added to the cluster architecture. After the target server receives the message from the gateway device, it stores the message from the same gateway device in a queue (queue) according to the reporting time in the message middleware (RocketMQ), and then caches it in Redis (i.e., the first type of middleware) through the message processing component. When the storage time of the data in Redis exceeds the preset delay time (i.e., the preset time), the data carried in the message that needs to be persistently stored is stored in the database MySQL (i.e., the second type of middleware) that supports permanent storage through other message processing components. The data that needs to be persistently stored can be determined based on the frequency with which the data is accessed. For example, when the access frequency is less than the preset frequency, it is determined as data that needs to be persistently stored. Through the method provided in the embodiment of the present application, the information reported by the gateway device is preferentially stored in the high-speed Redis in real time, and all query operations are preferentially performed in the cache, thereby improving query efficiency and reducing the number of database queries; at the same time, by setting a delayed warehousing time (i.e., a preset duration), only the cached data is updated during the delay period, and the data records in the database are updated after the delay is reached; this method greatly reduces the frequency of database operations, effectively solves the problem of decreased service performance of the gateway management platform due to frequent database operations, and improves the speed of message processing.

[0055] Through the above steps, when forwarding messages, the message capacity of the gateway management platform can be taken into consideration, and the messages reported by the gateway device can be forwarded to the gateway management platform with more remaining message capacity, thereby improving the message processing speed and reducing the risk of message blocking; through the hierarchical caching mechanism, the messages reported by the FTTR device in real time are preferentially stored in the high-speed temporary storage device, and all query operations are preferentially performed in the temporary storage device, thereby improving the query efficiency and reducing the number of database queries; at the same time, by setting the delayed warehousing time, only the cached data in the temporary storage device is updated during the delay period, and the data in the database (permanent storage device) is updated after the delay is reached; the frequency of database operations is greatly reduced, and the problem of decreased service performance of the gateway management platform due to frequent database operations is effectively solved.

[0056] Figure 6 is a structural diagram of a data processing device provided according to an embodiment of the present application, such as Figure 6As shown, the data processing device includes: a detection module 60, which is used to determine the first message capacity of each gateway management platform deployed in the cluster when a message reported by a gateway device is detected, wherein the first message capacity is used to indicate the message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the message processing capability; a determination module 62, which is used to determine a target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time-series processing and hierarchical caching on the message reported by the gateway device.

[0057] The above-mentioned data processing device includes a cloud server, a gateway device and a gateway management platform, wherein a database with a data storage function is deployed on the cloud server, and at least a memory database with a fast storage characteristic and a database supporting permanent storage are deployed; the cloud server, the gateway device and the gateway management platform form a cluster, and when the service operation of the cluster is stable, the detection module 60 detects in real time whether the gateway device reports a message, and when it is detected that the gateway device reports a message, the (first) message capacity of each gateway management platform in the cluster is obtained, wherein the (first) message capacity of the gateway management platform is used to indicate the message processing capability of the gateway management platform, such as the number of messages processed per second; next, the determination module 62 determines the message processing capacity of the gateway management platform according to the gateway management platform The message capacity of the platform and the connection relationship between the gateway management platform and the server determine the server (i.e., the target server) used to receive and process the messages reported by the gateway device; in the embodiment of the present application, the cluster adopts a resource hierarchical forwarding strategy to select the target server, and the target server performs time-series processing and hierarchical caching on the messages reported by the gateway device, wherein the time-series processing refers to storing the messages reported by the gateway device with the same MAC address as a sequence in the order defined according to the reporting time; the hierarchical caching refers to storing the messages / data reported by the gateway device in a storage device with fast storage characteristics, and after the storage time reaches the preset time, the messages / data that need to be persistently stored will be transferred to the storage device that supports permanent storage.

[0058] It should be noted that Figure 6 The preferred implementation of the illustrated embodiment can be found in Figure 2 The relevant description of the illustrated embodiment will not be repeated here.

[0059] An embodiment of the present application further provides a non-volatile storage medium, in which a computer program is stored, wherein the above data processing method is executed by running the computer program on a device where the non-volatile storage medium is located.

[0060] The above-mentioned non-volatile storage medium is used to store programs that perform the following functions: when a message reported by a gateway device is detected, determine the first message capacity of each gateway management platform deployed in the cluster, wherein the first message capacity is used to indicate the message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the message processing capability; determine the target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time-series processing and hierarchical caching on the message reported by the gateway device.

[0061] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above data processing method through the computer program.

[0062] The processor in the above-mentioned electronic device is used to run a program that performs the following functions: when a message reported by a gateway device is detected, determine the first message capacity of each gateway management platform deployed in the cluster, wherein the first message capacity is used to indicate the message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the message processing capability; determine the target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time-series processing and hierarchical caching on the message reported by the gateway device.

[0063] The embodiment of the present application also provides a computer program product, including computer instructions, which implement the steps of the above data processing method when executed by a processor.

[0064] It should be noted that the various modules in the above-mentioned data processing device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0065] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0066] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0067] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0068] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0069] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0070] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0071] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: In the case of detecting that the gateway device reports a message, determining a first message capacity of each gateway management platform deployed in the cluster, wherein the first message capacity is used to indicate a message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the message processing capability; A target server is determined in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time-sequential processing and hierarchical caching on the message reported by the gateway device.

2. The method according to claim 1, characterized in that Determining a target server in the cluster according to the first message capacity includes: Comparing the first message capacities of all the gateway management platforms deployed in the cluster to obtain a comparison result; When the comparison result indicates that the first message capacities of all the gateway management platforms are the same, determining the target server in the cluster according to the first message forwarding strategy; When the comparison result indicates that the first message capacities of all the gateway management platforms are not exactly the same, the target server is determined in the cluster according to the second message forwarding strategy, wherein the methods for determining the reception probability in the first message forwarding strategy and the second message forwarding strategy are different, and the reception probability is the probability that the server receives the message reported by the gateway device.

3. The method according to claim 2, characterized in that Determining the target server in the cluster according to the first message forwarding strategy includes: For each of the servers, determining the number of clients of the server, wherein the number of clients is the number of the gateway management platforms connected to the server; Determine a product of the number of clients and the first message capacity, and determine a first ratio of the product to a total number, wherein the total number is the number of all the gateway management platforms included in the cluster, and the first ratio is used to indicate a reception probability of the server, and the first ratio is positively correlated with the reception probability; The target server is determined according to a plurality of the first ratios corresponding to a plurality of the servers.

4. The method according to claim 3, characterized in that Determining the target server according to a plurality of the first ratios corresponding to a plurality of the servers includes: Obtain a random number, and accumulate multiple first ratios in order from large to small, wherein each time the accumulation is performed, the accumulation result is compared with the random number until the accumulation result is greater than or equal to the random number, and the accumulation is stopped; and the server corresponding to the last first ratio accumulated is determined as the target server.

5. The method according to claim 2, characterized in that: Determining the target server in the cluster according to the second message forwarding strategy includes: For each of the servers, determining a second message capacity of the server according to the number of the gateway management platforms connected to the server and the first message capacity of each of the gateway management platforms, wherein the second message capacity is used to indicate the message processing capability of the server, and the message processing capability of the server is positively correlated with the second message capacity; Determining a third message capacity of the cluster according to the first message capacity of all the gateway management platforms and each of the gateway management platforms deployed in the cluster, wherein the third message capacity is used to indicate the message processing capability of the cluster, and the message processing capability of the cluster is positively correlated with the third message capacity; Determine a second ratio of the second message capacity to the third message capacity, wherein the second ratio is used to indicate a reception probability of the server, and the second ratio is positively correlated with the reception probability; The target server is determined according to a plurality of the second ratios corresponding to a plurality of the servers.

6. The method according to claim 5, characterized in that Determining the target server according to a plurality of the second ratios corresponding to the plurality of the servers includes: Obtain a random number, and accumulate multiple second ratios in order from large to small, wherein each time the accumulation is performed, the accumulation result is compared with the random number until the accumulation result is greater than or equal to the random number, and the accumulation is stopped; and the server corresponding to the last accumulated second ratio is determined as the target server.

7. The method according to claim 1, characterized in that Performing time-sequential processing on the message reported by the gateway device, including: For each of the messages, obtaining an address mapping value carried by the message, and determining a reporting time of the message, wherein the address mapping value is a mapping value of a media access control MAC address of a gateway device that reports the message; The messages containing the same address mapping value are stored as a message sequence in order of the reporting time from earliest to latest.

8. The method according to claim 1, characterized in that The cluster further includes: a plurality of middlewares, wherein the middlewares include at least: a first type of middleware for temporarily storing messages reported by the gateway device, and a second type of middleware for permanently storing messages reported by the gateway device; The message reported by the gateway device is cached in a hierarchical manner, including: After receiving the message reported by the gateway device, storing all the messages in the first type of middleware; When the time length during which the message is stored in the first-type middleware is greater than a preset time length, the message is dumped into the second-type middleware.

9. A data processing device, characterized in that: include: A detection module, configured to determine a first message capacity of each gateway management platform deployed in the cluster when a message reported by a gateway device is detected, wherein the first message capacity is used to indicate a message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the message processing capability; A determination module is used to determine a target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time-sequential processing and hierarchical caching on the message reported by the gateway device.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the data processing method according to any one of claims 1 to 8 is executed by running the computer program in the device where the non-volatile storage medium is located.

11. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to execute the data processing method according to any one of claims 1 to 8 through the computer program.

12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the data processing method described in any one of claims 1 to 8 are implemented.

Images