Data processing method and device, nonvolatile storage medium and electronic equipment

Abstract

The application discloses a data processing method and device, a nonvolatile storage medium and an electronic device. The method comprises the following steps: in the case that a gateway device reports a message, determining a first message capacity of each gateway management platform deployed in a cluster, wherein the first message capacity is used to indicate the message processing capacity of the gateway management platform, and the message processing capacity of the gateway management platform is positively correlated with the message processing capacity; determining a target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time sequence processing and hierarchical caching on the message reported by the gateway device. The application solves the technical problems of long message processing delay time and message blocking when the gateway device message processing scheme of the related art is applied to a high-concurrency message processing scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent gateway, in particular to a data processing method and device, a nonvolatile storage medium and an electronic device. BACKGROUND

[0002] With the development of Internet technology, the traditional network access method can no longer meet the growing demand for network quality and speed. Fiber To The Room (FTTR) equipment, which extends directly to the user's home or enterprise through optical fiber, significantly improves the stability and transmission speed of network signals, and becomes an optimal networking solution to improve network quality, meet high bandwidth demand and promote industrial upgrading. For a centralized Information and Communication Technology (ICT) management platform, a large number of FTTR equipment (i.e. gateway equipment) are deployed in the platform, and the gateway equipment will generate a large amount of reporting information. For example, the gateway equipment reports state information every 5 seconds (s), and the system will generate more than 200,000 throughput, and the processing capacity may be higher in abnormal conditions. In related technologies, for the above-mentioned high-concurrency message processing scene, there are problems of message processing delay and blocking.

[0003] At present, no effective solution has been proposed for the above problems. SUMMARY

[0004] The embodiments of the present application provide a data processing method and device, a nonvolatile storage medium and an electronic device, to at least solve the technical problems of long message processing delay time and message blocking when the gateway equipment message processing scheme of related technologies is applied to a high-concurrency message processing scene.

[0005] According to an aspect of an embodiment of the present application, a data processing method is provided, comprising: in the case of detecting that a gateway device reports a message, determining a first message capacity of each gateway management platform deployed in a cluster, wherein the first message capacity is used to indicate the message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the first message capacity; determining a target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time sequence processing and hierarchical caching on the message reported by the gateway device.

[0006] Optionally, determining the target server in the cluster according to the first message capacity comprises: comparing the first message capacities of all gateway management platforms deployed in the cluster to obtain a comparison result; in a case where the comparison result indicates that the first message capacities of all gateway management platforms are the same, determining the target server in the cluster according to a first message forwarding strategy; in a case where the comparison result indicates that the first message capacities of all gateway management platforms are not the same, determining the target server in the cluster according to a second message forwarding strategy, wherein the method for determining the receiving probability is different in the first message forwarding strategy and the second message forwarding strategy, and the receiving probability is the probability of the server receiving the message reported by the gateway device.

[0007] Optionally, determining the target server in the cluster according to the first message forwarding strategy comprises: for each server, determining the number of clients of the server, wherein the number of clients is the number of gateway management platforms connected to the server; determining the product of the number of clients and the first message capacity, and determining the first ratio of the product to the total number, wherein the total number is the number of all gateway management platforms contained in the cluster, the first ratio is used to indicate the receiving probability of the server, and the first ratio is positively correlated with the receiving probability; and determining the target server according to the plurality of first ratios corresponding to the plurality of servers.

[0008] Optionally, determining the target server according to the plurality of first ratios corresponding to the plurality of servers comprises: obtaining a random number, and sequentially accumulating the plurality of first ratios in descending order of the first ratios, wherein each time the accumulation is performed, the accumulation result is compared with the random number until the accumulation result is greater than or equal to the random number, and the accumulation is stopped; and determining the server corresponding to the last accumulated first ratio as the target server.

[0009] Optionally, determining the target server in the cluster according to the second message forwarding strategy comprises: for each server, determining the second message capacity of the server according to the number of gateway management platforms connected to the server and the first message capacity of each gateway management platform, wherein the second message capacity is used to indicate the message processing capability of the server, and the message processing capability of the server is positively correlated with the second message capacity; determining the third message capacity of the cluster according to all gateway management platforms deployed in the cluster and the first message capacity of each gateway management platform, wherein the third message capacity is used to indicate the message processing capability of the cluster, and the message processing capability of the cluster is positively correlated with the third message capacity; determining the second ratio of the second message capacity to the third message capacity, wherein the second ratio is used to indicate the receiving probability of the server, and the second ratio is positively correlated with the receiving probability; and determining the target server according to the plurality of second ratios corresponding to the plurality of servers.

[0010] Optionally, the target server is determined according to the plurality of second ratios corresponding to the plurality of servers, including: obtaining a random number, and sequentially accumulating the plurality of second ratios in descending order of the second ratios, wherein, at each time of accumulation, the accumulation result is compared with the random number, and the accumulation is stopped until the accumulation result is greater than or equal to the random number; and the server corresponding to the last accumulated second ratio is determined as the target server.

[0011] Optionally, the message reported by the gateway device is processed in a time sequence, including: for each message, obtaining an address mapping value carried by the message, and determining a reporting time of the message, wherein the address mapping value is a mapping value of a media access control (MAC) address of the gateway device reporting the message; and messages containing the same address mapping value are stored as a message sequence in descending order of the reporting times.

[0012] Optionally, the cluster further includes: a plurality of middlewares, wherein the middlewares at least include: a first type of middleware for temporarily storing the message reported by the gateway device, and a second type of middleware for permanently storing the message reported by the gateway device; and the message reported by the gateway device is cached in a hierarchical manner, including: after receiving the message reported by the gateway device, storing the message in the first type of middleware; and in a case where the message is stored in the first type of middleware for a time greater than a preset time, transferring the message to the second type of middleware.

[0013] According to another aspect of the embodiments of the present application, a data processing apparatus is further provided, including: a detection module configured to determine a first message capacity of each gateway management platform deployed in a cluster in a case where a message reported by a gateway device is detected, wherein the first message capacity is used to indicate a message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the first message capacity; and a determination module configured to determine a target server in the cluster according to the first message capacity, wherein the target server is configured to receive the message reported by the gateway device, and the target server is configured to process the message reported by the gateway device in a time sequence and cache the message in a hierarchical manner.

[0014] According to another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, and the non-volatile storage medium stores a computer program, wherein a device in which the non-volatile storage medium is located executes the data processing method described above by running the computer program.

[0015] According to another aspect of the embodiments of the present application, an electronic device is further provided, including a memory and a processor, the memory stores a computer program, and the processor is configured to execute the data processing method described above by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is also provided, which comprises computer instructions for implementing the steps of the data processing method described above when executed by a processor.

[0017] In the embodiments of the present application, the first message capacity of each gateway management platform deployed in the cluster is determined when it is detected that the gateway device reports a message, wherein the first message capacity is used to indicate the message processing capacity of the gateway management platform, and the message processing capacity of the gateway management platform is positively correlated with the first message capacity; the target server is determined in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server processes the message reported by the gateway device in a time sequence and in a hierarchical cache manner, and the server for processing the message reported by the gateway device is determined in the cluster by a resource hierarchical forwarding method when the message reported by the gateway device is forwarded, wherein the resource hierarchical forwarding algorithm takes into account the ability of the gateway management platform connected to the server to process the message, so that the message reported by the gateway device can be forwarded to the server with stronger message processing capacity for processing, thereby achieving the technical effect of improving the processing speed of the message reported by the gateway device; in addition, when the message reported by the gateway device is processed, the hierarchical cache strategy is adopted, which can effectively avoid message blocking, thereby achieving the technical effect of improving the message transmission efficiency, and further solving the technical problems of long message processing delay time and message blocking when the gateway device message processing scheme of the related art is applied to a high-concurrency message processing scenario. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and the description thereof, and do not constitute improper limitations on the present application. In the drawings:

[0019] Figure 1 is a hardware structure block diagram of a computer terminal for implementing the data processing method according to the embodiments of the present application;

[0020] Figure 2 is a step flowchart of the data processing method according to the embodiments of the present application;

[0021] Figure 3 is an information interaction flowchart of a gateway device and a gateway management platform according to the embodiments of the present application;

[0022] Figure 4 is an architecture schematic diagram of a cluster according to the embodiments of the present application;

[0023] Figure 5is a flow diagram of a server processing a message reported by a gateway device according to an embodiment of the present application;

[0024] Figure 6 is a structural diagram of a data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0028] Message Queuing Telemetry Transport (MQTT): is a client-server based message publishing / subscription transmission protocol.

[0029] In the related art, only the message reported by the gateway device is forwarded to the message broker server (Broker) directly connected with the gateway device, and then the server forwards the message to the gateway management platform connected therewith, so that in the high-concurrency message processing scenario, there are problems of message processing delay blocking and low message transmission efficiency. In order to solve the problem, the related solutions in the embodiments of the present application are provided, which are described in detail as follows.

[0030] According to the embodiments of the present application, a method embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] The method embodiment provided by the embodiments of the present application can be executed in a mobile terminal, a computer terminal or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing the data processing method is shown. As shown in the figure, Figure 1 the computer terminal 10 can include one or more (shown in the figure as 102a, 102b, …, 102n) processors 102 (the processor 102 can include but not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports in the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that, Figure 1 the structure shown in the figure is only a schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can include more or less components than shown in the figure, or have a different configuration from that shown in the figure. Figure 1 Figure 1

[0032] It should be noted that the one or more processors 102 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any other combination. In addition, the data processing circuit can be a single independent processing module, or any one of the other elements combined into the computer terminal 10 in whole or in part. As referred to in the embodiments of the present application, the data processing circuit as a kind of processor control (for example, the selection of the variable resistance terminal path connected with the interface).

[0033] ​​The memory 104 can be used to store software programs of application software and modules, such as program instructions / data storage devices corresponding to the data processing method in the embodiments of the present application, and the processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the data processing method described above. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to be able to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0035] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the computer terminal 10.

[0036] The embodiments of the present application provide a data processing method which can run in the above-mentioned running environment, Figure 2 is a step flow chart of the data processing method provided by the embodiments of the present application, as Figure 2 shown, the method comprises the following steps:

[0037] In step S202, in the case of detecting that the gateway device reports a message, the first message capacity of each gateway management platform deployed in the cluster is determined, wherein the first message capacity is used to indicate the message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the first message capacity.

[0038] Figure 3 is an information interaction flow diagram of the gateway device and the gateway management platform, and the method provided by the embodiments of the present application can be applied in a high-concurrency message processing scenario, for example, applied in a centralized management information and communication technology function management platform of multiple FTTR devices, as Figure 3 shown, such as Figure 3As shown, in the information and communication technology function management platform for centralized management of multiple FTTR devices, multiple FTTR devices and multiple gateway management platforms (Data management platform, DMP) and multiple message broker servers (MQTT Broker) complying with the message queue telemetry transfer protocol are deployed; the DMP interacts with the FTTR device through the message broker server (MQTT Broker), for processing the state, performance related information of the gateway reported by the FTTR device, and issuing the configuration command for the FTTR device. Since the method provided in the embodiment of the application comprehensively considers the data processing capacity of the gateway platform to determine how to forward the message reported by the gateway device, in step S202, after the server (MQTT Broker) detects the message reported by the FTTR device, it first detects the message capacity (i.e., the first message capacity) of each gateway management platform contained in the cluster at the current time, wherein the (first) message capacity of the gateway management platform represents the data processing capacity of the gateway management platform, for example, the number of messages processed by the gateway management platform per second, and the greater the value of the (first) message capacity of the gateway management platform, the stronger the data processing capacity of the gateway management platform. Figure 4 is an architecture diagram of a cluster, the cluster contains multiple FTTR devices, multiple gateway management platforms and multiple message broker servers (MQTT Broker), wherein, as shown in Figure 4 As shown, the node B representing the message broker server (MQTT Broker) simultaneously acts as a cluster head (for acting as a local controller) and a cluster gateway (for acting as a bridge between two clusters), the node C representing the gateway management platform is connected with the message broker server (MQTT Broker), for receiving the message reported by the FTTR device, and generating the configuration command for the FTTR by processing the message reported by the FTTR device, as shown in Figure 4 As shown in the cluster, there is no loose node, i.e., the message reported by the FTTR can be forwarded to different message broker servers, and then can be forwarded to any one of the gateway management platforms connected with the message broker server. In addition, the number of messages that can be received by each gateway management platform is proportional to its capacity (i.e., the first message capacity), and the gateway management platform will synchronously reduce its remaining capacity after receiving the message reported by the FTTR. After processing the message, the remaining capacity increases.

[0039] In step S204, the target server is determined in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time sequence processing and hierarchical caching on the message reported by the gateway device.

[0040] In step S204, after determining the message capacity of each gateway management platform end in the cluster (i.e., the first message capacity), the server (i.e., the target server) for receiving the message reported by the FTTR device in this cluster is determined according to the connection relationship between the gateway management platform and the server (MQTT Broker), wherein the selected server (i.e., the target server) will follow the time sequence and process the message reported by the FTTR device in turn according to the time from early to late when the message is reported by the FTTR, and the message is classified and cached according to the attribute, wherein the attribute of the message includes temporary storage message and long-term storage message.

[0041] Optionally, determining the target server in the cluster according to the first message capacity includes: comparing the first message capacities of all the gateway management platforms deployed in the cluster to obtain a comparison result; in the case that the comparison result indicates that the first message capacities of all the gateway management platforms are the same, determining the target server in the cluster according to the first message forwarding strategy; in the case that the comparison result indicates that the first message capacities of all the gateway management platforms are not the same, determining the target server in the cluster according to the second message forwarding strategy, wherein the method for determining the receiving probability is different in the first message forwarding strategy and the second message forwarding strategy, and the receiving probability is the probability of the server receiving the message reported by the gateway device.

[0042] The method provided in the embodiment of the application adopts the resource hierarchical forwarding strategy to select the server for receiving the message reported by the gateway device in the cluster, and the resource hierarchical forwarding strategy mainly determines the server (i.e., the target server) for receiving the message reported by the FTTR device in this cluster according to the connection relationship between the gateway management platform and the server (MQTT Broker), mainly considering the message capacity of the gateway management platform connected to each server, specifically, in this embodiment, two different message forwarding strategies (i.e., the first message forwarding strategy) and (the second message forwarding strategy) are set according to whether the message capacities of all the gateway management platforms contained in the cluster are the same, the difference between the two message forwarding strategies is that the algorithms for determining the probability of each server as the server (i.e., the target server) for receiving the message reported by the gateway device are different, wherein the probability of each server as the target server is related to the message capacity of the gateway management platform connected to each server, when the message capacities of all the gateway device management platforms in the cluster are the same, the first message forwarding strategy corresponding thereto is used to determine the target server in the cluster; when the message capacities of all the gateway device management platforms in the cluster are not the same, the second message forwarding strategy corresponding thereto is used to determine the target server in the cluster.

[0043] According to an optional embodiment of the present application, the target server is determined in the cluster according to the first message forwarding strategy, comprising: determining the number of clients of each server, wherein the number of clients is the number of gateway management platforms connected to the server; determining the product of the number of clients and the first message capacity, and determining the first ratio of the product to the total number, wherein the total number is the number of all gateway management platforms included in the cluster, and the first ratio is used to indicate the receiving probability of the server, and the first ratio is positively correlated with the receiving probability; and determining the target server according to the plurality of first ratios corresponding to the plurality of servers.

[0044] As mentioned in the above embodiments, when the message capacities of all gateway device management platforms in the cluster are the same, the probability of each server being selected as the target server in the cluster is determined in the following manner to further select the target server in the cluster. In this embodiment, when the message capacities of all gateway device management platforms in the cluster are the same, if the number of gateway management platforms connected to the server is the same, the message processing capacity of the server is also completely the same; according to the formula determining the probability of each server being selected as the target server , wherein k represents the kth server, m represents the number of gateway management platforms connected to the kth server (i.e. the number of clients), t represents the message capacity of the gateway device management platform (i.e. the first message capacity), and n represents the total number of gateway device management platforms included in the cluster; in this case, the number of gateway management platforms connected to the server (Broker) is positively correlated with the number of messages that can be processed by the server. After determining the probability of each server being selected as the target server in the cluster by the above method, the target server for receiving the message reported by the gateway device is further determined according to the probability corresponding to each server.

[0045] Optionally, the target server is determined according to the plurality of first ratios corresponding to the plurality of servers, comprising: obtaining a random number, and sequentially accumulating the plurality of first ratios in descending order of the first ratios, wherein each time the accumulation is performed, the accumulation result is compared with the random number until the accumulation result is greater than or equal to the random number, and the accumulation is stopped; and the server corresponding to the last first ratio accumulated is determined as the target server.

[0046] In the determination of the target server for receiving the message reported by the gateway device according to the probability corresponding to each server (i.e., the first ratio), the probability polling method can be used to determine the target server, or the probability accumulation method can be used to determine the target server. In the probability polling method, the probability values are arranged in a sequence in ascending or descending order according to the probability values (i.e., the first ratio), in the first selection, a probability value is randomly selected from the sequence generated by the arrangement of the probability values, and the server corresponding to the probability value is selected as the target server, in the next selection, the server corresponding to the probability value on the left or right of the probability value selected in the last selection is selected as the target server, and the round is ended after all the probability values are selected once, in each round, each server is selected as the target server only once. In the probability accumulation method, a random number with a value in the interval [0, 1] is obtained, in the sequence generated by arranging the probability values (i.e., the first ratio) in descending order, the probability values are sequentially accumulated from the first probability value, after each probability value is accumulated, the size relationship between the random number and the accumulated result is compared, and the accumulation is stopped immediately when the accumulated result is greater than or equal to the random number, at this time, the server corresponding to the last accumulated probability value (i.e., the first ratio) is selected as the target server for receiving the information reported by the gateway device.

[0047] According to another optional embodiment of the present application, the target server is determined in the cluster according to the second message forwarding strategy, including: for each server, determining the second message capacity of the server according to the number of gateway management platforms connected to the server and the first message capacity of each gateway management platform, wherein the second message capacity is used to indicate the message processing capacity of the server, and the message processing capacity of the server is positively correlated with the second message capacity; determining the third message capacity of the cluster according to all the gateway management platforms deployed in the cluster and the first message capacity of each gateway management platform, wherein the third message capacity is used to indicate the message processing capacity of the cluster, and the message processing capacity of the cluster is positively correlated with the third message capacity; determining the second ratio of the second message capacity to the third message capacity, wherein the second ratio is used to indicate the receiving probability of the server, and the second ratio is positively correlated with the receiving probability; and determining the target server according to the plurality of second ratios corresponding to the plurality of servers.

[0048] In the present embodiment, when the message capacities of all the gateway device management platforms in the cluster are not completely the same, the message processing capacity of each server needs to be determined when the target server is selected. Specifically, the message capacity (i.e., the second message capacity) of each server is determined according to the formula wherein k represents the kth server, ​Here, m represents the message capacity (i.e., the first message capacity) of each gateway management platform j, and m is the total number of gateway management platforms connected to server k. In the resource hierarchical forwarding strategy adopted in this embodiment, the larger the server's message capacity (i.e., the second message capacity), the more messages it can process per second, indicating a stronger message processing capability. The server's message processing capability is related to the gateway management platforms it connects to. Since the message capacities of the gateway device management platforms are not entirely the same in this embodiment, the server's message processing capabilities are also not entirely the same. In this embodiment, according to the formula... Determine the probability (i.e., the second ratio) of each server being selected as the target server, where, This represents the total message processing capacity (i.e., the third message capacity) of all gateway management platforms deployed in the cluster, where i represents the i-th gateway management platform and n represents the total number of gateway management platforms deployed in the centralized system. This represents the message capacity of the i-th gateway management platform. After determining the probability of each server in the cluster being selected as the target server using the above method, the target server for receiving messages reported by the gateway device is further determined based on the probability corresponding to each server.

[0049] Optionally, determining the target server based on multiple second ratios corresponding to multiple servers includes: obtaining a random number and sequentially accumulating multiple second ratios in descending order, wherein each time the accumulation occurs, the accumulation result is compared with the random number until the accumulation result is greater than or equal to the random number, at which point the accumulation stops; and determining the server corresponding to the last second ratio to be accumulated as the target server.

[0050] When servers have different message processing capabilities, and the target server for receiving messages reported by the gateway device is further determined based on the probability (i.e., the second ratio) corresponding to each server, a probability accumulation method is adopted. Specifically, in this embodiment, a random number with a value in the interval [0,1] is generated. Starting from the first probability value, the probability values ​​are accumulated sequentially in a sequence generated by arranging multiple probability values ​​(i.e., the second ratio) from largest to smallest. After accumulating a probability value each time, the size relationship between the random number and the accumulated result is compared until the accumulated result is greater than or equal to the random number. At this time, the accumulation is stopped immediately. Then, the server corresponding to the last accumulated probability value (i.e., the second ratio) is selected as the (target) server for receiving information reported by the gateway device.

[0051] According to some optional embodiments of the present application, the message reported by the gateway device is processed in time sequence, including: for each message, obtaining an address mapping value carried by the message, and determining the reporting time of the message, wherein the address mapping value is a mapping value of a Media Access Control (MAC) address of the gateway device reporting the message; storing the messages containing the same address mapping value in the order of reporting time from early to late as a message sequence.

[0052] The method provided by the embodiments of the present application also optimizes the message processing flow. Specifically, a data buffering channel is added in the architecture of the cluster, which is used to perform a load balancing strategy on the messages reported by the gateway device; the load balancing strategy refers to storing the messages reported by the same gateway device as a message sequence; in the embodiments, after receiving the message reported by the gateway device, the target server identifies the address mapping value carried in the message, wherein the address mapping value carried by each message is generated by mapping the Media Access Control (MAC) address of the gateway device reporting the message, for example, the address mapping value can be a hash value generated by hashing the MAC address, and the same MAC address mapping generates the same hash value; then the target server can determine which messages come from the same gateway device by identifying the address mapping value carried in the message; further, after identifying the messages from the same gateway device, the messages are stored in the order of reporting time from early to late as a message sequence, thereby completing the time sequence processing of the messages reported by the gateway device. For example, the target server identifies that the hash code carried in the message is h, and then according to the formula i=MOD (h, n), the identification i of the message sequence to which the message is forwarded can be determined, wherein n represents the total number of queues included in the middleware for storing messages, and MOD represents the "modulus operation". Through the method of the embodiments, all messages reported by the gateway devices with the same MAC address are forwarded to the same queue, and are arranged according to the reporting time, thereby ensuring the time sequence of a message processing.

[0053] According to some optional embodiments of the present application, the cluster further includes: a plurality of middlewares, wherein the middlewares at least include: a first type of middleware for temporarily storing the messages reported by the gateway device, and a second type of middleware for permanently storing the messages reported by the gateway device; the messages reported by the gateway device are processed in a hierarchical cache, including: after receiving the message reported by the gateway device, storing the message in the first type of middleware; in the case that the time length of the message stored in the first type of middleware is greater than a preset time length, transferring the message to the second type of middleware.

[0054] Figure 5 is a flowchart of the server processing the messages reported by the gateway device, as Figure 5As shown, in addition to adding a data buffer channel (msg-sink) to the server, a memory database (Remote Dictionary Server, Redis) with fast storage characteristics is added in the architecture of the cluster to support a relational database (MySQL) for permanently storing data. After the target server receives the message of the gateway device, the message from the same gateway device is stored in a queue in the message middleware (RocketMQ) according to the reporting time, and then the message is cached in Redis (i.e., the first type of middleware) through a message processing component. When the storage time of the data in Redis exceeds the preset delay time (i.e., the preset time), the message carrying the data that needs to be persistently stored is stored in the database MySQL (i.e., the second type of middleware) supporting persistent storage through other message processing components. The data that needs to be persistently stored can be determined according to the frequency of data access, for example, when the access frequency is less than the preset frequency, the data is determined to be persistently stored. Through the method provided in the embodiment of the application, the information reported by the gateway device is preferentially stored in the high-speed Redis in real time, and all query operations are preferentially performed in the cache, thereby improving the query efficiency and reducing the number of database queries. At the same time, by setting the delay storage time (i.e., the preset time), only the cache data is updated during the delay period, and the data record in the database is updated only after the delay is reached. This method greatly reduces the database operation frequency, effectively solves the problem of service performance degradation of the gateway management platform caused by frequent database operations, and improves the message processing speed.

[0055] Through the above steps, when forwarding the message, the message capacity of the gateway management platform is considered, the message reported by the gateway device is forwarded to the gateway management platform with more remaining message capacity, the message processing speed is improved, and the message blocking risk is reduced. Through the hierarchical caching mechanism, the message reported by the FTTR device in real time is preferentially stored in the high-speed temporary storage device, all query operations are preferentially performed in the temporary storage device, thereby improving the query efficiency and reducing the number of database queries. At the same time, by setting the delay storage time, only the cache data in the temporary storage device is updated during the delay period, and the data in the database (permanent storage device) is updated only after the delay is reached. The database operation frequency is greatly reduced, and the problem of service performance degradation of the gateway management platform caused by frequent database operations is effectively solved.

[0056] Figure 6 is a structural diagram of a data processing device according to an embodiment of the application, as Figure 6As shown, the data processing apparatus comprises a detection module 60 configured to determine a first message capacity of each gateway management platform deployed in the cluster in a case where it is detected that the gateway device reports a message, wherein the first message capacity is used to indicate a message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the first message capacity; and a determination module 62 configured to determine a target server in the cluster according to the first message capacity, wherein the target server is configured to receive the message reported by the gateway device, and the target server performs time sequence processing and hierarchical caching on the message reported by the gateway device.

[0057] The data processing apparatus described above comprises a cloud server, a gateway device and a gateway management platform, wherein the cloud server is deployed with a database having a data storage function, at least a memory database having a fast storage feature and a database supporting permanent storage; the cloud server, the gateway device and the gateway management platform form a cluster, when the service of the cluster runs stably, the detection module 60 detects whether the gateway device reports a message in real time, when it is detected that the gateway device reports a message, the (first) message capacity of each gateway management platform in the cluster is obtained, wherein the (first) message capacity of the gateway management platform is used to indicate the message processing capability of the gateway management platform, such as the number of messages processed per second; next, the determination module 62 determines a server (i.e. a target server) for receiving and processing the message reported by the gateway device according to the message capacity of the gateway management platform and the connection relationship between the gateway management platform and the server; in the embodiment of the application, the target server is selected by the cluster using a resource hierarchical forwarding strategy, and the target server performs time sequence processing and hierarchical caching on the message reported by the gateway device, wherein the time sequence processing means that the messages reported by the gateway device with the same MAC address are stored as a sequence according to the order defined according to the reporting time; the hierarchical caching means that the messages / data reported by the gateway device are stored in a storage device having a fast storage feature, and after the storage duration reaches a preset duration, the messages / data that need to be persistently stored are transferred to a storage device supporting permanent storage.

[0058] It should be noted that, Figure 6 The preferred implementation of the embodiment shown can be seen from Figure 2 The related description of the embodiment shown will not be repeated here.

[0059] The embodiment of the application also provides a non-volatile storage medium, and the non-volatile storage medium stores a computer program, wherein a device in which the non-volatile storage medium is located executes the above data processing method by running the computer program.

[0060] The nonvolatile storage medium is configured to store a program for performing the following functions: determining a first message capacity of each gateway management platform deployed in a cluster in a case where it is detected that a gateway device reports a message, wherein the first message capacity is used to indicate a message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the first message capacity; and determining a target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time series processing and hierarchical caching on the message reported by the gateway device.

[0061] The embodiment of the present application further provides an electronic device, including a memory and a processor, the memory stores a computer program, and the processor is configured to execute the data processing method by the computer program.

[0062] The processor in the electronic device is configured to execute a program for performing the following functions: determining a first message capacity of each gateway management platform deployed in a cluster in a case where it is detected that a gateway device reports a message, wherein the first message capacity is used to indicate a message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the first message capacity; and determining a target server in the cluster according to the first message capacity, wherein the target server is used to receive the message reported by the gateway device, and the target server performs time series processing and hierarchical caching on the message reported by the gateway device.

[0063] The embodiment of the present application further provides a computer program product, including computer instructions, and the computer instructions are executed by a processor to implement the steps of the data processing method.

[0064] It should be noted that each module in the data processing apparatus described above can be a program module (for example, a set of program instructions for implementing a certain specific function) or a hardware module, and for the latter, it can be in the following form, but is not limited to: the form of each module is a processor, or the functions of each module are implemented by a processor.

[0065] The serial numbers of the embodiments of the present application described above are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0066] In the above embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0067] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other ways. Among them, the above-described device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0068] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0069] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0070] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the essential part or all or part of the technical solutions that make contributions to the related art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0071] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method of processing data, characterized by, Comprise: In the case of detecting the gateway device reported message, determine the first message capacity of each gateway management platform deployed in the cluster, wherein the first message capacity is used to indicate the message processing capacity of the gateway management platform, and the message processing capacity of the gateway management platform is positively correlated with the first message capacity; According to the first message capacity in the cluster to determine the target server, including: when the first message capacity of all gateway device management platforms in the cluster is the same, using the first message forwarding strategy Determine the target server in the cluster, including: for each server, determine the number of clients of the server, wherein the number of clients is the number of gateway management platforms connected to the server; according to the number of clients and the first message capacity, determine the probability of each server being selected as the target server, and determine the target server according to the probability; When the first message capacity of all gateway device management platforms in the cluster is not completely the same, using the second message forwarding strategy to determine the target server in the cluster, including: determining the second message capacity of each server, and determining the third message capacity of the cluster according to the first message capacity; according to the second message capacity and the third message capacity, determine the receiving probability of the server, and determine the target server according to the corresponding receiving probability of each server; Wherein, the target server is used to receive the message reported by the gateway device, and the target server performs time sequence processing and hierarchical caching on the message reported by the gateway device.

2. The method of claim 1, wherein, According to the first message capacity in the cluster to determine the target server, including: Compare the first message capacity of all gateway management platforms deployed in the cluster to obtain a comparison result; In the case that the comparison result indicates that the first message capacity of all gateway management platforms is the same, determine the target server in the cluster according to the first message forwarding strategy; In the case that the comparison result indicates that the first message capacity of all gateway management platforms is not completely the same, determine the target server in the cluster according to the second message forwarding strategy, wherein the method for determining the receiving probability in the first message forwarding strategy and the second message forwarding strategy is different, and the receiving probability is the probability that the server receives the message reported by the gateway device.

3. The method of claim 1, wherein, According to the number of clients and the first message capacity, determine the probability of each server being selected as the target server, including: Determine the product of the number of clients and the first message capacity, and determine the first ratio of the product to the total number, wherein the total number is the number of all gateway management platforms contained in the cluster, and the first ratio is used to indicate the receiving probability of the server, and the first ratio is positively correlated with the receiving probability.

4. The method of claim 3, wherein, According to a plurality of first ratios corresponding to a plurality of servers, determine the target server, including: Obtaining a random number, and sequentially accumulating a plurality of the first ratios in descending order according to the first ratios, wherein, at each time of accumulation, comparing the accumulation result with the random number until the accumulation result is greater than or equal to the random number, and stopping the accumulation; determining the server corresponding to the last accumulated first ratio as the target server.

5. The method of claim 1, wherein, Determining a second message capacity of each of the servers, comprising: for each of the servers, determining the second message capacity of the server according to the number of the gateway management platforms connected with the server and the first message capacity of each of the gateway management platforms, wherein the second message capacity is used to indicate the message processing capacity of the server, and the message processing capacity of the server is positively correlated with the second message capacity; Determining a third message capacity of the cluster according to the first message capacity, comprising: determining the third message capacity of the cluster according to all the gateway management platforms deployed in the cluster and the first message capacity of each of the gateway management platforms, wherein the third message capacity is used to indicate the message processing capacity of the cluster, and the message processing capacity of the cluster is positively correlated with the third message capacity; Determining a receiving probability of the server according to the second message capacity and the third message capacity, comprising: determining a second ratio of the second message capacity and the third message capacity, wherein the second ratio is used to indicate the receiving probability of the server, and the second ratio is positively correlated with the receiving probability.

6. The method of claim 5, wherein, Determining the target server according to a plurality of second ratios corresponding to a plurality of servers, comprising: Obtaining a random number, and sequentially accumulating a plurality of the second ratios in descending order according to the second ratios, wherein, at each time of accumulation, comparing the accumulation result with the random number until the accumulation result is greater than or equal to the random number, and stopping the accumulation; determining the server corresponding to the last accumulated second ratio as the target server.

7. The method of claim 1, wherein, Performing timing processing on the messages reported by the gateway device, comprising: For each of the messages, obtaining an address mapping value carried by the message, and determining a reporting time of the message, wherein the address mapping value is a mapping value of a media access control (MAC) address of the gateway device reporting the message; Storing the messages containing the same address mapping value as a message sequence in descending order according to the reporting times.

8. The method of claim 1, wherein, The cluster further comprises a plurality of middlewares, wherein the middlewares at least include: a first type of middleware for temporarily storing the messages reported by the gateway device, and a second type of middleware for permanently storing the messages reported by the gateway device; Performing hierarchical caching on the messages reported by the gateway device, comprising: After receiving the message reported by the gateway device, storing the message in the first type of middleware; In the case that the duration of storing the message in the first type of middleware is greater than a preset duration, transferring the message to the second type of middleware.

9. A data processing device, characterized by Comprising: The detection module is configured to determine a first message capacity of each gateway management platform deployed in the cluster when detecting that the gateway device reports a message, wherein the first message capacity is used to indicate a message processing capability of the gateway management platform, and the message processing capability of the gateway management platform is positively correlated with the first message capacity. The determination module is configured to determine a target server in the cluster according to the first message capacity, including: when the first message capacities of all the gateway device management platforms in the cluster are the same, determining the target server in the cluster by using a first message forwarding strategy, including: determining a client quantity of each server, wherein the client quantity is a quantity of the gateway management platforms connected to the server; determining a probability of each server being selected as the target server according to the client quantity and the first message capacity, and determining the target server according to the probability; when the first message capacities of all the gateway device management platforms in the cluster are not completely the same, determining the target server in the cluster by using a second message forwarding strategy, including: determining a second message capacity of each server, and determining a third message capacity of the cluster according to the first message capacities; determining a receiving probability of the server according to the second message capacity and the third message capacity, and determining the target server according to the receiving probability corresponding to each server; wherein the target server is used to receive the message reported by the gateway device, and the target server performs time sequence processing and hierarchical caching on the message reported by the gateway device.

10. A non-volatile storage medium, comprising: The non-volatile storage medium stores a computer program, wherein a device in which the non-volatile storage medium is located executes the data processing method of any one of claims 1 to 8 by running the computer program.

11. An electronic device comprising a memory and a processor, characterized in that The memory stores a computer program, and the processor is configured to execute the data processing method of any one of claims 1 to 8 by using the computer program.

12. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the steps of the data processing method of any one of claims 1 to 8.

Images