Data acquisition method, system and related device

Abstract

The invention discloses a data acquisition method, a data acquisition system and a related device, which can be used for realizing stable data transmission and processing under concurrent connection of more than million levels. The method comprises the steps that an MQTT server receives and caches a data message; the MQTT server sends a query request to an application registration proxy server, the application registration proxy server is used for managing an application acquisition server cluster, and basic parameters of each application acquisition server are pre-stored in the application registration proxy server; when the application registration proxy server receives the query request, dynamically determining a target application acquisition server according to basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster; the application registration proxy server sends a query result to the MQTT server; and the MQTT server pushes the data message to the target application acquisition server according to the query result.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to methods, systems and related devices for data acquisition. Background Art

[0002] Currently, Internet of Things (IoT) technology is rapidly becoming a core part of people's daily lives and industrial operations. Common IoT devices include smart home devices and industrial sensors. These data are very important to enterprises and are the foundation of enterprise IoT data centers. As the number of devices grows exponentially, the amount of data they generate is also increasing significantly, which places higher and higher performance requirements on data acquisition systems.

[0003] In the prior art, most data acquisition architectures use data acquisition devices to directly establish connections with data application servers. However, in high-concurrency situations, a single server can easily become a bottleneck, limiting the scalability and stability of the entire system. In order to alleviate the pressure on a single server, some architectures introduce proxy servers between data acquisition devices and data application servers. The proxy servers can temporarily store data and then forward it to the server in a unified manner, thereby dispersing the load of a single server. However, when faced with millions of concurrent connections, the proxy server cannot handle the huge network traffic due to its own bandwidth limitations, resulting in delays or losses in data transmission, and it is impossible to achieve stable millions of high-load concurrency. Summary of the invention

[0004] The present application provides a data collection method, system and related devices for realizing stable data transmission and processing under more than one million concurrent connections.

[0005] The first aspect of the present application provides a method for data collection, comprising:

[0006] When the application registration proxy server receives a query request sent by the MQTT server, it dynamically determines the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster. The application registration proxy server is used to manage the application acquisition server cluster. The query request is a request sent by the MQTT server after receiving and caching the data message.

[0007] The application registration proxy server sends the query result to the MQTT server, so that the MQTT server pushes the data message to the target application acquisition server.

[0008] Optionally, the method further includes:

[0009] The application registration proxy server receives a registration request sent by a new application collection server, wherein the registration request includes basic parameters of the new application collection server, and the basic parameters include a server address, a port, a service type, and a weight parameter;

[0010] The application registration proxy server adds the new application acquisition server to the application acquisition server cluster according to the registration request, and dynamically monitors the load of the new application acquisition server.

[0011] Optionally, when the application registration proxy server receives the query request sent by the MQTT server, dynamically determining the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster includes:

[0012] When the application registration proxy server receives the query request sent by the MQTT server, it obtains the list of active servers and real-time load conditions in the application collection server cluster;

[0013] The application registration proxy server dynamically determines a target application acquisition server in the active server list based on basic parameters and real-time load conditions of each application acquisition server in the active server list and in combination with a preset access priority strategy.

[0014] Optionally, the application acquisition server cluster includes a customized application acquisition server cluster and a general application acquisition server cluster. Before dynamically determining the target application acquisition server according to basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster, the method further includes:

[0015] The application registration proxy server determines the type of the data message according to the query request;

[0016] The step of dynamically determining the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster includes:

[0017] If the type of the data message is a real-time message, the application registration proxy server dynamically determines a target application acquisition server in the customized application acquisition server cluster according to basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster;

[0018] If the type of the data message is a subscription message, the application registration proxy server dynamically determines a target application acquisition server in the general application acquisition server cluster according to basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster.

[0019] A second aspect of the present application provides a method for data collection, comprising:

[0020] The MQTT server receives and caches data messages;

[0021] The MQTT server sends a query request to the application registration proxy server, so that the application registration proxy server dynamically determines the target server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster;

[0022] The MQTT server receives the query result returned by the application registration proxy server;

[0023] The MQTT server pushes the data message to the target application acquisition server according to the query result.

[0024] Optionally, after the MQTT server receives and caches the data message, the method further includes:

[0025] The MQTT server obtains subscription configuration information of the client associated with the data message, and determines the type of the data message according to the subscription configuration information, where the type of the data message includes a real-time message and a subscription message;

[0026] The MQTT server pushes the data message to the target application acquisition server according to the query result, including:

[0027] If the type of the data message is a real-time message, the MQTT server pushes the data message to the target application acquisition server in real time according to the query result;

[0028] If the type of the data message is a subscription message, the MQTT server pushes the data message to the target application acquisition server according to the query result and the subscription push strategy.

[0029] Optionally, the MQTT server pushes the data message to the target application acquisition server according to the query result, including:

[0030] The MQTT server pushes the data message to the target application acquisition server according to the query result and the link authentication priority.

[0031] The third aspect of the present application provides a method for data collection, characterized in that the method comprises:

[0032] The MQTT server receives and caches data messages;

[0033] The MQTT server sends a query request to the application registration proxy server, the application registration proxy server is used to manage the application acquisition server cluster, and the application registration proxy server pre-stores the basic parameters of each application acquisition server;

[0034] When the application registration proxy server receives the query request, it dynamically determines a target application acquisition server in the application acquisition server cluster according to the basic parameters and the real-time load situation;

[0035] The application registration proxy server sends the query result to the MQTT server;

[0036] The MQTT server pushes the data message to the target application acquisition server according to the query result.

[0037] A fourth aspect of the present application provides an application registration proxy server, including:

[0038] A determination unit, configured to dynamically determine a target application acquisition server according to basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster when receiving a query request sent by the MQTT server, wherein the application registration proxy server is configured to manage the application acquisition server cluster, and the query request is a request sent by the MQTT server after receiving and caching a data message;

[0039] The first sending unit is used to send the query result to the MQTT server, so that the MQTT server pushes the data message to the target application collection server.

[0040] Optionally, the application registration proxy server further includes a registration unit, configured to:

[0041] Receiving a registration request sent by a new application acquisition server, wherein the registration request includes basic parameters of the new application acquisition server, wherein the basic parameters include a server address, a port, a service type, and a weight parameter;

[0042] The new application acquisition server is added to the application acquisition server cluster according to the registration request, and the load condition of the new application acquisition server is dynamically monitored.

[0043] Optionally, the determining unit is specifically configured to:

[0044] When the application registration proxy server receives the query request sent by the MQTT server, it obtains the list of active servers and real-time load conditions in the application collection server cluster;

[0045] The application registration proxy server dynamically determines a target application acquisition server in the active server list based on basic parameters and real-time load conditions of each application acquisition server in the active server list and in combination with a preset access priority strategy.

[0046] Optionally, the application acquisition server cluster includes a customized application acquisition server cluster and a general application acquisition server cluster, and the determining unit is further configured to:

[0047] Determine the type of the data message according to the query request;

[0048] If the type of the data message is a real-time message, the application registration proxy server dynamically determines a target application acquisition server in the customized application acquisition server cluster according to basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster;

[0049] If the type of the data message is a subscription message, the application registration proxy server dynamically determines a target application acquisition server in the general application acquisition server cluster according to basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster.

[0050] A fifth aspect of the present application provides an MQTT server, including:

[0051] A first receiving unit, used for the MQTT server to receive and cache data messages;

[0052] A second sending unit is used to send a query request to the application registration proxy server, so that the application registration proxy server dynamically determines the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster;

[0053] A second receiving unit, configured to receive the query result returned by the application registration proxy server;

[0054] A push unit is used to push the data message to the target application acquisition server according to the query result.

[0055] Optionally, the MQTT server further includes an acquisition unit, configured to:

[0056] Acquire subscription configuration information of a client associated with the data message, and determine the type of the data message according to the subscription configuration information, where the type of the data message includes a real-time message and a subscription message;

[0057] The push unit is specifically used for:

[0058] If the type of the data message is a real-time message, the data message is pushed to the target application acquisition server in real time according to the query result;

[0059] If the type of the data message is a subscription message, the data message is pushed to the target application acquisition server according to the query result and the subscription push strategy.

[0060] Optionally, the pushing unit is further used for:

[0061] The data message is pushed to the target application acquisition server according to the query result and the link authentication priority.

[0062] A sixth aspect of the present application provides a data collection system, the system comprising: an MQTT server, an application registration proxy server, and an application collection server cluster;

[0063] The MQTT server is used to receive and cache data messages;

[0064] The MQTT server is used to send a query request to the application registration proxy server, the application registration proxy server is used to manage the application acquisition server cluster, and the application registration proxy server pre-stores basic parameters of each application acquisition server;

[0065] The application registration proxy server is used to dynamically determine the target application acquisition server in the application acquisition server cluster according to the basic parameters and the real-time load situation when receiving the query request;

[0066] The application registration proxy server is used to send the query result to the MQTT server;

[0067] The MQTT server is used to push the data message to the target application acquisition server according to the query result.

[0068] A seventh aspect of the present application provides a data collection device, the device comprising:

[0069] Processor, memory, input-output unit, and bus;

[0070] The processor is connected to the memory, the input and output unit, and the bus;

[0071] The memory stores a program, and the processor calls the program to execute any optional data collection method in the first aspect, the second aspect, or the third aspect.

[0072] In an eighth aspect, the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program performs any optional data collection method in the first aspect, the second aspect, or the third aspect.

[0073] It can be seen from the above technical solutions that this application has the following advantages:

[0074] Through the MQTT server and the application registration proxy server, an efficient and stable high-concurrency data collection solution is built. First, the MQTT server caches the received data messages. By using its lightweight protocol characteristics, it can effectively alleviate the impact of instantaneous high traffic on the system, thereby significantly improving the stability of data transmission. The application registration proxy server is responsible for managing the application collection server cluster, which pre-stores the basic parameters of each application collection server. When the MQTT server receives the data message, it will send a query request to the application registration proxy server, which will dynamically determine the target application collection server that is most suitable for processing the data message based on the basic parameters and real-time load conditions of each application collection server. Finally, the MQTT server pushes the data message to the specified target application collection server to ensure that the data message can be processed quickly and efficiently. The centralized management of the application collection server by the application registration proxy server enhances the scalability and stability of the system, supports the horizontal expansion of the application collection server, and can achieve stable data transmission and processing under more than one million concurrent connections. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0076] Figure 1 A schematic diagram of an embodiment of a method for data collection provided in this application;

[0077] Figure 2 A flowchart of an embodiment of a registration mechanism in the data collection method provided by the present application;

[0078] Figure 3 A schematic diagram of a flow chart of an embodiment of dynamically determining a target application acquisition server in a data acquisition method provided by the present application;

[0079] Figure 4 A schematic diagram of another embodiment of the data collection method provided by the present application;

[0080] Figure 5 A schematic diagram of a system framework based on message type optimization in the data collection method provided in this application;

[0081] Figure 6 A schematic diagram of the structure of an embodiment of the application registration proxy server provided in this application;

[0082] Figure 7 A schematic diagram of the structure of an embodiment of the MQTT server provided in this application;

[0083] Figure 8 A schematic diagram of the structure of an embodiment of a data acquisition system provided by the present application;

[0084] Fig. 9 A schematic diagram of the structure of an embodiment of a data acquisition device provided in this application. DETAILED DESCRIPTION

[0085] The present application provides a data collection method, system and related devices, which can achieve stable data transmission and processing under more than one million concurrent connections.

[0086] It should be noted that the data collection method provided in this application can be applied to a terminal or a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as the execution subject for example.

[0087] See also Figure 1 , Figure 1 An embodiment of a method for data collection provided by the present application includes:

[0088] S101, the MQTT server receives and caches the data message;

[0089] In the entire data collection process, the client is the source of data messages. The client includes various smart devices, sensors, industrial equipment or control systems. The client is connected to the data collection system through the network and sends the collected data messages in the form of data transmission requests. However, in a high-load, high-concurrency network environment, directly passing the data messages sent by the client to the back-end server may cause back-end resource overload and affect system stability. Therefore, a reverse proxy server can be introduced as a front-end entry. The reverse proxy server receives data messages from various clients and then forwards the data messages to the back-end MQTT server for further processing.

[0090] The reverse proxy server can not only hide the real address of the backend server, provide a unified access point for the client, and avoid direct attacks on the backend server, but also act as a security gateway in network communications. By processing SSL / TLS, that is, symmetric or asymmetric encryption and decryption operations, at the reverse proxy layer, the reverse proxy server can be responsible for the management of encrypted communications between the client and the backend server, thereby reducing the burden on the backend server and improving the overall performance of the system. In addition, as business needs increase, the reverse proxy server can add additional MQTT servers or other types of servers, and dynamically adjust its routing rules to distribute data messages to different backend servers to meet the needs of system expansion and high availability. By centrally monitoring traffic, the reverse proxy server can also effectively manage and optimize network traffic, and provide comprehensive data support for system analysis and troubleshooting.

[0091] MQTT (Message Queuing Telemetry Transport) is a lightweight message transmission protocol based on a publish / subscribe model, designed for low-bandwidth, high-latency or unreliable network environments. An MQTT server refers to a server-side component that supports the MQTT protocol and is used to receive, process and forward data messages. In this embodiment, after receiving the data messages forwarded by the reverse proxy server, the MQTT server first caches the data messages to ensure that the system can efficiently handle high-concurrency data message streams.

[0092] Specifically, the MQTT server can cache data messages through the Redis cache mechanism combined with the NoSQL database. As an in-memory database, Redis can handle a large number of read and write requests with extremely low latency. Its high performance enables the MQTT server to quickly access data in the cache, avoiding the bottleneck of direct access to disk storage. Redis supports multiple data structures and can flexibly store and operate different types of data messages to meet various business needs. In addition to the Redis cache, the MQTT server also uses a NoSQL type database. The NoSQL database supports large-scale data storage and can process unstructured or semi-structured data. It is especially suitable for storing diversified data generated by IoT devices. In addition, the NoSQL database has good scalability and can be horizontally expanded as the amount of data grows, ensuring that the system can handle the increasing data load. By combining Redis with the NoSQL database, the MQTT server can efficiently cache and manage data messages in a high-concurrency environment, optimize data storage and processing performance, and ensure that the system can handle large-scale data traffic.

[0093] S102, the MQTT server sends a query request to the application registration proxy server, the application registration proxy server is used to manage the application collection server cluster, and the application registration proxy server pre-stores basic parameters of each application collection server;

[0094] After caching the data message, the MQTT server also needs to send a query request to the application registration proxy server to determine which specific application acquisition server these data messages should be pushed to for processing. In this embodiment, the application registration proxy server is responsible for managing the application acquisition server cluster, which is composed of a number of application acquisition server nodes, which work together to improve the processing power and reliability of the system. The application registration proxy server also pre-stores the basic parameters of each application acquisition server and records the basic information of all application acquisition servers to effectively manage all application acquisition servers in the system.

[0095] S103, when the application registration proxy server receives the query request sent by the MQTT server, it dynamically determines the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster;

[0096] The application registration proxy server dynamically selects the target application acquisition server that is most suitable for processing the current data message based on the basic parameters and real-time load conditions of the application acquisition servers it manages. The application registration proxy server pre-stores the basic parameters of all application acquisition servers. These basic parameters include the server's business type, processing capacity, network connection information, and other static features. The basic parameters determine the type and capacity of data that each server can process. Therefore, the application registration proxy server can preliminarily screen out a possible batch of application acquisition servers based on the basic parameters. After the preliminary screening, the application registration proxy server further combines the real-time load conditions to finally determine the target application acquisition server. The real-time load conditions include dynamic data such as the current number of processing requests, CPU and memory usage, and network latency of each application acquisition server.

[0097] After comprehensively considering the basic parameters and real-time load conditions, the application registration proxy server can determine the target application collection server that is most suitable for processing the data message. The application registration proxy server returns the detailed information of the target application collection server, such as IP address and port information, to the MQTT server for subsequent data message push.

[0098] S104, the application registration proxy server sends the query result to the MQTT server;

[0099] After determining the target application collection server, the application registration proxy server sends the query result including the address or identification information of the target application collection server to the MQTT server. In addition, the query result may also include information such as port number or protocol type, which is not limited here.

[0100] S105. The MQTT server pushes the data message to the target application collection server according to the query result.

[0101] After receiving the query result returned by the application registration proxy server, the MQTT server establishes a connection with the target application acquisition server based on the query result, reads the data message from the cache and pushes the data message to the target application acquisition server through the established connection, ensuring that the data message can be correctly sent to the target application acquisition server for further processing.

[0102] In this process, the MQTT server does not need to manually configure the push address. The MQTT server only needs to automatically push the data message to the specified target application collection server according to the instructions of the application registration proxy server. The MQTT server also does not need to handle the switching logic of access failure separately. The application registration proxy server will manage the health status of all application collection servers in real time. If a server is unavailable, the application registration proxy server will distribute the request to other available servers, thereby simplifying the push configuration and fault handling of the MQTT server and optimizing the system's high concurrent access efficiency and data transmission efficiency.

[0103] In this embodiment, an efficient and stable high-concurrency data acquisition solution is constructed through the MQTT server and the application registration proxy server. First, the MQTT server caches the received data messages, and by using its lightweight protocol characteristics, it can effectively alleviate the impact of instantaneous high traffic on the system, thereby significantly improving the stability of data transmission. The application registration proxy server is responsible for managing the application acquisition server cluster, in which the basic parameters of each application acquisition server are pre-stored. When the MQTT server receives the data message, it will send a query request to the application registration proxy server, and the application registration proxy server will dynamically determine the target application acquisition server that is most suitable for processing the data message in combination with the basic parameters and real-time load conditions of each application acquisition server. Finally, the MQTT server pushes the data message to the specified target application acquisition server to ensure that the data message can be processed quickly and efficiently. Through the centralized management of the application acquisition server by the application registration proxy server, the scalability and stability of the system are enhanced, the horizontal expansion of the application acquisition server is supported, and stable data transmission and processing can be achieved under concurrent connections of more than one million.

[0104] The following describes the registration process for the application collection server. Figure 2 , Figure 2 In an embodiment of registering an application acquisition server on an application registration proxy server, the method includes:

[0105] S201, the application registration proxy server receives a registration request sent by a new application collection server, the registration request includes basic parameters of the new application collection server, the basic parameters include server address, port, service type and weight parameters;

[0106] When the application acquisition server is started or the configuration is changed, it will send a registration request to the application registration proxy server. The purpose of the registration request is to update the current status and configuration of the new application acquisition server to the application registration proxy server so that the application registration proxy server can correctly manage and allocate resources. The registration request contains the basic parameters of the application acquisition server, which specifically include but are not limited to: server address, used to identify the location of the application acquisition server in the network, so that data messages can be correctly routed to the server; port, used to define the specific network port on which the application acquisition server listens and receives data at its address; business type, describing the specific business functions handled by the application acquisition server, such as data processing type, application scenario, etc., which helps the application registration proxy server to make appropriate resource allocation according to business needs; weight parameter, used to measure the priority or processing capacity of the application acquisition server in the load balancing strategy. The weight parameter can reflect the server's processing capacity, response speed, load bearing capacity, etc., so as to more reasonably allocate traffic during the load balancing process.

[0107] S202: The application registration proxy server adds the new application collection server to the application collection server cluster according to the registration request, and dynamically monitors the load of the new application collection server.

[0108] After receiving the registration request and parsing the basic parameters therein, the application registration proxy server adds the new application acquisition server to the application acquisition server cluster. The application registration proxy server can specifically maintain a registry, store the basic information of the new application acquisition server in the registry, and ensure that the server can be queried and accessed. In addition, the application registration proxy server should also continuously monitor the load of each application acquisition server in the cluster, including real-time tracking of the server's request processing volume, the current number of active connections, resource usage (such as CPU, memory utilization), etc. Based on this data, the application registration proxy server can dynamically adjust the load distribution strategy to ensure the overall load balancing and performance optimization of the system.

[0109] In this embodiment, the horizontal expansion of any application acquisition server can be achieved through the registration mechanism provided by the application registration proxy server. You only need to complete the registration in the application registration proxy server, and the newly added application acquisition server will be automatically included in the management of the cluster. And this registration mechanism is not only applicable to the registration of new servers, but also to the update registration of server configuration changes (such as upgrading hardware, adjusting business types, etc.). This process can achieve seamless expansion to cope with growing business needs without modifying or interrupting the existing application acquisition server or business logic, which improves the scalability and flexibility of the system, and ensures that the data collection process can continue to run stably when increasing computing resources or responding to sudden traffic.

[0110] In the above step S104, the application registration proxy server can pre-set an access priority policy based on multiple factors to dynamically allocate the target application collection server for push. Figure 3 , Figure 3 An embodiment of dynamically determining a target application collection server for an application registration proxy server includes:

[0111] S301. When the application registration proxy server receives a query request sent by the MQTT server, it obtains a list of active servers and real-time load conditions in the application collection server cluster;

[0112] When the application registration proxy server receives a query request from the MQTT server, it first obtains a list of all currently active servers from the managed application collection server cluster. The active server specifically refers to the server that can currently process requests normally, that is, the server that is currently available and the load is within the preset acceptance range, thereby excluding the failed or overloaded servers. During this process, the application registration proxy server also collects the real-time load status of each active server, including the current request processing volume, resource utilization (such as CPU, memory, etc.), number of connections, etc. Obtaining real-time load status helps the application registration proxy server understand the current working status of each server and provide data support for the selection of the target application collection server.

[0113] S302: The application registration proxy server dynamically determines a target application collection server in the active server list based on basic parameters and real-time load conditions of each application collection server in the active server list and in combination with a preset access priority policy.

[0114] After obtaining the active server list, the application registration proxy server will dynamically determine the target application collection server that is most suitable for processing the data message based on the basic parameters of each server in the active server list (such as server address, port, service type, weight parameter, etc.) and real-time load data, combined with the preset access priority strategy. Specifically, the preset access priority strategy includes the following factors:

[0115] Server business type: Give priority to servers with business types corresponding to the current data message to ensure that the message can be processed efficiently and accurately.

[0116] Real-time server load: By monitoring the number of requests currently being processed by each server, servers with fewer requests are given priority to avoid overloading certain servers, thereby balancing the overall system load.

[0117] Server weight parameters: allocation is based on the weight parameters configured in the server basic parameters.

[0118] Specific access content: For specific types of data messages or specific clients, you may need to select a specific server to process them to maintain business continuity and data consistency;

[0119] Session persistence: To ensure a continuous session with a client, it may be necessary to send subsequent requests to the server that previously handled the session.

[0120] Through comprehensive consideration of these factors, for example: first, according to the business type of the data message, the request is assigned to the server that can handle the business type, initially reducing unnecessary loads concentrated on a certain type of server, ensuring that the request can be processed by the appropriate server. Secondly, among the servers with the same business type, the application registration proxy server will use a load balancing algorithm to distribute requests based on the real-time load of each server to prevent some servers from affecting performance due to excessive load. If multiple qualified servers are encountered during the selection process, the application registration proxy server can also finally determine the target application acquisition server based on factors such as server weight, specific access content, and session retention, ensuring that high-priority requests can be processed first, while also supporting session continuity and reducing request delays. As a result, the application registration proxy server can intelligently allocate the target application acquisition servers to be pushed, ensuring the efficiency and stability of the system in a high-concurrency environment.

[0121] In this embodiment, through real-time load monitoring and dynamic allocation, the application registration proxy server can reasonably distribute a large number of concurrent requests to multiple servers instead of concentrating them on a single or a few servers. This decomposition ensures that the load of each server is within a controllable range, optimizes the resource utilization of the entire system, and ensures that the system can still run stably and efficiently in high-concurrency scenarios.

[0122] In other specific embodiments, the data collection method can be further optimized according to the message type. Figure 4 , Figure 4 Another embodiment of the data collection method provided by the present application includes:

[0123] S401, the MQTT server receives and caches the data message;

[0124] In this embodiment, step S401 is similar to step S101 in the above embodiment and will not be described again here.

[0125] S402, the MQTT server obtains subscription configuration information of the client associated with the data message, and determines the type of the data message according to the subscription configuration information, where the type of the data message includes a real-time message and a subscription message;

[0126] In this embodiment, the subscription configuration information of each client is stored in the MQTT server, and the subscription configuration information specifically includes the subject, priority, processing logic, etc. of the message. In the subscription configuration, the client may set the priority of certain topics, or set the time sensitivity requirements of the message. For example, some messages may need to be processed in real time and require strong timeliness, while other messages can be processed later. According to the priority or time sensitivity in the subscription configuration, the MQTT service can classify messages as real-time messages or subscription messages. Among them, real-time messages refer to messages that must be processed and pushed immediately. These messages usually have a higher priority and need to be quickly forwarded to the target application acquisition server to ensure the timeliness of the data. Subscription messages refer to messages pushed according to the client's subscription configuration. These messages do not need to be processed immediately and can be batch processed or delayed according to subscription policies or conditions.

[0127] Therefore, when the MQTT server receives a new data message, it will determine the client associated with the data message and obtain the client's subscription configuration information. The client's subscription configuration information defines the client's expectations for the data message, including topic, priority, processing logic, time requirements, etc. The MQTT server classifies the data message according to these expectations, thereby determining how to subsequently process the data message.

[0128] It should be noted that the MQTT server supports real-time modification or addition of client subscription configuration. Any subscription configuration changes can take effect immediately without restarting or reloading the service. Therefore, users can adjust subscription policies at any time to cope with changes in business needs.

[0129] S403, the MQTT server sends a query request to the application registration proxy server, the application registration proxy server is used to manage the application collection server cluster, and the application registration proxy server pre-stores the basic parameters of each application collection server;

[0130] In this embodiment, step S403 is similar to step S102 in the above embodiment and will not be described again here.

[0131] It should be noted that the query request includes the type of data message.

[0132] It should be noted that, in this embodiment, when the MQTT server sends a query request to the application registration proxy server, it needs to send the type of the data message to the application registration proxy server.

[0133] S404: When the application registration proxy server receives the query request sent by the MQTT server, it determines the type of the data message according to the query request, and dynamically determines the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster;

[0134] See also Figure 5 , Figure 5 This is a schematic diagram of the system framework optimized based on message types. In order to better handle different types of messages, the application collection server cluster can be divided into two sub-clusters:

[0135] Customized application collection server cluster: Specialized in processing data messages with real-time requirements, such as messages that need to be responded to or processed immediately, with higher processing capabilities and the ability to handle a large number of data requests in a short period of time;

[0136] General application collection server cluster: specializes in processing subscription messages, which can usually be processed delayed or in batch push mode. It is usually used to process lower priority data and does not require an immediate response.

[0137] When the application registration proxy server receives a query request, it first determines the subcluster based on the type of data message. If the data message is identified as a real-time message, the application registration proxy server will limit the query scope to the customized application acquisition server cluster. If the data message is identified as a subscription message, the application registration proxy server will limit the query scope to the general application acquisition server cluster. That is, the application registration proxy server will dynamically determine the target application acquisition server in the corresponding subcluster based on the type of data message. The dynamic determination process will also take into account the basic parameters and real-time load of the server to ensure that the message is pushed to the most appropriate server. By assigning different types of messages to different server clusters, the system can more effectively handle high-concurrency access requests and avoid overloading a single server. In addition, real-time messages are preferentially assigned to high-performance customized application acquisition servers, while subscription messages are assigned to general application acquisition servers, achieving efficient use of system resources.

[0138] S405, the application registration proxy server sends the query result to the MQTT server;

[0139] In this embodiment, step S405 is similar to step S104 in the above embodiment and will not be described again here.

[0140] S406, the MQTT server pushes the data message to the target application collection server according to the query result and the link authentication priority;

[0141] After determining the target application collection server based on the query results, the MQTT server will use different push strategies to push the data message to the predetermined target application collection server based on different message types. Specifically, if the data message type is a real-time message, the MQTT server will push the data message to the target application collection server in real time. If the data message type is a subscription message, the MQTT server will push the data message to the target application collection server according to the subscription push strategy. The subscription push strategy depends on the parameters in the client subscription configuration information, such as the priority of the message, the push frequency, etc. The subscription push strategy determines when and how to push the data message to the target application collection server.

[0142] Furthermore, in order to optimize system performance and security in a high-concurrency environment, the MQTT server will give priority to target application collection servers that have completed link authentication when pushing data messages. Link authentication refers to the process of establishing a secure communication link between the MQTT server and the target application collection server, which usually involves operations such as verifying the identity of the target application collection server, encrypting the communication channel, and ensuring data integrity. When the MQTT server needs to push a data message, it will first check the link status between itself and the target application collection server, and give priority to target application collection servers that have passed authentication and have stable links for data push. Since these links have been security verified, data can be transmitted directly, avoiding repeated authentication steps and thus reducing delays. By managing link authentication priorities, the MQTT server can improve the efficiency of pushing data messages while ensuring data security.

[0143] S407, the MQTT server regularly clears cached data messages according to a preset time interval.

[0144] In this embodiment, in order to prevent the data in the cache from accumulating continuously and occupying the memory resources of the server, the MQTT server needs to regularly clean up the data messages in the cache according to a preset time interval. The preset time interval is used to trigger the cache cleaning operation, which can be set according to business needs and system load conditions, for example, set to 2 hours. The cleaning operation specifically includes checking the timestamps of each data message in the cache to determine whether they have expired or are no longer needed. If the message exceeds the set retention time, or has been pushed to the target application acquisition server and no longer needs to be retransmitted, it will be removed from the cache. By regularly cleaning up cached data messages, the MQTT server can effectively manage memory resources and ensure the system's ability to operate persistently.

[0145] See also Figure 6 , Figure 6 An embodiment of an application registration proxy server provided by the present application, the application registration proxy server includes:

[0146] The determining unit 601 is used to dynamically determine the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster when receiving the query request sent by the MQTT server. The application registration proxy server is used to manage the application acquisition server cluster. The query request is a request sent by the MQTT server after receiving and caching the data message;

[0147] The first sending unit 602 is used to send the query result to the MQTT server, so that the MQTT server pushes the data message to the target application collection server.

[0148] Optionally, the application registration proxy server further includes a registration unit 603, which is used to:

[0149] The application registration proxy server receives a registration request sent by a new application collection server, where the registration request includes basic parameters of the new application collection server, including server address, port, service type, and weight parameter;

[0150] The application registration proxy server adds the new application collection server to the application collection server cluster according to the registration request, and dynamically monitors the load of the new application collection server.

[0151] Optionally, the determining unit 601 is specifically configured to:

[0152] When the application registration proxy server receives the query request sent by the MQTT server, it obtains the list of active servers and real-time load conditions in the application collection server cluster;

[0153] The application registration proxy server dynamically determines the target application collection server in the active server list based on the basic parameters and real-time load conditions of each application collection server in the active server list and in combination with a preset access priority strategy.

[0154] Optionally, the application acquisition server cluster includes a customized application acquisition server cluster and a general application acquisition server cluster, and the determining unit 601 is further specifically configured to:

[0155] determining the type of the data message according to the query request;

[0156] If the type of the data message is a real-time message, the application registration proxy server dynamically determines the target application acquisition server in the customized application acquisition server cluster according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster;

[0157] If the type of the data message is a subscription message, the application registration proxy server dynamically determines the target application acquisition server in the general application acquisition server cluster according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster.

[0158] In this embodiment, the functions of each unit are the same as those described above. Figures 1 to 4 The steps in the method embodiment shown correspond to each other and will not be repeated here.

[0159] See also Figure 7 , Figure 7 An embodiment of the MQTT server provided by the present application includes:

[0160] The first receiving unit 701 is used for the MQTT server to receive and cache data messages;

[0161] The second sending unit 702 is used to send a query request to the application registration proxy server, so that the application registration proxy server dynamically determines the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster;

[0162] The second receiving unit 703 is used to receive the query result returned by the application registration proxy server;

[0163] The push unit 704 is used to push the data message to the target application collection server according to the query result.

[0164] Optionally, the MQTT server further includes an acquisition unit 705, configured to:

[0165] Obtaining subscription configuration information of a client associated with a data message, and determining the type of the data message according to the subscription configuration information, where the type of the data message includes a real-time message and a subscription message;

[0166] The push unit 704 is specifically used for:

[0167] If the data message type is a real-time message, the data message is pushed to the target application collection server in real time according to the query result;

[0168] If the type of the data message is a subscription message, the data message is pushed to the target application collection server according to the query result and the subscription push strategy.

[0169] Optionally, the push unit 704 is further configured to:

[0170] The data message is pushed to the target application collection server based on the query results and link authentication priority.

[0171] In this embodiment, the functions of each unit are the same as those described above. Figures 1 to 4 The steps in the method embodiment shown correspond to each other and will not be repeated here.

[0172] The following is an explanation of the data collection system provided by this application. Figure 8 , Figure 8 An embodiment of a system for data collection provided by the present application includes:

[0173] MQTT server 801, application registration proxy server 802 and application collection server cluster 803;

[0174] The MQTT server 801 is used to receive and cache data messages;

[0175] The MQTT server 801 is used to send a query request to the application registration proxy server 802, and the application registration proxy server 802 is used to manage the application collection server cluster 803. The application registration proxy server 802 pre-stores the basic parameters of each application collection server;

[0176] The application registration proxy server 802 is used to dynamically determine the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster 803 when receiving a query request sent by the MQTT server;

[0177] The application registration proxy server 802 is used to send the query result to the MQTT server 801;

[0178] The MQTT server 801 is used to push data messages to the target application collection server according to the query results.

[0179] In the system of this embodiment, the functions of the MQTT server 801, the application registration proxy server 802 and the application collection server cluster 803 are the same as those described above. Figures 1 to 4 The steps in the method embodiment shown correspond to each other and will not be repeated here.

[0180] This application also provides a data collection device, see Fig. 9 , Fig. 9 An embodiment of a data collection device provided in the present application includes:

[0181] Processor 901, memory 902, input and output unit 903, bus 904;

[0182] The processor 901 is connected to the memory 902, the input and output unit 903 and the bus 904;

[0183] The memory 902 stores a program, and the processor 901 calls the program to execute any of the above data collection methods.

[0184] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above data collection methods.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0189] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc. Various media that can store program codes.

Claims

1. A method for data collection, characterized in that: The method comprises: When the application registration proxy server receives a query request sent by the MQTT server, it dynamically determines the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster. The application registration proxy server is used to manage the application acquisition server cluster. The query request is a request sent by the MQTT server after receiving and caching the data message. The application registration proxy server sends the query result to the MQTT server, so that the MQTT server pushes the data message to the target application acquisition server.

2. The method according to claim 1, characterized in that The method further comprises: The application registration proxy server receives a registration request sent by a new application collection server, wherein the registration request includes basic parameters of the new application collection server, and the basic parameters include a server address, a port, a service type, and a weight parameter; The application registration proxy server adds the new application acquisition server to the application acquisition server cluster according to the registration request, and dynamically monitors the load of the new application acquisition server.

3. The method according to claim 1, characterized in that: When the application registration proxy server receives the query request sent by the MQTT server, the target application acquisition server is dynamically determined according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster, including: When the application registration proxy server receives the query request sent by the MQTT server, it obtains the list of active servers and real-time load conditions in the application collection server cluster; The application registration proxy server dynamically determines a target application acquisition server in the active server list based on basic parameters and real-time load conditions of each application acquisition server in the active server list and in combination with a preset access priority strategy.

4. The method according to any one of claims 1 to 3, characterized in that The application acquisition server cluster includes a customized application acquisition server cluster and a general application acquisition server cluster. Before dynamically determining the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster, the method further includes: The application registration proxy server determines the type of the data message according to the query request; The step of dynamically determining the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster includes: If the type of the data message is a real-time message, the application registration proxy server dynamically determines a target application acquisition server in the customized application acquisition server cluster according to basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster; If the type of the data message is a subscription message, the application registration proxy server dynamically determines a target application acquisition server in the general application acquisition server cluster according to basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster.

5. A method for data collection, characterized in that: The method comprises: The MQTT server receives and caches data messages; The MQTT server sends a query request to the application registration proxy server, so that the application registration proxy server dynamically determines the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster; The MQTT server receives the query result returned by the application registration proxy server; The MQTT server pushes the data message to the target application acquisition server according to the query result.

6. The method according to claim 5, characterized in that After the MQTT server receives and caches the data message, the method further includes: The MQTT server obtains subscription configuration information of the client associated with the data message, and determines the type of the data message according to the subscription configuration information, where the type of the data message includes a real-time message and a subscription message; The MQTT server pushes the data message to the target application acquisition server according to the query result, including: If the type of the data message is a real-time message, the MQTT server pushes the data message to the target application acquisition server in real time according to the query result; If the type of the data message is a subscription message, the MQTT server pushes the data message to the target application acquisition server according to the query result and the subscription push strategy.

7. A method for data collection, characterized in that: The method comprises: The MQTT server receives and caches data messages; The MQTT server sends a query request to the application registration proxy server, the application registration proxy server is used to manage the application acquisition server cluster, and the application registration proxy server pre-stores the basic parameters of each application acquisition server; When the application registration proxy server receives the query request, it dynamically determines the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster; The application registration proxy server sends the query result to the MQTT server; The MQTT server pushes the data message to the target application acquisition server according to the query result.

8. An application registration proxy server, characterized in that: The application registration proxy server includes: A determination unit, configured to dynamically determine a target application acquisition server according to basic parameters and real-time load conditions of each application acquisition server in an application acquisition server cluster when receiving a query request sent by an MQTT server, wherein the application registration proxy server is configured to manage the application acquisition server cluster, and the query request is a request sent by the MQTT server after receiving and caching a data message; The first sending unit is used to send the query result to the MQTT server, so that the MQTT server pushes the data message to the target application collection server.

9. An MQTT server, characterized in that: The MQTT server includes: A first receiving unit, used for the MQTT server to receive and cache data messages; A second sending unit is used to send a query request to the application registration proxy server, so that the application registration proxy server dynamically determines the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster; A second receiving unit, configured to receive the query result returned by the application registration proxy server; A push unit is used to push the data message to the target application acquisition server according to the query result.

10. A data collection system, characterized in that: The system includes: an MQTT server, an application registration proxy server and an application acquisition server cluster; The MQTT server is used to receive and cache data messages; The MQTT server is used to send a query request to the application registration proxy server, the application registration proxy server is used to manage the application acquisition server cluster, and the application registration proxy server pre-stores basic parameters of each application acquisition server; The application registration proxy server is used to dynamically determine the target application acquisition server according to the basic parameters and real-time load conditions of each application acquisition server in the application acquisition server cluster when receiving the query request; The application registration proxy server is used to send the query result to the MQTT server; The MQTT server is used to push the data message to the target application acquisition server according to the query result.

Images