A message processing method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses a message processing method, apparatus, device and storage medium. The method includes: obtaining a first device message sent by an Internet of Things device, the first device message including a target message transmission topic name and target device data generated by the Internet of Things device; obtaining topic mapping relationship configuration information in a configuration database, each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name; determining a target stream processing topic name according to the target message transmission topic name and the topic mapping relationship configuration information; generating a second device message based on the target device data and the target stream processing topic name, and sending the second device message to a stream processing server so that the stream processing server processes the second device message. Through the technical solution of the embodiment of the present invention, efficient docking between the acquisition server and the stream processing server can be achieved, thereby improving the message processing performance.

Description

Technical Field

[0001] Embodiments of the present invention relate to the fields of Internet of Things and medical health technology, and in particular to a message processing method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of computer technology, more and more IoT devices are being developed and applied to different application scenarios, such as blood glucose devices, blood pressure devices, and blood oxygen devices in the medical and health field. The data messages generated by each IoT device can be used between devices or by business servers to implement functions such as security auditing, traffic billing, data statistics, and notification triggering. Each data message goes through the process of "collection-processing-storage-analysis-application".

[0003] Currently, IoT devices send generated messages to a collection server to collect and distribute messages in the collection server, for example, by using the MQTT (Message Queuing Telemetry Transport) protocol to send messages to an MQTT broker.

[0004] However, in the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] Since the collection server only collects and distributes messages, it does not perform stream processing and does not support asynchronous processing. It cannot store or reprocess events for a long time, which reduces the message processing performance. Summary of the invention

[0006] The embodiments of the present invention provide a message processing method, apparatus, device and storage medium to achieve efficient connection between a collection server and a stream processing server, so that message processing can be performed in the stream processing server, thereby improving message processing performance.

[0007] In a first aspect, an embodiment of the present invention provides a message processing method, which is applied to a collection server, comprising:

[0008] Acquire a first device message sent by an IoT device, where the first device message includes a target message transmission topic name and target device data generated by the IoT device;

[0009] Obtaining topic mapping relationship configuration information in a configuration database, wherein each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name;

[0010] Determine the target stream processing topic name corresponding to the IoT device according to the target message transmission topic name and the topic mapping relationship configuration information;

[0011] Based on the target device data and the target stream processing topic name, a second device message is generated, and the second device message is sent to the stream processing server so that the stream processing server processes the second device message.

[0012] In a second aspect, an embodiment of the present invention further provides a message processing device, which is integrated in a collection server, and includes:

[0013] A first device message acquisition module, used to acquire a first device message sent by an IoT device, wherein the first device message includes a target message transmission topic name and target device data generated by the IoT device;

[0014] A mapping relationship configuration information acquisition module, used to acquire topic mapping relationship configuration information in a configuration database, wherein each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name;

[0015] A target stream processing topic name determination module is used to determine the target stream processing topic name corresponding to the Internet of Things device according to the target message transmission topic name and the topic mapping relationship configuration information;

[0016] The second device message generating module is used to generate a second device message based on the target device data and the target stream processing topic name, and send the second device message to the stream processing server so that the stream processing server processes the second device message.

[0017] In a third aspect, an embodiment of the present invention further provides a message processing system, characterized in that the system comprises: an Internet of Things device, a collection server and a stream processing server; wherein,

[0018] The acquisition server is used to implement the message processing method provided by any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0020] one or more processors;

[0021] A memory for storing one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the message processing method provided by any embodiment of the present invention.

[0023] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the message processing method provided by any embodiment of the present invention is implemented.

[0024] One embodiment of the above invention has the following advantages or beneficial effects:

[0025] By pre-configuring the topic mapping relationship configuration information, flexible mapping between message transmission topics and stream processing topics can be achieved, and each configured stream processing topic name corresponds to at least one message transmission topic name, so that one or more message transmission topic names can be mapped to the same stream processing topic name, solving the problem of the small number of connection construction times in the stream processing server. The acquisition server can quickly and easily determine the target stream processing topic name corresponding to the Internet of Things device based on the target message transmission topic name in the first device message sent by the Internet of Things device and the topic mapping relationship configuration information in the configuration database, and send the second device message generated based on the target device data and the target stream processing topic name to the stream processing server, so that the stream processing server processes the second device message, thereby reducing the complexity of docking, ensuring efficient and fast message forwarding, and realizing efficient docking between the acquisition server and the stream processing server, so that the device message can be stored, processed and forwarded in the stream processing server, thereby improving the message processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 is a flow chart of a message processing method provided by an embodiment of the present invention;

[0028] Figure 2 This is an example of the architecture of a message processing system involved in one embodiment of the present invention;

[0029] Figure 3 is a flowchart of another message processing method provided by an embodiment of the present invention;

[0030] Figure 4 It is a structural diagram of a message processing device provided by an embodiment of the present invention;

[0031] Figure 5 It is a structural diagram of a message processing system provided by an embodiment of the present invention;

[0032] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0034] Figure 1 This is a flow chart of a message processing method provided by an embodiment of the present invention. This embodiment is applicable to the case where the device messages collected by the collection server are reported to the stream processing server. The method can be executed by a message processing device, which can be implemented by software and / or hardware and integrated in the collection server. Figure 1 As shown, the method specifically comprises the following steps:

[0035] S110. Acquire a first device message sent by the IoT device, where the first device message includes a target message transmission topic name and target device data generated by the IoT device.

[0036] Among them, the Internet of Things device refers to any smart device that can communicate with the Internet of Things. For example, the Internet of Things device may include but is not limited to smart devices worn on the body, such as any type of wearable devices such as fitness trackers, headphones and smart watches. The first device message may refer to the data message generated in the Internet of Things device at the current moment. The target message transmission topic name may refer to the data transmission topic name Topic corresponding to the target device data generated by the Internet of Things device at the current moment. For example, when transmitting device messages through the MQTT protocol, the target message transmission topic name may refer to the MQTT Topic corresponding to the Internet of Things device. The message transmission topic name may refer to the message type. After the subscriber subscribes, he can receive the message content of the topic.

[0037] Specifically, the IoT device that currently needs to process the message can collect the target device data generated by its own sensor, and determine the target message transmission topic name based on its own device parameters. For example, if the IoT device is a blood glucose monitoring device, the device manufacturer in English is JZYQ, the device model is HB3, and the device serial number SN is ADGX0000001, then the target message transmission topic name corresponding to the device is: / DATA / DEVICE / SEND / JZYQ-HB3 / ADGX0000001. The IoT device can generate a first device message based on the target message transmission topic name and the generated target device data, and send the first device message to the collection server, so that the collection server can obtain the first device message reported by the IoT device in real time.

[0038] It should be noted that in this embodiment, there may be only one collection server, such as an MQTT broker, so that the same collection server can be used to process device messages sent by all IoT devices. There may also be a collection server cluster, such as an MQTT Cluster, which includes multiple MQTT brokers, so that the collection server cluster can be used to process device messages sent by all IoT devices, ensuring high availability of message processing and supporting linear expansion.

[0039] Exemplarily, S110 may include: acquiring a first device message sent by the IoT device by calling a message monitoring module in a preloaded message processing plug-in.

[0040] Among them, the message processing plug-in can be implemented by the hook mechanism provided by the acquisition server, that is, hook declaration, hook registration and hook call, so that stability and instability can be isolated from the code design. The framework kernel is relatively stable because it has weak relevance to the business or is irrelevant. The message processing plug-in implements specific business code, which is essentially subject to change at any time as business needs change, so it is relatively unstable. The message monitoring module can be a functional module for monitoring whether the connected IoT device sends device messages.

[0041] Specifically, each IoT device can pre-load a message processing plug-in and initialize the plug-in, so that the loaded message processing plug-in can be used for efficient message docking and processing, thereby realizing the decoupling of message collection and message processing. When an IoT device connects to a collection server, it can trigger the connection monitoring module in the message processing plug-in in the collection server, such as by monitoring the connection operation of the IoT device through the on_connection(Client) function. After the IoT device is successfully connected, the generated first device message can be automatically reported to the collection server. The collection server calls the message monitoring module in the message processing plug-in, such as by monitoring the first device message sent by the IoT device through the on_publish(Message) function.

[0042] For example, Figure 2 An architectural example of a message processing system is given. Figure 2 Each collection server in the collection server cluster is pre-loaded with a message processing plug-in. There is no association between plug-ins, which is equivalent to module decoupling in the code, thereby improving stability. It should be noted that loading a message processing plug-in inside the collection server can reduce network resource consumption once compared to setting up an additional message docking device, so that data is directly forwarded to the stream processing server after conversion inside the collection server, thereby improving the docking efficiency between the collection server and the stream processing server.

[0043] S120. Obtain topic mapping relationship configuration information in a configuration database, wherein each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name.

[0044] Among them, the configuration database can be used to store the configuration information configured by the user. The topic mapping relationship configuration information can be the mapping relationship between the stream processing topic name and the information transmission topic name configured based on business needs. The stream processing topic name can refer to the topic name Topic in the stream processing server. For example, when the stream processing server is an ApacheKafka broker, the stream processing topic name can refer to a Kafka Topic, such as stream-hiot-device-misc--online.

[0045] Each stream processing topic name can correspond to one or more message transmission topic names, so that one or more message transmission topic names can be mapped to the same stream processing topic name, thereby solving the problem of low number of connection establishment times in the stream processing server. A stream processing topic name can refer to a message collection. Each stream processing topic name can have multiple producers sending messages to it, and multiple consumers consuming messages from it. Effectively combine the acquisition server and the stream processing server to build a reliable and secure IoT infrastructure.

[0046] Specifically, the subject mapping relationship configuration information can be configured in the configuration database based on business needs, thereby realizing dynamic and flexible configuration of the mapping relationship, so that when adding new IoT devices, only the corresponding subject mapping relationship configuration information needs to be modified, such as adding a new mapping relationship or changing the original mapping relationship, without modifying other business logic, thereby simplifying development operations, improving development efficiency and maintenance costs. In addition, business access is isolated, and the success or failure of new business access will not affect previously connected devices, and businesses do not affect each other.

[0047] Exemplarily, S120 may include: acquiring topic mapping relationship configuration information in a configuration database by calling a rule engine module in a message processing plug-in.

[0048] Specifically, Figure 2 As shown, the collection server can quickly obtain the topic mapping relationship configuration information from the corresponding configuration database based on the connection information of the configuration database by calling the rule engine module in the message processing plug-in.

[0049] S130. Determine the target stream processing topic name corresponding to the IoT device according to the target message transmission topic name and topic mapping relationship configuration information.

[0050] Specifically, if the correspondence between the message transmission topic name and the stream processing topic name is directly stored in the topic mapping relationship configuration information, the stream processing topic name corresponding to the target message transmission topic name in the topic mapping relationship configuration information can be directly determined as the target stream processing topic name. Alternatively, if the regular expression corresponding to each stream processing topic name is stored in the topic mapping relationship configuration information, the target message transmission topic name is matched with each regular expression to determine whether the corresponding regular expression is satisfied, and the stream processing topic name corresponding to the satisfied regular expression is determined as the target stream processing topic name, so that the target stream processing topic name corresponding to the IoT device can be determined more conveniently based on the topic mapping relationship configuration information, thereby reducing the complexity of the calculation and ensuring efficient and fast mapping of message forwarding.

[0051] It should be noted that if there is no successful match with the target message transmission topic name in the topic mapping relationship configuration information, the pre-set default stream processing topic name can be determined as the target stream processing topic name corresponding to the IoT device, so that the device message can be forwarded to the stream processing server corresponding to the default stream processing topic name, thereby avoiding message loss.

[0052] Exemplarily, S130 may include: determining the target flow processing topic name corresponding to the IoT device according to the target message transmission topic name and topic mapping relationship configuration information by calling a rule engine module in the message processing plug-in.

[0053] Specifically, Figure 2 As shown, the collection server can quickly match the target message transmission topic name with the acquired topic mapping relationship configuration information by calling the rule engine module in the message processing plug-in to obtain the target stream processing topic name corresponding to the IoT device.

[0054] S140. Generate a second device message based on the target device data and the target stream processing topic name, and send the second device message to the stream processing server so that the stream processing server processes the second device message.

[0055] Among them, the stream processing server can be a platform for streaming messages. The stream processing server is but not limited to Apache Kafka, Apache Heron, Flink, Apache Pulsar, RabbitMQ, ActiveMQ. Among them, Apache Kafka is a highly scalable distributed streaming platform that combines messaging, storage, and data processing to build a highly scalable, reliable, secure, and real-time infrastructure. Users who use Kafka usually use Kafka Connect to integrate with any source or receiver. Kafka streams are also useful because they allow continuous stream processing. From the perspective of the Internet of Things, the advantages of Apache Kafka are: stream processing, not just queuing; high throughput; large; high availability; long-term storage and buffering; event reprocessing; good integration with other departments of the enterprise; hybrid, multi-cloud, and global deployment. The disadvantages of Kafka are: not built for tens of thousands of connections; requires a stable network and solid infrastructure; lacks IoT-specific features such as keep alive, last will and testament, and covenant. The advantages of MQTT are: widely adopted; lightweight; simple API interface; built for poor connection and high latency scenarios; supports multiple client connections (tens of thousands per MQTT server). The disadvantages of MQTT are that it is just queuing, not stream processing; it cannot handle usage surges (no buffering); most MQTT servers do not support high scalability; asynchronous processing (usually offline for a long time); lack of good integration with other departments of the enterprise; single infrastructure (usually somewhere on the edge); unable to reprocess events. It can be seen that MQTT is not built for high scalability, longer storage time or easy integration with legacy systems, but focuses on communication between clients and servers, quickly receiving and processing messages from a large number of smart devices. Apache Kafka is a highly scalable distributed streaming platform. Kafka ingests, stores, processes and forwards large amounts of data from thousands of IoT devices, thereby combining Apache Kafka and MQTT to build a scalable, reliable and secure IoT infrastructure, further improving message processing performance.

[0056] Specifically, the integration server can generate a second device message based on the target device data and the target flow processing topic name, and send the second device message to the flow processing server so that the second device message can be processed in the flow processing server, such as message storage, processing and forwarding operations, thereby improving message processing performance.

[0057] Exemplarily, "sending the second device message to the stream processing server so that the stream processing server processes the second device message" in S140 may include: sending the second device message to the stream processing server cluster so that the target stream processing server in the stream processing server cluster processes the second device message.

[0058] Specifically, this embodiment can be connected to only one stream processing server, or it can be connected to a stream processing server cluster, such as Kafka Cluster, to improve message processing efficiency and scalability. The acquisition server forwards the second device message to the stream processing server cluster, and the second device message can be stored in the high-performance stream processing server cluster. The target stream processing server corresponding to the target stream processing topic name processes the second device message, such as message storage, processing and forwarding operations, such as Figure 2 The target device data is published to the consumer who subscribes to the target stream processing topic name for processing, thereby realizing efficient connection between the collection server cluster and the stream processing server cluster.

[0059] Exemplarily, S140 may include: generating a second device message based on target device data and a target stream processing topic name by calling a stream processing client module in a message processing plug-in, and sending the second device message to a stream processing server.

[0060] Specifically, Figure 2 As shown, the acquisition server can generate a second device message based on the target device data and the target stream processing topic name by calling the stream processing client module in the message processing plug-in, and send the second device message to the stream processing server, so that the stream processing client module can be used to quickly forward messages, realize efficient docking between the acquisition server cluster and the stream processing server cluster, and thus ensure high availability and high scalability of the production environment. When the scale of users using a certain type of device increases, if the bottleneck level of the MQTT Cluster is reached, the MQTT Broker can be expanded. If the Kafka Cluster is under great pressure, the scale of the Kafka Cluster can be adjusted dynamically. The expansion and contraction of the MQTT Cluster has no effect on the business. The realization of the business does not rely heavily on specific technologies, and the two routes of system stability and rapid business development can be promoted in parallel.

[0061] New servers can be added to the cluster dynamically, thus enhancing the performance of the cluster; the cluster uses server redundancy to prevent clients from easily encountering out of service warnings. In a cluster, the same service can be provided by multiple servers. If a server fails, another server will take over the failed server. The cluster's ability to recover from a failed server to another server enhances the availability of the application. In order to have scalability and high availability, the cluster must have the following two capabilities: load balancing (load balancing can distribute tasks more evenly to the computing and network resources in the cluster environment); fault recovery (for some reason, the resource executing a task fails, and the resource executing the same task in another server continues to complete the task. This process of transparently continuing to complete the task in another server because the resources in one server cannot work is called fault recovery), internal communication (in order to work together, achieve load balancing and error recovery, the cluster servers must communicate frequently, such as the load balancer's communication of server heartbeat test information and task execution context information between servers).

[0062] It should be noted that directly loading the message processing plug-in in the collection server can reduce the number of direct conversions from data collection to data using different services. IoT devices generally have the characteristics of large amount of collected data, high reporting frequency, and occupying a lot of network resources. In order to respond to and feedback to users in a timely manner for subsequent business processing, it is necessary to efficiently transmit messages to the stream processing server cluster to ensure the timeliness and sequence of messages. The received data in a timely manner can greatly reduce the cost of data processing calculations, shorten the processing flow, and enhance the timeliness of information feedback. The more timely the information, the higher its value. Therefore, the time intervals between information collection, storage, processing, transmission, and use should be shortened as much as possible to increase the value of information.

[0063] Exemplarily, multiple stream processing server clusters, such as a Kafka cluster, and a mapping relationship between stream processing topic names and stream processing server clusters can also be configured in the configuration database, so that device messages with different stream processing topic names can be forwarded to different stream processing server clusters, thereby ensuring high availability, high scalability and flexibility of message forwarding through multiple stream processing server clusters across computer rooms.

[0064] For example, the collection server can provide cache internally. Based on the cache mechanism and hard disk cache mechanism, the probability of data loss caused by network instability when pushing messages to the Kafka cluster can be reduced, while being compatible with jitter and temporary instability of the MQTT Cluster and KafkaCluster communication network.

[0065] The technical solution of this embodiment can realize flexible mapping between message transmission topics and stream processing topics by pre-configuring topic mapping relationship configuration information, and each configured stream processing topic name corresponds to at least one message transmission topic name, so that one or more message transmission topic names can be mapped to the same stream processing topic name, solving the problem of small number of connection establishment times in the stream processing server. The acquisition server can quickly and easily determine the target stream processing topic name corresponding to the Internet of Things device based on the target message transmission topic name in the first device message sent by the Internet of Things device and the topic mapping relationship configuration information in the configuration database, and send the second device message generated based on the target device data and the target stream processing topic name to the stream processing server, so that the stream processing server processes the second device message, thereby reducing the complexity of docking, ensuring efficient and fast message forwarding, and realizing efficient docking between the acquisition server and the stream processing server, so that the device message can be stored, processed and forwarded in the stream processing server, thereby improving the message processing performance.

[0066] On the basis of the above technical solution, after S110, it may also include: determining the allowed acquisition server corresponding to the first device message based on the target message transmission topic name and the allowed message transmission topic name corresponding to each acquisition server in the acquisition server cluster; if the target acquisition server is not the allowed acquisition server, sending the first device message to the allowed acquisition server so that the allowed acquisition server processes the first device message.

[0067] Specifically, in this embodiment, the collection server currently executing message processing is the target collection server in the collection server cluster. Each collection server in the collection server cluster can be pre-configured with the allowed message transmission topic name that is allowed to be processed. For the target collection server, that is, the collection server that executes message processing at the current moment, after receiving the first device message, the target message transmission topic name in the first device message can be matched with the allowed message transmission topic name corresponding to the collection server by calling the message routing module, and the collection server that successfully matches is determined as the allowed collection server corresponding to the first device message. Detect whether the allowed collection server is its own target collection server. If so, the message processing can continue in the target collection server. If not, the first device message is routed to the allowed collection server, so that the first device message is processed in the allowed collection server, so that the load can be balanced and high availability can be achieved.

[0068] Exemplarily, "generating a second device message based on the target device data and the target stream processing topic name" in S140 may include: obtaining the target message encoding configuration method corresponding to the target stream processing topic name; encoding the target device data based on the target message encoding configuration method to determine the target device encoding data; generating a second device message based on the target device encoding data and the target stream processing topic name.

[0069] Among them, the message encoding configuration method can be a message data compression format pre-configured for each stream processing topic name, such as binary encoding method, hexadecimal encoding method, etc., so as to achieve flexible encoding to meet different business needs.

[0070] Specifically, the target message encoding configuration mode corresponding to the target stream processing topic name can be determined according to the mapping relationship between the stream processing topic name and the message encoding configuration mode pre-configured in the configuration database. The target device data is encoded and processed using the target message encoding configuration mode, the target device encoding data is determined, and the second device message is generated based on the target device encoding data and the target stream processing topic name, so that the second device message can be encoded and compressed according to the target message encoding configuration mode, so that a suitable data encoding mode can be configured to reduce network transmission volume and improve transmission efficiency.

[0071] Figure 3 This is a flowchart of another message processing method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment further optimizes the step of "determining the target flow processing topic name corresponding to the IoT device according to the target message transmission topic name and the topic mapping relationship configuration information". The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0072] See also Figure 3 Another message processing method provided in this embodiment specifically includes the following steps:

[0073] S310: Acquire a first device message sent by the IoT device, where the first device message includes a target message transmission topic name and target device data generated by the IoT device.

[0074] S320. Obtain topic mapping relationship configuration information in a configuration database, wherein each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name.

[0075] S330: Match the target message transmission topic name with the message transmission topic prefix information in the topic mapping relationship configuration information to determine the successfully matched target message transmission topic prefix information.

[0076] Among them, the message transmission topic prefix information can be the prefix information of the message topic name that is pre-set based on business needs. If a more precise mapping match is required, more detailed message transmission topic prefix information can be set. The topic mapping relationship configuration information may include a one-to-one mapping relationship between the message transmission topic prefix information and the stream processing topic name. For example, the message transmission topic prefix information can be: / DATA / DEVICE / SEND / JZYQ-HB3 / *, then the data messages sent by the same device model, i.e., the JZYQ-HB3 device, can be distributed to the same stream processing topic name.

[0077] Specifically, the target message transmission topic name may be matched with each message transmission topic prefix information in the topic mapping relationship configuration information, and the message transmission topic prefix information with a successful prefix match may be determined as the target message transmission topic prefix information.

[0078] It should be noted that if the target message transmission topic name fails to match all the message transmission topic prefix information in the topic mapping relationship configuration information, the pre-set default stream processing topic name can be determined as the target stream processing topic name corresponding to the IoT device, so that the device message can be forwarded to the stream processing server corresponding to the default stream processing topic name, thereby avoiding message loss.

[0079] S340. Determine the target stream processing topic name corresponding to the Internet of Things device based on the stream processing topic name corresponding to the target message transmission topic prefix information in the topic mapping relationship configuration information.

[0080] Specifically, if there is only one successfully matched target message transmission topic prefix information in the topic mapping relationship configuration information, the stream processing topic name corresponding to the target message transmission topic prefix information in the topic mapping relationship configuration information can be directly determined as the target stream processing topic name corresponding to the Internet of Things device.

[0081] Exemplarily, S340 may include: if there are at least two target message transmission topic prefix information, obtaining the priority configuration information corresponding to each target message transmission topic prefix information; based on the priority configuration information, determining the target matching prefix information from each target message transmission topic prefix information; determining the stream processing topic name corresponding to the target matching prefix information in the topic mapping relationship configuration information as the target stream processing topic name corresponding to the Internet of Things device.

[0082] The priority configuration information may be a matching priority configured in advance for each message transmission topic prefix information. In this embodiment, a numerical value may be used to represent the priority configuration information, and a smaller numerical value indicates a higher matching priority.

[0083] Specifically, when there are at least two successfully matched target message transmission topic prefix information, the priority configuration information corresponding to each target message transmission topic prefix information can be obtained from the configuration database, and the various priority configuration information can be compared, and the target message transmission topic prefix information with the highest priority can be used as the target matching prefix information, and the stream processing topic name corresponding to the target matching prefix information in the topic mapping relationship configuration information can be determined as the target stream processing topic name, so that the target device data can be forwarded to the target stream processing topic name with the highest priority. By utilizing the priority, the flexibility of the configuration can be further improved to meet personalized business needs.

[0084] Exemplarily, when the MQTT Cluster is connected to the Kafka Cluster for data connection, the table fields in the mapping table in the configuration database may include: configuration identifier "id", message transmission topic prefix information "|mqtt_topic_prefix", stream processing topic name "kafka_topic", priority configuration information "match_priority", deletion identifier "is_delete" (1 means deleted, 0 means normal), valid identifier "is_active" (1 means valid, 0 means invalid), creation timestamp "create_time", and message encoding method "content_type".

[0085] S350. Generate a second device message based on the target device data and the target stream processing topic name, and send the second device message to the stream processing server so that the stream processing server processes the second device message.

[0086] The technical solution of this embodiment determines the successfully matched target message transmission topic prefix information by matching the target message transmission topic name with the message transmission topic prefix information in the topic mapping relationship configuration information, and determines the target stream processing topic name corresponding to the Internet of Things device based on the stream processing topic name corresponding to the target message transmission topic prefix information in the topic mapping relationship configuration information, thereby mapping multiple message transmission topic names to one stream processing topic name by prefix matching, thereby improving the topic mapping efficiency and further improving the data forwarding efficiency.

[0087] The following is an embodiment of a message processing device provided by an embodiment of the present invention. The device and the message processing methods of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the message processing device, reference can be made to the embodiment of the above message processing method.

[0088] Figure 4This is a schematic diagram of the structure of a message processing device provided by an embodiment of the present invention. This embodiment is applicable to the case where the device messages collected by the collection server are reported to the stream processing server. Figure 4 As shown, the apparatus is integrated in the acquisition server, and specifically includes: a first device message acquisition module 410, a mapping relationship configuration information acquisition module 420, a target stream processing topic name determination module 430 and a second device message generation module 440.

[0089] Among them, the first device message acquisition module 410 is used to obtain the first device message sent by the Internet of Things device, and the first device message includes the target message transmission topic name and the target device data generated by the Internet of Things device; the mapping relationship configuration information acquisition module 420 is used to obtain the topic mapping relationship configuration information in the configuration database, wherein each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name; the target stream processing topic name determination module 430 is used to determine the target stream processing topic name corresponding to the Internet of Things device according to the target message transmission topic name and the topic mapping relationship configuration information; the second device message generation module 440 is used to generate a second device message based on the target device data and the target stream processing topic name, and send the second device message to the stream processing server so that the stream processing server processes the second device message.

[0090] The technical solution of this embodiment can realize flexible mapping between message transmission topics and stream processing topics by pre-configuring topic mapping relationship configuration information, and each configured stream processing topic name corresponds to at least one message transmission topic name, so that one or more message transmission topic names can be mapped to the same stream processing topic name, solving the problem of small number of connection construction times in the stream processing server. The acquisition server can quickly and easily determine the target stream processing topic name corresponding to the Internet of Things device based on the target message transmission topic name in the first device message sent by the Internet of Things device and the topic mapping relationship configuration information in the configuration database, and send the second device message generated based on the target device data and the target stream processing topic name to the stream processing server, so that the stream processing server processes the second device message, thereby reducing the complexity of docking, ensuring efficient and fast message forwarding, and realizing efficient docking between the acquisition server and the stream processing server, so that the device message can be stored, processed and forwarded in the stream processing server, thereby improving the message processing performance.

[0091] Optionally, the target stream processing topic name determination module 430 includes:

[0092] A prefix matching unit, used to match the target message transmission topic name with the message transmission topic prefix information in the topic mapping relationship configuration information, and determine the target message transmission topic prefix information that matches successfully;

[0093] A target stream processing topic name determination unit is used to determine the target stream processing topic name corresponding to the Internet of Things device based on the stream processing topic name corresponding to the target message transmission topic prefix information in the topic mapping relationship configuration information.

[0094] Optionally, the target stream processing topic name determination unit is specifically used to:

[0095] If there are at least two target message transmission topic prefix information, obtain the priority configuration information corresponding to each target message transmission topic prefix information; based on the priority configuration information, determine the target matching prefix information from each target message transmission topic prefix information; and determine the stream processing topic name corresponding to the target matching prefix information in the topic mapping relationship configuration information as the target stream processing topic name corresponding to the Internet of Things device.

[0096] Optionally, the second device message generating module 440 is specifically configured to:

[0097] Obtain a target message encoding configuration method corresponding to the target stream processing topic name; based on the target message encoding configuration method, encode the target device data to determine the target device encoding data; based on the target device encoding data and the target stream processing topic name, generate a second device message.

[0098] Optionally, the first device message acquisition module 410 is specifically used to: acquire the first device message sent by the IoT device by calling a message monitoring module in a pre-loaded message processing plug-in;

[0099] The mapping relationship configuration information acquisition module 420 and the target stream processing topic name determination module 430 are specifically used to: obtain the topic mapping relationship configuration information in the configuration database by calling the rule engine module in the message processing plug-in; determine the target stream processing topic name corresponding to the IoT device according to the target message transmission topic name and the topic mapping relationship configuration information;

[0100] The second device message generation module 440 is specifically used to generate a second device message based on the target device data and the target stream processing topic name by calling the stream processing client module in the message processing plug-in, and send the second device message to the stream processing server.

[0101] Optionally, the acquisition server is a target acquisition server in an acquisition server cluster connected to the Internet of Things device;

[0102] The device also includes: a routing module, which is used to determine the allowed acquisition server corresponding to the first device message based on the target message transmission topic name and the allowed message transmission topic name corresponding to each acquisition server in the acquisition server cluster after obtaining the first device message sent by the Internet of Things device; if the target acquisition server is not the allowed acquisition server, the first device message is sent to the allowed acquisition server so that the allowed acquisition server processes the first device message.

[0103] Optionally, the second device message generating module 440 is further specifically configured to: send the second device message to a stream processing server cluster, so that a target stream processing server in the stream processing server cluster processes the second device message.

[0104] The message processing device provided in the embodiment of the present invention can execute the message processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the message processing method.

[0105] It is worth noting that in the embodiment of the above-mentioned message processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0106] Figure 5 This is a schematic diagram of a message processing system provided by an embodiment of the present invention. This embodiment is applicable to the case where device messages collected by a collection server are reported to a stream processing server. Figure 5 As shown, the system includes: an Internet of Things device 510, a collection server 520 and a stream processing server 530.

[0107] There may be multiple IoT devices 510. The acquisition server 520 may exist in the form of a cluster; the stream processing server 530 may also exist in the form of a cluster. The acquisition server 520 is used to implement the message processing method provided by any embodiment of the present invention.

[0108] In the message processing system of this embodiment, the acquisition server can quickly and easily determine the target stream processing topic name corresponding to the IoT device based on the target message transmission topic name in the first device message sent by the IoT device and the topic mapping relationship configuration information in the configuration database, and send the second device message generated based on the target device data and the target stream processing topic name to the stream processing server, so that the stream processing server processes the second device message, thereby reducing the complexity of docking, ensuring efficient and fast message forwarding, and realizing efficient docking between the acquisition server and the stream processing server, so that the device message can be stored, processed and forwarded in the stream processing server, thereby improving the message processing performance.

[0109] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 6 A block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 6 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0110] like Figure 6 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0111] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0112] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0113] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0114] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0115] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0116] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a message processing method provided by the embodiment of the present invention, the method comprising:

[0117] Acquire a first device message sent by an IoT device, where the first device message includes a target message transmission topic name and target device data generated by the IoT device;

[0118] Obtaining topic mapping relationship configuration information in a configuration database, wherein each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name;

[0119] Determine the target stream processing topic name corresponding to the IoT device according to the target message transmission topic name and the topic mapping relationship configuration information;

[0120] Based on the target device data and the target stream processing topic name, a second device message is generated, and the second device message is sent to the stream processing server so that the stream processing server processes the second device message.

[0121] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the message processing method provided by any embodiment of the present invention.

[0122] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the message processing method steps provided in any embodiment of the present invention are implemented. The method includes:

[0123] Acquire a first device message sent by an IoT device, where the first device message includes a target message transmission topic name and target device data generated by the IoT device;

[0124] Obtaining topic mapping relationship configuration information in a configuration database, wherein each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name;

[0125] Determine the target stream processing topic name corresponding to the IoT device according to the target message transmission topic name and the topic mapping relationship configuration information;

[0126] Based on the target device data and the target stream processing topic name, a second device message is generated, and the second device message is sent to the stream processing server so that the stream processing server processes the second device message.

[0127] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0128] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0129] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0130] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0132] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A message processing method, characterized in that: Applied to the acquisition server, including: Obtain a first device message sent by an IoT device, wherein the first device message includes a target message transmission topic name and target device data generated by the IoT device; wherein the target message transmission topic name refers to a data transmission topic name Topic corresponding to the target device data generated by the IoT device at the current moment; Obtain topic mapping relationship configuration information in a configuration database, wherein each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name; the stream processing topic name refers to a topic name Topic in a stream processing server; Determine the target stream processing topic name corresponding to the IoT device according to the target message transmission topic name and the topic mapping relationship configuration information; Based on the target device data and the target stream processing topic name, a second device message is generated, and the second device message is sent to the stream processing server so that the stream processing server processes the second device message.

2. The method according to claim 1, characterized in that The determining, according to the target message transmission topic name and the topic mapping relationship configuration information, the target stream processing topic name corresponding to the Internet of Things device comprises: Matching the target message transmission topic name with the message transmission topic prefix information in the topic mapping relationship configuration information to determine the target message transmission topic prefix information that matches successfully; Based on the stream processing topic name corresponding to the target message transmission topic prefix information in the topic mapping relationship configuration information, determine the target stream processing topic name corresponding to the Internet of Things device.

3. The method according to claim 2, characterized in that The determining the target stream processing topic name corresponding to the Internet of Things device based on the stream processing topic name corresponding to the target message transmission topic prefix information in the topic mapping relationship configuration information includes: If there are at least two target message transmission topic prefix information, obtaining the priority configuration information corresponding to each target message transmission topic prefix information; Based on the priority configuration information, determining target matching prefix information from each of the target message transmission topic prefix information; The stream processing topic name corresponding to the target matching prefix information in the topic mapping relationship configuration information is determined as the target stream processing topic name corresponding to the Internet of Things device.

4. The method according to claim 1, characterized in that: The generating a second device message based on the target device data and the target stream processing topic name includes: Obtain the target message encoding configuration method corresponding to the target stream processing topic name; Based on the target message encoding configuration mode, encoding the target device data to determine the target device encoding data; A second device message is generated based on the target device encoding data and the target stream processing topic name.

5. The method according to claim 1, characterized in that The obtaining of a first device message sent by the IoT device includes: Obtain a first device message sent by the IoT device by calling a message monitoring module in a pre-loaded message processing plug-in; The acquiring of the topic mapping relationship configuration information in the configuration database; and determining the target stream processing topic name corresponding to the IoT device according to the target message transmission topic name and the topic mapping relationship configuration information, include: By calling the rule engine module in the message processing plug-in, the topic mapping relationship configuration information in the configuration database is obtained; according to the target message transmission topic name and the topic mapping relationship configuration information, the target flow processing topic name corresponding to the Internet of Things device is determined; The step of generating a second device message based on the target device data and the target stream processing topic name, and sending the second device message to the stream processing server includes: By calling the stream processing client module in the message processing plug-in, a second device message is generated based on the target device data and the target stream processing topic name, and the second device message is sent to the stream processing server.

6. The method according to claim 1, characterized in that The acquisition server is a target acquisition server in the acquisition server cluster connected to the Internet of Things device; After obtaining the first device message sent by the IoT device, the method further includes: Determine the allowed acquisition server corresponding to the first device message based on the target message transmission topic name and the allowed message transmission topic name corresponding to each acquisition server in the acquisition server cluster; If the target acquisition server is not the allowed acquisition server, the first device message is sent to the allowed acquisition server, so that the allowed acquisition server performs message processing on the first device message.

7. The method according to any one of claims 1 to 6, characterized in that: The sending the second device message to a stream processing server so that the stream processing server processes the second device message includes: The second device message is sent to a stream processing server cluster, so that a target stream processing server in the stream processing server cluster processes the second device message.

8. A message processing device, characterized in that: Integrated in the acquisition server, including: The first device message acquisition module is used to acquire the first device message sent by the IoT device. The first device message includes a target message transmission topic name and target device data generated by the IoT device; wherein the target message transmission topic name refers to the data transmission topic name Topic corresponding to the target device data generated by the IoT device at the current moment; A mapping relationship configuration information acquisition module is used to acquire topic mapping relationship configuration information in a configuration database, wherein each stream processing topic name in the topic mapping relationship configuration information corresponds to at least one message transmission topic name; the stream processing topic name refers to the topic name Topic in the stream processing server; A target stream processing topic name determination module is used to determine the target stream processing topic name corresponding to the Internet of Things device according to the target message transmission topic name and the topic mapping relationship configuration information; The second device message generating module is used to generate a second device message based on the target device data and the target stream processing topic name, and send the second device message to the stream processing server so that the stream processing server processes the second device message.

9. A message processing system, characterized in that: The system includes: an Internet of Things device, a collection server and a stream processing server; wherein, The acquisition server is used to implement the message processing method as described in any one of claims 1-7.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the message processing method as described in any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the message processing method as described in any one of claims 1 to 7 is implemented.

Images