Message processing method and device and electronic equipment

Abstract

The invention discloses a message processing method and device and electronic equipment. The method comprises the following steps: receiving a plurality of original network device messages from a plurality of network devices respectively; according to a target attribute in the plurality of original network device messages, the plurality of original network device messages are distributed to a plurality of partitions of the message middleware, each partition corresponds to a subnet, and the target attribute is used for indicating a network manager and a subnet to which the original network device messages belong; and concurrently processing the original network equipment messages respectively corresponding to the plurality of partitions by adopting the processing threads respectively corresponding to the plurality of partitions. The technical problems that the processing delay is high when a single-thread message processing mode is adopted, and the time sequence is damaged due to the fact that the messages of the same subnet are distributed to different processing units are solved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a message processing method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of 5G, IoT, and cloud computing technologies, the amount of data in optical networks is experiencing explosive growth. Traditional single-threaded message processing methods face the following problems when handling high concurrency and large-scale data:

[0003] Performance Bottlenecks: Single-threaded processing cannot meet real-time requirements, resulting in processing latency of up to 1 second. Especially when encountering update deadlock issues, it will block the entire system, severely impacting the normal processing of other services. Message Out-of-Order: Messages from the same subnet may be assigned to different processing units, disrupting temporal order and causing causal misalignment. Low Supplemental Acquisition Efficiency: Synchronous supplemental acquisition mechanisms block the main thread, reducing overall system efficiency. Poor Scalability: It is difficult to dynamically adapt to changes in data scale; from millions of messages per day previously to tens of millions per day currently, single-threaded processing latency reaches up to 5 seconds.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a message processing method, apparatus, and electronic device to at least solve the technical problems of high latency in single-threaded message processing and the disruption of timing caused by messages from the same subnet being assigned to different processing units.

[0006] According to one aspect of the embodiments of this application, a message processing method is provided, comprising: receiving multiple original network device messages from multiple network devices respectively; distributing the multiple original network device messages to multiple partitions of a message middleware according to a target attribute in the multiple original network device messages, wherein each partition corresponds to a subnet, and the target attribute is used to indicate the network management system and subnet to which the original network device message belongs; and using processing threads corresponding to the multiple partitions respectively to concurrently process the original network device messages corresponding to the multiple partitions respectively.

[0007] Optionally, the multiple original network device messages are distributed to multiple partitions of the message middleware based on the target attributes in the multiple original network device messages, including: standardizing the multiple original network device messages to obtain multiple standard network device messages, wherein the data format of the standard network device messages is a format adapted by the message middleware; for each standard network device message, determining the network management identifier and subnet identifier carried in the standard network device message as target attributes; converting the target attributes of each standard network device message into a hash key corresponding to each standard network device message through a target hash algorithm; and distributing the multiple standard network device messages to multiple partitions of the message middleware based on the hash keys corresponding to the multiple standard network device messages respectively.

[0008] Optionally, the multiple standard network device messages are distributed to multiple partitions of the message middleware according to the hash keys corresponding to the multiple standard network device messages, including: determining the total number of partitions of the message middleware; performing partition mapping according to the total number of partitions and the hash keys corresponding to the multiple standard network device messages to obtain a mapping result, wherein the mapping result is used to indicate the hash key corresponding to each partition; and distributing the multiple standard network device messages to multiple partitions of the message middleware according to the mapping result.

[0009] Optionally, multiple processing threads corresponding to each partition are used to concurrently process the original network device messages corresponding to each partition, including: allocating a processing thread for each partition; using the processing threads corresponding to each partition to concurrently process all standard network device messages in the corresponding partition in chronological order according to the timestamps in the standard network device messages, and verifying the standard network device messages before processing each message to obtain a verification result; if the verification result indicates that the target standard network device message has missing data, data supplementation processing is performed, wherein the target standard network device message is any one of the multiple standard network device messages.

[0010] Optionally, before processing each standard network device message, the standard network device message is validated to obtain a validation result, including: performing a first validation on the target standard network device message to obtain a first validation result, wherein the first validation verifies the attributes and data format of the target standard network device message; performing a second validation on the target standard network device message to obtain a second validation result, wherein the second validation verifies the association relationship of the target standard network device message; determining a validation result based on the first validation result and the second validation result, wherein if at least one of the first validation result and the second validation result fails the validation, it is determined that the target standard network device message has missing data.

[0011] Optionally, if the verification result indicates that there is data missing in the target standard network device message, data supplementation processing is performed, including: generating supplementation tasks for all target standard network device messages with data missing, and forming all supplementation tasks into a priority task queue; assigning priority to each supplementation task in the priority task queue according to the data importance corresponding to the target standard network device message, and executing the supplementation tasks in the priority task queue through threads in an asynchronous thread pool in order of priority from high to low.

[0012] Optionally, the initial number of threads in the asynchronous thread pool is determined based on the number of CPU cores on the server. During the execution of the supplementary data collection task, the initial number of threads is adjusted according to the number of supplementary data collection tasks in the priority task queue. If the number of supplementary data collection tasks in the priority task queue exceeds the monitoring threshold built into the asynchronous thread pool, an alarm message is generated and pushed to the administrator.

[0013] According to another aspect of the embodiments of this application, a message processing apparatus is also provided, comprising: a receiving module, configured to receive multiple original network device messages from multiple network devices respectively; an allocation module, configured to allocate the multiple original network device messages to multiple partitions of a message middleware according to a target attribute in the multiple original network device messages, wherein each partition corresponds to a subnet, and the target attribute is used to indicate the network management system and subnet to which the original network device message belongs; and a processing module, configured to concurrently process the original network device messages corresponding to the multiple partitions using processing threads corresponding to the multiple partitions respectively.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, and the program controls the device where the non-volatile storage medium is located to execute the above message processing method when it runs.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program executes the above-described message processing method during runtime.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the above-described message processing method.

[0017] In this embodiment, multiple raw network device messages from multiple network devices are received. Based on the target attributes in the raw network device messages, the messages are distributed to multiple partitions of a message middleware, where each partition corresponds to a subnet. The target attributes indicate the network management system and subnet to which the raw network device message belongs. Processing threads corresponding to each partition are used to process the raw network device messages concurrently. By distributing the raw network device messages to multiple partitions of the message middleware, each partition corresponding to a subnet, and then using processing threads corresponding to each partition to process the raw network device messages concurrently, a multi-threaded message processing method is achieved, ensuring that messages from a subnet are only assigned to one partition. This solves the technical problems of high latency and time-sequence disruption caused by assigning messages from the same subnet to different processing units when using a single-threaded message processing method. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a message processing method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a message processing method provided according to an embodiment of this application;

[0021] Figure 3 This is a message hash routing class diagram provided according to an embodiment of this application;

[0022] Figure 4 This is a partitioning and multithreading processing class diagram provided according to an embodiment of this application;

[0023] Figure 5 This is a real-time verification and supplementary sampling class diagram provided according to an embodiment of this application;

[0024] Figure 6 This is a flowchart of another message processing method provided according to an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of a message processing apparatus provided according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0030] Hash Routing: Data is distributed to different processing units using a hash algorithm, ensuring that data with the same key value is assigned to the same unit.

[0031] Partition: A logical unit in message middleware that divides data into pieces for storage and processing, with data within each partition ordered.

[0032] Asynchronous thread pool: A concurrent processing mechanism that manages asynchronous tasks through a thread pool, improving resource utilization and task processing efficiency.

[0033] In related technologies, single-threaded message processing methods face the following problems when dealing with high concurrency and large-scale data:

[0034] Performance Bottlenecks: Single-threaded processing cannot meet real-time requirements, resulting in processing latency of up to 1 second. Especially when encountering update deadlock issues, it will block the entire system, severely impacting the normal processing of other services. Message Out-of-Order: Messages from the same subnet may be assigned to different processing units, disrupting timeliness and causing causal misalignment. Low Supplementation Efficiency: Synchronous supplementation mechanisms block the main thread, reducing overall system efficiency. Poor Scalability: It is difficult to dynamically adapt to changes in data scale. From millions of messages per day previously to tens of millions per day currently, single-threaded processing latency reaches up to 5 seconds. Therefore, there are technical problems with single-threaded message processing, including high latency and the disruption of timeliness caused by messages from the same subnet being assigned to different processing units. To solve these problems, this application provides relevant solutions, which are detailed below.

[0035] According to an embodiment of this application, an embodiment of a message processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a message processing method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the message processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned message processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0040] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0041] Under the above operating environment, this application provides an embodiment of a message processing method, such as... Figure 2 As shown, a flowchart of a message processing method according to an embodiment of this application includes:

[0042] Step S202: Receive multiple raw network device messages from multiple network devices.

[0043] In some embodiments of this application, the message processing method is applicable to high-throughput, low-latency optical network data management scenarios. First, step S202 is performed, executing the message processing system of the message processing method of this application embodiment. This system receives multiple raw network device messages from multiple network devices through a message-converter service. This service, as the entry point of the message processing system, is mainly responsible for receiving raw network device messages from different manufacturers and performing standardized processing and intelligent distribution. Its core functions include message format conversion, key attribute extraction, hash key calculation, and message partitioning and routing. Multiple raw network device messages originate from multiple network devices, which may be manufactured by different companies. The raw network device messages contain information such as device status, alarm information, performance data, and configuration changes. Network devices are physical or virtual devices in a network, such as routers, switches, and firewalls. A subnet refers to a group of devices or network segments in a network. These devices are logically considered as a single unit, sharing the same network address range or management policies. A network management system is a software platform used to monitor, manage, and control network devices. The network administrator manages and monitors subnets. A subnet is the logical unit managed by the network administrator, and each subnet contains one or more network devices.

[0044] Step S204: Distribute the multiple original network device messages to multiple partitions of the message middleware according to the target attributes in the multiple original network device messages.

[0045] In the technical solution provided in step S204, each partition corresponds to a subnet, and the target attribute is used to indicate the network management system and subnet to which the original network device message belongs. There are several ways to allocate multiple original network device messages to multiple partitions of the message middleware based on the target attributes in the multiple original network device messages. For example: standardize the multiple original network device messages to obtain multiple standard network device messages, wherein the data format of the standard network device messages is a format adapted by the message middleware; for each standard network device message, determine the network management system identifier and subnet identifier carried in the standard network device message as the target attribute; convert the target attribute of each standard network device message into a hash key corresponding to each standard network device message using a target hash algorithm; and allocate the multiple standard network device messages to multiple partitions of the message middleware based on the hash keys corresponding to the multiple standard network device messages respectively.

[0046] In the above steps, there are several ways to allocate multiple standard network device messages to multiple partitions of the message middleware based on the hash keys corresponding to each of the multiple standard network device messages. For example: determine the total number of partitions in the message middleware; perform partition mapping based on the total number of partitions and the hash keys corresponding to each of the multiple standard network device messages to obtain the mapping result, wherein the mapping result is used to indicate the hash key corresponding to each partition; allocate multiple standard network device messages to multiple partitions of the message middleware based on the mapping result. The following details the process of allocating multiple original network device messages to multiple partitions of the message middleware based on the target attributes in multiple original network device messages in step S204.

[0047] Due to significant differences in message formats among various equipment manufacturers (including XML, JSON, binary, and other formats), the message receiving service first performs unified standardization processing on the messages. Multiple raw network device messages are standardized through format validation, field mapping, data cleaning, and format conversion operations to obtain multiple standard network device messages. The field mapping operation relies on a field mapping table, which defines the correspondence between fields in the raw network device messages and fields in the standard messages. Different message formats and different device types correspond to different mapping rules. Then, the field mapping table is used to map the fields in the raw network device messages to their corresponding fields in the standard message format (involving field name conversion, data type adaptation, and unit unification), ensuring that all messages use consistent field naming and data format. The format conversion operation converts the messages, after format validation, field mapping, and data cleaning, into a selected target format (a format adapted by the message middleware). In some embodiments of this application, the message middleware is Kafka (Kafka is an open-source distributed stream processing platform).

[0048] After obtaining multiple standard network device messages, the network management identifier and subnet identifier carried in each standard network device message are extracted (i.e., the field content of the network management identifier and subnet identifier carried in each standard network device message is read). The network management identifier and subnet identifier are determined as the target attributes of the standard network device message, because they can uniquely identify the source of the message and the subnet to which it belongs, ensuring that messages from the same subnet are distributed to the same partition. The two identifiers in the target attributes of each standard network device message are combined into a string and used as input to the target hash algorithm (e.g., a consistent hash algorithm, such as Kafka's default partitioning strategy, which hashes based on the target attributes of the message and performs a modulo operation with the number of partitions to ensure that the mapping from hash key to partition is consistent and avoids load imbalance). Then, the selected target hash algorithm is used to hash the string to generate a hash key corresponding to each standard network device message. This hash key is the standard network device message key field.

[0049] The next step is to determine the total number of partitions in the message middleware (in Kafka, the number of partitions for a topic is determined at creation time; the total number of partitions is greater than the total number of current subnets, and this greater number is determined based on both future subnet growth needs and performance and resource constraints (assessing whether current hardware and infrastructure can support the predetermined number of partitions)). Then, the generated hash keys are mapped to specific partitions. Partition mapping is performed based on the total number of partitions and the hash keys corresponding to multiple standard network device messages. Specifically, the hash value of the key field is moduloed with the total number of partitions to determine the partition position of the message, resulting in the mapping result: the partition position of each standard network device message = hashValue%numPartitions, where numPartitions is the total number of partitions and hashValue is the hash key (i.e., hash value). The partition positions of all standard network device messages are organized into a mapping result. Based on the mapping result, multiple standard network device messages are distributed to multiple partitions of the message middleware. At this time, standard network device messages with the same partition position will be assigned to the same partition. Because the hash key is determined based on the network management identifier and the subnet identifier, this Kafka partitioning mechanism ensures that messages from the same subnet enter the same partition. That is, one partition corresponds to one subnet and contains all standard network device messages of one subnet.

[0050] Step S206: Processing threads corresponding to multiple partitions are used to process the original network device messages corresponding to the multiple partitions concurrently.

[0051] In the technical solution provided in step S206, there are several ways to implement the concurrent processing of original network device messages corresponding to multiple partitions using processing threads corresponding to each partition. For example: allocating one processing thread for each partition; using processing threads corresponding to multiple partitions to concurrently process all standard network device messages in the corresponding partition in the order of the timestamps in the standard network device messages, and verifying the standard network device messages before processing each message to obtain the verification result; and performing data supplementation processing when the verification result indicates that the target standard network device message has missing data, wherein the target standard network device message is any one of the multiple standard network device messages.

[0052] In the above steps, the standard network device message is validated before processing each message. There are several ways to obtain the validation result. For example, a first validation is performed on the target standard network device message to obtain a first validation result, where the first validation verifies the attributes and data format of the target standard network device message; a second validation is performed on the target standard network device message to obtain a second validation result, where the second validation verifies the association relationship of the target standard network device message; and a validation result is determined based on the first and second validation results. If at least one of the first and second validation results fails the validation, it is determined that the target standard network device message has missing data.

[0053] When the verification result indicates that there is missing data in the target standard network device message, there are several ways to perform data supplementation processing. For example, generate supplementation tasks for all target standard network device messages with missing data, and form all supplementation tasks into a priority task queue; assign priority to each supplementation task in the priority task queue according to the data importance corresponding to the target standard network device message, and execute the supplementation tasks in the priority task queue through threads in an asynchronous thread pool in order of priority from high to low.

[0054] The initial number of threads in the asynchronous thread pool is determined based on the number of CPU cores on the server. During the execution of supplementary data collection tasks in the priority task queue, the initial number of threads is adjusted according to the number of supplementary data collection tasks in the priority task queue. If the number of supplementary data collection tasks in the priority task queue exceeds the monitoring threshold built into the asynchronous thread pool, an alarm message is generated and pushed to the administrator. Step S206 is described in detail below.

[0055] After distributing multiple standard network device messages to multiple partitions in the message middleware, an independent processing thread is assigned to each partition (Kafka partition). This fully utilizes the server's multi-core CPU capabilities to improve message processing speed. Each thread processes only standard network device messages from a fixed partition, avoiding data contention among multiple threads and reducing waiting time. Specifically, the processing threads for each partition concurrently process all standard network device messages in their respective partitions in chronological order according to the timestamps in the messages, ensuring that messages within the same subnet are processed sequentially and avoiding out-of-order issues. It's important to note that during message processing, the correlation and integrity of each message are monitored in real time. Each partition verifies each standard network device message before processing it, obtaining the verification result. If the verification passes, the message is processed normally. The processing thread parses each standard network device message, extracting key attributes and data fields. These include information such as message ID, timestamp, device status, and performance metrics, depending on the message content and business requirements. Based on the extracted message attributes, the corresponding business logic is executed. For example, in network management scenarios, this includes analyzing device status, calculating changes in performance parameters, triggering alarm mechanisms, and updating database records.

[0056] If the verification result indicates that the target standard network device message contains missing data, data supplementation will be performed. If the supplementation task fails multiple times consecutively (e.g., 3 times), it will be automatically escalated to an abnormal task, and the operation and maintenance personnel will be notified to intervene. All supplementation tasks and their execution status will be logged in detail, supporting post-event auditing and problem tracing.

[0057] Before processing each standard network device message, the message is validated. The validation process employs a multi-level checking strategy: first, a rapid basic validation (the first validation mentioned above) is performed, followed by deep correlation analysis (the second validation mentioned above). This automatically identifies issues such as missing data, abnormal formatting, or broken relationships, ensuring that no data problems are overlooked while maintaining processing efficiency. The validation process is detailed below:

[0058] The first validation step involves verifying the attributes and data format of the target standard network device message: This check confirms that the message uses the correct encoding and contains the necessary fields. For example, it verifies that the message includes critical information such as timestamp, network management identifier, subnet identifier, and device status. Attribute integrity check: Ensures that all required attributes exist and are not empty. Any missing critical attributes will mark the message as invalid. Timestamp format validation: Verifies that the timestamp conforms to the predetermined time format and range to avoid data processing errors caused by incorrect timestamps. Failure to pass any of the above steps results in the first validation failing.

[0059] The second verification checks the relationships between target standard network device messages: Historical data correlation verification: Compare the current target standard network device messages with historical messages stored in the database to check for unexpected state transitions or data anomalies. Cross-message correlation verification: If target standard network device message processing involves tracking the status of multiple devices or subnets, the logical relationships between these related messages need to be checked to ensure there are no breaks in the relationships. Business rule verification: Apply preset business rules to further check the validity of target standard network device messages, such as traffic threshold checks, device status consistency checks, and network operation compliance reviews. If any of the above fails, the second verification fails.

[0060] The results of the first and second verifications are taken as the final verification results. If at least one of the first and second verification results fails, it is determined that the target standard network device message has missing data. When the verification results indicate missing data in the target standard network device message, data re-acquisition processing is performed on that message. All target standard network device messages with missing data automatically generate re-acquisition tasks. All re-acquisition tasks form a priority task queue. Priority is assigned to each re-acquisition task in the priority task queue based on the data importance corresponding to the target standard network device message (the data importance of the original network device message corresponding to each standard network device message is pre-set). Then, re-acquisition tasks in the priority task queue are executed in descending order of priority through threads in an asynchronous thread pool. The re-acquisition tasks are executed through an independent asynchronous thread pool, completely decoupled from the main processing flow, ensuring that the main thread's processing performance is not affected. Metadata for the re-acquisition tasks is also recorded for subsequent tracking and analysis.

[0061] The asynchronous thread pool employs a dynamic resource allocation strategy. The initial thread count is automatically calculated based on the number of server CPU cores. It features built-in task queue monitoring (with a set monitoring threshold). If the number of tasks in the priority task queue exceeds the pool's built-in monitoring threshold, it indicates that the current thread count is insufficient to handle the backlog. An alarm is generated and pushed to administrators, informing them of the specific reason (e.g., a task timeout warning), preventing individual long-running tasks from consuming resources. During the execution of tasks in the priority task queue, the initial thread count is adjusted based on the number of tasks in the queue, taking into account system load to avoid resource waste or overload. For example, if the task length in the priority task queue exceeds a preset adjustment threshold (less than the monitoring threshold), and the system load has not reached a preset warning line, the thread count can be increased. The increase in thread count is determined based on the queue length increase ratio or the expected processing speed improvement. When the priority task queue length decreases significantly and the system load is low, the thread count can be reduced accordingly to free up resources for other services or tasks.

[0062] The message processing method of this application embodiment is also applicable to other high-throughput, low-latency distributed message processing scenarios, such as network management: real-time monitoring of network management status, processing alarms and performance data; subnet monitoring: ensuring orderly processing of messages within the same subnet and avoiding timing errors; and real-time data analysis: supporting high-throughput data stream processing, such as traffic analysis and fault detection. The multi-threaded parallel processing architecture of this application embodiment, combined with intelligent load balancing, increases system data throughput by more than 50%, and stably controls processing latency within 200ms, meeting high real-time business requirements. A hash-key-based hash routing precise partitioning strategy ensures that messages within the same subnet are always stored in the same partition, strictly maintaining the time sequence of data and avoiding timestamp out-of-order issues caused by cross-partition distribution, ensuring the strict sequential execution of business logic. An asynchronous data acquisition mechanism reduces the completion time of acquisition tasks by more than 60% without affecting the main process performance, ensuring high system availability and balancing data processing efficiency and integrity. Based on a Kafka-based partition replication mechanism, even if a single node fails, seamless switching to other replicas ensures no message loss and uninterrupted service. In addition, it supports dynamically adjusting the number of partitions and thread resources, enabling flexible and smooth scaling from millions to tens of millions of data points, adapting to the needs of rapid business growth. It supports dynamically expanding the number of partitions and threads to accommodate data scales from millions to tens of millions.

[0063] The message processing method in this application significantly improves system performance and operational efficiency while ensuring strong data consistency. Actual deployment tests show that this solution, through intelligent hashing (partition allocation via hash keys) and parallel partitioning, significantly improves data processing throughput and reduces unit data processing costs under the same hardware conditions, while keeping data processing latency within 200ms, meeting real-time requirements. The system employs a multi-threaded collaborative mechanism, effectively improving resource utilization and reducing computational resource waste. Unique real-time verification and automatic data acquisition functions reduce the need for manual maintenance, making the system more stable and reliable. These technological improvements enable the system to maintain high performance while possessing good scalability and fault tolerance, making it particularly suitable for real-time data processing needs in large-scale optical network environments.

[0064] Figure 3 This is a message hash routing class diagram provided according to an embodiment of this application, which shows the main classes and relationships involved in the message hash routing mechanism (the process of allocating multiple standard network device messages to multiple partitions of the message middleware based on the hash keys corresponding to the multiple standard network device messages in step S204).

[0065] This mechanism ensures that multiple standard network device messages are accurately routed to the appropriate partitions based on their key attributes (such as network manager ID and subnet ID), thereby achieving ordered and parallel message processing. The MessageConverter class includes methods for receiving raw network device messages (+receiveRawMessage()), extracting key attributes for hash calculation from the raw network device messages (+extractHashKey()), and distributing messages to specific partitions based on the calculated hash key (+distributeMessage()). The HashKey class includes two attributes: network manager ID (-networkManager) and subnet ID (-subnet), with methods for obtaining the network manager ID (+getNetworkManager()) and obtaining the subnet ID (+getSubnet()). The RawMessage class includes the content of the raw network device message (-messageContent) and methods for obtaining the raw message content (+getMessageContent()). The ConsistentHashing class includes a method (+hash(HashKey)) that receives a HashKey object and calculates its hash value, and a Kafka Partition object determined by the hash calculation result. The Kafka Partition class includes a unique identifier for the partition (-partitionId) and a method (+getPartitionId()) that retrieves the partition ID. The MessageConverter class receives raw network device messages from the RawMessage class. The HashKey class extracts and normalizes the raw network device messages received by the MessageConverter class, inputs them for hashing using the ConsistentHashing class, obtains the hash key for each raw network device message, and finally outputs the hash value to distribute it to the corresponding partition.

[0066] Figure 4This is a class diagram of partitioning and multithreaded processing provided in an embodiment of this application. It illustrates the classes involved in the multithreaded parallel processing of standard network device messages in a Kafka partition in step S206. The message handler (MessageHandlerService) class is responsible for processing all standard network device messages assigned to a specific partition. It includes a consumer object (-kafkaConsumer: kafkaConsumer, used to subscribe to and consume messages in the Kafka partition), a thread pool manager (-threadPool: ThreadPoolManager, used to manage the threads required for message processing), and a message processing method (+handleMessages()). The message processing thread (MessageProcessorThread) class includes the Kafka partition number corresponding to the thread (-partition: int, an integer), -kafkaConsumer: KafkaConsumer, +run() (+run() is a basic Java thread method that starts the thread to execute message processing), and a method for processing the message list in a specific partition (+processMessages(messages: List)). The Message class includes a timestamp attribute (-timestamp: long) and a subnet ID attribute (-subnet: String). The KafkaConsumer class contains methods for consuming messages from a specified Kafka partition, returning a list of standard network device messages in the partition (+consumeMessages(partition: int): List). The message processor creates MessageProcessorThread instances, each responsible for processing messages for one partition. It then uses KafkaConsumer to read messages from a specific partition. MessageProcessorThread processes messages.

[0067] Figure 5This is a class diagram for real-time verification and data reacquisition provided according to an embodiment of this application. It illustrates the classes involved in verification and data reacquisition in step S206. The MessageProcessor class is responsible for processing standard network device messages and includes message processing methods, methods for executing message business logic processing (+processMessage()), methods for checking message integrity (+checkMessageIntegrity()), and methods for triggering data reacquisition (+triggerReacquisition()). The Message class represents the data entity of a standard network device message and includes a string used to indicate the association between messages (-association: String) and the specific data content of the message, which can be any type of object (-data: Object). The ReacquisitionMechanism class contains methods for executing reacquisition tasks (+executeReacquisitionTask()). The AsyncThreadPoolManager class contains the current thread pool size (-poolSize: int), a priority queue for tasks to be processed (-taskQueue: Queue), a method to configure the thread pool size (+configureThreadPoolSize()), and a method to monitor the task queue (+monitorTaskQueue()) for detecting queue length and adjusting the thread pool size or issuing alerts. The MessageProcessor processes messages and triggers the ReacquisitionMechanism to execute reacquisition tasks. The ReacquisitionMechanism uses the AsyncThreadPoolManager to execute these reacquisition tasks.

[0068] Figure 6This is a flowchart of another message processing method provided according to an embodiment of this application. First, the message is received by the optical network management system (i.e., the aforementioned management system) and sent to the message receiving service (message-converter). Then, a hash algorithm is used to distribute the message to the specified Kafka partition (i.e., step S204 above). The next step is to perform message processing service (message-handler): multi-threaded processing (each Kafka partition corresponds to an independent processing thread), and real-time verification is performed simultaneously (checking message integrity and correlation) to determine whether missing data is found. If so, a supplementary data acquisition task is triggered, and supplementary data acquisition is performed through an independent asynchronous thread pool (i.e., step S206 above).

[0069] Figure 7 This is a schematic diagram of a message processing apparatus according to an embodiment of this application, comprising:

[0070] The receiving module 702 is used to receive multiple raw network device messages from multiple network devices.

[0071] The allocation module 704 is used to allocate multiple original network device messages to multiple partitions of the message middleware based on the target attributes in the multiple original network device messages. Each partition corresponds to a subnet, and the target attributes are used to indicate the network management system and subnet to which the original network device message belongs.

[0072] The allocation module 704 is also used to allocate multiple original network device messages to multiple partitions of the message middleware based on the target attributes in the multiple original network device messages: standardizing the multiple original network device messages to obtain multiple standard network device messages, wherein the data format of the standard network device messages is a format adapted by the message middleware; for each standard network device message, determining the network management identifier and subnet identifier carried in the standard network device message as target attributes; converting the target attributes of each standard network device message into a hash key corresponding to each standard network device message using a target hash algorithm; and allocating the multiple standard network device messages to multiple partitions of the message middleware based on the hash keys corresponding to the multiple standard network device messages respectively.

[0073] The allocation module 704 is also used to allocate multiple standard network device messages to multiple partitions of the message middleware according to the hash keys corresponding to the multiple standard network device messages respectively: determine the total number of partitions of the message middleware; perform partition mapping according to the total number of partitions and the hash keys corresponding to the multiple standard network device messages respectively to obtain the mapping result, wherein the mapping result is used to indicate the hash key corresponding to each partition; and allocate multiple standard network device messages to multiple partitions of the message middleware according to the mapping result.

[0074] The processing module 706 is used to concurrently process the raw network device messages corresponding to the multiple partitions using processing threads corresponding to the multiple partitions.

[0075] The processing module 706 is also used to allocate a processing thread for each partition; to use the processing threads corresponding to multiple partitions to process all standard network device messages in the corresponding partition in the order of the timestamps in the standard network device messages, and to verify the standard network device messages before processing each standard network device message to obtain the verification result; if the verification result indicates that the target standard network device message has missing data, to perform data supplementation processing, wherein the target standard network device message is any one of the multiple standard network device messages.

[0076] The processing module 706 is further configured to perform a verification on each standard network device message before processing it, and obtain a verification result: perform a first verification on the target standard network device message to obtain a first verification result, wherein the first verification is to verify the attributes and data format of the target standard network device message; perform a second verification on the target standard network device message to obtain a second verification result, wherein the second verification is to verify the association relationship of the target standard network device message; determine the verification result based on the first verification result and the second verification result, wherein if at least one of the first verification result and the second verification result fails the verification, it is determined that the target standard network device message has missing data.

[0077] The processing module 706 is also used to perform data acquisition processing when the verification result indicates that there is data missing in the target standard network device message: generate acquisition tasks for all target standard network device messages with data missing, and form all acquisition tasks into a priority task queue; assign priority to each acquisition task in the priority task queue according to the data importance corresponding to the target standard network device message, and execute the acquisition tasks in the priority task queue through threads in the asynchronous thread pool in order of priority from high to low.

[0078] It should be noted that, Figure 7 The message processing device shown is used to perform Figure 2 The message processing method shown, therefore Figure 2 The explanations and descriptions regarding the message processing methods in the document also apply to the message processing device, and will not be repeated here.

[0079] It should be noted that each module in the above message processing device can be a program module (e.g., a set of program instructions that implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0080] This application also provides a non-volatile storage medium, which includes a stored program, wherein, during program execution, the device where the non-volatile storage medium is located executes the message processing method of any of the above embodiments.

[0081] This application also provides an electronic device, which includes a processor for running a program, wherein the message processing method of any of the above embodiments is executed during program execution.

[0082] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the message processing method of any of the above embodiments.

[0083] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0084] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0088] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A message processing method, characterized in that, include: Receive multiple raw network device messages from multiple network devices; Based on the target attributes in the multiple original network device messages, the multiple original network device messages are distributed to multiple partitions of the message middleware, wherein each partition corresponds to a subnet, and the target attributes are used to indicate the network management system and subnet to which the original network device message belongs; The processing threads corresponding to the multiple partitions are used to process the original network device messages corresponding to the multiple partitions concurrently.

2. The method according to claim 1, characterized in that, The step of allocating the multiple original network device messages to multiple partitions of the message middleware based on the target attributes in the multiple original network device messages includes: The multiple original network device messages are standardized to obtain multiple standard network device messages, wherein the data format of the standard network device messages is the format adapted by the message middleware; For each of the aforementioned standard network device messages, the network management identifier and subnet identifier carried in the standard network device message are determined as the target attributes; The target attribute of each standard network device message is converted into a hash key corresponding to each standard network device message using a target hash algorithm; The multiple standard network device messages are distributed to multiple partitions of the message middleware based on the hash keys corresponding to each message.

3. The method according to claim 2, characterized in that, The step of allocating the multiple standard network device messages to multiple partitions of the message middleware based on the hash keys corresponding to the multiple standard network device messages includes: Determine the total number of partitions for the message middleware; Based on the total number of partitions and the hash keys corresponding to the multiple standard network device messages, partition mapping is performed to obtain a mapping result, wherein the mapping result is used to indicate the hash key corresponding to each partition; Based on the mapping result, the multiple standard network device messages are distributed to multiple partitions of the message middleware.

4. The method according to claim 2, characterized in that, The step of using processing threads corresponding to the multiple partitions to concurrently process the original network device messages corresponding to the multiple partitions includes: Allocate one processing thread for each partition; Multiple partitions are used to process all standard network device messages in their respective partitions concurrently according to the timestamps in the standard network device messages, in order from first to last. Before processing each standard network device message, the standard network device message is verified to obtain the verification result. If the verification result indicates that there is missing data in the target standard network device message, data supplementation processing is performed, wherein the target standard network device message is any one of multiple standard network device messages.

5. The method according to claim 4, characterized in that, Before processing each standard network device message, the standard network device message is validated to obtain the validation result, including: The target standard network device message is subjected to a first verification to obtain a first verification result, wherein the first verification is to verify the attributes and data format of the target standard network device message; A second verification is performed on the target standard network device message to obtain a second verification result, wherein the second verification is to verify the association relationship of the target standard network device message; The verification result is determined based on the first verification result and the second verification result. If at least one of the first verification result and the second verification result fails the verification, it is determined that the target standard network device message has missing data.

6. The method according to claim 4, characterized in that, If the verification result indicates that there is missing data in the target standard network device message, data supplementation processing is performed, including: Generate supplementary acquisition tasks for all target standard network device messages with missing data, and organize all supplementary acquisition tasks into a priority task queue; Based on the data importance corresponding to the target standard network device message, each supplementary data acquisition task in the priority task queue is assigned a priority, and the supplementary data acquisition tasks in the priority task queue are executed by threads in the asynchronous thread pool in descending order of priority.

7. The method according to claim 6, characterized in that, The initial number of threads in the asynchronous thread pool is determined based on the number of CPU cores on the server. During the execution of the supplementary data acquisition task, the initial number of threads is adjusted according to the number of supplementary data acquisition tasks in the priority task queue. If the number of supplementary data acquisition tasks in the priority task queue exceeds the monitoring threshold built into the asynchronous thread pool, an alarm message is generated and pushed to the administrator.

8. A message processing apparatus, characterized in that, include: The receiving module is used to receive multiple raw network device messages from multiple network devices. The allocation module is used to allocate the multiple original network device messages to multiple partitions of the message middleware according to the target attributes in the multiple original network device messages, wherein each partition corresponds to a subnet, and the target attributes are used to indicate the network management system and subnet to which the original network device message belongs; The processing module is used to concurrently process the original network device messages corresponding to the multiple partitions using the processing threads corresponding to each of the multiple partitions.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device where the non-volatile storage medium is located to execute the message processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the message processing method according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the message processing method according to any one of claims 1 to 7.

Images