Data packet processing method and device

Abstract

The invention provides a data packet processing method and device.According to the method, data packets from a data transmission platform are received, the data transmission platform is used for storing the data packets sent by an associated system, the data packets are stored in a file receiving directory in a file form, and the data packets comprise associated system identifiers; the design enables the data packets to be accurately identified and classified. Then, the data packets are placed in the corresponding association queues according to the association system identifiers of the data packets, and effective classification of the data packets is achieved. And sequentially taking out the data packets from the associated queues by adopting a preset scheduling mode, and putting the data packets into a waiting queue. And finally, the data packets are sequentially taken out from the waiting queue and are put into a consumer pool, and the consumer pool comprises one or more than two consumers. In conclusion, according to the data packet processing method provided by the invention, rapid, accurate and reliable processing of the data packet is realized, and the response speed and the processing capability of the system are improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and more specifically, to a data packet processing method and device. Background Art

[0002] Driven by modern information technology, business data exchanges between systems are becoming more and more frequent. This interaction is usually achieved in the form of data packet files, which allow different systems to exchange information in a structured and standardized way. However, with the continuous expansion of business and the deepening of digital transformation, the magnitude and frequency of data exchanges between systems have shown explosive growth.

[0003] With the explosive growth of data volume and the significant increase in interaction frequency, distributed systems have encountered unprecedented challenges in data processing capabilities. Faced with high data flow and huge data volumes, the system is prone to processing chaos. Therefore, how to efficiently and accurately process massive data packet files in this environment has become a key problem that distributed systems need to overcome. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a data packet processing method and device to improve the performance of the distributed system. The specific solution is as follows:

[0005] In a first aspect, a method for processing a data packet is provided, comprising:

[0006] Receiving a data packet sent by a data transmission platform, and placing the data packet in the form of a file into a file receiving directory, wherein the data transmission platform is used to store data packets sent by an associated system, and the data packet includes an associated system identifier;

[0007] Acquire a data packet in the file receiving directory, and put the data packet into an associated queue according to an associated system identifier of the data packet, wherein the associated queue corresponds to the associated system;

[0008] The data packets of the associated queue are sequentially taken out in a preset scheduling mode, and are sequentially put into the waiting queue;

[0009] The data packets in the waiting queue are taken out in sequence, and the data packets are put into a consumer pool, and the data packets are processed by consumers in the consumer pool, wherein the consumer pool includes more than one consumer.

[0010] In a possible design, in another implementation of the first aspect of the embodiment of the present application, after sequentially taking out the data packets in the waiting queue, the method further includes:

[0011] Copying the data packet to a reply queue, wherein the reply queue is used to store the data packet being processed by the consumer pool;

[0012] Determine whether the data packet has been processed, and if so, remove the data packet from the reply queue;

[0013] If the processing is not completed and an operation error occurs in the consumer pool, the data packets in the reply queue are copied to the waiting queue after the consumer pool is restored.

[0014] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the data packet processing method is applied to a distributed system, further comprising:

[0015] When the distributed system crashes and restarts, the data packets transmitted from the data transmission platform and received in the file receiving directory during the crash of the distributed system are placed in the corresponding associated queue according to the associated system identifier.

[0016] In a possible design, in another implementation of the first aspect of the embodiment of the present application, obtaining a data packet in a file receiving directory and placing the data packet into an associated queue according to an associated system identifier of the data packet includes:

[0017] The listener thread is called to monitor the file receiving directory, and when a new data packet is monitored, the new data packet is placed into an associated queue corresponding to the associated system according to an associated system identifier of the new data packet.

[0018] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the scheduling mode includes polling scheduling and priority scheduling;

[0019] The method of sequentially taking out the data packets from the associated queue in a preset manner and sequentially putting them into the waiting queue includes:

[0020] Call the scheduling thread, use polling to take out the data packets in the associated queue in turn, and put the data packets into the waiting queue in turn;

[0021] or,

[0022] The scheduling thread is called to take out the data packets in the associated queue in sequence according to the preset priority, and put the data packets into the waiting queue in sequence.

[0023] In a possible design, in another implementation of the first aspect of the embodiment of the present application, sequentially taking out the data packets in the waiting queue and putting the data packets into the consumer pool includes:

[0024] The consumer thread is called, and the data packets in the waiting queue are taken out in sequence by using the consumer thread, and the data packets are sent to consumers in the consumer pool.

[0025] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, the listener thread includes more than two, and the process of calling the listener thread to put the newly added data packet into the associated queue corresponding to the associated system when listening to the newly added data packet includes:

[0026] When any listener thread listens to a new data packet, it accesses the distributed lock of the file receiving directory. If the lock is acquired successfully, it executes the process of placing the new data packet into the associated queue corresponding to the associated system. The distributed lock of the file receiving directory is used to ensure that only one listener thread runs at the same time.

[0027] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the scheduling thread includes more than two, and the calling scheduling thread sequentially takes out the data packets in the associated queue in a polling manner, and sequentially puts the data packets into the waiting queue, or, the calling scheduling thread sequentially takes out the data packets in the associated queue according to a preset priority, and sequentially puts the data packets into the waiting queue, including:

[0028] Any scheduling thread accesses the distributed lock of the associated queue. If the lock is acquired successfully, the process of taking out the data packets in the associated queue in turn by polling and putting the data packets into the waiting queue in turn is executed. The distributed lock of the associated queue is used to ensure that only one scheduling thread runs at the same time;

[0029] or,

[0030] Any scheduling thread accesses the distributed lock of the associated queue. If the lock is acquired successfully, the process of taking out the data packets in the associated queue in sequence according to the preset priority and placing the data packets in the waiting queue in sequence is executed. The distributed lock of the associated queue is used to ensure that only one scheduling thread runs at the same time.

[0031] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the consumer threads include more than two, and calling the consumer threads and using the consumer threads to sequentially take out the data packets in the waiting queue includes:

[0032] Any consumer thread accesses the distributed lock of the waiting queue. If the lock is acquired successfully, the process of using the consumer thread to sequentially take out the data packets in the waiting queue is executed. The distributed lock of the waiting queue is used to ensure that only one consumer thread runs at the same time.

[0033] In a second aspect, a data packet processing device is provided, comprising:

[0034] A data packet transmission unit, used to receive a data packet sent by a data transmission platform, and put the data packet into a file receiving directory in the form of a file, wherein the data transmission platform is used to store a data packet sent by an associated system, and the data packet includes an associated system identifier;

[0035] A data packet distribution unit, used for acquiring data packets in the file receiving directory, and placing the data packets into an associated queue according to an associated system identifier of the data packets, wherein the associated queue corresponds to the associated system;

[0036] A data packet scheduling unit, used to sequentially take out the data packets of the associated queue using a preset scheduling method, and sequentially put them into the waiting queue;

[0037] The data packet processing unit is used to sequentially take out the data packets in the waiting queue, put the data packets into a consumer pool, and process the data packets using consumers in the consumer pool, wherein the consumer pool includes more than one consumer.

[0038] By means of the above technical scheme, the data packet processing method proposed in the present application first receives data packets from a data transmission platform, wherein the data transmission platform is used to store data packets sent by an associated system, and these data packets are stored in a file receiving directory in the form of files, and the data packets contain associated system identifiers. This design enables data packets to be accurately identified and classified, laying a solid foundation for subsequent processing procedures. Then, the data packets are placed in the corresponding associated queues according to their associated system identifiers, thereby achieving effective classification of data packets. Each associated queue corresponds to a specific associated system. This design makes the processing of data packets more orderly and reduces confusion and errors in the processing process. Further, a preset scheduling method is adopted to sequentially take out data packets from the associated queues and put them into the waiting queue. This scheduling mechanism avoids delays and accumulation of data packet processing, and also improves the response speed and processing capacity of the system. Finally, data packets are taken out from the waiting queues in turn and put into the consumer pool. The consumer pool contains one or more consumers, and these consumers process data packets in parallel. In this way, the system can make full use of the advantages of multi-threaded processing and significantly improve the processing speed and throughput of data packets. In summary, the data packet processing method proposed in this application realizes fast, accurate and reliable processing of data packets, improves the response speed and processing capability of the system, and provides an efficient and reliable solution for data interaction between systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0040] Figure 1 A schematic diagram of a data packet processing method provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of another data packet processing method provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of a data packet processing system provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of a distributed lock application structure provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of a data packet processing device provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] Before introducing this application, the English terms involved in this article are explained first:

[0047] Distributed lock: A mechanism for coordinating multiple nodes' access to shared resources in a distributed system, ensuring that only one node can operate resources at any time to prevent data inconsistency. It must have mutual exclusion, high availability, high performance, reentrancy, and security. Common implementation methods include those based on databases, cache systems (such as Redis), ZooKeeper, and etcd. The implementation method should be selected based on the specific scenario and system architecture.

[0048] Message Queue: refers to the use of highly reliable message delivery mechanisms for platform-independent data interaction and the integration of distributed systems based on data communication. Through message delivery and message queuing, it can provide application decoupling, elastic scaling, redundant storage, traffic peak shaving, asynchronous communication, data synchronization and other functions in a distributed environment.

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The present application provides a data packet processing method, which first receives a data packet file containing an associated system identifier stored in a data transmission platform to ensure accurate identification and classification of the data packet. Then, the data packet is placed in a corresponding associated queue according to the identifier to achieve effective classification. Then, through a preset scheduling method, the data packet is moved from the associated queue to a waiting queue, and finally sent to a consumer pool containing multiple consumers. This method improves the speed, accuracy and reliability of data packet processing and enhances the system response and processing capabilities.

[0051] The present application solution can be implemented based on a terminal with data processing capabilities, which can be a mobile phone, computer, server, etc.

[0052] Next, see Figure 1 , a flow chart of a data packet processing method provided by the present application, the data packet processing method of the present application can be implemented by a data packet processing system deployed on a terminal, as described below, specifically including the following steps:

[0053] Step S100: receiving a data packet sent by a data transmission platform, and placing the data packet in the form of a file into a file receiving directory.

[0054] Specifically, when the associated systems need to interact with the current system for business data, they will interact with the packet processing system through the data transmission platform. The data transmission platform sends data packets to the packet processing system, which are encapsulated into files during the transmission process to ensure the integrity and security of the data. When the packet processing system receives the data packets, they will be uniformly stored in the pre-set file receiving directory.

[0055] The file receiving directory is a specific storage area used to temporarily store all data packet files sent from the associated system through the data transmission platform. In the file receiving directory, each data packet contains the associated system identification. This identification information is used to distinguish data packets sent by different associated systems to ensure that the data packets can be correctly identified and processed. In addition, each data packet in the file receiving directory follows a specific format specification, which contains the identification information of the associated system to which it belongs and the unique identification information of the data packet. This information is crucial for the subsequent data packet processing process and is used to guide how the data packet is further processed and distributed.

[0056] Step S110: Acquire a data packet in the file receiving directory, and put the data packet into an associated queue according to an associated system identifier of the data packet.

[0057] Specifically, the data packets in the file receiving directory are obtained. Each data packet has an associated system identifier, which is used to distinguish which specific associated system the data packet comes from. By parsing these identifiers, the system can identify the source of each data packet and perform subsequent classification operations accordingly. The "associated queue" here is actually a message queue, which is an advanced data structure used to manage and transmit messages in a distributed system.

[0058] Each associated queue is set up specifically to receive data packets from a specific associated system, ensuring that the data packets can be correctly routed and processed. This one-to-one correspondence not only improves the efficiency of data processing, but also enhances the manageability and scalability of the system. In this way, the data packets of each associated system are effectively organized and managed, providing a solid foundation for subsequent data processing and analysis.

[0059] Step S120: taking out the data packets from the associated queue in sequence according to a preset scheduling method, and putting them into the waiting queue in sequence.

[0060] Specifically, the system will take out data packets from each associated queue in turn according to the preset scheduling method, and put them into the waiting queue. The waiting queue contains data packets from different associated queues, which are managed uniformly and wait to be scheduled to run, thereby avoiding delays and accumulation of data packet processing, and also improving the system's response speed and processing capabilities.

[0061] Step S130, taking out the data packets in the waiting queue in sequence, and putting the data packets into the consumer pool, and using the consumers in the consumer pool to process the data packets.

[0062] Specifically, the system takes out packets from the waiting queue in turn. The waiting queue acts as a buffer area, which collects packets from different associated queues. These packets have been preliminarily sorted and sorted and are ready to enter the next processing stage.

[0063] After taking out the data packets, the system puts them into the consumer pool. The consumer pool is composed of multiple consumers, which work concurrently and are jointly responsible for the final processing of the data packets. Each consumer is a "consumer" in the message queue. They pull data packets from the waiting queue and independently perform business logic processing on these data packets. The design of the consumer pool allows the system to process multiple data packets in parallel, which can improve the efficiency of data processing and speed up the entire data flow. This concurrent processing mechanism is an important means to improve system throughput and response speed. In this way, the system can ensure that data packets are processed in a timely and effective manner to meet the needs of real-time data processing.

[0064] The data packet processing method proposed in the present application first receives data packets from a data transmission platform, wherein the data transmission platform is used to store data packets sent by an associated system. These data packets are stored in a file receiving directory in the form of files, and the data packets contain associated system identifiers. This design enables data packets to be accurately identified and classified, laying a solid foundation for subsequent processing procedures. Then, the data packets are placed in corresponding associated queues according to their associated system identifiers, thereby achieving effective classification of data packets. Each associated queue corresponds to a specific associated system. This design makes the processing of data packets more orderly and reduces confusion and errors in the processing process. Further, a preset scheduling method is adopted to sequentially take out data packets from the associated queues and put them into a waiting queue. This scheduling mechanism avoids delays and accumulation in data packet processing, and also improves the response speed and processing capacity of the system. Finally, data packets are taken out from the waiting queues in turn and put into a consumer pool. The consumer pool contains one or more consumers, which process data packets in parallel. In this way, the system can make full use of the advantages of multi-threaded processing and significantly improve the processing speed and throughput of data packets. In summary, the data packet processing method proposed in this application realizes fast, accurate and reliable processing of data packets, improves the response speed and processing capability of the system, and provides an efficient and reliable solution for data interaction between systems.

[0065] Further, in some embodiments of the present application, the data packet processing system may also include a reply queue, referring to Figure 2 , Figure 2 A schematic diagram of another data packet processing method provided in an embodiment of the present application, the specific process is as follows:

[0066] Step S200: receiving a data packet sent by a data transmission platform, and placing the data packet in the form of a file into a file receiving directory.

[0067] Step S210: Acquire a data packet in the file receiving directory, and put the data packet into an associated queue according to an associated system identifier of the data packet.

[0068] Step S220: taking out the data packets from the associated queue in sequence according to a preset scheduling method, and putting them into the waiting queue in sequence.

[0069] Specifically, the above steps S200-S220 correspond one-to-one to the steps S100-S120 in the aforementioned embodiment, and please refer to the above introduction for details, which will not be repeated here.

[0070] Step S230, sequentially taking out the data packets in the waiting queue, copying the data packets to the response queue, and putting the data packets into the consumer pool, and using the consumers in the consumer pool to process the data packets.

[0071] Specifically, when the system takes packets from the waiting queue, it copies and stores them in the reply queue. The reply queue acts as a temporary storage area to record which packets are currently being processed by consumers in the consumer pool. In this way, the system can keep accurate track of the packet processing status.

[0072] After taking out the data packets, the system puts them into the consumer pool. The consumer pool is composed of multiple consumers, which work concurrently and are jointly responsible for the final processing of the data packets. Each consumer is a "consumer" in the message queue. They pull data packets from the waiting queue and independently perform business logic processing on these data packets. The design of the consumer pool allows the system to process multiple data packets in parallel, which can improve the efficiency of data processing and speed up the entire data flow. This concurrent processing mechanism is an important means to improve system throughput and response speed. In this way, the system can ensure that data packets are processed in a timely and effective manner to meet the needs of real-time data processing.

[0073] Step S240, determine whether the data packet has been processed. If it has been processed, execute step S250; if it has not been processed, execute step S260.

[0074] Specifically, for the data packets that have been copied to the reply queue, the system needs to perform a judgment process to determine their processing status. The system checks each data packet in the reply queue to determine whether they have been completely processed by consumers in the consumer pool.

[0075] Step S250: remove the data packet in the reply queue.

[0076] Specifically, if the system determines that a data packet has been processed, the data packet will be removed from the reply queue. This step is to keep the reply queue accurate and up-to-date, ensuring that the queue only contains data packets that are being processed or waiting to be processed.

[0077] Step S260: A running error occurs in the consumer pool. After the consumer pool is restored, the data packets in the reply queue are copied to the waiting queue.

[0078] Specifically, if the system finds that a data packet has not been processed during the inspection process, and an operation error occurs in the consumer pool during this process, the system will take appropriate recovery measures. Once the consumer pool resumes normal operation, the system will copy the unprocessed data packets in the reply queue back to the waiting queue so that they can re-enter the processing flow and wait to be taken out and processed again by consumers in the consumer pool. This process ensures that even in the event of an error, the data packet will not be lost and can be correctly rescheduled for processing, thereby ensuring the continuity and integrity of data processing.

[0079] In this embodiment, the system takes out the data packets from the waiting queue in sequence and puts them into the consumer pool, and the system copies the data packets taken out from the waiting queue to the reply queue. This temporary storage design enables the system to accurately track the real-time processing status of each data packet, providing a solid foundation for subsequent data processing. By checking each data packet in the reply queue, the system can determine whether they have been completely processed by the consumers in the consumer pool. If the data packets have been processed, these data packets are removed from the reply queue, which not only maintains the accuracy and latest status of the reply queue, but also releases storage resources and optimizes the performance of the system. However, if the consumer pool has an operation error during the processing, the system provides an effective recovery measure. The system will re-copy the unprocessed data packets in the reply queue to the waiting queue. Once the consumer pool resumes normal operation, these data packets can re-enter the processing flow, waiting to be taken out and processed again by the consumers in the consumer pool. This process not only ensures that the data packets will not be lost due to system failures, but also guarantees the continuity and integrity of data processing, and even in the face of operation errors, the stability and reliability of data processing can be ensured.

[0080] In some embodiments of the present application, in the data packet processing method described in the present application, in particular for the operating characteristics of the distributed system, a processing mechanism for coping with system downtime and restart may also be adopted. The processing mechanism is introduced below:

[0081] Optionally, when a system downtime event occurs and then restarts, the system will execute a specific recovery process. In this process, the system first identifies the data packets sent to the file receiving directory through the data transmission platform during the downtime. These data packets were not processed in time when the system was down, so they accumulated in the file receiving directory.

[0082] After the system restarts, it will check the file receiving directory and collect all the data packets received during the downtime. Then, the system will classify them according to the associated system identifier of each data packet and put them into the corresponding associated queue. This step ensures that each data packet can be correctly assigned to its corresponding processing queue and prepare for subsequent data processing.

[0083] In this way, this embodiment can ensure that no data packets will be lost even in a downtime event, and all data packets received during the downtime can be properly managed and processed. This mechanism improves the robustness of the system, ensures the continuity and integrity of data processing, and maintains the orderly flow of data even in the face of unexpected system failures.

[0084] In some embodiments of the present application, a method for placing a data packet into an associated queue using a listener thread is provided, and the method is introduced below.

[0085] Optionally, a listener thread can be used to continuously monitor the status of the file receiving directory. This file receiving directory is the designated location where the system receives data packets from the data transmission platform. The main task of the listener thread is to detect whether there are new data packets arriving in the directory.

[0086] When the listener thread detects that a new data packet has been added to the file receiving directory, it will start a processing flow. This process first involves identifying the associated system identifier of the newly added data packet. Based on this identifier, the listener thread can determine which associated system each new data packet should belong to. After determining the associated system, the listener thread will put the new data packet into the associated queue corresponding to the associated system. Each associated system has its own associated queue for temporarily storing data packets from the system. This design allows the system to classify and organize data packets to prepare for subsequent data processing and consumption processes. At the same time, in order to optimize storage resources, the listener thread can also delete the corresponding data packet file in the receiving directory after the data packet is successfully transferred to save disk space and optimize the use of storage resources.

[0087] The listener thread in this embodiment ensures the continuity and accuracy of receiving, identifying, classifying and initially processing data packets, and provides a solid foundation for subsequent data packet processing. This process is automated, ensuring the timeliness and accuracy of data packet processing, and improving the efficiency and reliability of the entire data processing flow.

[0088] In some embodiments of the present application, the present application provides a flexible scheduling mechanism to ensure that data packets can be processed efficiently and fairly. Scheduling methods may include round-robin scheduling and priority scheduling. These two scheduling methods can also be implemented through scheduling threads, which are key components in the system responsible for managing and allocating data packets. The following is an introduction to the process:

[0089] Optionally, in the polling scheduling mode, the scheduling thread takes out data packets from each associated queue in turn according to the preset order and puts them into the waiting queue. This polling mechanism ensures that each associated system can get equal processing opportunities, effectively avoiding the situation where some queues wait for processing for a long time, thereby enhancing the fairness and responsiveness of the system. Polling scheduling can be used in scenarios where fairness of data packet processing needs to be guaranteed because of its simplicity and effectiveness.

[0090] Priority scheduling allows the system to process data packets according to preset priority rules. Specifically, the system can use a variety of strategies including weight, packet size, processing time, etc. to determine the priority of data packets according to the actual situation. In this mode, the scheduling thread takes data packets from the associated queue and puts them into the waiting queue based on the urgency of the data packet or other priority indicators related to business logic. This mechanism enables high-priority data packets to be processed faster and is suitable for scenarios with strict requirements on data processing timeliness.

[0091] The flexible scheduling mechanism in this embodiment enables the system to adapt to different business needs and processing pressures, improves the efficiency of data processing and the stability of the system. Through this design, it can ensure that in a distributed system, the processing of data packets is both fair and efficient, meeting the different requirements for data processing in different business scenarios.

[0092] In some embodiments of the present application, a consumer thread may be used to put a data packet into a consumer pool. The process is described below:

[0093] Optionally, after the packets are successfully transferred from the associated queue to the waiting queue, the consumer threads can be used to further process the packets. Consumer threads are a key component in the system. They are responsible for taking packets from the waiting queue in sequence and sending them to consumers in the consumer pool for final processing.

[0094] The design of consumer threads allows the system to process multiple data packets in parallel, which can improve the efficiency of data processing and speed up the entire data flow. In actual operation, the system monitors the waiting queue, and once a new data packet is in the queue, it triggers the consumer thread to take out and process the data packet. The consumer thread can allocate the data packet to the most suitable consumer for processing based on the type, size or processing requirements of the data packet. This allocation mechanism ensures the efficiency and balance of data processing and prevents some consumers in the consumer pool from being overloaded while other consumers are idle. There can be multiple consumers in the consumer pool, and they can independently perform business logic processing on these data packets, and they will also work together to ensure that the data packets can be processed in a timely and effective manner.

[0095] This embodiment can realize the rapid processing of a large number of data packets through the cooperative work of the consumer thread and the consumer pool, while ensuring the accuracy and reliability of data processing. This mechanism not only improves the efficiency of data processing, but also enhances the stability of the system.

[0096] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a data packet processing system disclosed in this embodiment. Figure 3 To introduce:

[0097] The figure includes key components such as an associated system, a data transmission platform, a file receiving directory, an associated queue, a waiting queue, a reply queue, and a consumer pool, among which the file receiving directory, the associated queue, the waiting queue, the reply queue, and the consumer pool can constitute a data packet processing system.

[0098] Optionally, the associated system is an external system that interacts with the data packet processing system. The associated system generates data packets and sends them to the data packet processing system through the data transmission platform. As an intermediary, the data transmission platform is responsible for receiving data packets sent from different associated systems A, B, C, etc., and storing these data packets in the file receiving directory in the form of files. These data packets contain the associated system identification, which is the key information for identifying and classifying data packets.

[0099] In the file receiving directory, the data packets are then captured by the listener thread. The listener thread continuously monitors the directory and once a new data packet is detected, it distributes the data packet to the corresponding associated queue according to its associated system ID. Each associated queue is dedicated to one associated system, ensuring that the data packet can be correctly classified and initially processed.

[0100] Next, the scheduling thread takes out the data packets from each associated queue and puts them into the waiting queue according to the preset scheduling strategy. The scheduling strategy here can be polling scheduling or priority scheduling to adapt to different business needs. Polling scheduling ensures that each associated queue can be processed fairly, while priority scheduling allows the system to process data packets based on the urgency of the data packet or other priority indicators related to business logic.

[0101] Once a data packet is placed in the waiting queue, the consumer thread is called. The consumer thread is responsible for taking data packets from the waiting queue and sending them to consumers in the consumer pool for processing. The consumer pool consists of multiple consumers that work in parallel, improving the efficiency and speed of data processing.

[0102] In addition, in order to improve the fault tolerance of the system, the present application also includes a reply queue for storing data packets being processed by the consumer pool. The system will periodically check the data packets in the reply queue to determine whether they have been processed. If the data packets are processed, they will be removed from the reply queue. If a running error occurs in the consumer pool during processing, the system will put the unprocessed data packets in the reply queue back to the waiting queue after recovery to ensure the continuity and integrity of data processing.

[0103] The schematic diagram of the data packet processing structure shown in this embodiment achieves significant technical effects through the mutual cooperation of various components. The system first receives data packets from different associated systems through the file receiving directory, and uses the associated queue to classify and temporarily store the data packets, ensuring the orderliness of data processing. Secondly, a combination of polling scheduling and priority scheduling is adopted to transfer data packets from the associated queue to the waiting queue through the scheduling thread, thereby improving the flexibility and efficiency of scheduling. The design of the consumer thread pool allows multiple consumers to process data packets in parallel, significantly improving the concurrent processing capability of the system. In addition, the introduction of the reply queue provides a fault-tolerant mechanism for the system, ensuring that when a consumer pool fails, the data packet will not be lost and can be rescheduled for processing. Overall, this system structure achieves efficient, stable and reliable processing of data packets through the collaborative work of various components, and is particularly suitable for data processing requirements in high concurrency and distributed environments.

[0104] In some embodiments of the present application, it is further provided to coordinate mutually exclusive access to resources using distributed locks, see Figure 4 , Figure 4 This is a schematic diagram of the distributed lock application structure. The following is an introduction to the process:

[0105] Optionally, since system applications can be deployed in a distributed cluster, there may be multiple Pod applications running at the same time to provide services, and the file receiving directory, associated queue, and waiting queue in the overall packet processing solution are mutually exclusive resources, so distributed locks are needed to coordinate the work between different threads. Distributed locks can be applied to file receiving directories, associated queues, waiting queues, and reply queues.

[0106] Figure 4 It includes key components such as file receiving directory, associated queue, waiting queue and reply queue. The system uses distributed locks (lock 1, lock 2, lock 3) to ensure that the processing of data packets is mutually exclusive and safe in a high-concurrency environment. The file listener thread monitors the file receiving directory. After detecting a new data packet, it ensures uniqueness through distributed lock 1 and then assigns the data packet to the corresponding associated queue. The scheduling thread manages the associated queue through distributed lock 2 and transfers the data packet to the waiting queue according to the preset strategy. The consumer thread takes the data packet from the waiting queue through distributed lock 3 and processes it. The entire process is implemented through independent running units such as Pod1, Pod2, and Pod3. Each unit contains listeners, scheduling and consumer threads to jointly maintain the efficiency and stability of data processing. The following is a detailed introduction to the distributed locks of the file receiving directory, associated queue, waiting queue and reply queue.

[0107] Distributed lock of the file receiving directory: When inputting data packets from the file receiving directory to the associated queue, the listener thread is responsible for monitoring the file receiving directory to capture newly arrived data packets in time. When any listener thread detects a new data packet, it first attempts to access and acquire the distributed lock of the file receiving directory. This step is the key to ensure mutual exclusivity of the data packet processing process. It prevents multiple listener threads from writing to the file receiving directory at the same time, thereby avoiding data conflicts and potential system errors.

[0108] Once the listener thread successfully acquires the distributed lock, it will perform the classification and distribution process of the data packet. Specifically, the listener thread will accurately place the data packet into the associated queue corresponding to the associated system according to the associated system identifier in the newly added data packet. This process not only ensures that the data packet can be correctly classified, but also lays the foundation for the subsequent data processing flow. The distributed lock of the file receiving directory plays a vital role in the whole process. It ensures that at any given moment, only one listener thread can perform the data packet placement operation, thereby maintaining the stability of the system and the consistency of the data.

[0109] Distributed lock of associated queue: The scheduling thread is responsible for taking data packets from the associated queue and putting them into the waiting queue so that the consumer thread pool can further process these data packets. In order to ensure the thread safety and data consistency of this process, the system introduces a distributed lock mechanism for the associated queue.

[0110] When any scheduling thread is ready to perform its task, it first tries to acquire the distributed lock of the associated queue. The purpose of this lock is to ensure that only one scheduling thread can operate on the associated queue at the same time. If the scheduling thread successfully acquires the lock, it will process the data packet according to the system's preset scheduling policy. In polling scheduling mode, the scheduling thread will visit each associated queue in turn, take out the data packets fairly, and put them into the waiting queue. This polling mechanism ensures that all associated queues can be processed evenly, avoiding the situation where some queues wait for a long time to be processed. Alternatively, if the system is configured with priority scheduling mode, the scheduling thread will take out the data packets according to the preset priority of the data packet. This means that urgent or important data packets can be processed first, thus meeting the business needs that have strict requirements on the timeliness of data processing. The distributed lock of the associated queue plays a key role in this process. It not only ensures the mutual exclusion of the scheduling process, but also ensures the order of data packet processing and the overall stability of the system.

[0111] Distributed lock of waiting queue: The consumer thread is responsible for taking out the data packet from the waiting queue and processing it. In order to ensure the thread safety of this process and avoid repeated processing or loss of data packets, the system adopts a distributed lock mechanism of the waiting queue.

[0112] When any consumer thread is ready to take a data packet from the waiting queue, it will first try to acquire the distributed lock of the waiting queue. The purpose of this lock is to ensure that only one consumer thread can operate on the waiting queue at the same time. This mutual exclusion mechanism is necessary because in a distributed system, multiple consumer threads may try to access the waiting queue at the same time. If there is no proper synchronization measure, it may cause confusion in data packet processing.

[0113] If the consumer thread successfully acquires the distributed lock, it will execute the process of taking out the data packet. The consumer thread will take out the data packets from the waiting queue one by one. This operation is orderly, ensuring that the data packets are processed in the order in which they are placed in the waiting queue. The distributed lock of the waiting queue not only ensures the mutual exclusivity of the consumer thread when taking out the data packet, but also ensures the consistency of data packet processing and the stability of the system.

[0114] Distributed lock of reply queue: When the consumer thread takes the data packets from the waiting queue and starts processing, the system copies these data packets to the reply queue. Distributed lock ensures that at any given time, only one consumer thread can write to the reply queue, preventing multiple threads from modifying the reply queue at the same time, thereby avoiding duplicate processing or loss of data packets.

[0115] The distributed lock proposed in this embodiment plays a vital role in ensuring the stability and reliability of the system in a high-concurrency environment. By acquiring the distributed lock before key operations, the system can achieve mutually exclusive access to shared resources and prevent data conflicts and inconsistencies. The distributed lock is combined with the scheduling strategy to support load balancing and fault recovery, improving the fault tolerance and high availability of the system.

[0116] The following is a description of a data packet processing device provided in an embodiment of the present application. The data packet processing device described below and the data packet processing method described above can be referenced to each other.

[0117] See also Figure 5 , Figure 5 A schematic diagram of the structure of a data packet processing device disclosed in an embodiment of the present application.

[0118] like Figure 5 As shown, the device may include:

[0119] The data packet transmission unit 11 is used to receive a data packet sent by a data transmission platform, and put the data packet into a file receiving directory in the form of a file. The data transmission platform is used to store data packets sent by an associated system, and the data packet includes an associated system identifier;

[0120] The data packet distribution unit 12 is used to obtain the data packet in the file receiving directory and put the data packet into the associated queue according to the associated system identifier of the data packet, and the associated queue corresponds to the associated system;

[0121] The data packet scheduling unit 13 is used to sequentially take out the data packets of the associated queue using a preset scheduling method, and sequentially put them into the waiting queue;

[0122] The data packet processing unit 14 is used to sequentially take out the data packets in the waiting queue, put the data packets into the consumer pool, and process the data packets using consumers in the consumer pool, wherein the consumer pool includes more than one consumer.

[0123] Optionally, a data packet processing device in an embodiment of the present application further includes:

[0124] a data packet duplication unit, used for duplicating data packets to a reply queue, wherein the reply queue is used for storing data packets being processed by the consumer pool;

[0125] a determination unit, for determining whether the data packet has been processed, and if so, removing the data packet from the reply queue;

[0126] The data packet recovery unit is used to copy the data packets in the reply queue to the waiting queue after the consumer pool is recovered if the data packets are not processed completely and an operation error occurs in the consumer pool.

[0127] Optionally, a data packet processing device in an embodiment of the present application further includes:

[0128] The retransmission unit is used to place the data packets transmitted from the data transmission platform received in the file receiving directory during the distributed system downtime into the corresponding associated queue according to the associated system identifier after the distributed system crashes and restarts.

[0129] In a possible implementation, the process in which the data packet distributing unit 12 obtains the data packet in the file receiving directory and puts the data packet into the associated queue according to the associated system identifier of the data packet includes:

[0130] The listener thread is called to monitor the file receiving directory, and when a new data packet is monitored, the new data packet is placed into an associated queue corresponding to the associated system according to an associated system identifier of the new data packet.

[0131] In a possible implementation, the process of the data packet scheduling unit 13 sequentially taking out the data packets from the associated queue in a preset manner and sequentially putting them into the waiting queue includes:

[0132] Call the scheduling thread, use polling to take out the data packets in the associated queue in turn, and put the data packets into the waiting queue in turn; or, call the scheduling thread, take out the data packets in the associated queue in turn according to the preset priority, and put the data packets into the waiting queue in turn.

[0133] In a possible implementation, the process in which the data packet processing unit 14 is used to sequentially take out the data packets in the waiting queue and put the data packets into the consumer pool includes:

[0134] The consumer thread is called, and the data packets in the waiting queue are taken out in sequence by using the consumer thread, and the data packets are sent to consumers in the consumer pool.

[0135] In a possible implementation, the listener threads include more than two, and the process in which the data packet distribution unit 12 calls the listener thread to put the newly added data packet into the associated queue corresponding to the associated system when listening to the newly added data packet includes:

[0136] When any listener thread listens to a new data packet, it accesses the distributed lock of the file receiving directory. If the lock is acquired successfully, it executes the process of placing the new data packet into the associated queue corresponding to the associated system. The distributed lock of the file receiving directory is used to ensure that only one listener thread runs at the same time.

[0137] In a possible implementation, the scheduling thread includes more than two, and the data packet scheduling unit 13 calls the scheduling thread, sequentially takes out the data packets in the associated queue in a polling manner, and sequentially puts the data packets into the waiting queue, or calls the scheduling thread, sequentially takes out the data packets in the associated queue according to the preset priority, and sequentially puts the data packets into the waiting queue, including:

[0138] Any scheduling thread accesses the distributed lock of the associated queue. If the lock is acquired successfully, the process of taking out the data packets in the associated queue in turn by polling and putting the data packets into the waiting queue in turn is executed. The distributed lock of the associated queue is used to ensure that only one scheduling thread runs at the same time;

[0139] or,

[0140] Any scheduling thread accesses the distributed lock of the associated queue. If the lock is acquired successfully, the process of taking out the data packets in the associated queue in sequence according to the preset priority and placing the data packets in the waiting queue in sequence is executed. The distributed lock of the associated queue is used to ensure that only one scheduling thread runs at the same time.

[0141] In a possible implementation, the consumer threads include more than two, and the data packet processing unit 14 calls the consumer threads, and uses the consumer threads to sequentially take out the data packets in the waiting queue, including:

[0142] Any consumer thread accesses the distributed lock of the waiting queue. If the lock is acquired successfully, the process of using the consumer thread to sequentially take out the data packets in the waiting queue is executed. The distributed lock of the waiting queue is used to ensure that only one consumer thread runs at the same time.

[0143] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, wearable devices, etc. Figure 6The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0144] like Figure 6 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603 to implement the data packet processing method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0145] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0146] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the data packet processing method of the aforementioned embodiment of the present application.

[0147] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0148] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0150] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0151] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A data packet processing method, characterized in that: include: Receiving a data packet sent by a data transmission platform, and placing the data packet in the form of a file into a file receiving directory, wherein the data transmission platform is used to store data packets sent by an associated system, and the data packet includes an associated system identifier; Acquire a data packet in the file receiving directory, and put the data packet into an associated queue according to an associated system identifier of the data packet, wherein the associated queue corresponds to the associated system; The data packets of the associated queue are sequentially taken out in a preset scheduling mode, and are sequentially put into the waiting queue; The data packets in the waiting queue are taken out in sequence, and the data packets are put into a consumer pool, and the data packets are processed by consumers in the consumer pool, wherein the consumer pool includes more than one consumer.

2. The method according to claim 1, characterized in that After sequentially taking out the data packets in the waiting queue, the method further includes: Copying the data packet to a reply queue, wherein the reply queue is used to store the data packet being processed by the consumer pool; Determine whether the data packet has been processed, and if so, remove the data packet from the reply queue; If the processing is not completed and an operation error occurs in the consumer pool, the data packets in the reply queue are copied to the waiting queue after the consumer pool is restored.

3. The method according to claim 1, characterized in that: The data packet processing method is applied to a distributed system, and further comprises: When the distributed system crashes and restarts, the data packets transmitted from the data transmission platform and received in the file receiving directory during the crash of the distributed system are placed in the corresponding associated queue according to the associated system identifier.

4. The method according to claim 1, characterized in that The step of obtaining a data packet in a file receiving directory and placing the data packet into an associated queue according to an associated system identifier of the data packet includes: The listener thread is called to monitor the file receiving directory, and when a new data packet is monitored, the new data packet is placed into an associated queue corresponding to the associated system according to an associated system identifier of the new data packet.

5. The method according to claim 1, characterized in that The scheduling methods include polling scheduling and priority scheduling; The method of sequentially taking out the data packets from the associated queue in a preset manner and sequentially putting them into the waiting queue includes: Call the scheduling thread, use polling to take out the data packets in the associated queue in turn, and put the data packets into the waiting queue in turn; or, The scheduling thread is called to take out the data packets in the associated queue in sequence according to the preset priority, and put the data packets into the waiting queue in sequence.

6. The method according to claim 1, characterized in that The method of sequentially taking out the data packets in the waiting queue and placing the data packets into the consumer pool includes: The consumer thread is called, and the data packets in the waiting queue are taken out in sequence by using the consumer thread, and the data packets are sent to consumers in the consumer pool.

7. The method according to claim 4, characterized in that If the listener threads include more than two, the process of calling the listener thread to put the newly added data packet into the associated queue corresponding to the associated system when the newly added data packet is listened to includes: When any listener thread listens to a new data packet, it accesses the distributed lock of the file receiving directory. If the lock is acquired successfully, it executes the process of placing the new data packet into the associated queue corresponding to the associated system. The distributed lock of the file receiving directory is used to ensure that only one listener thread runs at the same time.

8. The method according to claim 5, characterized in that The scheduling threads include more than two, and the calling scheduling thread sequentially takes out the data packets in the associated queue in a polling manner, and sequentially puts the data packets into the waiting queue, or the calling scheduling thread sequentially takes out the data packets in the associated queue according to the preset priority, and sequentially puts the data packets into the waiting queue, including: Any scheduling thread accesses the distributed lock of the associated queue. If the lock is acquired successfully, the process of taking out the data packets in the associated queue in turn by polling and putting the data packets into the waiting queue in turn is executed. The distributed lock of the associated queue is used to ensure that only one scheduling thread runs at the same time; or, Any scheduling thread accesses the distributed lock of the associated queue. If the lock is acquired successfully, the process of taking out the data packets in the associated queue in sequence according to the preset priority and placing the data packets in the waiting queue in sequence is executed. The distributed lock of the associated queue is used to ensure that only one scheduling thread runs at the same time.

9. The method according to claim 6, characterized in that The consumer threads include more than two, and the calling of the consumer threads and using the consumer threads to sequentially take out the data packets in the waiting queue include: Any consumer thread accesses the distributed lock of the waiting queue. If the lock is acquired successfully, the process of using the consumer thread to sequentially take out the data packets in the waiting queue is executed. The distributed lock of the waiting queue is used to ensure that only one consumer thread runs at the same time.

10. A data packet processing device, characterized in that: include: A data packet transmission unit, used to receive a data packet sent by a data transmission platform, and put the data packet into a file receiving directory in the form of a file, wherein the data transmission platform is used to store a data packet sent by an associated system, and the data packet includes an associated system identifier; A data packet distribution unit, used for acquiring data packets in the file receiving directory, and placing the data packets into an associated queue according to an associated system identifier of the data packets, wherein the associated queue corresponds to the associated system; A data packet scheduling unit, used to sequentially take out the data packets of the associated queue using a preset scheduling method, and sequentially put them into the waiting queue; The data packet processing unit is used to sequentially take out the data packets in the waiting queue, put the data packets into a consumer pool, and process the data packets using consumers in the consumer pool, wherein the consumer pool includes more than one consumer.

Images