Multi-channel and multi-thread data stream processing method and system

Abstract

The invention provides a multi-channel and multi-thread data stream processing method and system, and relates to the technical field of data stream processing.The method comprises the steps that a storage cache pool used for data storage and a resolving cache pool used for data resolving are created; synchronously writing the received data flow into the channels of the storage cache pool and the calculation cache pool through an independently running data receiving thread; when any channel reaches a threshold, a corresponding state identifier is triggered, and the next channel is switched to continue working; through the data storage thread and the data resolving thread which are mutually independent and run in parallel, the state identification of each channel in each cache pool is independently polled and monitored, and once an effective state is identified, the data in the corresponding storage channel is immediately stored in the storage medium or the data in the corresponding resolving channel is resolved and analyzed. The problem of data loss caused by intense system resource competition and processing flow blockage under high-speed data flow due to too tight coupling of data receiving, storage and resolving is solved.

Description

Technical Field

[0001] This invention relates to the field of data stream processing technology, and in particular to a multi-channel, multi-threaded data stream processing method and system. Background Technology

[0002] With the rapid development of aerospace, aviation, and weaponry, telemetry technology is facing unprecedented data pressure. As data transmission performance and speed have significantly improved, the parameters of the measured objects have become increasingly complex. This has led to an exponential increase in the amount of data requiring real-time reception, storage, and processing, posing extreme challenges to the real-time performance and reliability of telemetry software. Traditional solutions primarily address high-speed data streams by expanding hardware capacity. For example, using multi-channel hardware to distribute data transmission or using multiple external computers for distributed reception, storage, and processing. However, while these methods can alleviate data pressure to some extent, they inevitably lead to a significant increase in hardware costs, system complexity, and the difficulty of subsequent maintenance. Therefore, given fixed hardware conditions, alleviating data pressure through software architecture has become an inevitable trend in the industry.

[0003] Existing software architectures are mostly single-threaded sequential execution or simple multi-threaded models. There is a strong coupling between the three data processing stages: data reception, data storage, and data processing. However, this strong coupling between stages can lead to many problems: (1) Blocking problem: When the system is performing time-consuming data processing or slow disk storage operations, the high-speed data receiving thread will be blocked and unable to continue receiving new data, resulting in a very high risk of data loss.

[0004] (2) Resource competition: Computation tasks (CPU intensive) and storage tasks (I / O intensive) compete for system resources and interfere with each other, resulting in low overall efficiency and failure to fully utilize the performance of multi-core CPUs and high-speed I / O hardware.

[0005] (3) Poor flexibility: The system is difficult to adapt to different working scenarios. For example, in scenarios where only data storage is required, the calculation task will still consume resources; while in scenarios where complex real-time data calculation is required, the pressure of the storage task may affect the calculation speed.

[0006] Therefore, there is an urgent need for a multi-channel, multi-threaded data stream processing method and system that will not lose data under various high-speed conditions. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a multi-channel, multi-threaded data stream processing method and system, which solves the technical problem in the prior art where the tight coupling between data receiving, storage, and processing stages leads to intense competition for system resources and blockage of the processing flow under high-speed data streams, resulting in data loss.

[0008] In a first aspect, the present invention provides a multi-channel, multi-threaded data stream processing method, the method comprising: Create a storage cache pool for data storage and a computation cache pool for data computation; the storage cache pool and the computation cache pool are independent of each other and each includes multiple independent channels; The received data stream is synchronously written into the channels of the storage cache pool and the solution cache pool through an independently running data receiving thread; when any channel reaches the threshold, the corresponding status flag is triggered and the system switches to the next channel to continue working. The data storage thread and data processing thread run independently and in parallel, and independently poll and listen to the status flags of each channel in their respective cache pools. Once a valid status is identified, the data in the corresponding storage channel is immediately stored into the storage medium or the data in the corresponding processing channel is processed and analyzed.

[0009] Furthermore, when any channel reaches its threshold, a corresponding status indicator is triggered, and the system switches to the next channel to continue operating, including: A storage status identifier is created for each storage channel of the storage cache pool; when the amount of data in the current storage channel in the storage cache pool reaches the storage threshold, the storage status identifier corresponding to the current storage channel is set to a valid state, and the data receiving thread is switched to write the data to be stored into the next storage channel of the storage cache pool. A solution status identifier is created for each solution channel in the solution cache pool. When the amount of data in the current solution channel in the solution cache pool reaches the solution threshold, the solution status identifier corresponding to the current solution channel is set to a valid state, and the data receiving thread is switched to write the data to be stored into the next solution channel of the solution cache pool.

[0010] Furthermore, the data storage thread polls and listens to the status flags of each channel in the storage cache pool, including: when the storage status flag of a certain storage channel is found to be valid, the data in the storage channel is stored into the storage medium; after storage is completed, the storage status flag of the storage channel is reset to invalid.

[0011] Furthermore, the data processing thread polls and listens to the status flags of each channel in the processing cache pool, including: when the processing status flag of a certain processing channel is found to be valid, processing and analysis are performed on the data in that processing channel; after processing is completed, the processing status flag of that processing channel is reset to invalid.

[0012] Furthermore, the data receiving thread is: Receive data to be stored from an external interface, obtain the storage channel number of the current storage channel for data storage, write the data to be stored into the current storage channel in sequence, and determine whether the amount of data in the current storage channel has reached the storage threshold. If the storage threshold is reached, the storage status flag of the current storage channel is set to valid, the storage channel number for data storage is set to the next storage channel number, and the current processing channel number for data processing is obtained; if the storage threshold is not reached, the current processing channel number for data processing is obtained directly. After obtaining the current solution channel number for data solution, the data to be solved is written into the current solution channel in sequence, and it is determined whether the amount of data in the current solution channel has reached the solution threshold. If the calculation threshold is reached, the calculation status flag of the current calculation channel is set to valid, and the calculation channel number for data calculation is set to the next calculation channel number. Then, the next data to be stored is received from the external interface, and the next data reception is performed. If the storage threshold is not reached, the next data to be stored is received directly from the external interface, and the next data reception is performed.

[0013] Furthermore, the data storage thread is: Obtain the storage channel number of the current storage channel for data storage, and determine whether the storage status identifier of the current storage channel is valid; If the state is invalid, continue to obtain the storage channel number of the current storage channel for data storage; if the state is valid, store the data in the current storage channel into the storage medium. After the data is stored, the storage status flag of the storage channel is reset to an invalid state, the data of the storage channel is cleared, the storage channel number for data storage is set as the next storage channel number, and the storage channel number for data storage is obtained again. This process is repeated continuously to store data.

[0014] Furthermore, the data processing thread is: Obtain the current solution channel number for the data solution channel and determine whether the solution status identifier of the current solution channel is valid; If the state is invalid, the current solution channel number for the current solution channel will be obtained; if the state is valid, the data in the solution channel will be solved and analyzed. After the calculation is completed, the calculation status flag of the calculation channel is reset to invalid, the data of the calculation channel is cleared, the calculation channel number for data calculation is set as the next calculation channel number, and the calculation channel number for data calculation is obtained again. This process is repeated continuously to solve the data.

[0015] Furthermore, the method also includes: disabling the computation cache pool and the data computation thread in scenarios where only data storage is required; and disabling the storage cache pool and the data storage thread in scenarios where only data computation is required.

[0016] Furthermore, the number of channels in the storage cache pool and the solution cache pool, as well as the size of each channel, are dynamically configured based on the actual data volume and computer performance.

[0017] In a second aspect, the present invention provides a multi-channel, multi-threaded data stream processing system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0018] In summary, this invention provides a multi-channel, multi-threaded data stream processing method and system. Compared with the prior art, the technical solution conceived in this invention can achieve the following beneficial effects: (1) This invention completely decouples the three tasks of data reception, data storage, and data processing—which have different system resource requirements—by creating independent storage cache pools and processing cache pools. On the one hand, the data reception thread can run at full speed, synchronously writing data to the two cache pools. Once a cache channel is full, the reception thread will immediately switch to the next idle channel to continue working, thus completely avoiding blocking caused by waiting for storage or processing. On the other hand, the data storage thread and the data processing thread run in parallel, each processing the ready cache channels. This architecture ensures the continuity of data reception, not only reducing the impact of data storage and data processing on data reception and avoiding the time-consuming problem caused by writing too much data to a single file or writing too many files at once, but also fundamentally eliminating data loss and achieving extremely high data throughput.

[0019] (2) This invention separates data processing and data storage into independent threads for execution; these two threads can be scheduled by different CPU cores and utilize the computing resources of different disks simultaneously, achieving true parallel processing and greatly reducing resource competition and mutual waiting between threads; in addition, the data storage thread and the data processing thread work by polling and listening to the status flags. When there are no tasks to execute, the threads can be in a waiting state instead of idling and consuming CPU resources; compared with the traditional busy query, this event-driven triggering mechanism significantly reduces the idle overhead of the system, making computing resources more concentrated on actual data processing tasks and effectively optimizing the utilization efficiency of system resources.

[0020] (3) The number of channels and the size of each channel in the storage cache pool and the computation cache pool of the present invention can be dynamically loaded according to user configuration. This means that on high-performance servers with ample memory, more and larger cache channels can be configured to cope with extreme data flows; while on resource-constrained industrial control computers, a simplified configuration can be performed, ensuring the robustness of the system on different hardware platforms. In addition, separating data computation and data storage into independent threads allows the architecture to support feature pruning. In scenarios where only data storage is required, the data computation thread and related cache pool can be disabled; in scenarios where only data computation is required, the data storage thread can be disabled. This configurable and prudent feature makes the present invention highly flexible and adaptable, and able to cope with various scenarios.

[0021] (4) In this invention, both the data storage thread and the data processing thread perform batch operations using complete cache channels as units. The fine-grained process control of the data receiving thread ensures that data is distributed to the corresponding cache channels in an orderly and correct manner, laying the foundation for high-quality processing by the data storage thread and the data processing thread. For storage, batch storage reduces the operating system overhead and disk seek time associated with frequently opening and closing files and storing small data packets, improving efficiency and reducing the risk of errors or data corruption due to frequent operations, thus enhancing the reliability of data storage. For processing, processing the entire block of data helps maintain the contextual relevance of data packets, making the parsing of complex data structures and parameter calculations smoother and more accurate. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1This is a schematic diagram of the method steps of a multi-channel, multi-threaded data stream processing method and system provided by the present invention; Figure 2 This is a schematic diagram of the data receiving thread of a multi-channel, multi-threaded data stream processing method and system provided by the present invention; Figure 3 This is a schematic diagram of the data storage thread of a multi-channel, multi-threaded data stream processing method and system provided by the present invention; Figure 4 This is a schematic diagram of the data processing thread of a multi-channel, multi-threaded data stream processing method and system provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method, step, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the method, step, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the method, step, or apparatus that includes that element.

[0026] Firstly, to address the technical problem in existing technologies where tight coupling between data reception, storage, and processing stages leads to intense competition for system resources and processing flow blockage under high-speed data streams, resulting in data loss, this invention provides a multi-channel, multi-threaded data stream processing method. Specifically, as follows: Figure 1 As shown, the method includes: S100: Creates a storage cache pool for data storage and a computation cache pool for data computation; the independent storage cache pool and computation cache pool each include multiple independent channels.

[0027] As an example, the number of channels in the storage cache pool and the computation cache pool, as well as the size of each channel, are dynamically configured based on the actual amount of data and computer performance.

[0028] For example, a multi-channel storage cache pool can be created, where each channel is a storage channel, the number of channels is N, and the size of each channel is M megabytes. Both N and M can be configured according to actual needs, as long as they do not exceed the computer's available memory. The storage cache pool is a cache space allocated by the software to buffer the received data stream for data storage. The purpose of the storage cache pool is to temporarily store the received data as quickly as possible for subsequent storage on the storage medium, avoiding data reception being blocked due to slow disk I / O speeds.

[0029] Create a multi-channel solution cache pool. Each channel is a solution channel, with X channels and Y megabytes in size. Both X and Y can be configured according to actual needs, as long as they do not exceed the computer's available memory. The solution cache pool is a cache space allocated by the software to buffer the received data stream for data solution. The purpose of the solution cache pool is to temporarily store data to be solved, ensuring that complex calculation processes do not affect the speed of data reception and storage.

[0030] It should be noted that the independent storage cache pool and computation cache pool, as well as the number and size of channels, can be flexibly configured according to the actual data volume and computer performance. This is suitable not only for complex scenarios that require simultaneous storage and computation, but also for simple scenarios that only require storage or only computation. Even with industrial control computers with poor performance, increasing the number of cache channels can provide sufficient buffer time for slow disk I / O or CPU computation, thereby adapting to high-speed data streams and demonstrating good robustness.

[0031] S200: Through an independently running data receiving thread, the received data stream is synchronously written into the channels of the storage cache pool and the calculation cache pool; when any channel reaches the threshold, the corresponding status flag is triggered, and the system switches to the next channel to continue working.

[0032] The data receiving thread is the sole entry point for the data stream, simultaneously writing data to be stored into both the current storage channel and the processing channel. When the data volume in any channel reaches a preset threshold, it performs a critical operation: either setting the corresponding status flag for that channel to valid and notifying the downstream thread (data storage thread or data processing thread) to handle it; or immediately switching to the next available channel to continue writing data. This approach ensures that the data receiving thread is never blocked while waiting for storage or processing, achieving continuity in data reception.

[0033] It should be noted that a data stream refers to a continuous, high-speed, and non-reproducible data sequence. Data streams can be remote sensing data streams (such as engineering parameter data, mission payload data, audio and command data, etc.), network communication data streams (such as user behavior logs or events generated by search engines, social platforms, etc.), Internet of Things and Industrial Internet data streams (such as continuous data generated by smart meters, environmental monitoring points, industrial production line sensors, etc.), multimedia data streams (such as audio, video, etc.), etc.

[0034] The threshold is either a storage threshold or a processing threshold. As an example, the storage threshold and the processing threshold are preferably configured to be 70% to 80% of the total capacity of their respective cache channels.

[0035] As an example, when the data receiving thread writes data to the channels of the storage cache pool and the computation cache pool, it can use memory copying to write the data packets received in a single transaction to the corresponding cache channels of the two cache pools simultaneously. This ensures data synchronization and consistency, avoids the risk of data inconsistency that may be caused by asynchronous distribution, and improves the reliability of data processing.

[0036] In addition, a storage status flag (for storing channels) or a solution status flag (for solving channels) is set for each channel. These flags are used to determine whether a channel is currently full and ready to be processed by downstream threads.

[0037] As one embodiment, when any channel reaches a threshold, a corresponding status flag is triggered, and the system switches to the next channel to continue operating, including: Create a storage status identifier for each storage channel in the storage cache pool; when the amount of data in the current storage channel in the storage cache pool reaches the storage threshold, set the storage status identifier corresponding to the current storage channel to a valid state, switch the data receiving thread, and write the data to be stored into the next storage channel in the storage cache pool. Create a solution status identifier for each solution channel in the solution cache pool; when the amount of data in the current solution channel in the solution cache pool reaches the solution threshold, set the solution status identifier corresponding to the current solution channel to the valid state, switch the data receiving thread, and write the data to be stored into the next solution channel in the solution cache pool.

[0038] Furthermore, such as Figure 2 As shown, the data receiving thread is: Receive data to be stored from an external interface, obtain the storage channel number of the current storage channel for data storage, write the data to be stored into the current storage channel in sequence, and determine whether the amount of data in the current storage channel has reached the storage threshold. If the storage threshold is reached, the storage status flag of the current storage channel is set to valid, the storage channel number for data storage is set to the next storage channel number, and the current processing channel number for data processing is obtained; if the storage threshold is not reached, the current processing channel number for data processing is obtained directly. After obtaining the current solution channel number for data solution, the data to be solved is written into the current solution channel in sequence, and it is determined whether the amount of data in the current solution channel has reached the solution threshold. If the calculation threshold is reached, the calculation status flag of the current calculation channel is set to valid, and the calculation channel number for data calculation is set to the next calculation channel number. Then, the next data to be stored is received from the external interface, and the next data reception is performed. If the storage threshold is not reached, the next data to be stored is received directly from the external interface, and the next data reception is performed.

[0039] The external interface is either a gigabit Ethernet interface or a fiber optic interface.

[0040] As a specific embodiment, the data stream is received, the sequence number of the storage channel for which data needs to be written is obtained as 1, the data is stored in storage channel 1, and it is determined whether the data capacity of storage channel 1 has reached the storage threshold of 75%.

[0041] When the total capacity of the channel is reached to more than 75%, after the current data is written, the writing of data to storage channel 1 will stop, the status flag of writing files to storage channel 1 will be set to valid, and the storage channel number that needs to be written will be set to 2. In this way, the next data will be directly written to storage channel 2.

[0042] When the data capacity of storage channel 2 reaches 75% of the storage threshold, after the current data is written, the writing of data to storage channel 2 will stop, the status flag for writing files to storage channel 2 will be set to valid, and the storage channel number for which data needs to be written will be set to 3. In this way, the next data will be written directly to storage channel 3.

[0043] This process continues until all storage channels have been written with data sufficient to meet the required capacity. Once the last storage channel has been written with the required capacity, storage will resume from storage channel 1 again.

[0044] Simultaneously, whenever data is written to the storage cache pool, it is also synchronously written to the computation cache pool. That is, the received data stream is sequentially stored into the computation channels according to the above method. The sequence number of the computation channel that needs to write data is obtained as 1, the data is stored in computation channel 1, and it is determined whether the data capacity of computation channel 1 has reached the computation threshold of 75%.

[0045] When the total capacity of the channel is reached to more than 75%, after the current data is written, the writing of data to the solution channel 1 will stop, the status flag of writing to the file in the solution channel 1 will be set to valid, and the sequence number of the solution channel that needs to be written will be set to 2. In this way, the next data will be directly written to the solution channel 2.

[0046] When the data capacity of solution channel 2 reaches 75% of the storage threshold, after the current data is written, stop writing data to solution channel 2, set the status flag of writing file to solution channel 2 to valid, and set the solution channel number that needs to be written to 3, so that the next data will be directly written to solution channel 3.

[0047] This process continues until all solution channels have been written with data that meets the required capacity. Once the last solution channel has been written with data that meets the required capacity, storage will resume from solution channel 1.

[0048] S300: Through independent and parallel data storage threads and data processing threads, each thread independently polls and listens to the status flags of each channel in its respective cache pool. Once a valid status is detected, the data in the corresponding storage channel is immediately stored in the storage medium or the data in the corresponding processing channel is processed and analyzed.

[0049] In other words, a data storage thread that runs independently and in parallel with the data receiving thread stores the data in the corresponding storage channel into the storage medium; at the same time, a data processing thread that runs independently and in parallel with the data receiving thread processes and analyzes the data in the corresponding processing channel.

[0050] It's important to note that the data storage thread and the data processing thread execute in complete parallel; there is no specific order between them. They are independent of each other, each retrieving data from its dedicated cache pool and performing subsequent operations. The data storage thread operates on the data storage channel, while the data processing thread operates on the data processing channel.

[0051] As an example, the data storage thread polls and listens to the status flags of each channel in the storage cache pool, including: when the storage status flag of a certain storage channel is found to be valid, the data in the storage channel is stored into the storage medium; after the storage is completed, the storage status flag of the storage channel is reset to invalid.

[0052] In other words, the data storage thread continuously polls and checks the storage status flags of each storage channel. When a flag is found to be valid, it indicates that the data in that channel is ready to be saved, and all the data in that channel is stored into the storage medium at once. This batch storage operation is more efficient than frequently storing small amounts of data. After storage is complete, the status flag is cleared, and the next channel is polled. This separates the time-consuming disk I / O operation from the high-speed data reception process, avoiding data reception being blocked due to slow disk I / O speed.

[0053] Furthermore, such as Figure 3 As shown, the data storage thread is: Obtain the storage channel number of the current storage channel for data storage, and determine whether the storage status identifier of the current storage channel is valid; If the state is invalid, continue to obtain the storage channel number of the current storage channel for data storage; if the state is valid, store the data in the current storage channel into the storage medium. After the data is stored, the storage status flag of the storage channel is reset to an invalid state, the data of the storage channel is cleared, the storage channel number for data storage is set as the next storage channel number, and the storage channel number for data storage is obtained again. This process is repeated continuously to store data.

[0054] As a specific embodiment, the channel number for which data storage operation needs to be performed is obtained as 1. At this time, the data storage thread checks the status flag of storage channel 1. If it is in a valid state, that is, data storage can be performed, then all data of storage channel 1 is immediately stored into the storage medium. After storage is completed, the status flag corresponding to storage channel 1 is cleared to zero, that is, reset to an invalid state, and then the storage channel number for data storage is set to 2.

[0055] The status flag of storage channel 2 is checked to determine whether data storage is permitted. If it is valid, the storage condition is met, and all data from storage channel 2 is stored in the storage medium. After storage is complete, the status flag of storage channel 2 is cleared to zero, i.e., reset to invalid, and the storage channel number for data storage is set to 3.

[0056] Similarly, the same method is used to determine whether the next storage channel is in a valid state. If it is in a valid state, the storage operation is performed immediately, and the corresponding identifier status of the channel is cleared after storage is completed.

[0057] By polling status flags instead of passively waiting, ready storage channels can be accurately captured and batch storage can be performed immediately. This completely decouples time-consuming storage operations from high-speed data receiving processes, ensuring that storage tasks do not affect the continuous operation of the receiving thread, and greatly improving the overall throughput and reliability of the system.

[0058] As an example, the data processing thread polls and listens to the status flags of each channel in the processing cache pool, including: when the processing status flag of a certain processing channel is found to be valid, the data in the processing channel is processed and analyzed; after the processing is completed, the processing status flag of the processing channel is reset to invalid.

[0059] The data processing thread operates on the same logic as the data storage thread, but its operations target the processing channels. That is, the data processing thread continuously polls and checks the processing status flags of each processing channel, processing data from channels with valid flags, such as unpacking, calculating, and displaying. After processing is complete, the status flags are cleared, and the thread polls the next channel. This separates CPU-intensive computational tasks from data receiving and data storage tasks, ensuring that complex computational processes do not affect the speed of data receiving and storage.

[0060] Furthermore, such as Figure 4 As shown, the data storage thread is: Obtain the current solution channel number for the data solution channel and determine whether the solution status identifier of the current solution channel is valid; If the state is invalid, the current solution channel number for the current solution channel will be obtained; if the state is valid, the data in the solution channel will be solved and analyzed. After the calculation is completed, the calculation status flag of the calculation channel is reset to invalid, the data of the calculation channel is cleared, the calculation channel number for data calculation is set as the next calculation channel number, and the calculation channel number for data calculation is obtained again. This process is repeated continuously to solve the data.

[0061] In a specific implementation, the channel number for which data processing is required is set to 1. The data processing thread then checks the status flag of channel 1. If it is valid, meaning data processing can proceed, the processing of channel 1 is immediately initiated, including disassembling the data structure, calculating parameter coefficients, and displaying parameter content. After processing is complete, the flag status of channel 1 is cleared to zero, i.e., reset to invalid, and the channel number for data processing is then set to 2.

[0062] The status flag of solution channel 2 is checked to determine whether data processing can be performed. If it is valid, the processing conditions are met, and the processing of solution channel 2 is started immediately. After processing is completed, the status flag of solution channel 2 is cleared to zero, that is, reset to invalid state, and the sequence number of the solution channel for data processing is set to 3.

[0063] Similarly, the same method is used to determine whether the next solution channel is in a valid state. If it is in a valid state, the solution operation is performed immediately. After the solution is completed, the corresponding identifier state of the channel is cleared to zero.

[0064] By driving the status flag and cyclically switching the channel sequence number, the decomposition task is completely decoupled from the receiving and storage tasks. The data decomposition thread runs independently and is only triggered when the channel data is ready, avoiding idle time consumption. After the decomposition is completed, the flag is immediately reset and the channel is switched to ensure the efficient and orderly execution of the decomposition task, thereby providing continuous and stable processing capabilities for high-speed data streams.

[0065] It's important to note that the data receiving thread, data storage thread, and data processing thread run in parallel under the operating system's scheduling, operating independently of each other. The data receiving thread simultaneously stores a copy of the original data into both the storage channel and the processing channel, providing the material basis for the parallel operation of the data storage and processing threads. The startup and operation of the data storage and processing threads do not depend on the completion status of the other, but are triggered by their respective status flags. Their simultaneous operation fully utilizes the computing power of the multi-core CPU and disk I / O bandwidth, enabling computationally intensive data processing and I / O-intensive data storage to be performed concurrently. Furthermore, a brief delay or fluctuation in any thread will not directly affect the normal operation of the other thread, greatly improving the overall system throughput and stability under high pressure. In addition, if the two operations are executed sequentially, slow disk writes may block data processing, or complex data processing may delay data storage; parallel operation completely eliminates this risk.

[0066] As one embodiment, the method further includes: disabling the computation cache pool and data computation thread in scenarios where only data storage is required; and disabling the storage cache pool and data storage thread in scenarios where only data computation is required.

[0067] In other words, by disabling unnecessary cache pools and threads, the system can concentrate computing and memory resources on the current core task. In storage-only scenarios, memory can be entirely allocated to the storage cache pool to handle data surges, and the CPU does not need to be preempted by computation tasks; conversely, in computation-only scenarios, the interference of disk I / O operations on real-time computing performance can be avoided. This maximizes resource utilization and optimizes task execution efficiency.

[0068] In a second aspect, the present invention also provides a multi-channel, multi-threaded data stream processing system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0069] The specific technical features of the system are consistent with those of the method, and will not be elaborated here.

[0070] In summary, this invention decouples high-speed, continuous data reception tasks from relatively low-speed, time-consuming data storage and processing tasks by creating independent storage and processing cache pools. This asynchronous pipelined architecture completely avoids data reception blockage caused by slow data processing or storage speeds, fundamentally eliminating data loss. Furthermore, separating data storage and processing maximizes system resource utilization and significantly improves overall throughput. In addition, this architecture is highly flexible, adaptable to different hardware platforms and diverse application scenarios by configuring the number of channels, demonstrating strong versatility and robustness.

[0071] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0072] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0074] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-channel, multi-threaded data stream processing method, characterized in that, The method includes: Create a storage cache pool for data storage and a computation cache pool for data computation; the storage cache pool and the computation cache pool are independent of each other and each includes multiple independent channels; The received data stream is synchronously written into the channels of the storage cache pool and the solution cache pool through an independently running data receiving thread; when any channel reaches the threshold, the corresponding status flag is triggered and the system switches to the next channel to continue working. The data storage thread and data processing thread run independently and in parallel, and independently poll and listen to the status flags of each channel in their respective cache pools. Once a valid status is identified, the data in the corresponding storage channel is immediately stored into the storage medium or the data in the corresponding processing channel is processed and analyzed.

2. The multi-channel, multi-threaded data stream processing method according to claim 1, characterized in that, When any channel reaches its threshold, a corresponding status indicator is triggered, and the system switches to the next channel to continue operating, including: A storage status identifier is created for each storage channel of the storage cache pool; when the amount of data in the current storage channel in the storage cache pool reaches the storage threshold, the storage status identifier corresponding to the current storage channel is set to a valid state, and the data receiving thread is switched to write the data to be stored into the next storage channel of the storage cache pool. A solution status identifier is created for each solution channel in the solution cache pool. When the amount of data in the current solution channel in the solution cache pool reaches the solution threshold, the solution status identifier corresponding to the current solution channel is set to a valid state, and the data receiving thread is switched to write the data to be stored into the next solution channel of the solution cache pool.

3. The multi-channel, multi-threaded data stream processing method according to claim 2, characterized in that, The data storage thread polls and listens to the status flags of each channel in the storage cache pool, including: when the storage status flag of a certain storage channel is found to be valid, the data in the storage channel is stored into the storage medium; after the storage is completed, the storage status flag of the storage channel is reset to invalid.

4. The multi-channel, multi-threaded data stream processing method according to claim 2, characterized in that, The data processing thread polls and listens to the status flags of each channel in the processing cache pool, including: when the processing status flag of a certain processing channel is found to be valid, the data in the processing channel is processed and analyzed; after the processing is completed, the processing status flag of the processing channel is reset to invalid.

5. A multi-channel, multi-threaded data stream processing method according to any one of claims 2 to 4, characterized in that, The data receiving thread is: Receive data to be stored from an external interface, obtain the storage channel number of the current storage channel for data storage, write the data to be stored into the current storage channel in sequence, and determine whether the amount of data in the current storage channel has reached the storage threshold. If the storage threshold is reached, the storage status flag of the current storage channel is set to valid, the storage channel number for data storage is set to the next storage channel number, and the current processing channel number for data processing is obtained; if the storage threshold is not reached, the current processing channel number for data processing is obtained directly. After obtaining the current solution channel number for data solution, the data to be solved is written into the current solution channel in sequence, and it is determined whether the amount of data in the current solution channel has reached the solution threshold. If the calculation threshold is reached, the calculation status flag of the current calculation channel is set to valid, and the calculation channel number for data calculation is set to the next calculation channel number. Then, the next data to be stored is received from the external interface, and the next data reception is performed. If the storage threshold is not reached, the next data to be stored is received directly from the external interface, and the next data reception is performed.

6. A multi-channel, multi-threaded data stream processing method according to any one of claims 5, characterized in that, The data storage thread is: Obtain the storage channel number of the current storage channel for data storage, and determine whether the storage status identifier of the current storage channel is valid; If the state is invalid, continue to obtain the storage channel number of the current storage channel for data storage; If the status is valid, the data in the current storage channel will be stored in the storage medium. After the data is stored, the storage status flag of the storage channel is reset to an invalid state, the data of the storage channel is cleared, the storage channel number for data storage is set as the next storage channel number, and the storage channel number for data storage is obtained again. This process is repeated continuously to store data.

7. A multi-channel, multi-threaded data stream processing method according to any one of claims 5, characterized in that, The data processing thread is: Obtain the current solution channel number for the data solution channel and determine whether the solution status identifier of the current solution channel is valid; If the state is invalid, continue to obtain the current solution channel number for the current solution channel to perform data solution; If it is in a valid state, then the data in the solution channel will be solved and analyzed; After the calculation is completed, the calculation status flag of the calculation channel is reset to invalid, the data of the calculation channel is cleared, the calculation channel number for data calculation is set as the next calculation channel number, and the calculation channel number for data calculation is obtained again. This process is repeated continuously to solve the data.

8. A multi-channel, multi-threaded data stream processing method according to any one of claims 1 to 7, characterized in that, The method further includes: disabling the computation cache pool and the data computation thread in scenarios where only data storage is required; and disabling the storage cache pool and the data storage thread in scenarios where only data computation is required.

9. The multi-channel, multi-threaded data stream processing method according to claim 1, characterized in that, The number of channels in the storage cache pool and the solution cache pool, as well as the size of each channel, are dynamically configured based on the actual data volume and computer performance.

10. A multi-channel, multi-threaded data stream processing system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

Images