A multi-modal data stream adaptive scheduling method

Abstract

This invention belongs to the field of multimodal data fusion and scheduling technology, and in particular to an adaptive scheduling method for multimodal data streams. The method includes the following steps: Step 1, data access; Step 2, classification and caching: multiple digital data streams are written into corresponding cache channels according to service type, priority, or transmission requirements, and the data in different channels are temporarily stored and decoupled in terms of rate to form independent queues of data streams to be scheduled, providing a queued input basis for subsequent status acquisition and scheduling decisions; Step 3, status acquisition: the channel status parameters of each queue of data streams to be scheduled are acquired, and the channel status parameters include at least the data flow status, historical backlog status, and latency urgency status; Step 4, scheduling decision; Step 5, unified output.

Description

Technical Field

[0001] This invention belongs to the field of multimodal data fusion and scheduling technology, and in particular to an adaptive scheduling method for multimodal data streams. Background Technology

[0002] With the development of multimodal information sensing technology and embedded processing technology, more and more application scenarios require the simultaneous input of multiple modalities of data in the same processing node, and their unified processing and output. However, due to the significant differences in data rate, frame length, arrival period, real-time requirements, and importance of different modal data, the transmission and unified scheduling of multimodal data in the same platform has gradually become a key issue in data processing systems.

[0003] Existing data scheduling methods mostly employ fixed-priority scheduling, ordinary polling scheduling, or static time-slot allocation scheduling. While fixed-priority scheduling can guarantee the priority transmission of high-priority services to a certain extent, it easily leads to long waiting times for low-priority data, thus affecting overall transmission efficiency. Ordinary polling scheduling is simple to implement but struggles to reflect the differences in real-time performance and importance among different data streams, and is prone to service imbalance in variable-length data frame scenarios. Static time-slot allocation scheduling can achieve a certain degree of time sequence isolation, but it is difficult to adjust resource allocation in a timely manner when data traffic changes dynamically, buffer backlogs fluctuate, or sudden service interruptions occur, leading to decreased link utilization. In summary, when multiple data streams with different priorities, rates, and latency requirements share the same output resource, existing scheduling methods struggle to achieve adaptive perception and dynamic arbitration of the status of each channel, easily resulting in insufficient real-time performance of critical data streams, increased backlog of ordinary data streams, and decreased link utilization during the unified output process. Therefore, we propose a multimodal data stream adaptive scheduling method to address the aforementioned core scheduling problems.

[0004] Chinese patent publication number "CN114827040B" is titled "A Method for Planning and Scheduling Transmission-Constrained Services." This method maintains a dynamic service planning and scheduling table and combines bandwidth allocation interval constraints and a priority round-robin mechanism to schedule and control transmission-constrained services, thereby reducing service scheduling latency and FPGA storage resource consumption to some extent. However, this method is mainly aimed at transmission-constrained service scenarios, and it still lacks a targeted adaptive processing mechanism for the differentiated scheduling problems of different data streams in terms of priority, traffic status, buffer backlog status, and latency urgency status under multimodal data stream concurrent input conditions. Summary of the Invention

[0005] To address the shortcomings of existing multimodal data stream scheduling methods, which struggle to adaptively adjust to different data stream priorities, traffic states, buffer backlogs, and latency urgency, leading to insufficient real-time performance of critical data, unbalanced service for ordinary data streams, and reduced efficiency of unified output, as well as the inadequacy of existing fixed-weight polling and static credit allocation methods for dynamic traffic scenarios, this invention proposes a TWDRR multimodal data stream adaptive scheduling algorithm. Within a time-slot isolation framework, this algorithm introduces a byte credit adaptive update mechanism for service queues within general time slots, and combines traffic states, historical backlog states, and latency urgency states to achieve dynamic scheduling. This enables adaptive scheduling and unified output of multimodal data streams, improving the real-time performance of critical data streams while maintaining the transmission efficiency of ordinary data streams.

[0006] To achieve the above objectives, the present invention specifically adopts the following technical solution: An adaptive scheduling method for multimodal data streams includes the following steps: Step 1, Data Access: Receive data input in five modalities: video data, audio data, temperature data, smoke data, and alarm data, and convert the different modal information into multiple digital data streams that can be transmitted and processed within the processing platform; Step 2, Categorized Cache: Write multiple digital data streams into corresponding cache channels according to service type, priority or transmission requirements, and temporarily store and decouple the data of different channels to form independent queues of data streams to be scheduled, providing a queued input basis for subsequent status acquisition and scheduling decisions; Step 3, Status Acquisition: Acquire the channel status parameters of each data stream queue to be scheduled. The channel status parameters include at least the data flow status, historical backlog status, and latency urgency status. Step 4, Scheduling decision: Based on the channel status parameters, the TWDRR adaptive scheduling algorithm proposed in this invention is used to adaptively schedule each data stream to be scheduled, and to determine the target output channel and its transmission order in the current scheduling period; Step 5, Unified Output: Based on the scheduling decision results, the selected target channel data stream is sequentially output and time-series multiplexed to form a unified output data stream and output to the subsequent module; Furthermore, in step 1, the accessed data streams include video data streams, audio data streams, temperature data streams, smoke data streams, and alarm data streams; wherein, the video data stream is used to represent image information within the monitoring area, the audio data stream is used to represent sound information within the monitoring area, the temperature data stream is used to represent ambient temperature changes within the monitoring area, the smoke data stream is used to represent smoke concentration changes within the monitoring area, and the alarm data stream is used to represent alarm information triggered by abnormal flames, abnormal temperatures, or abnormal smoke; after the above types of data are accessed through the corresponding input interfaces, they are converted into digital data streams that can be cached, scheduled, and output within the processing platform; Furthermore, in step 2, the multiple digital data streams accessed in step 1 are written into corresponding buffer channels according to service type, priority or transmission requirements. Among them, the high-timeliness alarm data stream is written into the high-priority channel, and the video data stream, audio data stream, temperature data stream and smoke data stream are written into the ordinary priority channel to form a multi-channel data stream queue to be scheduled. Furthermore, in step 3, the obtained channel status parameters include data flow status, historical backlog status, and latency urgency status; wherein, data flow status is used to characterize the current channel input flow size, historical backlog status is used to characterize the current channel buffer backlog degree, and latency urgency status is used to characterize the waiting time or latency urgency degree of the current channel queue head data frame. Furthermore, in step 4, the TWDRR adaptive scheduling algorithm proposed in this invention is used for scheduling decisions. The TWDRR adaptive scheduling algorithm uses a combination of time slot isolation and byte credit adjustment to construct a total scheduling time slot and divide the total scheduling time slot into dedicated time slots and general time slots. The dedicated time slots are used to send high-priority data frames, and the general time slots are used to send other priority data frames. Within the general time slots, the byte credit value of each traffic channel is adaptively updated based on the data traffic status, historical backlog status, and latency urgency status of each traffic channel. The target output channel and sending order are determined based on the updated byte credit value and the current queue head frame length. After the byte credit value of each traffic channel is updated, the corresponding byte credit value is added to the deficit counter of the corresponding channel. When the current deficit counter value of a channel is greater than or equal to the length of the first frame of the current queue of that channel, the first frame of the current queue of that channel is allowed to be sent. After the first frame of the current queue is sent, the corresponding deficit counter value is deducted according to the length of the first frame of the current queue. When the deficit counter value after deduction still meets the sending conditions, subsequent data frames continue to be sent until the current channel buffer is empty or the remaining credit is insufficient to support the sending of the first frame of the current queue.

[0007] Furthermore, in step 5, according to the target output channel and sending order determined in step 4, the selected data stream is sequentially output and time-series multiplexed to form a unified output data stream, and the unified output data stream is sent to the subsequent processing module.

[0008] Compared with existing technologies, this invention provides a multimodal data stream adaptive scheduling method, which has the following beneficial effects: (1) By setting a scheduling method that combines dedicated time slots and general time slots, the present invention prioritizes the scheduling and transmission of high-priority alarm data streams, thereby ensuring the timeliness of transmission of key data under the condition of concurrent input of multiple data streams and reducing the transmission waiting time of high-priority data streams.

[0009] (2) In general time slots, the present invention combines the data flow status, historical backlog status and time delay urgency status of each flow channel to adaptively update the byte credit value, thereby avoiding the problem of long-term waiting or service imbalance of ordinary data streams under fixed priority scheduling or static polling mode, and improving the scheduling flexibility and overall transmission efficiency of multiple ordinary data streams.

[0010] (3) In view of the differences in traffic volume, arrival rate, frame length and latency requirements of video data stream, audio data stream, temperature data stream, smoke data stream and alarm data stream, the present invention adopts a state-aware adaptive scheduling method, which can dynamically adjust the scheduling results according to the real-time status of each channel, thereby better adapting to complex scenarios such as traffic fluctuations, buffer backlog and sudden alarms.

[0011] (4) By sequentially outputting and time-series multiplexing of the selected target channel data stream, the present invention forms a unified output data stream from multiple data streams to be scheduled, which facilitates unified reception and processing by the subsequent processing module, communication link or storage module, thereby reducing the complexity of the subsequent interface processing and improving the overall output organization capability of the system.

[0012] (5) The TWDRR adaptive scheduling algorithm proposed in this invention can achieve more reasonable link resource allocation under variable length data frame conditions, and improve the scheduling flexibility and time slot utilization of ordinary services. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the workflow of a multimodal data stream adaptive scheduling method according to the present invention. Figure 2 This is a flowchart of the adaptive credit adjustment of the present invention; Figure 3 This is a schematic diagram of the time slot division and output of the adaptive scheduling algorithm described in this invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] like Figure 1 As shown in the figure, an embodiment of the present invention proposes a multimodal data stream adaptive scheduling method, the steps of which include: Step 1, Data Access: Receive data input in five modalities: video data, audio data, temperature data, smoke data, and alarm data, and convert the different modal information into multiple digital data streams that can be transmitted and processed within the processing platform; The video data stream is used to acquire continuous image information within the monitoring area. This video data stream primarily reflects the status of on-site targets, scene changes, and abnormal image features, and is a large-volume continuous service stream within the system. The audio data stream is used to acquire sound information within the monitoring area. This audio data stream primarily reflects changes in ambient sound, abnormal acoustic events, or voice information, and is a continuous service stream within the system. The temperature data stream is used to acquire ambient temperature change information within the monitoring area. This temperature data stream primarily reflects the ambient temperature status and its changes, and is a low-rate status monitoring data stream within the system. The smoke data stream is used to acquire smoke concentration change information within the monitoring area. This smoke data stream primarily reflects the ambient smoke status and its changing trends, and is a low-rate status monitoring data stream within the system. The alarm data stream is used to transmit alarm flag information or alarm event information generated by flame anomaly detection, temperature threshold exceeding detection, or smoke threshold exceeding detection. This alarm data stream is used for rapid reporting when an abnormal status is detected and is a high-priority, high-timeliness data stream within the system. To facilitate information supervision and management, as well as the formulation of orderly scheduling strategies, it is necessary to process the raw input information of different modalities into a unified data stream that can be transmitted, cached, and scheduled within the processing platform. Specifically, the information of different modalities is first digitized or accessed through digital interfaces, and then formed into a data stream with time sequence, modal identifier, and boundary identifier according to their respective data organization rules, thereby providing a unified input basis for subsequent classification caching, status acquisition, and adaptive scheduling. Video information is typically output as a pixel matrix continuously over time after acquisition. A single frame of video can be represented as: in, Indicates frame width. Indicates frame height, Indicates the first Frame coordinates The pixel value at that location. If the video frame rate is... The pixel bit width is The uncompressed video data rate can then be expressed as: Direct calculation of digital video data volume reveals that video services are characterized by large data volume and strong continuity.

[0016] Video data is transmitted in the format of AXI4-Stream Video. Line boundaries and frame boundaries are marked by additional control signals. The frame video stream is organized as follows: in, For the first One video data word, For the start of frame marker, This is a line-end marker. A time sequence marker is also called a timestamp. For the first The number of data words in a frame. After the above processing, the original image information is converted into a continuous video data stream with frame boundaries, line boundaries, and temporal order.

[0017] The audio information is first converted from analog to digital to obtain discrete sample values. Let the audio sampling frequency be... Quantization bit width is The number of channels is Then the raw PCM data rate per unit time can be expressed as: Audio data is output continuously as a serial bit stream, following the bit clock and word select signal, as follows: in, Indicates the first PCM sample values, Indicates the audio channel identifier. Indicates the sampling time or time sequence marker. This represents the number of sampling points. After this processing, the raw audio information is organized into a continuously buffered and scheduled audio data stream. Audio traffic is typically lower than video traffic, but it still belongs to a continuously arriving data stream.

[0018] Temperature and smoke information are typically not output continuously at full rate, but rather according to the sampling period. Periodic data acquisition. Let the first... The sample value is Then the discrete sampling sequence can be represented as The temperature sequence can be written as The smoke concentration sequence can be written as To facilitate subsequent supervision, management, and scheduling, each sampled value needs to be associated with the sampling time. Therefore, a single sensor sample is encapsulated as follows: in, For modal identification, For sampled values, For sampling timestamps, This serves as a channel identifier. This, in turn, forms a sensor data stream, which is an information stream with a time sequence and channel origin: Alarm messages are not periodic, high-volume data, but rather event-triggered data. Their triggering is based on conditions such as abnormal flames, excessive temperature, or excessive smoke. Taking threshold-based judgment as an example, the abnormal triggering conditions can be defined as follows: in, Temperature threshold The smoke threshold, This indicates the result of flame anomaly detection. An alarm event is generated when any condition is met.

[0019] For alarm data triggered in order to ensure that the subsequent scheduling process can identify its high-priority attributes, it is encapsulated as follows: in, For warning signs, Alarm type For the trigger time, This identifies the channel or event source. Multiple alarm events are organized in chronological order to form an alarm data stream. This alarm data stream is characterized by its suddenness and high timeliness requirements, so it can be used as a high-priority data stream in subsequent steps for dedicated time slot scheduling.

[0020] Step 2, Categorized Cache: Write multiple digital data streams into corresponding cache channels according to service type, priority or transmission requirements, and temporarily store and decouple the data of different channels to form independent queues of data streams to be scheduled, providing a queued input basis for subsequent status acquisition and scheduling decisions; Furthermore, in step 2, the video data stream, audio data stream, temperature data stream, smoke data stream, and alarm data stream received in step 1 are written into corresponding cache channels according to service type, priority, or transmission requirements to form independent queues of data streams to be scheduled. By classifying and caching data streams of different modalities, it is possible to avoid direct mixed transmission of multiple heterogeneous data streams after they enter the processing platform, reduce the processing complexity caused by scattered data directly entering subsequent modules, and provide a unified queued input basis for subsequent status acquisition and adaptive scheduling.

[0021] High-timeliness alarm data streams are written to high-priority cache channels, while video, audio, temperature, and smoke data streams are written to ordinary-priority cache channels. Alarm data streams, used for rapid reporting of abnormal events, are given priority in subsequent scheduling. Video and audio data streams participate in regular scheduling as continuous service streams, while temperature and smoke data streams participate in regular scheduling as low-rate status monitoring service streams. Each cache channel is independently configured to store the data to be sent corresponding to different modal data streams, ensuring that different modal data maintain their respective time order and data boundary information during the caching phase. When a modal data stream arrives, it is written to the corresponding cache channel. Multiple-channel FIFO caching, cache read / write control, or a large-capacity cache module are used to temporarily store and decouple the data from different channels. Once each channel's cache reaches its preset conditions, the data to be sent in each cache channel is used as input for subsequent status acquisition and scheduling decisions.

[0022] Data in each cache channel is continuously written to the cache according to its own arrival pattern. The processing platform records the data writing status, cache occupancy, and queue head waiting status of each cache channel to form channel status parameters for subsequent steps. By setting up a categorized cache module and a large-capacity cache module, the cache management and input organization capabilities of multiple data streams under shared output resource conditions can be improved.

[0023] Step 3, Status Acquisition: Acquire the channel status parameters of each data stream queue to be scheduled. The channel status parameters include at least the data flow status, historical backlog status, and latency urgency status. Further, in step 3, the channel status parameters of each data stream queue to be scheduled are obtained. The channel status parameters include data flow status, historical backlog status, and latency urgency status. Among them, the data flow status is used to characterize the data arrival status and input flow size of the current channel in the current scheduling period, the historical backlog status is used to characterize the degree of data backlog in the current channel buffer, and the latency urgency status is used to characterize the waiting time of the data frame at the head of the current channel queue and its latency urgency.

[0024] To characterize the data traffic status of each channel within the current scheduling period, the mean square error of the traffic for each channel is first calculated to determine the degree of dispersion in the traffic volume across multiple channels. Firstly, within the traffic sensing period, the mean square error of the traffic for each channel is calculated. Wheel to obtain channel The number of data read requests is , No. The length of each arriving data frame is Then the channel of that wheel Cumulative arrival traffic for The mean square error of the traffic in the k-th round of scheduling for .in, for The average value of the traffic of each channel, i.e. Finally, the data adjustment coefficients for the flow rate of each channel. .

[0025] Based on the remaining byte length in the cache, the historical backlog state of the quantization channel is determined. In the... After the round of scheduling is completed, a channel is set up. The remaining byte length is The maximum capacity of a single channel buffer is Then the channel In the The cache backlog factor in round-robin scheduling is .when If there is no result from the previous round of scheduling, then... This means there is no cache backlog.

[0026] The latency urgency of traffic is quantified based on the first frame wait time of the channel queue. Let the... When round scheduling begins, the channel The current team's first frame wait time is The maximum allowed waiting time is Then the channel Delay urgency coefficient .

[0027] Step 4, Scheduling decision: Based on the channel status parameters, the TWDRR adaptive scheduling algorithm proposed in this invention is used to adaptively schedule each data stream to be scheduled, and to determine the target output channel and its transmission order in the current scheduling period; Furthermore, in step 4, an adaptive scheduling method combining the time slot allocation strategy of the TAS mechanism with byte credit adjustment is adopted for scheduling decisions. The total scheduling time slot is constructed based on the real-time transmission requirements of high-priority data streams, and this total scheduling time slot is divided into dedicated time slots and general time slots. The dedicated time slots are used to send high-priority data frames, and the general time slots are used to send other priority data frames. Within the general time slots, it is first determined whether the buffer space of each traffic channel is empty. If data exists in the current buffer space, the statistics of the corresponding channel are updated. After all traffic channels have been traversed, the current round byte credit value of each traffic channel is calculated and updated based on the data traffic status, historical backlog status, and latency urgency status obtained in step 3. First, the channel state parameters in step 3 are normalized: Then, based on the obtained normalized parameters, the current-round byte credit value of each traffic channel is adaptively updated. The specific process is as follows: Figure 2 As shown, let the first... The total byte credit budget available for allocation within a typical time slot is Then the channel In the Byte credit value in the round : After each traffic channel completes the calculation of its current byte credit value, the byte credit value is accumulated into the deficit counter of the corresponding channel. Then, the scheduler sequentially checks whether there is data in the current buffer space according to the channel order. If there is data in the current channel, it further checks whether the current deficit counter is greater than or equal to the length of the first frame of the current queue in that channel. If the sending condition is met, the first frame of the current queue in that channel is sent, and the deficit counter value is deducted according to the length of the first frame of the current queue. If there is still data in the current channel after sending and the deficit counter still meets the sending condition, subsequent data frames are sent until the current channel buffer space is empty or the deficit counter is insufficient to support the sending of the first frame of the current queue. If there is no data in the current channel, or the deficit counter is insufficient to send the first frame of the current queue, the system switches to the next traffic channel to continue the judgment.

[0028] Finally, the target output channel and transmission order are determined based on the updated byte credit value and the current queue first frame length: aisle In the The deficit counter value at the start of round scheduling is The updated deficit counter value is Set up a channel. The length of the first frame of the current team is Then when the following conditions are met When this occurs, the first frame of the current queue on this channel is allowed to be sent; after the transmission is completed, the deficit counter is updated to [value]. .like If so, the current remaining credit value will be retained and accumulated in subsequent rounds. At that time, the initial deficit counter is taken .

[0029] Once all traffic channels have completed this round of traversal, the system resets the index and the statistics for this round, and enters the next round of scheduling. At the same time, it continues to count whether there is data in each traffic channel, which serves as the basis for subsequent traffic awareness and credit adaptive allocation.

[0030] Step 5, Unified Output: Based on the scheduling decision results, the selected target channel data stream is sequentially output and time-series multiplexed to form a unified output data stream and output to the subsequent module.

[0031] Based on the target output channel and transmission order determined in step 4, the selected target channel data stream is output sequentially, and time-series multiplexing is performed in the time dimension to form a unified output data stream. Specific scheduling is as follows: Figure 3 As shown; wherein, high-priority data streams are output first in dedicated time slots, and ordinary-priority data streams are sent sequentially in general time slots according to the updated byte credit value and scheduling order, thereby realizing the conversion of multi-channel parallel input data into single-channel continuous output data; the unified output data stream is output to the subsequent processing module, communication link or storage module to realize the orderly scheduling and unified output of multiple heterogeneous data streams.

[0032] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for adaptive scheduling of multi-modal data streams, the method comprising: This step includes: ​ Step 1: Receive data input in five modalities: video data, audio data, temperature data, smoke data, and alarm data; and convert the different modal information into multiple digital data streams that can be transmitted and processed within the processing platform. Step 2: Write the multiple digital data streams into the corresponding buffer channels according to the service type, priority or transmission requirements, and temporarily store and decouple the data of different channels to form independent queues of data streams to be scheduled, so as to provide a queued input basis for subsequent status acquisition and scheduling decisions. Step 3: Obtain the channel status parameters of each data stream queue to be scheduled. The channel status parameters include at least the data traffic status, historical backlog status, and latency urgency status. Step 4: Based on the channel status parameters, the TWDRR adaptive scheduling algorithm proposed in this invention is used to adaptively schedule each data stream to be scheduled, and to determine the target output channel and its transmission order in the current scheduling period. Step 5: Based on the scheduling decision results, the selected target channel data stream is sequentially output and time-series multiplexed to form a unified output data stream and output to the subsequent module. 2.The multi-modal data flow adaptive scheduling method of claim 1, wherein: In step 1, the video data stream is used to acquire image information of the monitoring area, the audio data stream is used to acquire sound information of the monitoring area, and the temperature data stream is used to acquire temperature change information of the monitoring area. 3.The multi-modal data flow adaptive scheduling method of claim 1, wherein: In step 2, the multiple digital data streams generated in step 1 are written into the corresponding cache channels according to the service type, priority or latency requirements. Among them, high-time-sensitive alarm data streams are mapped to high-priority channels, and video data streams, audio data streams, temperature data streams and smoke data streams are mapped to ordinary priority channels to form a multi-channel data stream queue to be scheduled.

4. The method of claim 1, wherein: In step 4, the TWDRR adaptive scheduling algorithm proposed in this invention is used to make scheduling decisions, and the total scheduling time slot is divided into a dedicated time slot for sending high-priority data frames and a general time slot for sending other priority data frames.

5. The method of claim 4, wherein: Within the general time slot, based on the pending data in the buffer of each traffic channel and its data traffic status, historical backlog status, and time delay urgency status, the current round byte credit value of each channel is determined, and the byte credit value is accumulated into the deficit counter of the corresponding channel; then, based on the current buffer status of each channel, the deficit counter value, and the length of the first frame, a transmission determination is made to determine the target output channel and transmission order.

6. The method of claim 2, wherein the smoke The data stream is used to acquire information on changes in smoke concentration in the monitored area, and the alarm data stream is used to transmit alarm information generated by flame anomaly detection, temperature threshold over-limit detection, or smoke threshold over-limit detection. Different modal data are converted into digital data streams that can be transmitted and processed within the processing platform via corresponding input interfaces.

Images