Multi-queue scheduler and scheduling method

Abstract

The invention provides a multi-queue scheduler and a scheduling method, and the multi-queue scheduler comprises a queue type distinguishing module which is used for classifying data packets; the port scheduler module is used for calculating a port feature value of a port to which the data packet information belongs when receiving the data; when dequeuing is scheduled, an optimal port is calculated; the queue scheduler module is used for calculating a queue characteristic value and enqueue data packet information when receiving data; and when dequeueing is scheduled, calculating an optimal queue, and scheduling dequeueing. The method has the advantages that the scheduling algorithm can be independently modified and replaced, other parts of the scheduler are not affected, and the method has good flexibility, expandability and maintainability; customization and expansion can be performed according to requirements and characteristics of a system, the number of queues is increased or modified according to actual conditions, and a priority rule and a scheduling algorithm are adjusted. The state of the characteristic value called by the queue scheduler is maintained in the BRAM, the expandability is high, the expansion number is within the depth of the BRAM, and the resource consumption is basically unchanged.

Description

Technical Field

[0001] The present invention belongs to the field of data communication technology, and in particular relates to programmable switching devices such as FPGA-based programmable switches or ASIC-based programmable switching chips, and relates to a multi-queue scheduler and a scheduling algorithm, and specifically relates to a multi-queue scheduler and a scheduling method. Background Art

[0002] In high-speed network switching and application layer multicast, there are multiple input ports and multiple output ports, and there is a problem of mismatch between input traffic and output bandwidth. When there is a burst of input traffic, the input traffic size exceeds the port output capacity in a short period of time, and the output port hardware queue may overflow, resulting in packet loss. Traditional schedulers usually use a single queue to manage all tasks. However, with the development of computer systems and the popularity of multi-core processors, single-queue schedulers have gradually exposed some problems:

[0003] 1. Fairness issues caused by a single queue: In a traditional single-queue scheduler, all incoming packets are placed in the same queue. However, different data flows may require different priorities. For example, some data flows may have high real-time requirements, while others may not have strict restrictions on bandwidth consumption. A single queue cannot meet the diverse needs of different data flows.

[0004] 2. Performance issues caused by a single queue: When processing a large number of data flows, the performance of a single queue scheduler may be affected. Because all packets need to wait in the queue, the waiting time may increase as the number of data flows increases, which may cause delays for some packets.

[0005] 3. Unable to handle burst traffic: In some cases, such as large data transmission, video streaming, or gaming, data traffic may burst. A single-queue scheduler may not be able to handle this burst traffic effectively because it does not provide a dedicated channel for burst traffic.

[0006] 4. Lack of fine-grained control: A single-queue scheduler cannot provide fine-grained control over each data flow. For example, it cannot set higher priority for specific data flows, nor can it provide lower latency for certain data flows. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the existing scheduler in terms of poor fairness, low performance, inability to effectively handle burst traffic and lack of fine-grained control.

[0008] In order to achieve the above object, the present invention proposes a multi-queue scheduler, which includes: a queue type distinction module, a port scheduler module and a queue scheduler module; wherein,

[0009] A queue type distinguishing module, configured to classify data packets according to the data packet description information when the multi-queue scheduler receives the data packet description information;

[0010] The port scheduler module is used to calculate the port characteristic value of the port to which the incoming data packet description information belongs according to the port scheduling strategy when receiving data, and update the port information; when scheduling out of the queue, the port with the best port characteristic value is obtained by comparison and the port information is updated;

[0011] The queue scheduler module is used to calculate the queue characteristic value of the data packet description information according to the queue scheduling strategy and queue type when receiving data, and put the data packet description information into the queue; when scheduling out of the queue, the head element of the queue with the best characteristic value under the port with the best characteristic value is selected by comparison, and then the queue is scheduled out of the queue.

[0012] As an improvement of the above-mentioned multi-queue scheduler, the port scheduler module includes logical resources; the port number and the corresponding port characteristic value are maintained in the logical resources; the port scheduler selects a suitable port by comparing the characteristic values ​​of each port generated according to the port scheduling strategy.

[0013] As an improvement of the above multi-queue scheduler, the port scheduler module further includes a plurality of comparators at the same level, each comparator being responsible for scheduling a portion of ports.

[0014] As an improvement of the above multi-queue scheduler, the queue scheduler module includes on-chip memory and logic resources, uses the port number as the address index, and stores the queue characteristic value of each queue data packet description information under each port in the on-chip memory.

[0015] As an improvement of the above multi-queue scheduler, the multi-queue scheduler further includes a port scheduling policy module for storing the port scheduling policy; the port scheduler module obtains the port scheduling policy from the port scheduling policy module.

[0016] As an improvement of the above-mentioned multi-queue scheduler, the multi-queue scheduler also includes a queue scheduling policy module for storing queue scheduling policies; the queue scheduler module obtains the queue scheduling policies from the queue scheduling policy module.

[0017] The present invention also provides a multi-queue scheduling method, which is implemented based on the multi-queue scheduler, and the method includes:

[0018] When the multi-queue scheduler receives the data packet description information, the queue type differentiation module classifies the data packet according to the data packet description information;

[0019] The port scheduler module calculates the port characteristic value of the port to which the incoming data packet description information belongs according to the port scheduling strategy and updates the port information;

[0020] The queue scheduler module calculates the queue characteristic value of the data packet description information according to the queue scheduling strategy and queue type, and enters the data packet description information into the queue.

[0021] As an improvement of the above method, the method further comprises:

[0022] When scheduling out of the queue, the port scheduler module obtains the port with the best port characteristic value through comparison;

[0023] The queue scheduler module compares and selects the head element of the queue with the best characteristic value under the port with the best characteristic value, and schedules it out of the queue;

[0024] The port scheduler module updates the port information.

[0025] As an improvement of the above method, in the port scheduler module, the update of the port information occurs when the data packet information is queued and dequeued;

[0026] When a queue ID is added to a port that was previously empty, the port information is updated and the port participates in the port scheduling competition; when a data packet description information is dequeued, the queue head element mapped to it is also scheduled away, and the port storing the queue ID of the data packet description information deletes the queue ID information. If there is still data in the queue, it needs to be re-queued.

[0027] Compared with the prior art, the advantages of the present invention are:

[0028] 1. The present invention realizes the decoupling of the scheduler and the scheduling algorithm, and the scheduling algorithm can be modified and replaced independently without affecting other parts of the scheduler, and has good flexibility, scalability and maintainability;

[0029] 2. The two-level scheduling structure can be customized and expanded according to the needs and characteristics of the system, the number of queues can be increased or modified according to actual conditions, and the priority rules and scheduling algorithms can be adjusted;

[0030] 3. The characteristic value state called by the queue scheduler is maintained in BRAM, which has strong scalability. The resource consumption remains basically unchanged when the number of expansions is within 512. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Shown is the overall framework diagram of the multi-queue scheduler;

[0032] Figure 2 Shown is a diagram of the architecture of a multi-queue scheduling module;

[0033] Figure 3 Shown is a diagram of the multi-comparator architecture of the port scheduler module;

[0034] Figure 4 The figure shows a schematic diagram of the signal format when the data packet descriptor is enqueued;

[0035] Figure 5 The figure shows a schematic diagram of the format of the data packet descriptor output by the multi-queue scheduler;

[0036] Figure 6 Shown is the flow chart of the port scheduler module information re-entry queue. DETAILED DESCRIPTION

[0037] In order to realize a multi-queue scheduler that can provide higher throughput and better cope with network traffic bursts, the present invention proposes a multi-queue scheduler and a scheduling method, which is suitable for switching devices such as FPGA-based programmable switches or ASIC-based programmable switching chips. The working process of the scheduler is as follows: after the data packet is stored in the cache, the data packet description information is transmitted to the scheduler. After the flow classification, the port scheduling module queues the corresponding port information and calculates the corresponding port characteristic value according to the port scheduling strategy; the queue scheduling module calculates the characteristic value of the queued data packet element according to the queue scheduling strategy. When scheduling out of the queue, two-level scheduling is performed, and the queue element with the highest priority under the port with the highest priority characteristic value is scheduled.

[0038] The multi-queue scheduler and scheduling method provided by the present invention include a queue type distinction module, a queue scheduler module, a port scheduler module, a port scheduling strategy module and a queue scheduling strategy module. The specific process is: the entire scheduling structure adopts two-level scheduling. When the data packet information enters the scheduler, it first enters the column type distinction module for feature analysis, and the port scheduling module calculates the characteristic value of the port to which the incoming data packet belongs according to the port scheduling strategy; the queue scheduling module also calculates the characteristic value of the data packet according to the queue scheduling strategy and the queue type. When scheduling out of the queue, the port scheduler first obtains the port with the best characteristic value through comparison, and the queue scheduler then selects the head element of the queue with the best characteristic value under the port with the best characteristic value through comparison, and schedules out of the queue.

[0039] The implementation steps of the queue type differentiation module are as follows: when using different scheduling algorithms, different feature values ​​need to be extracted and differentiated for the queues. When the multi-queue scheduler receives a data packet, it classifies the data packet according to the data packet description information. For example, distinguish between priority and non-priority, distinguish between order-preserving and non-order-preserving, etc. Feature value mapping needs to be performed according to the specific scheduling algorithm used.

[0040] The implementation steps of the queue scheduler module are as follows: the queue scheduler can allocate appropriate resources to different queues or tasks according to the availability and demand of resources. When receiving data, the queue scheduler module calculates the queue characteristic value of the data packet according to the queue scheduling strategy and queue type, and enters the data packet information into the queue; when scheduling out of the queue, the queue scheduler module compares and selects the head element of the queue with the best characteristic value under the port with the best characteristic value, and schedules out of the queue. The queue scheduler module includes on-chip memory BRAM and logic resources (logic gates, registers, lookup tables, etc.), and uses the port number as the address index to store the queue characteristic value of each queue data packet under each port in the on-chip memory.

[0041] The queue scheduler module ensures that each queue or task gets the computing, storage, and communication resources they need to achieve optimal performance and efficiency. At the same time, it can determine the order in which they are executed based on the priority of the queue or task. By setting different priorities for different queues or tasks, it can ensure that high-priority queues or tasks can be processed in a timely manner to meet the time sensitivity of critical tasks. Most importantly, the queue scheduler uses a specific scheduling algorithm to decide when to execute tasks in which queue. The scheduling algorithm can schedule and sort tasks based on different strategies and goals, such as fairness, response time, throughput, etc.

[0042] The implementation steps of the port scheduler module are as follows: The role of the port scheduler is to manage and allocate input / output (I / O) port resources to meet the needs of multiple devices or tasks for these ports. When receiving data, the port scheduler module calculates the port characteristic value of the port to which the incoming data packet belongs according to the port scheduling strategy and updates the port information; when scheduling out of the queue, the port scheduler module obtains the port with the best port characteristic value by comparison and updates the port information. The port scheduler module includes logical resources (registers or BlockRAM); the port number and the corresponding port characteristic value are maintained in the logical resources; by comparing the characteristic values ​​of each port generated according to the port scheduling strategy, the appropriate port is selected. The port scheduler module can also include multiple comparators of the same level, each comparator is responsible for the scheduling of a part of the ports.

[0043] The port scheduler can select appropriate ports to allocate to the requester based on predefined rules, priorities or policies to meet the needs of the device or task. When multiple tasks request the same port resources at the same time, conflicts may occur. The port scheduler can handle these conflicts and decide which device or task gets the required port resources first through reasonable algorithms or policies. This can avoid errors or performance degradation caused by resource contention and conflicts. In addition, the port scheduler uses a specific scheduling algorithm to decide when to allocate port resources to different devices or tasks. The scheduling algorithm can determine the order and method of port allocation based on different goals, such as fairness, priority, bandwidth requirements, etc. This allows resource management to be performed according to actual needs, achieve optimal performance in resource utilization, and release or reallocate idle port resources when needed. This can effectively utilize and manage port resources in the system to avoid resource waste and congestion.

[0044] The implementation steps of the port scheduling policy module are as follows: The scheduling algorithm of the port scheduler can vary according to the specific needs of the application and the characteristics of the system. For example, first come first served (FIFO) allocates ports in the order of requests. When multiple devices or tasks request port resources at the same time, ports are allocated in the order of their requests. This algorithm is relatively simple, but may lead to resource starvation and fairness problems. Round-Robin scheduling allocates port resources to each requester in turn in a cyclic manner, which can achieve fair resource allocation, but may cause some devices or tasks to wait too long. Priority scheduling assigns a priority to each requester and determines the order of port resource allocation based on the priority. Devices or tasks with high priority will get port resources first. This algorithm can meet the real-time requirements of critical tasks, but may lead to resource starvation problems for low-priority tasks. Dynamic weighted scheduling assigns a weight value to each requester and schedules and allocates port resources based on the weight value. Devices or tasks with high weight can obtain more port resources, while devices or tasks with low weight will obtain fewer resources. This algorithm can flexibly allocate port resources according to the importance or requirements of the device or task, etc. In addition to the common port scheduler scheduling algorithm, there may be other more complex scheduling algorithms and strategies in practice. In short, the scheduling algorithm can determine the order and method of port allocation based on different goals, such as fairness, priority, bandwidth requirements, etc. The specific choice of scheduling algorithm requires trade-offs and decisions based on specific application scenarios and performance requirements.

[0045] The implementation steps of the queue scheduling strategy module are as follows: the queue scheduling algorithm is applied to manage and schedule tasks or messages in the task queue or message queue. For example, First-In-First-Out (FIFO), tasks or messages are executed or processed in the order in which they arrive in the queue. The tasks or messages that arrive in the queue first will be processed first, and the tasks or messages that arrive later will wait in the queue. PQ (Priority Queuing), strictly schedules in the order of queue priority. Only after all high-priority ones are scheduled, low-priority ones can be scheduled. WFQ (Weighted Fair Queuing) allocates bandwidth fairly according to the weight of the data flow and the service quality requirements to ensure that each data flow can obtain the corresponding network resources. In the WFQ algorithm, each data flow is assigned a weight value. Data flows with higher weights will get more bandwidth shares, while data flows with lower weights will get less bandwidth shares. In this way, dynamic service quality scheduling can be performed according to the importance and requirements of the data flow. These queue scheduling algorithms have their own advantages and characteristics in different scenarios and applications. Choosing a suitable queue scheduling algorithm depends on the specific application requirements, task characteristics and system performance requirements. Some scheduling algorithms focus on response time, while others focus more on fairness and resource utilization.

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings. These examples are used to illustrate the present invention but not to limit the protection scope of the present invention. In the specific implementation, a programmable switch based on FPGA is used as an example, and on-chip state maintenance is used as an example of BRAM. These examples are used to illustrate the present invention but not to limit the protection scope of the present invention.

[0047] Figure 1The overall framework design diagram of the system is shown. First, the entire scheduler is implemented with a two-level scheduling structure, the first level is the port scheduler, and the second level is the queue scheduler. In the example, there are four ports, each with 8 queues. The port scheduling strategy is the token bucket algorithm with MTU=8192B, and the queue scheduling strategy is the LLQ algorithm with unlimited speed for priority queues. The queue IDs (queue IDs) of 0 to 5 in each port are all WFQ queues (weighted fair queues), and the queue IDs of 6 to 7 are priority queues, among which the queue with queue ID=7 has a higher priority. The specific description is as follows: After the data packet information enters the queue, it first enters the queue type distinction module, and distinguishes the WFQ queue and the priority queue according to the queue ID information. The data packet information of the priority queue and the data packet information of the WFQ queue are output from different channels of the queue type distinction module respectively. After the WFQ queue data packet information passes through the WFQ module, the scheduling time of the packet is calculated. Since the priority queue data packet does not need to be limited by the second layer, it only needs to be assigned a specific scheduling time value. Both enter the QueueScheduler (queue scheduler) to wait for scheduling. At the same time, when the data packet description information is queued in the multi-queue scheduler module, the corresponding port (port) information needs to be queued in the Port Scheduler (port scheduler). Each port is calculated by the speed limit token bucket module to obtain the sendtime (qualified time) that can be called for each port, and is passed to the PortScheduler to wait for scheduling. The Port Scheduler module determines the output of a certain port, and the port ID is returned to the QueueScheduler to schedule the queue element with the highest priority under the port. There is a global timer to calculate the current time, and all places that determine whether the packet is scheduled or the sendtime need to compare the time.

[0048] by Figure 2The queue scheduling module schematic diagram shown in the figure is described and explained. In the example, the Queue Scheduler module is actually mainly composed of BRAM, which is used to store the sendtime of the data packet elements of each queue under each port. Taking 2P6Q (2 priority queues, 6 WFQ queues) as an example, there are 8 queues under each port, and 8 BRAMs are used to store the sendtime of each queue element respectively. At the same time, 8 groups of register arrays are used to store whether there are packets in the eight queues, that is, when the sendtime of a queue is written to the corresponding address, its corresponding register needs to be set to 1. When the element is scheduled out of the queue, the register bit needs to be maintained and set to zero to prevent an element from being scheduled multiple times. When there are multiple ports, the port ID is used as the read and write address of the BRAM or register array. In this application scenario (four ports, 8 queues per port, 2P6Q), when an element needs to be queued, the sendtime (64bit) is written into the corresponding BRAM for the corresponding port, and the sendtime of the priority queue is a very small constant. When scheduling out of the queue, the Port Scheduler first selects a port according to the port speed limit condition. At this time, the port ID is used as the read address to read 8 BRAMs at the same time, and the 8 sendtimes of the 8 queue elements under the port are obtained respectively. The Priority Encoder compares the eight sendtimes with the highest priority, obtains the queue ID of the out-of-queue element, and then outputs the selected queue information.

[0049] by Figure 3The port scheduling module shown in FIG. 1 is described and explained. The port scheduler is composed of multiple small comparators, which is done to optimize the timing of the comparators. When scheduling out of the queue, the port scheduler first selects a port according to the port speed limit condition, and passes the port to the queue scheduler, and the queue scheduler selects the queue with the highest priority under the port for out of the queue. Inside the port scheduler, the number of comparators can be set. If the comparator NUM is greater than 1, the MUX (multiplexer) module is used, otherwise the fast channel is used, which can save the delay caused by the MUX module. When there are multiple comparators, each comparator is only responsible for the scheduling of a part of the port ID. For example, when the comparator NUM = 4 and 32 ports are scheduled, each comparator schedules 8 port ID elements. After the element enters the PortScheduler, the DEMUX (demultiplexer) module assigns it to the corresponding comparator according to its port ID range for enqueueing, and then waits for scheduling. When writing, the port ID information of all elements is written as 0, and the original portID is stored as data. The DEMUX here is not an entity module in the code. It directly selects the corresponding comparator module according to the high-bit information of the port ID. Each comparator has a corresponding counter module. When an element is queued into the comparator, the counter is increased by 1, and when an element is dequeued from the comparator, the counter is reduced by 1. When the comparator schedules the output, the dequeued port is always 0, because it is 0 when it is written. In addition, it is necessary to ensure that the current time is greater than the scheduling time of its element and that the comparator is non-empty. In addition, since the scheduling delay of the comparator is 2, when scheduling is enabled, the element can actually leave the comparator in the second clock cycle. In this process, the prefetch signal of the comparator will not change, so to avoid the situation where an element is scheduled multiple times, the intermittent scheduling is adopted, which ensures that there is no scheduling in the previous beat, and confirms that the current comparator is non-empty The state is the same as the previous time. This prevents the last element from being repeatedly scheduled and the empty comparator from being scheduled.

[0050] by Figure 4 The signal format diagram of the data packet descriptor when entering the queue is described in the figure. In the example, a total of 4 ports are set, and each port has 8 queues. The default queue number is 0 to 7, which belongs to the first port, and the queue number is 8 to 15, which belongs to the second port, and so on. Therefore, the input descriptor information only needs to include the queue ID and the data packet length information required by the token bucket algorithm.

[0051] by Figure 5The schematic diagram of the packet descriptor format of the scheduler scheduling output is described and explained. In the example, when the head element of a queue is scheduled, the scheduler only needs to output the queue ID and related handshake signals to the packet shared buffer module to complete the packet dequeue.

[0052] by Figure 6 The port scheduler port information re-queue flow chart shown in the figure is described for explanation. Under the port scheduler, the update of port information occurs when the descriptor information is queued and dequeued. When a queue ID queues a port that was previously empty, the port information needs to be updated and enters the token bucket to participate in the port scheduling competition; when a descriptor is dequeued, the queue head element mapped to it is also scheduled away, then the port storing the descriptor queue ID needs to delete the queue ID information, and if the queue is still not empty in the on-chip cache, it needs to be re-queued.

[0053] The FPGA-based multi-queue scheduler and scheduling method provided by the present application realize the decoupling of the scheduler and the scheduling algorithm, and they can be modified, replaced or upgraded independently without having much impact on other parts; and can be adjusted and optimized more flexibly without modifying the entire scheduler. When the system scale expands or the demand changes, the scheduling algorithm can be optimized or replaced without reconstructing the structure of the scheduler. This can better cope with the expansion and changes of the system and improve the maintainability and adaptability of the system. In addition, the present application adopts a two-level scheduling structure, which can be customized and expanded according to the needs and characteristics of the system, increase or modify the number of queues according to actual conditions, and adjust the priority rules and scheduling algorithms.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A multi-queue scheduler, characterized in that: The multi-queue scheduler includes: a queue type distinction module, a port scheduler module and a queue scheduler module; wherein, A queue type distinguishing module, configured to classify data packets according to the data packet description information when the multi-queue scheduler receives the data packet description information; The port scheduler module is used to calculate the port characteristic value of the port to which the incoming data packet description information belongs according to the port scheduling strategy when receiving data, and update the port information; when scheduling out of the queue, the port with the best port characteristic value is obtained by comparison and the port information is updated; The queue scheduler module is used to calculate the queue characteristic value of the data packet description information according to the queue scheduling strategy and queue type when receiving data, and put the data packet description information into the queue; when scheduling out of the queue, the head element of the queue with the best characteristic value under the port with the best characteristic value is selected by comparison, and then the queue is scheduled out of the queue.

2. The multi-queue scheduler according to claim 1, characterized in that: The port scheduler module includes logic resources; the port number and the corresponding port characteristic value are maintained in the logic resources; the port scheduler selects a suitable port by comparing the characteristic values ​​of each port generated according to the port scheduling strategy.

3. The multi-queue scheduler according to claim 2, characterized in that: The port scheduler module also includes a plurality of comparators at the same level, each comparator being responsible for scheduling a portion of ports.

4. The multi-queue scheduler according to claim 1, characterized in that: The queue scheduler module includes an on-chip memory and logic resources, takes the port number as the address index, and stores the queue characteristic value of each queue data packet description information under each port in the on-chip memory.

5. The multi-queue scheduler according to claim 1, characterized in that: The multi-queue scheduler also includes a port scheduling policy module for storing a port scheduling policy; the port scheduler module obtains the port scheduling policy from the port scheduling policy module.

6. The multi-queue scheduler according to claim 1, characterized in that: The multi-queue scheduler also includes a queue scheduling policy module for storing queue scheduling policies; the queue scheduler module obtains the queue scheduling policies from the queue scheduling policy module.

7. A multi-queue scheduling method, implemented based on any multi-queue scheduler according to claims 1-6, the method comprising: When the multi-queue scheduler receives the data packet description information, the queue type differentiation module classifies the data packet according to the data packet description information; The port scheduler module calculates the port characteristic value of the port to which the incoming data packet description information belongs according to the port scheduling strategy and updates the port information; The queue scheduler module calculates the queue characteristic value of the data packet description information according to the queue scheduling strategy and queue type, and enters the data packet description information into the queue.

8. The multi-queue scheduling method according to claim 7, characterized in that: The method further comprises: When scheduling out of the queue, the port scheduler module obtains the port with the best port characteristic value through comparison; The queue scheduler module compares and selects the head element of the queue with the best characteristic value under the port with the best characteristic value, and schedules it out of the queue; The port scheduler module updates the port information.

9. The multi-queue scheduling method according to claim 8, characterized in that: In the port scheduler module, the port information is updated when the data packet information is queued and dequeued; When a queue ID is added to a port that was previously empty, the port information is updated and the port participates in the port scheduling competition; when a data packet description information is dequeued, the queue head element mapped to it is also scheduled away, and the port storing the queue ID of the data packet description information deletes the queue ID information. If there is still data in the queue, it needs to be re-queued.

Images