Efficient management of queue entry state

Abstract

The invention relates to efficient management of queue entry states. Systems and techniques for tracking entry states in queues are disclosed. The techniques include providing a first entry of a queue to a first process and providing a second entry of the queue to a second process. The queue includes a plurality of segments. The techniques further include, in response to receiving the first signal from the first process, modifying a data structure associated with the queue for the first entry. A data structure associated with the queue includes a plurality of segments. Each segment includes one or more counters, each counter associated with one of the plurality of segments. The techniques also include, in response to receiving a second signal from a second process, modifying a data structure associated with the queue for a second entry. The techniques also include modifying a tail pointer of the queue based on a data structure associated with the queue.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to provisional patent application number 63 / 704,279 (filed October 7, 2024), the entirety of which is incorporated herein by reference. TECHNICAL FIELD

[0003] At least one embodiment relates to managing entries in a queue, and more specifically, to using metadata to track the state of entries in a queue. BACKGROUND

[0004] Parallel processing units (e.g., graphics processing units) can execute multiple tasks in parallel. In some cases, the tasks to be executed by a parallel processing unit can be represented as a graph of nodes. Each node can represent a unit of work to be performed on the parallel processing unit. Nodes can depend on other nodes, which can dictate an order of execution and control data flow between nodes. A producer node can produce data used by a consumer node. In some cases, a node can be both a consumer node (e.g., can use data produced by a previous node in a graph) and a producer node (e.g., can create data used by a next node in a graph). In some cases, data from a producer node can be stored in a queue associated with a consumer node. In some cases, multiple producer nodes can store data in the same queue for a particular consumer node. In some cases, multiple consumer nodes can use data from the same queue. BRIEF DESCRIPTION OF DRAWINGS

[0005] Figure 1 is a block diagram of an example system for tracking the state of entries in a queue, in accordance with at least one embodiment.

[0006] Figure 2 is a block diagram of an example entry queue, in accordance with at least one embodiment.

[0007] Figure 3 is a block diagram of an example queue with adjacent entries in different states, in accordance with at least one embodiment.

[0008] Figure 4A is a block diagram of an example queue while an entry is being consumed, in accordance with at least one embodiment.

[0009] Figure 4B is a block diagram of an example queue while an entry is being consumed, in accordance with at least one embodiment. is a block diagram of an example queue while an entry is being consumed, in accordance with at least one embodiment.

[0010] Figure 4C is a block diagram of an example queue while entries are being consumed, according to at least one embodiment.

[0011] Figure 4D is a block diagram of an example queue while entries are being consumed, according to at least one embodiment.

[0012] Figure 5A is a block diagram of a bitarray data structure for tracking the state of queue entries, according to at least one embodiment.

[0013] Figure 5B is a block diagram of a bitarray data structure for tracking the state of queue entries, according to at least one embodiment.

[0014] Figure 6 is a block diagram of a bitarray data structure for tracking the state of queue entries, according to at least one embodiment.

[0015] Figure 7A is a block diagram of a segmented bit queue data structure for tracking the state of queue entries, according to at least one embodiment.

[0016] Figure 7B is a block diagram of a segmented bit queue data structure for tracking the state of queue entries, according to at least one embodiment.

[0017] Figure 8A is a block diagram of a segmented bit queue data structure for tracking the state of queue entries that can be“partially deallocated”, according to at least one embodiment.

[0018] Figure 8B is a block diagram of a“partially deallocated” segmented bit queue data structure for tracking the state of queue entries, according to at least one embodiment.

[0019] Figure 9 is a flow diagram of an example method for tracking the state of entries in a queue, according to at least one embodiment.

[0020] Figure 10 is a block diagram of an example computer system, according to at least one embodiment of the present disclosure.

[0021] Figure 11A shows inference and / or training logic, according to at least one embodiment of the present disclosure.

[0022] Figure 11B shows inference and / or training logic, according to at least one embodiment.

[0023] Figure 12Training and deployment of neural networks are shown in accordance with at least one embodiment.

[0024] Figure 13 is an example dataflow graph of an advanced computing pipeline in accordance with at least one embodiment.

[0025] Figure 14 is a system diagram of an example system for training, tuning, instantiating, and deploying machine learning models in an advanced computing pipeline in accordance with at least one embodiment. DETAILED DESCRIPTION

[0026] It can be advantageous to track the state of items in a consumer node queue. For example, at a first time, a producer node can allocate space in the queue for a new item. That item can not be filled by the producer node and ready for consumption by a consumer node until a second time. On the consumer side, an item in the queue can be ready for consumption but can not yet have been consumed by a consumer node. Once a particular item is consumed, it can be eligible for purging (e.g., deallocation) from the queue, but some time can be required between when the item is ready to be released and when the item is actually released.

[0027] One or more pointers can be used to indicate where to allocate memory for new items in the queue, which items have been produced and are ready for consumption, which items have been consumed and are ready for release, which items have been released, and so on. In some cases, items in the queue can be ready for consumption and / or consumed out of order compared to adjacent items in the queue. In order to prevent data corruption or memory leaks, it is important to update the pointers associated with the queue only when all items between the old pointer location and the new pointer location have been processed (e.g., items that have been produced are actually ready for consumption, items that have been consumed are ready for release, and so on). This out-of-order behavior makes it difficult to update the pointers associated with the queue, and can require storing additional metadata about the queue and its items.

[0028] Aspects of the disclosure address the above-referenced deficiencies and others by providing systems and techniques for tracking the state of items in a queue and managing memory allocation and deallocation related to items in the queue. In some embodiments, a queue has a first head pointer and a first tail pointer associated with a producer side of the queue, and a second head pointer and a second tail pointer associated with a consumer side of the queue. In some embodiments, a data structure can be associated with the queue for storing metadata about the queue and its items. In some embodiments, more than one data structure can be associated with the queue. For example, there can be a first data structure for the producer side of the queue and a second data structure for the consumer side of the queue.

[0029] In a first embodiment, a data structure associated with a queue can include a maximum index and a current index. The data structure can be associated with a side of the queue (e.g., a producer side or a consumer side). A head pointer of the queue can be advanced when a new entry is allocated for the queue (e.g., on the producer side) or when an entry is provided for consumption (e.g., on the consumer side). For example, two entries can be ready for consumption, and the head pointer can be advanced over these entries. The entries can be provided to a processor (e.g., a parallel processing unit) for execution. In certain embodiments, the entries can be provided to the same processor, different threads of the same processor, different processors, etc. Once each entry is used (e.g., once the processor no longer needs to access a particular entry), the data structure associated with the queue can be updated.

[0030] In a first embodiment, after an entry has been consumed, the maximum index of the data structure can be updated based on the index of the consumed entry in the queue. For example, if the index of the consumed entry is greater than the current maximum index value of the data structure, the maximum index value can be modified to equal the index of the consumed entry. If the index of the consumed entry is less than or equal to the current maximum index value of the data structure, the maximum index value is not modified. In addition, the current index value of the data structure can be incremented for each entry that has been consumed. When the current index value of the data structure matches the maximum index value of the data structure, this indicates that all entries between the current position of the tail pointer and the maximum index value are ready to be released, and the tail pointer of the queue can be safely advanced to equal the maximum index value.

[0031] In a second embodiment, the data structure associated with the queue can include a bit array having one or more bits corresponding to indices in the queue. For example, each index in the queue can have a corresponding bit in the bit array. In some embodiments, the bit array can be divided into one or more sections, each section containing one tail indicator bit and a plurality of index bits (e.g., bits corresponding to indices in the queue). In some embodiments, a single bit array data structure can be used for the entire queue (e.g., the producer side of the queue and the consumer side of the queue can use the same bit array data structure). In this case, each set of head and tail pointers (e.g., head and tail pointers on the producer side, head and tail pointers on the consumer side, etc.) can use opposite bit values to indicate that a record is “ready” (e.g., ready to be consumed, ready to be released, etc.). For example, on the producer side, a bit value of 1 can indicate that a record is ready to be consumed, while on the consumer side, a bit value of 0 can indicate that a record is ready to be released. In some embodiments, a bit value of 0 can indicate that a record is ready to be consumed, and a bit value of 1 can indicate that a record is ready to be released. Bit values can be “flipped” (e.g., an exclusive-OR (XOR) operation is performed) to indicate that they are “ready” (e.g., ready to be consumed, ready to be released, etc.).

[0032] When a new entry is allocated for a queue, the head pointer on the producer side can be advanced to point to the new entry in the bit array data structure. Once the new entry in the queue is ready to be consumed, the bit value in the bit array data structure corresponding to that queue entry can be “flipped.” The tail pointer on the producer side can advance over a contiguous string of bit array entries having bit values indicating that the corresponding data entries are ready to be consumed (e.g., bit values of “1”).

[0033] The consumer side head pointer can advance over one or more bit array entries that are ready to be consumed (e.g., have bit values set to “1” in the data structure). Once a queue entry has been consumed and is ready to be released, the bit value in the bit array data structure corresponding to the consumed queue entry can be flipped. The consumer side tail pointer can advance over a contiguous string of bit array entries having bit values indicating that the corresponding data entries are ready to be released (e.g., bit values of “0”).

[0034] In a third embodiment, a queue can be divided into a plurality of segments, and a data structure associated with the queue can include a plurality of sections. Each section of the data structure can include one or more counters, each counter corresponding to a segment of the queue. In some embodiments, each section also includes one or more tail indicator bits and a local tail location bit. The data structure can be associated with one side of the queue (e.g., the producer side or the consumer side).

[0035] In allocating memory for a new entry in the queue, the producer-side head pointer can be advanced to point to the new entry. The new entry can be associated with a particular segment of the queue. Once the new entry in the queue is ready to be consumed, a counter associated with the segment of the new entry can be incremented in a data structure associated with the queue. The counter can be contained within a section of the data structure. Once the counter reaches a threshold, the producer-side tail pointer can be advanced over the entire segment of the queue associated with the counter.

[0036] Similarly, on the consumer side, when a new entry in the queue is provided to the consumer process, the consumer-side head pointer can be advanced to point to the new entry. The entry can be associated with a particular segment of the queue. Once the entry has been consumed and is ready to be released, a counter associated with the segment of the entry can be incremented in a data structure associated with the queue. The counter can be contained within a section of the data structure. Once the counter reaches a threshold, the consumer-side tail pointer can be advanced over the entire segment of the queue associated with the counter.

[0037] In some embodiments, if the head pointer and the tail pointer of a particular side of the queue are located in the same segment of the queue, the tail pointer can be partially advanced through the segment to prevent deadlock. In some embodiments, to prevent the head pointer from bypassing the queue to point to the same segment as the tail pointer, an initial gap can be introduced between the head pointer and the tail pointer to ensure that the head pointer lags behind the tail pointer by at least the length of one segment.

[0038] Advantages of the disclosed embodiments relative to the prior art include, but are not limited to, efficient memory management and initiation of parallel processing tasks.

[0039] Figure 1 is a block diagram of an example system 102 for tracking the status of entries in a queue, in accordance with at least one embodiment. The system 102 can include a memory 104 and one or more processors 108. The memory 104 can include read-only memory (ROM), flash memory, dynamic random-access memory (DRAM), such as synchronous DRAM (SDRAM), double data rate DRAM (DDR SDRAM), or DRAM (RDRAM), etc. The processors 108 can include one or more processing units, such as central processing units (CPUs), graphics processing units (GPUs), data processing units (DPUs), parallel processing units, accelerators, physical processing units (PPUs), etc. The memory 104 and the processors 108 can be connected by a queue and data structure management processing circuit 106.

[0040] The system 102 can be used to perform work defined in a node graph. Each node can represent a unit of work to be performed by one or more processors 108 of the system 102. Nodes can depend on other nodes, which can indicate an order of execution and control data flow between nodes. Producer nodes can produce data used by consumer nodes. In some cases, data from a producer node can be stored in a queue associated with a consumer node. Queues associated with each node can be stored in memory 104 by the queue and data structure management processing circuit 106. The memory 104 can also include data structures 112 associated with the queues 110 to track the status of entries within the queues 110.

[0041] In some embodiments, the work defined in the node graph is graphics rendering work. In some embodiments, the work defined in the node graph is artificial intelligence and / or machine learning work, such as training an artificial intelligence model and / or performing inference on an artificial intelligence model. Training and use of artificial intelligence models can be described in more detail with respect to Figure 11A 、 Figure 11B 、 Figure 12 、 Figure 13 and Figure 14 .

[0042] As an example of a simple work graph, a first node (e.g., a producer node) can be executed by a first processor (or by a first thread of a processor) of the processors 108. During execution, the first processor can signal the queue and data structure management processing circuit 106 to allocate memory for a new entry in a queue 110 associated with a particular consumer node. Memory is allocated to the entry, and the first processor can store data in the queue entry during execution of the first node. Once all data has been stored in the queue entry (e.g., once the data is ready to be consumed by a second node), the first processor can signal the queue and data structure management processing circuit 106. The queue and data structure management processing circuit 106 can update the data structure 112 associated with the queue 110 to indicate that the new entry is ready to be consumed, as described in more detail below.

[0043] The second node (e.g., a consumer node) may be executed by a second processor (or a second thread of the first processor) in processor 108. During execution, the second processor may signal the queue and data structure management processing circuitry 106 to request data stored in a queue entry associated with the consumer node. The queue and data structure management processing circuitry 106 may update the data structure 112 associated with queue 110 to indicate that the entry is being consumed. After the second node has used up the data stored in the entry, the second processor may signal the queue and data structure management processing circuitry 106, which may update the data structure 112 associated with queue 110 to indicate that the memory associated with the entry is available for release.

[0044] Each queue 110 may include one or more head pointers and tail pointers for tracking entries within the queue. The queue and data structure management processing circuitry 106 can modify the head and tail pointers as entries are allocated, produced, consumed, and released. The queue and data structure management processing circuitry 106 can track additional state associated with each queue 110 and its entries in the data structure 112 associated with that queue.

[0045] Figure 2 This is a block diagram of an example queue 202 of entries according to at least one embodiment. Queue 202 may be stored in memory (e.g., ...). Figure 1 The queue 202 may be a circular queue, located in memory 104. The queue 202 may include empty entries (e.g., empty queue slot 204, empty queue slot 212, etc.), allocated but not yet ready to be read entries (e.g., queue entries (in production) 206a, queue entries (in production) 206b, queue entries (in production) 206c, etc.), entries ready to be consumed (e.g., queue entries (ready) 208a, queue entries (ready) 208b), entries being consumed (e.g., queue entries (in consumption) 210a, queue entries (in consumption) 210b, queue entries (in consumption) 210c, queue entries (in consumption) 210d, etc.), and entries that have been consumed and are ready to be released, such as... Figure 3 As shown.

[0046] One or more pointers can be associated with the queue 202 to track the entries. One side of the queue can be associated with producer nodes and can have a producer-side head pointer and a producer-side tail pointer. For example, the producer-side head pointer 214 can indicate where the next entry in the queue is to be assigned (e.g., the next entry can be stored in the empty queue slot 204). The producer-side tail pointer 216 can indicate which entries are being produced but have not yet been prepared to be consumed. For example, the entries before the producer-side tail pointer 216 can not yet be prepared to be consumed, while the entries after the producer-side tail pointer 216 can be prepared to be consumed.

[0047] The other side of the queue can be associated with consumer nodes and can have a consumer-side head pointer and a consumer-side tail pointer. For example, the consumer-side head pointer 218 can indicate which entries are currently being consumed by one or more consumer nodes. The consumer-side tail pointer 220 can indicate which entries have been consumed and are ready to be released. For example, the entries before the consumer-side tail pointer 220 can not yet be ready to be released, while the entries after the consumer-side tail pointer 220 are ready to be released.

[0048] In some cases, the empty queue slot 212 previously stored data, was consumed by one or more consumer nodes, and was released after consumption.

[0049] Since the tail pointer acts as a boundary between “producing” entries and “produced” entries, and between “consuming” entries and “consumed” entries, it can only advance over all “produced” or “consumed” entries, as described in more detail below. Figure 3

[0050] The data structures associated with the queue 202 can track the status of the entries within the queue and can indicate when the tail pointer (e.g., the producer-side tail pointer, the consumer-side tail pointer, etc.) can be advanced over the entries of the queue.

[0051] Figure 3 is a block diagram of an example queue 302 in accordance with at least one embodiment, in which adjacent entries are in different states. The queue 302 can include one or more “ready” entries (e.g., queue entry (ready) 304), which are ready to be consumed by a consumer node, but have not yet been provided to the consumer node (e.g., a node being processed by a processor).

[0052] ​Queue 302 may include one or more “consuming” entries (e.g., queue entry (consuming) 306, queue entry (consuming) 310) that have been provided to consumer nodes and are currently being used. For example, an entry is a “consuming” entry if a node being processed by the processor is currently accessing or will access data stored in that entry.

[0053] Queue 302 may contain one or more "consumed" entries (e.g., queue entry (consumed) 308, queue entry (consumed) 312, queue entry (consumed) 314) that have been provided to consumer nodes and are no longer in use. For example, an entry is considered "consumed" if a node being processed by the processor previously accessed data stored in that entry but no longer needs to access that data. In some cases, the node has finished processing. In other cases, the node is still processing but no longer needs to access the data in that entry.

[0054] Queue 302 may also contain one or more empty slots (e.g., empty queue slot 316). In some cases, empty queue slot 316 may have previously stored data, which has been released because it has been consumed by consumer nodes.

[0055] The head pointer 318 can indicate which entries have been provided to the consumer node. When another consumer node begins execution, the head pointer 318 can advance to the queue entry (ready) 304 and provide it to the consumer node, thereby changing the queue entry (ready) 304 from the "ready" state to "consuming".

[0056] Each entry in queue 302 can be provided to and / or processed by different processors in parallel. Each entry in queue 302 can also store different amounts of data and / or have different sizes. Therefore, one entry in queue 302 may be consumed quickly after being provided to a processor, while another entry in queue 302 may take longer to be consumed. This can result in an entry having a different state compared to its surrounding entries.

[0057] For example, queue entry (consumed) 308 has been consumed, while queue entries (currently being consumed) 306 and 310 are still being consumed. As mentioned earlier, the tail pointer 320 can only advance on entries that have already been consumed. In this case, the tail pointer 320 can advance on queue entries (consumed) 314 and 312. However, the tail pointer 320 cannot move to queue entry (consumed) 308 because queue entry (currently being consumed) 310 is still being consumed. Once queue entry (currently being consumed) 310 switches from the "currently consuming" state to "consumed," the tail pointer 320 can advance on queue entries (currently being consumed) 310 and 308.

[0058] Queue 302 describes the consumer side of the queue, but it should be understood that the operation on the producer side is similar. A queue may include "empty slots" where new entries can be allocated, "entries being produced" (e.g., entries that have been allocated and are currently storing data), and "produced entries" (e.g., entries that have been written and are ready to be consumed). In some cases, a "produced entry" on the producer side of the queue is equivalent to a "ready" entry on the consumer side.

[0059] The diagram below illustrates different data structures that can be used to track the status of entries in a queue. Figure 4A to Figure 4D The first "maximum value + index" data structure is described. Figure 5A , Figure 5B and Figure 6 The second “bit array” data structure is described. Figure 7A and Figure 7B The third "segmented queue" data structure is described.

[0060] Figure 4A This is a block diagram of an example queue 402a according to at least one embodiment when an entry is consumed. Queue 402a has an associated data structure that tracks the “maximum index” (e.g., maximum value (max) 422a) of consumed nodes and the “current index” (e.g., index 424a) of consumed nodes.

[0061] and Figure 3 Similar to queue 302, queue 402a includes multiple entries: queue entry (ready) 404a, queue entry (consuming) 406a, queue entry (consuming) 408a, queue entry (consuming) 410a, queue entry (consumed) 412a, queue entry (consuming) 414a, and an empty queue slot 416a. The head pointer 418a of queue 402a points to queue entry (consuming) 406a at index 106, indicating that entries below index 106 have been provided to the consumer node.

[0062] likeFigure 4A As shown, queue entry (consumed) 412a has been consumed. When the processor of the consumer node that performed the consumption of queue entry (consumed) 412a completes the consumption of queue entry (consumed) 412a, it may have provided a signal to modify the data structure associated with queue 402a. For example, the maximum value 422a may represent the maximum index tracked by the data structure associated with queue 402a. Before queue entry (consumed) 412a completes consumption, the maximum value 422a may be equal to index 101 and may point to the same entry as the tail pointer 420a.

[0063] Once queue entry (consumed) 412a has been consumed, the maximum value 422a may have been updated based on the index of queue entry (consumed) 412a. For example, the current value of the maximum value 422a (e.g., 101) may have been compared with the index of queue entry (consumed) 412a (e.g., 103), and the maximum index value between the two (e.g., 103) may have been stored in the maximum value 422a.

[0064] Meanwhile, after queue entry (consumed) 412a has been consumed, index 424a may have been incremented by 1, from its previous value of index 101 to its current value of index 102, as shown below. Figure 4A As shown.

[0065] When the maximum value 422a equals index 424a, it means that all entries between the tail pointer 420a and the maximum value 422a have been consumed, and it is safe to advance the tail pointer 420a. Since the maximum value 422a equals index 103 and index 424a equals index 102, the tail pointer 420a cannot be advanced.

[0066] Figure 4B This is a block diagram of an example queue 402b when an entry is being consumed, according to at least one embodiment. Queue 402b may be the same as queue 402a, but located at some future time. Queue 402b may contain multiple entries: queue entry (ready) 404b, queue entry (consuming) 406b, queue entry (consuming) 408b, queue entry (consumed) 410b, queue entry (consumed) 412b, queue entry (consuming) 414b, and empty queue slot 416b.

[0067] Compared to queue 402a, queue entry (consumed) 410b has changed from "being consumed" to "consumed". When a processor that was previously consuming queue entry (consumed) 410b finishes accessing data in queue entry (consumed) 410b, it may cause the data structures associated with queue 402b to be updated. More specifically, index 424b may have been incremented by 1 from index 102 to index 103. Furthermore, the index of queue entry (consumed) 410b (e.g., 104) may have been compared with the maximum value 422b (e.g., 103). Since the index of queue entry (consumed) 410b is greater than the value of the maximum value 422b, the maximum value 422b may have been updated to be equal to index 104.

[0068] Since the maximum value 422b is still not equal to index 424b, the tail pointer 420b may still point to index 101. The head pointer 418b may not be modified and may continue to point to index 106.

[0069] Figure 4C This is a block diagram of an example queue 402c according to at least one embodiment when an entry is being consumed. Queue 402c may be the same as queue 402b, but located at some future time. Queue 402c may contain multiple entries: queue entry (ready) 404c, queue entry (consuming) 406c, queue entry (consuming) 408c, queue entry (consumed) 410c, queue entry (consumed) 412c, queue entry (consumed) 414c, and empty queue slot 416c.

[0070] Compared to queue 402b, queue entry (consumed) 414c has changed from "being consumed" to "consumed". When a processor that was previously consuming queue entry (consumed) 414c finishes accessing data in queue entry (consumed) 414c, it may cause the data structure associated with queue 402c to be updated. More specifically, index 424c may have been incremented by 1 from index 103 to index 104. Furthermore, the index of queue entry (consumed) 414c (e.g., 102) may have been compared with the maximum value 422c (e.g., 103). Since the index of queue entry (consumed) 414c is less than the value of the maximum value 422c, the maximum value 422c may not have been updated and may still be equal to 104.

[0071] The head pointer 418c can continue to point to index 106. However, since the maximum value 422c is now equal to index 424c, the tail pointer 420c can be advanced to index 104, as follows. Figure 4D As shown.

[0072] Figure 4DThis is a block diagram of an example queue 402d during the consumption of an entry, according to at least one embodiment. Queue 402d may be the same as queue 402c, but located at some future time. Queue 402d may contain multiple entries: queue entry (ready) 404d, queue entry (consuming) 406d, queue entry (consuming) 408d, empty queue slot 410d, empty queue slot 412d, empty queue slot 414d, and empty queue slot 416d. Head pointer 418d may continue to point to index 106.

[0073] Compared to queue 402c, empty queue slots 410d, 412d, and 414d have changed from "consumed" to "empty". Since the maximum value 422d is equal to index 424d, it is safe to advance the tail pointer 420d to index 104, and the entries at indices 102-104 can be released.

[0074] The process can continue as more entries are provided for consumption and are consumed, with the maximum index updated as entries are consumed and the current index incremented for each consumed entry. In some embodiments, very large (e.g., 64-bit) values ​​can be used to track the queue index, maximum index, and current index to avoid the need to loop back through the queue and perform modulo operations.

[0075] In some embodiments, instead of tracking the current index, a count of consumed records can be kept (e.g., a "maximum + count" data structure). For example, as entries are consumed, the maximum index can be determined as described above, and the count of consumed records can be incremented. The tail pointer can be safely pushed to the maximum index when the count of consumed records added to it equals the maximum index.

[0076] In some embodiments, signals from more than one consumer node can be combined, and the data structure can be updated once for multiple consumer nodes. For example, the first and second consumer nodes can (almost) complete consuming their respective queue entries simultaneously. One of the processors associated with a consumer node can provide a signal to update the data structure associated with the queue, incrementing the current index value by 2 instead of 1, and including the maximum index value of the two entries that have been consumed. The other processor may not need to provide a signal.

[0077] In some embodiments, multiple processors can access the same entry in a queue. In this case, the entry may contain the number of processors that need to access it. As processors complete their access to the entry, the number of processors that need to access it may decrease (e.g., decrement). Once the last processor has completed its access to the entry, it can provide a signal to update the data structure associated with the queue.

[0078] although Figure 4A , Figure 4B , Figure 4C and Figure 4D The consumer side of a queue has already been discussed, but it should be understood that a similar process can occur on the producer side of a queue. For example, the producer side of a queue can have its own head pointer, tail pointer, maximum index value, and current index value.

[0079] Figure 5A This is a block diagram of a bit array data structure 502 for tracking the status of queue entries, according to at least one embodiment. Bit array 502 may contain one or more bits corresponding to indices in the queue. For example, each index in the queue has a corresponding bit in bit array 502. In some embodiments, a single bit array data structure may be used for the entire queue (e.g., the same bit array data structure may be used on the producer side and the consumer side of the queue). In this case, each set of head and tail pointers (e.g., head and tail pointers on the producer side, head and tail pointers on the consumer side, etc.) may use opposite bit values ​​to indicate that a record is "ready" (e.g., ready to be consumed, ready to be released, etc.). For example, on the producer side, a bit value of 1 may indicate that a record is ready to be consumed, while on the consumer side, a bit value of 0 may indicate that a record is ready to be released. In some embodiments, a bit value of 0 may indicate that a record is ready to be consumed, and a bit value of 1 may indicate that a record is ready to be released. Bit values ​​may be "flipped" (e.g., by performing an XOR operation) to indicate that they are "ready" (e.g., ready to be consumed, ready to be released, etc.).

[0080] For example, bit array 502 can contain multiple bit values ​​corresponding to entries in the queue. Figure 5A It can depict a portion of bit array 502. Specifically, Figure 5A A head pointer and a tail pointer can be depicted on one side of the queue. For example, head pointer 504 could be the producer-side head pointer, and tail pointer 506 could be the producer-side tail pointer. A 0-bit value in bit array 502 can indicate a queue entry that has been allocated and is currently being filled (e.g., a "producing" entry). A 1-bit value in bit array 502 can indicate a queue entry that has been allocated, filled, and is available for consumption (e.g., a "produced" entry). As previously stated, entries can change state independently of adjacent entries, so some entries may be ready to be consumed while adjacent entries are still being produced.

[0081] When an entry changes from "in production" to "produced," the processor executing the producer node corresponding to that entry can issue a signal that causes the bit corresponding to that entry in bit array 502 to be flipped. After the bit flip, it can be evaluated whether there is a consecutive string of identical bit values ​​at the tail pointer 506 in bit array 502. For example, as...Figure 5A As shown, the tail pointer 506 points to the 0-bit value, and there are 3 1-bit values ​​to the left of the tail pointer 506. Once the entry corresponding to the 0-bit value that the tail pointer 506 is pointing to is flipped to a 1-bit value, the tail pointer 506 can advance on a continuous string of 1-bit values ​​and move 4 bits to the left.

[0082] Then, the tail pointer 506 can wait at the next 0-bit value until it is flipped. Once flipped, the tail pointer 506 can advance on the next consecutive sequence of 1-bit values. In some cases, the tail pointer 506 can only advance one bit at a time.

[0083] Unlike the "maximum value + index" data structure, the "bit array" data structure can provide granularity for updating its tail pointer at the level of a single entry.

[0084] On the consumer side, the tail pointer can wait at a 1-bit value, and once that value flips to a 0-bit value, the tail pointer can move forward on any consecutive string of 0-bit values.

[0085] As mentioned earlier, if multiple nodes complete the processing of entries (almost simultaneously), then the multiple nodes can coordinate the data structure update.

[0086] Figure 5B This is a block diagram of a bit array data structure 508 for tracking the state of queue entries according to at least one embodiment. In some embodiments, the bit array 508 may be divided into one or more segments, such as segments 510a, 510b, and 510c. Each segment may include a tail indicator (e.g., tail indicator bit 512) and one or more queue entry bits (e.g., queue entry bits 514). The tail indicator bit 512 may indicate whether a tail pointer exists in segment 510a of the bit array 508. For example, on the producer side, if the tail indicator bit 512 has a 1-bit value, the producer-side tail pointer may be located in queue entry bit 514 of segment 510a. On the consumer side, if the tail indicator bit 512 has a 0-bit value, the consumer-side tail pointer may be located in queue entry bit 514 of segment 510a. The specific position of the tail pointer can be determined based on the value in queue entry bit 514.

[0087] For example, starting from the right side of queue entry bit 514, the tail pointer could be exactly before the least significant bit indicating that a record is "ready". On the producer side, this could mean that when queue entry bit 514 is evaluated from right to left, the tail pointer is exactly at the 0 bit value before the first 1 bit value. On the consumer side, when queue entry bit 514 is evaluated from right to left, the tail pointer could be exactly at the 1 bit value before the first 0 bit value.

[0088] Because bit array 508 includes a tail indicator bit and queue entry bits, the number of bits in bit array 508 may exceed the number of entries in the corresponding queue. In some embodiments, bit array 508 may include one or more additional segments besides the segments required to represent entries in the corresponding queue, such as... Figure 6 As shown. One or more formulas can be used to calculate the bit array index corresponding to a specific queue index, and vice versa. In some embodiments, integer arithmetic can be used instead of floating-point arithmetic to optimize the calculation.

[0089] Because the tail indicator bits are distributed, bit array 508 can be updated efficiently and without locks. Since the tail indicator bits are stored adjacent to the queue entry bits, rather than in a separate area of ​​memory, no lock acquisition is required when modifying a segment of the bit array in memory. For example, bit array 508 can be loaded from memory. The queue entry bits and / or tail indicator bits of a specific segment can then be modified in a single instruction, improving the efficiency of memory operations and reducing the amount of time spent waiting to acquire and / or release locks. If it is necessary to modify the tail indicator bits of another segment (e.g., propagating the tail indicator bits from the first segment to the second segment), a second instruction can be executed.

[0090] Figure 6 This is a block diagram of a bit array data structure 602 for tracking the state of queue entries, according to at least one embodiment. The bit array 602 may include multiple segments, such as a first segment 604, a second segment 606, and a remaining segment 608. Each segment may have a tail bit indicator and multiple queue entry bits, as previously described. Figure 5B As shown. In some embodiments, bit array 602 may contain one or more additional segments in addition to the segments required to represent entries in the corresponding queue. For example, the queue corresponding to bit array 602 may have entries that can be adequately represented by the queue entry bits in the remaining segment 608. Bit array 602 may include a first segment 604 and a second segment 606 to ensure that the producer-side head and tail pointers 610 and the consumer-side head and tail pointers 612 do not simultaneously attempt to flip bits in the same segment. Figure 6 As shown, bit array 602 can be initialized such that the producer-side head pointer and tail pointer 610 lead the consumer-side head pointer and tail pointer 612 by one segment size. The initial state of the producer-side head pointer and tail pointer 610 can be all zeros in the queue entry bits and 1 in the tail pointer. The initial state of the consumer-side head pointer and tail pointer 612 can be all one in the queue entry bits and 0 in the tail pointer.

[0091] Figure 7AThis is a block diagram of a segmented bit queue data structure 736 for tracking the state of queue entries, according to at least one embodiment. As previously described, queue 702 may include multiple entries and may include a head pointer 716 and a tail pointer 718. In some embodiments, queue 702 may be associated with multiple data structures. For example, the producer side of queue 702 may be associated with a first data structure and a first set of head and tail pointers, while the consumer side of queue 702 may be associated with a second data structure and a second set of head and tail pointers.

[0092] exist Figure 7A In the discussion, head pointer 716 and tail pointer 718 can be discussed as the head pointer and tail pointer on the "consumer side", but it should be understood that the operation on the producer side of queue 702 is similar.

[0093] Queue 702 can be logically divided into multiple segments: segment 720, segment 722, segment 724, and segment 726. Each segment can be associated with a counter of the data structure associated with queue 702 (e.g., segmented queue 736). For example, segment 720 can be associated with counter 728, segment 722 with counter 730, segment 724 with counter 732, and segment 726 with counter 734.

[0094] As in other embodiments, when entries in queue 702 are provided to consumer nodes, the head pointer 716 can advance over those entries. Figure 7A As shown, entries currently being consumed are considered "in-use". For example, head pointer 716 may have advanced over all entries between tail pointer 718 and head pointer 716. Some of these entries are still being consumed, such as those in in-use regions 706 and 710. Some of these entries have already been consumed and are ready to be released, such as those in free regions 708 and 712. Entries after tail pointer 718 (e.g., entries in released region 714) may have been previously consumed and have already been released. Entries before head pointer 716 (e.g., entries in ready region 704) may be "ready entries" that have been produced and are ready to be consumed by one or more consumer nodes.

[0095] Once a specific entry is consumed, the processor executing the consumer node can provide a signal that causes the segment bit queue 736 to be modified. Specifically, this signal can cause the counter of the segment bit queue 736 associated with the segment of the consumed entry in queue 702 to be incremented. Once the specific counter equals a predetermined threshold, the tail pointer 718 can be advanced on the segment of queue 702 corresponding to that counter.

[0096] For example, assuming an entry is being consumed in the in-use region 710, after the consumer node finishes accessing the data in the in-use region 710, the processor executing the consumer node can provide a signal to modify the segment queue 736. Specifically, since the entry in the in-use region 710 is located within segment 724, a counter associated with segment 724 (e.g., counter 732) can be incremented. Incrementing counter 732 makes the value stored in counter 732 equal to a predetermined threshold. As a result, the tail pointer 718 can advance on segment 724 and can now point to the leftmost side of segment 724 within the free region 708. The queue entry in segment 724 can then be released.

[0097] Once the "consumed entries" belonging to segment 722 in the used area 706 have been consumed, the counter 730 can increment to equal its predetermined threshold, and the tail pointer 718 can advance on segment 722.

[0098] In some embodiments, the counter may be able to have a value greater than the number of queue entries in the corresponding queue segment. Therefore, a predetermined threshold for a particular counter may be equal to the number of queue entries in the segment corresponding to that counter to prevent the counter value from becoming too high. When the counter value equals the predetermined threshold, one or more "guard bits" within the counter may be toggled.

[0099] In some embodiments, in order to prevent the head pointer 716 from looping back into queue 702 and pointing to the same segment as the tail pointer 718, an initial gap may be introduced between the head pointer 716 and the tail pointer 718 to ensure that the head pointer 716 lags behind the tail pointer 718 by at least one segment length.

[0100] In some embodiments, updates to segmented bit queues 738 targeting multiple consumer nodes can be merged into a single update.

[0101] In some embodiments, updating the segmented bit queue 738 may require more than one atomic memory operation (e.g., to clear the counter of one segment and update the tail indicator bit in another segment).

[0102] In some embodiments, a single dedicated atomic memory operation can be used to perform an update to the segmented bit queue 738. The dedicated atomic memory operation may receive one or more values ​​as input to increment one or more segment counters (e.g., counter 728, counter 730, etc.) and a bit indicating whether the tail pointer bit needs to be propagated to the next segment. The dedicated atomic memory operation may return a value indicating whether the tail portion of the modified segment needs to be propagated to the next segment, and a value indicating how much the tail pointer has advanced due to the dedicated atomic memory operation.

[0103] For example, a segment of segmented bit queue 738 (e.g., segment 740a) can contain 64 bits of data, such as 1 bit for a tail indicator, 15 bits for a local tail position, and 4 bits for a 12-bit counter. A single 64-bit atomic memory operation can be used to update the value within the segment. If a dedicated atomic memory operation indicates that the tail portion needs to be propagated to the next segment, a second dedicated atomic memory operation can be performed on the next segment.

[0104] Similar to the bit array implementation discussed above, the segmented bit queue 738 can be updated efficiently and without locks due to the distributed tail indicator bit. Since the tail indicator bit is stored adjacent to the local tail set bit and counter bit, rather than in a separate area of ​​memory, no lock acquisition is required when modifying a specific segment of the segmented bit queue. For example, the segmented bit queue 738 can be loaded from memory. The counter bit, local tail set bit, and / or tail indicator bit of a specific segment can then be modified in a single instruction, improving the efficiency of memory operations and reducing the amount of time spent waiting to acquire and / or release locks. If it is necessary to modify the tail indicator bit of another segment (e.g., propagating the tail indicator bit from the first segment to the second segment), a second instruction can be executed.

[0105] In some embodiments, “wide” atomic memory operations can be used. For example, a wide atomic memory operation may be able to atomically modify 128 or 256 bits of memory. In this case, the segments of segmented bit queue 738 may be configured to match the size of the wide atomic. For example, if a wide atomic can modify 256 bits of memory in a single operation, then a segment of segmented bit queue 738 may contain 256 bits of data, such as one for a tail indicator, 16 for a local tail position, 47 for unused bits, and 16 for a 12-bit counter.

[0106] Figure 7BThis is a block diagram of a segmented bit queue data structure 738 for tracking the state of queue entries according to at least one embodiment. In some embodiments, the segmented bit queue 738 may be divided into one or more segments, such as segments 740a, 740b, and 740c. Each segment may include a tail bit indicator (e.g., tail indicator bit 742), one or more local tail position bits (e.g., local tail position 744), and one or more counters (e.g., counter 746, counter 748, counter 750, counter 752, etc.). The tail indicator bit 742 may indicate whether the tail pointer is present in one of the segments of the queue corresponding to the counter of segment 740a of the segmented bit queue 738. For example, on the producer side, if the tail indicator bit 742 has a 1-bit value, the producer-side tail pointer may be located in the segment corresponding to counter 746, counter 748, counter 750, or counter 752. To accurately identify the position of the tail pointer, the local tail position 744 may contain a value that represents the offset within the queue area corresponding to segment 740a, which identifies the position of the tail pointer.

[0107] Figure 8A This is a block diagram of a segmented bit queue data structure 830a for tracking the state of queue entries that can be "partially released," according to at least one embodiment. As previously described, queue 802a may contain multiple entries and may include a head pointer 810a and a tail pointer 812a. In some embodiments, queue 802a may be associated with multiple data structures. For example, the producer side of queue 802a may be associated with a first data structure and a first set of head and tail pointers, while the consumer side of queue 802a may be associated with a second data structure and a second set of head and tail pointers.

[0108] In discussion Figure 8A During the process, the head pointer 810a and the tail pointer 812a can be discussed as the head pointer and tail pointer on the "consumer side", but it should be understood that the operation on the producer side of queue 802a is similar.

[0109] Queue 802a can be logically divided into multiple segments: segment 814a, segment 816a, segment 818a, and segment 820a. Each segment can be associated with a counter of the data structure associated with queue 802a (e.g., segmented queue 830a). For example, segment 814a can be associated with counter 822a, segment 816a with counter 824a, segment 818a with counter 826a, and segment 820a with counter 828a.

[0110] As in other embodiments, when entries in queue 802a are provided to consumer nodes, the head pointer 810a can advance over those entries. For example, all entries between the tail pointer 812a and the head pointer 810a (e.g., entries in free area 806a) may have been provided to the consumer node and may have been consumed. When additional entries are provided to the consumer node, the head pointer 810a can move forward into ready area 804a. Entries that the tail pointer 812a has already moved forward over (e.g., entries in released area 808a) may have been released.

[0111] like Figure 8A As shown, entries in free region 806a may be ready to be released. However, since not all segments 814a are ready to be released, the tail pointer 812a may fail to advance on free entries and release them. This could lead to a deadlock.

[0112] To prevent deadlock, in some embodiments, if the head pointer 810a and the tail pointer 812a are located (e.g., pointing to) the same segment of the queue (e.g., segment 814a), the tail pointer 812a can partially advance through that segment. For example, if the partial advance criterion is met, the tail pointer 812a can be safely advanced to the same position as the head pointer 810a. In this case, the tail pointer 812a can track the entries that have been traversed (e.g., the index value of the corresponding entry). In some embodiments, the partial advance criterion is met when the number of entries between the head pointer 810a and the start position of the current segment (e.g., segment 814a) is equal to the value of the corresponding counter (e.g., counter 822a). In some embodiments, the partial advance criterion is met when the sum of the counters of segments within a segment located at or before the head pointer position is equal to the difference between the head pointer position and the start position of the segment. Therefore, the partial advance criterion can be based on segment granularity, segment granularity, etc.

[0113] Therefore, when the head pointer and tail pointer are in the same segment, entries within that segment can be released without waiting for the entire segment to be provided to the consumer node and marked as ready to be released.

[0114] Figure 8B This is a block diagram of a "partially released" segmented bit queue data structure 830b for tracking the state of queue entries, according to at least one embodiment. Queue 802b can be logically divided into multiple segments, each segment corresponding to a counter in segmented bit queue 830b. For example, segment 814b can correspond to counter 822b, segment 816b can correspond to counter 824b, segment 818b can correspond to counter 826b, and segment 820b can correspond to counter 828b.

[0115] Queue 802b can correspond to queue 802a after the partial release of queue 802a, such as in combination. Figure 8A The released area 808b of queue 802b can now be expanded to cover a portion of fragment 814b, and the head pointer 810b and tail pointer 812b can point to the same entry in queue 802b. As more entries are provided to consumer nodes, the head pointer 810b can advance into the ready area 804b.

[0116] In some cases, segment 814b can be partially freed again. For example, if the head pointer 810b advances over two entries, and those two entries have been consumed and are ready to be freed, then the counter 822b can be equal to the number of entries between the head pointer 810b and the starting position of segment 814b. The tail pointer 812b can advance over the same two entries and point to the same entries as the head pointer 810b. These two entries can then be freed.

[0117] Figure 9 This is a flowchart of an example method 900 for tracking the state of entries in a queue, according to at least one embodiment.

[0118] Method 900 can be executed using one or more processing units (e.g., CPU, GPU, accelerator, physical processing unit (PPU), data processing unit (DPU), etc.), which may include one or more storage devices (or communicate with one or more storage devices). In at least one embodiment, method 900 can be executed using one or more processing devices. In at least one embodiment, method 900 can use... Figure 1 The system 102's processing circuitry executes the method. In at least one embodiment, the processing unit executing method 900 can execute instructions stored on a non-transitory computer-readable storage medium. In at least one embodiment, method 900 can be executed using multiple processing threads (e.g., CPU threads and / or GPU threads), each thread executing one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, the processing threads implementing method 900 can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 900 can execute asynchronously relative to each other. Various operations of method 900 can be performed in accordance with... Figure 9 The operations of method 900 can be executed in different sequences. Some operations of method 900 can be executed concurrently with other operations. In at least one embodiment, Figure 9 One or more operations shown may not always be executed.

[0119] Figure 9This is a flowchart of an example method 900 for tracking the state of entries in a queue, according to at least one embodiment. At block 902, the processing unit executing method 900 may provide a first entry of the queue to a first process. At block 904, the processing unit may provide a second entry of the queue to a second process. At block 906, in response to receiving a first signal from the first process, the processing unit may modify the data structure associated with the queue for the first entry. At block 908, in response to receiving a second signal from the second process, the processing unit may modify the data structure associated with the queue for the second entry. At block 910, the processing unit may modify the tail pointer of the queue based on the data structure associated with the queue.

[0120] In some embodiments, additional queue entries may be provided to additional processes. When a signal is received from each process, the data structure associated with the queue may be modified for the queue entry provided to that process. Thus, queue entries may be provided to two or more processes, and the data structure associated with the queue may be modified for each queue entry in response to a signal received from the process that received the queue entry.

[0121] In some embodiments, the data structure associated with the queue includes a maximum index and a current index. Modifying the data structure associated with the queue for a first entry may include: modifying the maximum index based on the first index of the first entry in the queue, and incrementing the current index of the data structure. Modifying the data structure associated with the queue for a second entry may include: modifying the maximum index based on the second index of the second entry in the queue, and incrementing the current index of the data structure. Modifying the tail pointer of the queue based on the data structure associated with the queue may include: modifying the tail pointer of the queue to be equal to the maximum index in response to the maximum index being equal to the current index.

[0122] In some embodiments, the first process is a producer process that stores one or more data values ​​in the first entry of the queue. A first signal from the first process can indicate that the first entry of the queue is ready to be consumed by a consumer process. In some embodiments, the first process is a consumer process that performs one or more operations based on the data values ​​in the first entry of the queue. A first signal from the first process can indicate that the first entry of the queue is ready to be released.

[0123] In some embodiments, the data structure associated with the queue includes a bit array having one or more bits corresponding to indices in the queue. The bit array may also include one or more segments, each segment including a tail indicator bit and multiple queue entry bits. The one or more bits corresponding to indices in the queue may be included in the multiple queue entry bits. In some embodiments, a first subset of the one or more bits corresponding to indices in the queue may be included in the multiple queue entry bits of a first segment, and a second subset of the one or more bits corresponding to indices in the queue may be included in the multiple queue entry bits of a second segment.

[0124] Modifying the data structure associated with the queue for the first entry can include modifying the first bit of the bit array corresponding to the first queue index of the first entry. Modifying the data structure associated with the queue for the second entry can include modifying the second bit of the bit array corresponding to the second queue index of the second entry. Modifying the tail pointer of the queue based on the data structure associated with the queue can include modifying the tail pointer of the queue based on a consecutive sequence of values ​​of the bit array. For example, if the bit array entry corresponding to the queue entry pointed to by the tail pointer is flipped (e.g., from 0 to 1 or from 1 to 0), the tail pointer can advance on adjacent entries (e.g., a consecutive sequence of entries) that have the same value as the corresponding bit array entry after the flip. For example, if the bit array entry is flipped from 0 to 1, and there are 3 1-bit values ​​to the left (e.g., towards the head pointer), the tail pointer may advance on 3 entries in the queue and point to the next queue entry, which may have the corresponding bit array value 0.

[0125] In some embodiments, the queue comprises multiple segments. The data structure associated with the queue may include multiple sections, each section including one or more counters, each counter associated with one of the multiple segments. In some embodiments, a first section of the multiple sections includes a tail indicator bit, one or more tail address bits (e.g., a local tail position bit), a first counter, and a second counter. Modifying the data structure associated with the queue for a first entry may include incrementing the first counter. The first entry may be located in a first segment of the multiple segments of the queue, and the first counter may be associated with the first segment of the multiple segments. Modifying the data structure associated with the queue for a second entry may include incrementing the second counter. The second entry may be located in a second segment of the multiple segments of the queue, and the second counter may be associated with the second segment of the multiple segments. Modifying the tail pointer of the queue based on the data structure associated with the queue may include modifying the queue tail pointer to be equal to the end of the specific segment associated with that specific counter in response to a specific counter meeting a threshold criterion. For example, the queue tail pointer may point to the last entry of the specific segment associated with that specific counter (e.g., the last entry of that specific segment), or it may point to the first entry in a segment adjacent to that specific segment.

[0126] In some embodiments, the first counter and the second counter each have one or more protection bits for indicating whether a threshold criterion associated with the counter has been met. For example, if a particular counter has reached a threshold criterion, all the protection bits of that counter can be toggled to 1 (or 0).

[0127] Figure 10 This is a block diagram illustrating an exemplary computer system according to at least one embodiment of the present disclosure. The computer system 1000 may correspond to... Figure 1 System 102 is described herein. Computer system 1000 can operate as a server or endpoint machine in an endpoint server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. This machine can be a television, personal computer (PC), tablet computer, set-top box (STB), personal digital assistant (PDA), mobile phone, network device, server, network router, switch, or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) specifying the actions the machine should take. Furthermore, although only one machine is shown in the figure, the term "machine" should also be understood to include any set of machines that individually or jointly execute a set (or more) of instructions to perform one or more methods discussed herein.

[0128] Example computer system 1000 includes a processing device (processor) 1002, main memory 1004 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM), double data rate (DDRSDRAM) or DRAM (RDRAM), etc.), static memory 1006 (e.g., flash memory, static random access memory (SRAM), etc.) and data storage device 1016, which communicate with each other via bus 1028.

[0129] Processor (processing device) 1002 represents one or more general-purpose processing devices, such as microprocessors, central processing units, etc., and may include processing logic 1022. More specifically, processor 1002 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor that implements other instruction sets or a combination of instruction sets. Processor 1002 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. Processor 1002 is configured to execute instructions 1026 (e.g., for generating threat indicator alerts) to perform the operations described herein.

[0130] The computer system 1000 may also include a network interface device 1008. The computer system 1000 may also include a video display unit 1010 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an input device 1012 (e.g., a keyboard, alphanumeric keypad, motion-sensing input device, or touchscreen), a cursor control device 1014 (e.g., a mouse), and a signal generation device 1018 (e.g., a speaker). In some embodiments, the computer system 1000 may not include the video display unit 1010, the input device 1012, and / or the cursor control device 1014 (e.g., a headless configuration).

[0131] Data storage device 1016 may include a non-transitory machine-readable storage medium 1024 (also referred to as a computer-readable storage medium) storing one or more sets of instructions 1026 (e.g., for tracking the status of entries in a queue), which embody one or more methods or functions described herein. The instructions 1026 may also reside wholly or at least partially in main memory 1004 and / or processor 1002, and be executed by computer system 1000, which also constitute machine-readable storage media. These instructions may also be sent or received on network 1020 via network interface device 1008.

[0132] In one embodiment, instruction 1026 includes instructions for tracking the status of entries in the queue. While computer-readable storage medium 1024 (machine-readable storage medium) is shown as a single medium in the exemplary embodiment, the terms "computer-readable storage medium" and "machine-readable storage medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more sets of instructions. The terms "computer-readable storage medium" and "machine-readable storage medium" should also be understood to include any medium capable of storing, encoding, or carrying a set of machine-executable instructions and enabling a machine to perform one or more methods as described in this disclosure. Therefore, the terms "computer-readable storage medium" and "machine-readable storage medium" should be understood to include, but are not limited to, solid-state memory, optical media, and magnetic media.

[0133] Reasoning and training logic

[0134] Figure 11A Inference and / or training logic 1115 is shown for performing inference and / or training operations associated with one or more embodiments.

[0135] In at least one embodiment, the inference and / or training logic 1115 may include, but is not limited to, code and / or data storage 1101 for storing forward and / or output weights and / or input / output data, and / or other parameters of neurons or layers of a neural network trained for and / or used for inference in one or more embodiments. In at least one embodiment, the training logic 1115 may include graph code or other software for controlling timing and / or sequence (or coupled to code and / or data storage 1101 for storing graph code or other software for controlling timing and / or sequence), wherein weight and / or other parameter information is loaded to configure the processing unit, eight logic units, integer and / or floating-point units (collectively referred to as an arithmetic logic unit (ALU) or simple circuit). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, code and / or data storage 1101 is stored in the weight parameters and / or input / output data of each layer of a neural network trained or used in conjunction with one or more embodiments during the forward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data storage 1101 may be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0136] In at least one embodiment, any portion of the code and / or data storage 1101 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 1101 may be a cache memory, dynamic random-addressable memory (“DRAM”), static random-addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 1101 is internal or external to the processor, for example, or composed of DRAM, SRAM, flash memory, or some other storage type, may depend on the available on-chip or off-chip storage space, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0137] In at least one embodiment, the inference and / or training logic 1115 may include, but is not limited to, code and / or data storage 1105 for storing backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, during training and / or inference using one or more embodiments, the code and / or data storage 1105 stores weight parameters and / or input / output data for each layer of a neural network trained or used in one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 1115 may include (or coupled to) code and / or data storage 705 for storing graph code or other software for controlling timing and / or order, wherein weight and / or other parameter information is loaded to configure a processing unit, including logic units comprising integer and / or floating-point units (collectively referred to as an arithmetic logic unit (ALU)).

[0138] In at least one embodiment, code (such as graph code) causes the architecture of the neural network corresponding to that code to load weights or other parameter information into the processor ALU. In at least one embodiment, any portion of the code and / or data storage 1105 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 1105 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 1105 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice between the code and / or data storage 1105 being internal or external to the processor, for example, whether it consists of DRAM, SRAM, flash memory, or some other type of storage, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the data batch size used in the inference and / or training of the neural network, or some combination of these factors.

[0139] In at least one embodiment, code and / or code and / or data storage 1101 and code and / or data storage 1105 may be separate storage structures. In at least one embodiment, code and / or data storage 1101 and code and / or data storage 1105 may be combined storage structures. In at least one embodiment, code and / or data storage 1101 and code and / or data storage 1105 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data storage 1101 and code and / or data storage 1105 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0140] In at least one embodiment, the inference and / or training logic 1115 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 1110 (including integer and / or floating-point units) for performing logical and / or mathematical operations at least in part based on or instructed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from layers or neurons within a neural network) stored in activation storage 1120, which are functions of input / output and / or weight parameter data stored in code and / or data storage 1101 and / or code and / or data storage 1105. In at least one embodiment, activation is activated in response to execution instructions or other code, and linear algebraic and / or matrix-based mathematical generation performed by ALU 1110 is stored in activation storage 1120, wherein weight values ​​stored in code and / or data storage 1105 and / or code and / or data storage 1101 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, and any or all of these can be stored in code and / or data storage 1105 or code and / or code and / or data storage 1101 or another on-chip or off-chip memory.

[0141] In at least one embodiment, one or more processors or other hardware logic devices or circuits include one or more ALUs 1110, while in another embodiment, one or more ALUs 1110 may be located outside the processor or other hardware logic device or the circuitry using them (e.g., a coprocessor). In at least one embodiment, one or more ALUs 1110 may be included within an execution unit of a processor, or otherwise included in a group of ALUs accessible by the execution unit of the processor, which may be within the same processor or distributed among different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, code and / or data storage 1101, code and / or data storage 1105, and activation storage 1120 may share a processor or other hardware logic device or circuitry, while in another embodiment, they may be located in different processors or other hardware logic devices or circuitry, or in some combination of the same and different processors or other hardware logic devices or circuitry. In at least one embodiment, any portion of activation storage 1120 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be retrieved and / or processed using the processor’s fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0142] In at least one embodiment, the active memory 1120 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 1120 may be wholly or partially located inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory 1120 is internal to or external to the processor may depend on the available on-chip or off-chip storage, the latency requirements for training and / or inference functions, the batch size of data used in inference and / or training the neural network, or some combination of these factors. For example, it may include DRAM, SRAM, flash memory, or some other memory type.

[0143] In at least one embodiment, Figure 11A The inference and / or training logic 1115 shown can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 11A The inference and / or training logic 1115 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware (such as field programmable gate array (“FPGA”)).

[0144] Figure 11B An inference and / or training logic 1115 according to at least one embodiment is illustrated. In at least one embodiment, the inference and / or training logic 1115 may include, but is not limited to, hardware logic, wherein computational resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 11B The inference and / or training logic 1115 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 11BThe inference and / or training logic 1115 shown can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., field-programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 1115 includes, but is not limited to, code and / or data storage 1101 and code and / or data storage 1105, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 11B In at least one embodiment shown, each of code and / or data storage 1101 and code and / or data storage 1105 is associated with dedicated computing resources (e.g., computing hardware 1102 and computing hardware 1106), respectively. In at least one embodiment, each of computing hardware 1102 and computing hardware 1106 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) only on the information stored in code and / or data storage 1101 and code and / or data storage 1105, respectively, and the results of the function execution are stored in activation memory 1120.

[0145] In at least one embodiment, each of the code and / or data storage 1101 and 1105 and the corresponding computing hardware 1102 and 1106 corresponds to a different layer of the neural network, such that an activation obtained from a storage / computation pair 1101 / 1102 of the code and / or data storage 1101 and computing hardware 1102 provides input as input to the next storage / computation pair 1105 / 1106 of the code and / or data storage 1105 and computing hardware 1106, in order to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / computation pair 1101 / 1102 and 1105 / 1106 may correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic 1115 following or paralleling the storage / computation pairs 1101 / 1102 and 1105 / 1106.

[0146] Neural network training and deployment

[0147] Figure 12Training and deployment of a deep neural network according to at least one embodiment are illustrated. In at least one embodiment, an untrained neural network 1206 is trained using a training dataset 1202. In at least one embodiment, the training framework 1204 is the PyTorch framework, while in other embodiments, the training framework 1204 is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training frameworks. In at least one embodiment, the training framework 1204 trains the untrained neural network 1206 and enables it to be trained using the processing resources described herein to generate a trained neural network 1208. In at least one embodiment, the weights may be randomly selected or pre-trained using a deep belief network. In at least one embodiment, training may be performed in a supervised, partially supervised, or unsupervised manner.

[0148] In at least one embodiment, supervised learning is used to train an untrained neural network 1206, wherein the training dataset 1202 includes inputs paired with desired outputs for input, or wherein the training dataset 1202 includes inputs with known outputs and the neural network 1206 is manually graded output. In at least one embodiment, the untrained neural network 1206 is trained in a supervised manner, and inputs from the training dataset 1202 are processed, and the resulting outputs are compared with a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through the untrained neural network 1206. In at least one embodiment, a training framework 1204 adjusts the weights controlling the untrained neural network 1206. In at least one embodiment, the training framework 1204 includes tools for monitoring the degree to which the untrained neural network 1206 converges to a model (e.g., a trained neural network 1208) adapted to generate the correct answer (e.g., result 1214) based on input data (e.g., a new dataset 1212). In at least one embodiment, the training framework 1204 repeatedly trains the untrained neural network 1206 while adjusting the weights to improve the output of the untrained neural network 1206 using a loss function and tuning algorithm (e.g., stochastic gradient descent). In at least one embodiment, the training framework 1204 trains the untrained neural network 1206 until the untrained neural network 1206 reaches the desired accuracy. In at least one embodiment, the trained neural network 1208 can then be deployed to implement any number of machine learning operations.

[0149] In at least one embodiment, unsupervised learning is used to train an untrained neural network 1206, wherein the untrained neural network 1206 attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised learning training dataset 1202 will include input data without any associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 1206 can learn groupings within the training dataset 1202 and can determine how each input relates to the untrained dataset 1202. In at least one embodiment, unsupervised training can be used to generate a self-organizing graph in the trained neural network 1208, which is capable of performing operations useful for reducing the dimensionality of the new dataset 1212. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows the identification of data points in the new dataset 1212 that deviate from the normal patterns of the new dataset 1212.

[0150] In at least one embodiment, semi-supervised learning can be used, a technique in which the training dataset 1202 includes a mixture of labeled and unlabeled data. In at least one embodiment, the training framework 1204 can be used to perform incremental learning, for example, through transfer learning techniques. In at least one embodiment, incremental learning enables the trained neural network 1208 to adapt to the new dataset 1212 without forgetting the knowledge injected into the trained neural network 1208 during initial training.

[0151] Reference Figure 13 , Figure 13 This is an example data flow diagram of process 1300 for generating and deploying a processing and inference pipeline according to at least one embodiment. In at least one embodiment, process 1300 may be deployed to perform game name recognition analysis and inference on user feedback data at one or more facilities 1302, such as a data center.

[0152] In at least one embodiment, process 1300 may be executed within training system 1304 and / or deployment system 1306. In at least one embodiment, training system 1304 may be used to train, deploy, and implement machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use with deployment system 1306. In at least one embodiment, deployment system 1306 may be configured to offload processing and computing resources in a distributed computing environment to reduce infrastructure requirements of facility 1302. In at least one embodiment, deployment system 1306 may provide a pipeline platform for selecting, customizing, and implementing virtual instruments for use with computing devices at facility 1302. In at least one embodiment, virtual instruments may include software-defined applications for performing one or more processing operations on feedback data. In at least one embodiment, one or more applications in the pipeline may use or invoke services of deployment system 1306 (e.g., inference, visualization, computation, AI, etc.) during application execution.

[0153] In at least one embodiment, some applications used in the advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at facility 1302 using feedback data 1308 stored at facility 1302 (e.g., imaging data) or feedback data 1308 from another or more facilities, or a combination thereof. In at least one embodiment, training system 1304 may be used to provide applications, services, and / or other resources to generate a deployable machine learning model for the operation of deployment system 1306.

[0154] In at least one embodiment, the model registry 1324 may be supported by an object storage system that supports version control and object metadata. In at least one embodiment, it may be available from within a cloud platform via, for example, cloud storage (e.g., Figure 14 The system uses a cloud-compatible application programming interface (API) to access object storage. In at least one embodiment, machine learning models within the model registry 1324 can be uploaded, listed, modified, or deleted by the developer or partner of the system interacting with the API. In at least one embodiment, the API can provide access to methods that allow users with appropriate credentials to associate models with applications, enabling the models to be executed as part of the containerized instantiation of the application.

[0155] In at least one embodiment, training pipeline 1404 ( Figure 14This can include situations where facility 1302 is training its own machine learning model or has an existing machine learning model that needs optimization or updating. In at least one embodiment, feedback data 1308 can be received from various channels (such as forums, web forms, etc.). In at least one embodiment, once feedback data 1308 is received, AI-assisted annotation 1310 can be used to help generate annotations corresponding to the feedback data 1308 for use as ground truth data for the machine learning model. In at least one embodiment, AI-assisted annotation 1310 can include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that can be trained to generate annotations corresponding to certain types of feedback data 1308 (e.g., from certain devices), and / or certain types of anomalies in the feedback data 1308. In at least one embodiment, AI-assisted annotation 1310 can then be used directly, or it can be adjusted or fine-tuned using annotation tools to generate ground truth data. In at least one embodiment, in some examples, labeled data 1312 can be used as ground truth data for training the machine learning model. In at least one embodiment, AI-assisted annotations 1310, labeled data 1312, or a combination thereof can be used to train a machine learning model (e.g., via...). Figure 13 and / or Figure 14 The model is trained on ground-based real-world data (1314). In at least one embodiment, the trained machine learning model may be referred to as output model 1316 and may be used by deployment system 1306 as described herein.

[0156] In at least one embodiment, training pipeline 1404 ( Figure 14This may include situations where facility 1302 requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 1306, but facility 1302 may not currently have such a machine learning model (or may not have an efficient or effective model optimized for this purpose). In at least one embodiment, an existing machine learning model may be selected from model registry 1324. In at least one embodiment, model registry 1324 may include machine learning models trained to perform various inference tasks on imaging data. In at least one embodiment, the machine learning model in model registry 1324 may be trained on imaging data from a different facility (e.g., a remote facility) instead of facility 1302. In at least one embodiment, the machine learning model may have already been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a specific location (which may be in the form of feedback data 1308), training may be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of the imaging data off-site (e.g., in compliance with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model has been trained or partially trained at one location, a machine learning model can be added to model registry 1324. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be used in model registry 1324. In at least one embodiment, a machine learning model (referred to as output model 1316) can then be selected from model registry 1324 and used in deployment system 1306 to perform one or more processing tasks for one or more applications of the deployment system.

[0157] In at least one embodiment, training pipeline 1404 ( Figure 14This can be used in scenarios including facility 1302, which requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 1306, but facility 1302 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for this purpose). In at least one embodiment, the machine learning model selected from model registry 1324 may not be fine-tuned or optimized for the feedback data 1308 generated at facility 1302 due to population differences, genetic variation, robustness, anomalous diversity of the training data, and / or other problems with the training data used to train the machine learning model. In at least one embodiment, AI-assisted annotation 1310 can be used to help generate annotations corresponding to the feedback data 1308 for use as ground-based data for retraining or updating the machine learning model. In at least one embodiment, labeled data 1312 can be used as ground-based data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 1314. In at least one embodiment, model training 1314 may include data (e.g., AI-assisted annotations 1310, labeled data 1312, or a combination thereof) that can be used as ground-based data to retrain or update the machine learning model.

[0158] In at least one embodiment, deployment system 1306 may include software 1318, service 1320, hardware 1322, and / or other components, features, and functions. In at least one embodiment, deployment system 1306 may include a software "stack" such that software 1318 can be built on top of service 1320 and can be used to perform some or all of the processing tasks, and service 1320 and software 1318 can be built on top of hardware 1322 and can be used to perform processing, storage, and / or other computing tasks of deployment system 1306.

[0159] In at least one embodiment, software 1318 may include any number of different containers, each of which may perform an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in an advanced processing and inference pipeline. In at least one embodiment, for each type of computing device, there may be any number of containers that may perform data processing tasks on feedback data 1308 (or other data types, such as those described herein). In at least one embodiment, in addition to the containers that receive and configure imaging data for use by each container and / or for use by facility 1302 after processing through the pipeline, the advanced processing and inference pipeline may also be defined based on the selection of different containers desired or required for processing feedback data 1308 (e.g., to convert output back to a usable data type for storage and display in facility 1302). In at least one embodiment, a combination of containers within software 1318 (e.g., which constitute a pipeline) may be referred to as a virtual instrument (as described in more detail herein), and the virtual instrument may utilize service 1320 and hardware 1322 to perform some or all of the processing tasks of the application instantiated in the container.

[0160] In at least one embodiment, data may be preprocessed as part of a data processing pipeline to prepare it for processing by one or more applications. In at least one embodiment, post-processing may be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare output data for the next application and / or to prepare output data for user transmission and / or use (e.g., as a response to an inference request). In at least one embodiment, the inference task may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include the output model 1316 of the training system 1304.

[0161] In at least one embodiment, the tasks of the data processing pipeline can be encapsulated in one or more containers, each container representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing a machine learning model. In at least one embodiment, containers or applications can be published to a private (e.g., limited access) area of ​​a container registry (described in more detail herein), and trained or deployed models can be stored in a model registry 1324 and associated with one or more applications. In at least one embodiment, an image of the application (e.g., a container image) can be used in the container registry, and once a user selects an image from the container registry for deployment in the pipeline, that image can be used to generate containers for instantiation of the application for use by the user's system.

[0162] In at least one embodiment, the developer can develop, publish, and store an application (e.g., as a container) for performing processing and / or inference on the provided data. In at least one embodiment, a software development kit (SDK) associated with the system can be used to perform development, publication, and / or storage (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application can be tested locally using the SDK (e.g., at a first facility, on data from a first facility), the SDK serving as a system (e.g.,...). Figure 14 System 1400 may support at least some services 1320. In at least one embodiment, once verified by system 1400 (e.g., for accuracy, etc.), the application becomes available in the container registry for users (e.g., hospitals, clinics, laboratories, healthcare providers, etc.) to select and / or implement one or more processing tasks on data at the user's facility (e.g., a second facility).

[0163] In at least one embodiment, the developer can then share the application or container over a network for the system (e.g., Figure 14 The system 1400 allows for user access and use. In at least one embodiment, a completed and validated application or container may be stored in a container registry, and associated machine learning models may be stored in a model registry 1324. In at least one embodiment, a requesting entity (which provides an inference or image processing request) may browse the container registry and / or model registry 1324 for applications, containers, datasets, machine learning models, etc., select the desired combination of elements to include in the data processing pipeline, and submit a processing request. In at least one embodiment, the request may include input data necessary to execute the request, and / or may include a selection of the application and / or machine learning model to be executed when the request is processed. In at least one embodiment, the request may then be passed to one or more components of the deployment system 1306 (e.g., the cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 1306 may include referencing elements (e.g., applications, containers, models, etc.) selected from the container registry and / or model registry 1324. In at least one embodiment, once results are generated through the pipeline, the results may be returned to the user for reference (e.g., for viewing in a suite of viewing applications executed on a local machine, local workstation, or terminal).

[0164] In at least one embodiment, service 1320 may be utilized to assist in processing or executing applications or containers in the pipeline. In at least one embodiment, service 1320 may include computing services, collaborative content creation services, simulation services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, service 1320 may provide functionality common to one or more applications in software 1318, thus abstracting functionality into services that can be invoked or utilized by applications. In at least one embodiment, the functionality provided by service 1320 can operate dynamically and more efficiently, while also allowing applications to process data in parallel (e.g., using...). Figure 14 The parallel computing platform 1430 in the system can be scaled well. In at least one embodiment, it is not required that every application providing the same functionality as the shared service 1320 must have a corresponding instance of the service 1320, but rather the service 1320 can be shared between and among various applications. In at least one embodiment, as a non-limiting example, the service may include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included, which can provide the ability to train and / or retrain machine learning models.

[0165] In at least one embodiment, where service 1320 includes an AI service (e.g., an inference service), as part of application execution, one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) can be executed by invoking (e.g., as an API call) the inference service (e.g., an inference server) to execute one or more machine learning models or their processing. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, the application can invoke the inference service to execute the machine learning models for performing one or more processing operations associated with the segmentation task. In at least one embodiment, the software 1318 implementing the high-level processing and inference pipeline can be pipelined because each application can invoke the same inference service to execute one or more inference tasks.

[0166] In at least one embodiment, hardware 1322 may include a GPU, CPU, data processing unit (DPU), AI / deep learning system (e.g., an AI supercomputer, such as NVIDIA's DGX). TM(Supercomputer system), cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 may be used to provide efficient, specially built support for software 1318 and services 1320 in deployment system 1306. In at least one embodiment, GPU processing may be used to perform local processing (e.g., at facility 1302) within the AI / deep learning system, in the cloud system, and / or in other processing components of deployment system 1306 to improve the efficiency, accuracy, and performance of game name recognition.

[0167] In at least one embodiment, as a non-limiting example, regarding deep learning, machine learning and / or high-performance computing, simulation and visual computing, software 1318 and / or service 1320 may be optimized for GPU processing. In at least one embodiment, at least some of the computing environment in which system 1306 is deployed and / or training system 1304 can be in software with GPU optimization (e.g., NVIDIA DGX). TM The system's hardware and software combination is executed in a data center or one or more supercomputers or high-performance computing systems. In at least one embodiment, as described herein, hardware 1322 may include any number of GPUs that can be invoked to perform data processing in parallel. In at least one embodiment, the cloud platform may also include GPU-optimized execution for deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, AI / deep learning supercomputers and / or GPU-optimized software (e.g., such as NVIDIA's DGX) may be used. TM The system provides a hardware abstraction and scaling platform to execute cloud platforms (e.g., NVIDIA's NGC). TM In at least one embodiment, the cloud platform can integrate application container cluster systems or coordination systems (e.g., KUBERNETES) across multiple GPUs to achieve seamless scaling and load balancing.

[0168] Figure 14 This is a system diagram of an example system 1400 for generating and deploying a deployment pipeline according to at least one embodiment. In at least one embodiment, system 1400 can be used to implement Figure 13 The process 1300 and / or other processes include advanced processing and inference pipelines. In at least one embodiment, system 1400 may include training system 1304 and deployment system 1306. In at least one embodiment, training system 1304 and deployment system 1306 may be implemented using software 1318, service 1320 and / or hardware 1322, as described herein.

[0169] In at least one embodiment, system 1400 (e.g., training system 1304 and / or deployment system 1306) may be implemented in a cloud computing environment (e.g., using cloud 1426). In at least one embodiment, system 1400 may be implemented locally (regarding facilities) or as a combination of cloud computing resources and local computing resources. In at least one embodiment, access to the API in cloud 1426 may be restricted to authorized users by establishing security measures or protocols. In at least one embodiment, the security protocol may include a network token, which may be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and may carry appropriate authorization. In at least one embodiment, the API of the virtual instrument (described herein) or other instances of system 1400 may be restricted to a set of public Internet Service Providers (ISPs) that have been audited or authorized for interaction.

[0170] In at least one embodiment, the various components of system 1400 may communicate with each other and among themselves using any of a variety of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communication between facilities and components of system 1400 (e.g., for sending inference requests, for receiving the results of inference requests, etc.) may be transmitted via one or more data buses, wireless data protocols (e.g., Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0171] In at least one embodiment, similar to the description herein. Figure 13 As described, training system 1304 can execute training pipeline 1404. In at least one embodiment, where deployment system 1306 uses one or more machine learning models in deployment pipeline 1410, training pipeline 1404 can be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1406 (e.g., without retraining or updating). In at least one embodiment, as a result of training pipeline 1404, output model 1316 can be generated. In at least one embodiment, training pipeline 1404 can include any number of processing steps, AI-assisted annotation 1310, labeling or annotating feedback data 1308 to generate labeled data 1312, selecting a model from a model registry, model training 1314, training, retraining, or updating the model, and / or other processing steps. In at least one embodiment, DICOM adapter 1402a can be used to access DICOM data. In at least one embodiment, different training pipelines 1404 can be used for different machine learning models used by deployment system 1306. In at least one embodiment, similar to the...Figure 13 The training pipeline 1404 described in the first example can be used for the first machine learning model, similar to the one described above. Figure 13 The training pipeline 1404 described in the second example can be used for a second machine learning model, similar to the one described above. Figure 13 The training pipeline 1404 of the third example described can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1304 can be used according to the requirements of each respective machine learning model. In at least one embodiment, one or more machine learning models may have already been trained and are ready for deployment, so the training system 1304 may not perform any processing on the machine learning models, and the machine learning models may be implemented by the deployment system 1306.

[0172] In at least one embodiment, one or more output models 1316 and / or pre-trained models 1406 may comprise any type of machine learning model. In at least one embodiment, and not limited thereto, the machine learning model used by system 1400 may include models using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recursion, perceptrons, long short-term memory (LSTM), Bi-LSTM, Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0173] In at least one embodiment, training pipeline 1404 may include AI-assisted annotation. In at least one embodiment, labeled data 1312 (e.g., conventional annotation) can be generated using any number of techniques. In at least one embodiment, in some examples, labels or other annotations can be generated in drawing programs (e.g., annotation programs), computer-aided design (CAD) programs, tagging programs, another type of application suitable for generating annotations or labels for ground reality, and / or can be hand-drawn. In at least one embodiment, ground reality data can be synthetically generated (e.g., generated from computer models or renderings), realistically generated (e.g., designed and generated from real-world data), automatically generated by machines (e.g., extracting features from data using feature analysis and learning, and then generating labels), manually annotated (e.g., taggers or annotation experts, defining the placement of labels), and / or combinations thereof. In at least one embodiment, for each instance of feedback data 1308 (or other data types used by the machine learning model), there may be corresponding ground reality data generated by training system 1304. In at least one embodiment, AI-assisted annotation can be performed as part of deployment pipeline 1410; supplementing or replacing AI-assisted annotation included in training pipeline 1404. In at least one embodiment, system 1400 may include a multi-layer platform, which may include a software layer (e.g., software 1318) of a diagnostic application (or other application type) that can perform one or more medical imaging and diagnostic functions.

[0174] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or certified API that can invoke (e.g., call) an application or container from an external environment (e.g., facility 1302). In at least one embodiment, the application may then invoke or execute one or more services 1320 to perform computational, AI, or visualization tasks associated with their respective applications, and the software 1318 and / or service 1320 may utilize the hardware 1322 to perform processing tasks efficiently and effectively.

[0175] In at least one embodiment, deployment system 1306 may execute deployment pipeline 1410. In at least one embodiment, deployment pipeline 1410 may include any number of applications, which may be sequential, non-sequential, or otherwise applied to feedback data (and / or other data types) – including AI-assisted annotations, as described above. In at least one embodiment, as described herein, deployment pipeline 1410 for an individual device may be referred to as a virtual instrument for the device. In at least one embodiment, for a single device, there may be more than one deployment pipeline 1410, depending on the desired information from the data generated from the device.

[0176] In at least one embodiment, the applications that can be used to deploy pipeline 1410 may include any application that can perform processing tasks on feedback data or other data from the device. In at least one embodiment, since various applications can share common image operations, in some embodiments, data augmentation libraries (e.g., as one of services 1320) may be used to accelerate these operations. In at least one embodiment, to avoid bottlenecks in traditional processing methods that rely on CPU processing, parallel computing platform 1430 may be used for GPU acceleration of these processing tasks.

[0177] In at least one embodiment, deployment system 1306 may include a user interface 1414 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications to be included in deployment pipeline 1410, deploy applications, modify or change applications or their parameters or configurations, use and interact with deployment pipeline 1410 during setup and / or deployment, and / or otherwise interact with deployment system 1306. In at least one embodiment, although not shown with respect to training system 1304, UI 1414 (or a different user interface) can be used to select models to be used in deployment system 1306, to select models to be trained or retrained in training system 1304, and / or to otherwise interact with training system 1304.

[0178] In at least one embodiment, in addition to the application coordination system 1428, a pipeline manager 1412 may be used to manage interactions between applications or containers deployed through the pipeline 1410 and services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 may be configured to facilitate interactions from application to application, from application to service 1320, and / or from application or service to hardware 1322. In at least one embodiment, although shown as included in software 1318, this is not intended to be limiting, and in some examples, the pipeline manager 1412 may be included in service 1320. In at least one embodiment, the application coordination system 1428 (e.g., Kubernetes, DOCKER, etc.) may include a container coordination system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline 1410 (e.g., rebuilding applications, splitting applications, etc.) with individual containers, each application can execute in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0179] In at least one embodiment, each application and / or container (or its image) can be developed, modified, and deployed independently (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer). This allows focus on the tasks of a single application and / or container without being hindered by the tasks of other applications or containers. In at least one embodiment, the pipeline manager 1412 and the application coordination system 1428 can facilitate communication and collaboration between different containers or applications. In at least one embodiment, the application coordination system 1428 and / or the pipeline manager 1412 can facilitate communication and resource sharing between and within each application or container, provided that the expected inputs and / or outputs of each container or application are known to the system (e.g., based on the construction of the application or container). In at least one embodiment, since one or more applications or containers in the deployment pipeline 1410 can share the same services and resources, the application coordination system 1428 can coordinate, load balance, and determine the sharing of services or resources between and within the various applications or containers. In at least one embodiment, the scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. Therefore, in at least one embodiment, the scheduler can allocate resources to different applications and distribute resources between and among applications, taking into account the system's needs and availability. In some examples, the scheduler (and / or other components of the application coordination system 1428) can determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as Quality of Service (QoS), the urgency of data output (e.g., to determine whether to perform real-time processing or delayed processing), etc.

[0180] In at least one embodiment, service 1320, utilized and shared by applications or containers in deployment system 1306, may include computing service 1416, collaborative content creation service 1417, AI service 1418, simulation service 1419, visualization service 1420, and / or other service types. In at least one embodiment, an application may invoke (e.g., execute) one or more services 1320 to perform processing operations for the application. In at least one embodiment, an application may utilize computing service 1416 to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more computing services 1416 may be utilized to perform parallel processing (e.g., using parallel computing platform 1430) to process data substantially simultaneously through one or more applications and / or one or more tasks of a single application. In at least one embodiment, parallel computing platform 1430 (e.g., NVIDIA's...) General-purpose computing can be implemented on a GPU (GPGPU) (e.g., GPU / graphics 1422). In at least one embodiment, the software layer of the parallel computing platform 1430 can provide access to the GPU's virtual instruction set and parallel computing elements to execute computational kernels. In at least one embodiment, the parallel computing platform 1430 may include memory, and in some embodiments, memory may be shared between and within multiple containers, and / or between and within different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls can be generated for multiple containers and / or multiple processes within containers to use the same data from a shared memory segment of the parallel computing platform 1430 (e.g., where multiple different stages of one or more applications are processing the same information). In at least one embodiment, instead of copying data and moving it to different locations in memory (e.g., read / write operations), the same data in the same location in memory can be used for any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, this information about the new location of the data can be stored and shared between various applications because the data as a result of processing is used to generate new data. In at least one embodiment, the location of the data, and the location of the updated or modified data, can be part of the definition of how the payload in the container is understood.

[0181] In at least one embodiment, AI service 1418 may be used to perform an inference service for executing a machine learning model associated with the application (e.g., a task to perform one or more processing tasks of the application). In at least one embodiment, AI service 1418 may utilize AI system 1424 to execute a machine learning model (e.g., a neural network such as a CNN) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, the application deploying pipeline 1410 may use one or more output models 1316 from training system 1304 and / or other models from the application to perform inference on imaging data (e.g., DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.). For example, DICOM adapter 1402b may be used to access DICOM data. In at least one embodiment, two or more examples of inference using application coordination system 1428 (e.g., a scheduler) may be available. In at least one embodiment, a first category may include a high-priority / low-latency path that can implement a higher service level protocol, such as for performing inference on urgent requests in emergency situations or for radiologists during diagnostic procedures. In at least one embodiment, the second category may include standard priority paths, which can be used for requests that may not be urgent or for situations where analysis can be performed at a later time. In at least one embodiment, the application coordination system 1428 may allocate resources (e.g., services 1320 and / or hardware 1322) based on priority paths for different inference tasks of the AI ​​service 1418.

[0182] In at least one embodiment, shared memory may be installed into AI service 1418 in system 1400. In at least one embodiment, shared memory may operate as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a set of API instances of deployment system 1306 may receive the request and may select one or more instances (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request may be fed into a database, and if not already in the cache, a machine learning model may be located from model registry 1324. A verification step may ensure that an appropriate machine learning model is loaded into the cache (e.g., shared memory), and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not already running or there are not enough instances of the application, a scheduler (e.g., the scheduler of pipeline manager 1412) may be used to start the application referenced in the request. In at least one embodiment, if an inference server has not yet been started to execute the model, an inference server may be started. In at least one embodiment, any number of inference servers may be started for each model. In at least one embodiment, in a pull model that clusters inference servers, the model can be cached whenever load balancing is favorable. In at least one embodiment, the inference servers can be statically loaded into the corresponding distributed servers.

[0183] In at least one embodiment, an inference server running in a container can be used to perform inference. In at least one embodiment, an instance of the inference server can be associated with a model (and optionally multiple versions of the model). In at least one embodiment, if an instance of the inference server does not exist when a request to perform inference on the model is received, a new instance can be loaded. In at least one embodiment, when the inference server is started, a model can be passed to the inference server, allowing the same container to be used to serve different models, as long as the inference server runs as different instances.

[0184] In at least one embodiment, during application execution, an inference request for a given application can be received, and a container (e.g., an instance of a hosted inference server) can be loaded (if not already loaded), and a launcher can be invoked. In at least one embodiment, preprocessing logic within the container can (e.g., using a CPU and / or GPU) load, decode, and / or perform any additional preprocessing on the incoming data. In at least one embodiment, once the data is ready for inference, the container can infer the data as needed. In at least one embodiment, this can include a single inference call for an image (e.g., a hand X-ray) or can request inference for hundreds of images (e.g., a chest CT scan). In at least one embodiment, the application can summarize the results before completion, which may include, but is not limited to, a single confidence score, pixel-level segmentation, voxel-level segmentation, generating visualizations, or generating text to summarize the results. In at least one embodiment, different priorities can be assigned to different models or applications. For example, some models may have a real-time (turnaround time less than 1 minute) priority, while other models may have a lower priority (e.g., turnaround time less than 10 minutes). In at least one embodiment, model execution time can be measured from the requesting agency or entity, and may include cooperative network traversal time and inference service execution time.

[0185] In at least one embodiment, the transfer of requests between service 1320 and the inference application can be hidden behind a software development kit (SDK) and robust transfer can be provided via queues. In at least one embodiment, requests are placed in queues via an API for individual application / tenant ID combinations, and the SDK pulls requests from the queues and provides them to the application. In at least one embodiment, the name of the queue can be provided in the environment where the SDK picks up requests. In at least one embodiment, asynchronous communication via queues may be useful because it allows any instance of the application to pick up work as it becomes available. In at least one embodiment, results can be sent back via queues to ensure no data loss. In at least one embodiment, queues can also provide the ability to partition work, as the highest priority work can go into a queue connected to a majority of instances of the application, while the lowest priority work can go into a queue connected to a single instance that processes tasks in the order they are received. In at least one embodiment, the application can run on a GPU-accelerated instance generated in cloud 1426, and the inference service can perform inference on the GPU.

[0186] In at least one embodiment, visualization service 1420 can be used to generate visualizations for viewing the output of application and / or deployment pipeline 1410. In at least one embodiment, visualization service 1420 can utilize GPU / graphics 1422 to generate visualizations. In at least one embodiment, visualization service 1420 can implement rendering effects such as ray tracing or other light transport simulation techniques to generate higher quality visualizations. In at least one embodiment, visualizations can include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slicing, virtual reality display, augmented reality display, etc. In at least one embodiment, a virtualized environment can be used to generate virtual interactive displays or environments (e.g., virtual environments) for system users (e.g., doctors, nurses, radiologists, etc.) to interact with. In at least one embodiment, visualization service 1420 can include an internal visualizer, cinematic and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0187] In at least one embodiment, hardware 1322 may include a GPU / graphics 1422, an AI system 1424, a cloud 1426, and / or any other hardware for executing training system 1304 and / or deployment system 1306. In at least one embodiment, the GPU / graphics 1422 (e.g., NVIDIA's...) The GPU (and / or QUADRO GPU) may include any number of GPUs that can be used to perform processing tasks of any feature or function of computing service 1416, collaborative content creation service 1417, AI service 1418, simulation service 1419, visualization service 1420, other services and / or software 1318. For example, for AI service 1418, GPU / graphics 1422 may be used to perform preprocessing on imaging data (or other data types used by machine learning models), postprocessing on the output of machine learning models, and / or perform inference (e.g., to execute machine learning models). In at least one embodiment, other components of cloud 1426, AI system 1424, and / or system 1400 may use GPU / graphics 1422. In at least one embodiment, cloud 1426 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1424 may use GPUs, and one or more AI systems 1424 may be used to execute cloud 1426 (or tasks that are at least part of deep learning or inference). Similarly, although hardware 1322 is shown as a discrete component, this is not intended to be limiting, and any component of hardware 1322 can be combined with or utilized by any other component of hardware 1322.

[0188] In at least one embodiment, the AI ​​system 1424 may include a specially built computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, in addition to a CPU, RAM, memory, and / or other components, features, or functions, the AI ​​system 1424 (e.g., NVIDIA's DGX) TM It may also include GPU-optimized software (e.g., a software stack) that can be performed using multiple GPUs / graphics 1422. In at least one embodiment, one or more AI systems 1424 may be implemented in the cloud 1426 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1400.

[0189] In at least one embodiment, cloud 1426 may include GPU-accelerated infrastructure (e.g., NVIDIA's NGC). TM This can provide a GPU-optimized platform for performing processing tasks of system 1400. In at least one embodiment, cloud 1426 may include AI system 1424 for performing one or more AI-based tasks of system 1400 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1426 may be integrated with application coordination system 1428 utilizing multiple GPUs to achieve seamless scaling and load balancing between and within applications and services 1320. In at least one embodiment, as described herein, cloud 1426 may be responsible for performing at least some of the services 1320 of system 1400, including computing service 1416, AI service 1418, and / or visualization service 1420. In at least one embodiment, cloud 1426 may perform large and small batch inference (e.g., performing NVIDIA TensorRT). TM ), providing an accelerated parallel computing platform 1430 (e.g., NVIDIA's), The system 1428 executes an application coordination system (e.g., KUBERNETES), provides graphics rendering APIs and platforms (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality cinematic effects), and / or may provide other functionalities for the system 1400. In at least one embodiment, the parallel computing platform 1430 may include APIs.

[0190] In at least one embodiment, to protect patient confidentiality (e.g., in cases of off-site use of patient data or records), cloud 1426 may include a registry—e.g., a deep learning container registry. In at least one embodiment, the registry may store containers for instantiating applications that can perform preprocessing, post-processing, or other processing tasks on patient data. In at least one embodiment, cloud 1426 may receive data, including patient data and sensor data in containers, perform the requested processing only on the sensor data in those containers, and then forward the output and / or visualization to appropriate parties and / or devices (e.g., local medical devices for visualization or diagnosis), without extracting, storing, or otherwise accessing the patient data. In at least one embodiment, the confidentiality of patient data is maintained in accordance with HIPAA and / or other data regulations.

[0191] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0192] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar references, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (referring to a physical connection where not modified) should be interpreted as partially or wholly contained, attached to, or joined together, even with some intervention. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0193] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” means multiple items). In at least one embodiment, the number of items in the multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” or “at least based on” rather than “based on only”.

[0194] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is executed jointly on one or more processors via hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within transient signal transceivers. In at least one embodiment, code (e.g., executable code or source code) is stored on one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media lack all the code, but the multiple non-transitory computer-readable storage media collectively store all the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0195] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the operations of the processes described herein, either individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating in different ways, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0196] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not constitute a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0197] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference as if each reference were individually and specifically indicated to be incorporated herein by reference and the entire contents of which are described herein.

[0198] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0199] Unless otherwise expressly stated, in some embodiments, it will be understood that throughout the specification, terms such as “processing,” “computing,” “determining,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that process and / or convert data represented as physical quantities (e.g., electrons) in the registers and / or memory of the computing system into other data represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0200] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or intermittently, sequentially, or in parallel. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0201] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an application programming interface, or an inter-process communication mechanism.

[0202] While the description herein illustrates exemplary embodiments of the described technologies, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for descriptive purposes, various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0203] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A method comprising: The first entry of a queue is provided to the first process, wherein the queue contains multiple fragments; Provide the second entry of the queue to the second process; In response to receiving a first signal from the first process, modifying a data structure associated with the queue for the first entry, wherein the data structure associated with the queue comprises a plurality of segments, wherein each segment comprises one or more counters, each counter being associated with one of the plurality of segments, and wherein modifying the data structure associated with the queue for the first entry comprises incrementing a first counter; In response to receiving a second signal from the second process, the data structure associated with the queue is modified for the second entry, wherein modifying the data structure includes incrementing a second counter; as well as The tail pointer of the queue is modified based on the data structure associated with the queue.

2. The method according to claim 1, wherein: The first entry is located in the first segment of the plurality of segments; The first counter is associated with the first segment of the plurality of segments; The second entry is located in the second segment of the plurality of segments; and The second counter is associated with the second segment of the plurality of segments.

3. The method according to claim 2, wherein, Modifying the tail pointer of the queue based on the data structure associated with the queue includes: in response to a specific counter meeting a threshold criterion, modifying the tail pointer of the queue to be equal to the end of a specific segment associated with the specific counter.

4. The method according to claim 3, further comprising: Determining that the head pointer and the tail pointer of the queue point to the same segment of the queue; wherein modifying the tail pointer of the queue based on the data structure associated with the queue includes: modifying the tail pointer of the queue to a position equal to the head pointer in response to a counter corresponding to the same segment of the queue satisfying a partial forward criterion.

5. The method according to claim 2, wherein, The first segment of the plurality of segments includes a tail indicator bit, one or more tail address bits, the first counter, and the second counter.

6. The method according to claim 3, wherein, The first counter and the second counter each have a protection bit for indicating whether the threshold criterion has been met.

7. The method according to claim 1, wherein, The first process is a producer process that stores one or more data values ​​in the first entry of the queue.

8. A system comprising: A memory that stores queues and data structures associated with the queues, wherein the queues comprise multiple segments, and the data structures associated with the queues comprise multiple sections, and wherein each section comprises one or more counters, each counter being associated with one of the multiple segments; and Processing circuitry coupled to the memory, the processing circuitry being used for: Provide the first entry of the queue to the first process; Provide the second entry of the queue to the second process; In response to receiving a first signal from the first process, the data structure associated with the queue is modified for the first entry, wherein, in order to modify the data structure for the first entry, the processing circuit is used to increment a first counter; In response to receiving a second signal from the second process, the data structure associated with the queue is modified for the second entry, wherein, in order to modify the data structure for the second entry, the processing circuitry increments a second counter; and The tail pointer of the queue is modified based on the data structure associated with the queue.

9. The system according to claim 8, wherein: The first entry is located in the first segment of the plurality of segments; The first counter is associated with the first segment of the plurality of segments; The second entry is located in the second segment of the plurality of segments; and The second counter is associated with the second segment of the plurality of segments.

10. The system according to claim 9, wherein, In order to modify the tail pointer of the queue based on the data structure associated with the queue, the processing circuitry is configured to: modify the tail pointer of the queue to be equal to the end of a specific segment associated with the specific counter in response to a specific counter meeting a threshold criterion.

11. The system according to claim 10, wherein, The processing circuit is further configured to determine that the head pointer and the tail pointer of the queue point to the same segment of the queue, wherein, in order to modify the tail pointer of the queue based on the data structure associated with the queue, the processing circuit is configured to: modify the tail pointer of the queue to a position equal to the head pointer in response to a counter corresponding to the same segment of the queue satisfying a partial forward criterion.

12. The system according to claim 9, wherein, The first segment of the plurality of segments includes a tail indicator bit, one or more tail address bits, the first counter, and the second counter.

13. The system according to claim 10, wherein, The first counter and the second counter each have a protection bit for indicating whether the threshold criterion has been met.

14. The system according to claim 8, wherein, The first process is a consumer process that performs one or more operations based on the data value in the first entry of the queue.

15. A system comprising: First processor; Second processor; as well as Processing circuitry, coupled to the first processor and the second processor, the processing circuitry being used for: A first entry of a queue is provided to the first processor, wherein the queue comprises multiple segments; The second entry of the queue is provided to the second processor; In response to receiving a first signal from the first processor, the processing circuit modifies a data structure associated with the queue for the first entry, wherein the data structure associated with the queue comprises a plurality of segments, wherein each segment comprises one or more counters, each counter being associated with one of the plurality of segments, and wherein, in order to modify the data structure associated with the queue for the first entry, the processing circuit increments a first counter; In response to receiving a second signal from the second processor, the data structure associated with the queue is modified for the second entry, wherein, in order to modify the data structure associated with the queue for the second entry, the processing circuitry is configured to increment a second counter; and The tail pointer of the queue is modified based on the data structure associated with the queue.

16. The system according to claim 15, wherein: The first entry is located in the first segment of the plurality of segments; The first counter is associated with the first segment of the plurality of segments; The second entry is located in the second segment of the plurality of segments; and The second counter is associated with the second segment of the plurality of segments.

17. The system according to claim 16, wherein, In order to modify the tail pointer of the queue based on the data structure associated with the queue, the processing circuitry is configured to: modify the tail pointer of the queue to be equal to the end of a specific segment associated with the specific counter in response to a specific counter meeting a threshold criterion.

18. The system according to claim 17, wherein, The processing circuit is further configured to determine that the head pointer and the tail pointer of the queue point to the same segment of the queue, wherein, in order to modify the tail pointer of the queue based on the data structure associated with the queue, the processing circuit is configured to: modify the tail pointer of the queue to a position equal to the head pointer in response to a counter corresponding to the same segment of the queue satisfying a partial forward criterion.

19. The system according to claim 16, wherein, The first segment of the plurality of segments includes a tail indicator bit, one or more tail address bits, the first counter, and the second counter.

20. The system according to claim 17, wherein, The first counter and the second counter each have a protection bit for indicating whether the threshold criterion has been met.

Images