Work diagram queue structure

Abstract

The invention relates to a work graph queue structure. Systems and techniques for an efficient work graph queue structure are disclosed. The techniques include obtaining a first record lock and generating one or more records for consumption by one or more consumer processes. The techniques also include storing the one or more records in one or more queues. The first queue is associated with a first consumer process of the one or more consumer processes, and at least a first value associated with a first record of the one or more records is stored in the first queue. The techniques further include, in response to receiving the first signal from the first consumer process, deactivating the first value associated with the first record from the first queue. The techniques further include releasing the first recording lock.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of priority to provisional patent application number 63 / 704,874, filed on October 8, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] At least one embodiment relates to a queue structure for communication between nodes in a computation node graph, and more specifically, to managing record locks associated with the queue structure. Background Technology

[0004] Parallel processing units (such as graphics processing units) can execute multiple tasks in parallel. In some cases, the tasks to be executed by a parallel processing unit can be represented as a graph of nodes. Each node can represent a unit of work to be executed on the parallel processing unit. Nodes can depend on other nodes, which can indicate the order of execution and control the flow of data between nodes. Producer nodes can produce data for consumer nodes to use. In some cases, a node can be both a consumer node (e.g., using data produced by a previous node in the graph) and a producer node (e.g., creating data for the next node in the graph). In some cases, data from producer nodes can be stored in a queue associated with a consumer node. In some cases, multiple producer nodes can store data in the same queue for a specific consumer node. In some cases, multiple consumer nodes can use data from the same queue. Attached Figure Description

[0005] Figure 1 This is a block diagram of an example system for an efficient working graph queue structure according to at least one embodiment.

[0006] Figure 2 It is a block diagram of an example node and queue working graph according to at least one embodiment.

[0007] Figure 3 This is an example block diagram of a set of consumer nodes having associated record queues at a specific level in a work graph of nodes, according to at least one embodiment.

[0008] Figure 4 This is an example block diagram of a set of consumer nodes having an associated index queue pointing to a unified record queue at a specific level of the node working graph, according to at least one embodiment.

[0009] Figure 5This is an example block diagram of a set of consumer nodes having an associated record-index queue pointing to a counter queue at a specific level of the node working graph, according to at least one embodiment.

[0010] Figure 6 This is an example block diagram of a set of consumer nodes having an associated index queue pointing to a unified record-index queue at a specific level of the node working graph, according to at least one embodiment. The unified record-index queue points to a counter queue.

[0011] Figure 7 This is a flowchart of an example method for an efficient working graph queue structure according to at least one embodiment.

[0012] Figure 8 This is a block diagram of an exemplary computer system according to at least one embodiment of the present disclosure.

[0013] Figure 9A The inference and / or training logic according to at least one embodiment of this disclosure is illustrated.

[0014] Figure 9B The inference and / or training logic according to at least one embodiment is illustrated.

[0015] Figure 10 Training and deployment of a neural network according to at least one embodiment are illustrated.

[0016] Figure 11 This is an example data flow diagram of an advanced computing pipeline according to at least one embodiment.

[0017] Figure 12 This is a system diagram of an example system for training, tuning, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment. Detailed Implementation

[0018] Before a producer node generates an entry (e.g., a "record") to place in its queue, it may need to ensure that there is space in the consumer node's queue. In some cases, a producer node may acquire one or more locks (e.g., "reference counts") before generating an output entry for the consumer node's queue. For example, if a producer node is going to generate 2 output records, it may need to acquire 2 locks (e.g., 2 reference counts) before generating the output records.

[0019] After an entry has been consumed, the lock corresponding to that entry can be released. The producer node can then acquire the newly released lock, and another entry can be added to the queue.

[0020] In some cases, multiple producer nodes may exist at the same "level" in a node graph, generating records for one or more consumer nodes at the next "level" in the node graph. A "level" in the graph can be the depth of the nodes within the graph. The number of locks available to a producer node at a certain level of the node graph can be equal to the number of entries that can be stored in the queue of a consumer node at the next level of the node graph. Because all producer nodes at a certain level can put their output records into the same consumer queue, it may be necessary to adjust the size of all consumer queues so that any consumer queue can store all generated records.

[0021] The structure of one or more record queues determines how many locks need to be acquired before an entry is created, and also determines the amount of memory required for each queue. For example, the amount of memory required for a queue can be based on the number of slots in the queue that can store entries and the size of each slot / entry. Some queue structures may require a large amount of memory per queue and may not efficiently utilize the number of available locks.

[0022] The present disclosure addresses the above and other shortcomings by providing an efficient working graph queue structure. In a first embodiment, each consumer node may have a single queue for storing records. Each queue may have sufficient slots for storing all records generated by the producer nodes. The size of each slot may be equal to the expected record size for that particular consumer node. If multiple producer nodes are executed concurrently, each consumer node's queue may have sufficient slots for storing all records generated by all producer nodes.

[0023] When a record is generated, one or more producer nodes may need to acquire a lock for each record being created and added to the consumer node queue. In some embodiments, one or more producer nodes may acquire a lock sufficient to generate a maximum number of records, and may immediately release the lock if one or more producer nodes generate fewer records than that maximum number. Once a record in a consumer node's queue has been consumed, the record can be marked as "ready to be deallocated". Once a record is deallocated (e.g., "freed" from the queue), the lock corresponding to the record can be released.

[0024] In the second embodiment, a single unified record queue may exist for storing all records, and each consumer node may have an index queue that stores pointers to entries in the unified record queue. For example, a producer may generate three records, each for a different consumer node. All three records can be stored in the unified record queue, and pointer entries can be added to the queue of each consumer node, pointing to the corresponding entries in the unified record queue.

[0025] The number of slots in the unified record queue and the index queue of each consumer can be equal to the maximum number of records that can be generated by one or more producer nodes that are being executed. The size of the slots in the unified record queue can be equal to the maximum output record size of one or more producer nodes. The size of the slots in each index queue is sufficient to store the value used to identify an entry in the unified record queue. In some embodiments, the value stored in the index queue is a memory address (e.g., a pointer address). In some embodiments, the value stored in the index queue is an index or offset within the unified record queue.

[0026] When a record is generated, one or more producer nodes may need to acquire a lock for each record being created and stored in the unified record queue. Once a record has been consumed by a consumer node, both the record in the unified record queue and the index queue entry pointing to the unified record queue can be marked as "ready to be deallocated". Because unified record queue entries and per-node index queue entries can be added to their respective queues in any order, releasing the lock can be problematic.

[0027] The lock can be released when an entry in the unified record queue is deallocated; however, if the corresponding index queue entry has not yet been deallocated, there may not be enough space for a new index queue entry. In some embodiments, the lock is released only when both the unified record queue entry and the corresponding index queue entry have been deallocated. In some embodiments, the number of slots in each index queue is doubled to ensure that the lock can be released when a unified record queue entry is deallocated, while still guaranteeing that there is enough space in each index queue for a new index queue entry.

[0028] In the third embodiment, a single counter queue may exist for storing counts of the generated records, and each consumer node may have a record-index queue storing the generated records and pointers to entries in the counter queue. For example, a producer may generate 3 records, each for a different consumer node. The counter entry for the producer node with a value of 3 can be added to the counter queue. Each record may be stored in the corresponding consumer's record-index queue along with a pointer value identifying the producer's counter entry in the counter queue.

[0029] The number of slots in the counter queue can be equal to the number of producer nodes that can run concurrently. Each slot in the counter queue is large enough to store the pre-configured maximum counter value. The number of slots in the record-index queue for each consumer node can be equal to the maximum number of records a consumer node can receive from each producer multiplied by the maximum number of producer nodes (e.g., producer processes) that can be executed concurrently. In some embodiments, the number of slots in the record-index queue for each consumer node is doubled to avoid the issues discussed regarding the index queue in the second embodiment. Each slot in the record-index queue for each consumer node is large enough to store the consumer node's record and the value identifying the counter queue entry corresponding to that record (e.g., memory address pointer, index, offset, etc.).

[0030] When a record has been consumed, it can be marked as "ready to be deallocated," and the corresponding counter entry can be decremented by one. Once the counter entry reaches zero, it can be marked as "ready to be deallocated." Once the counter entry is deallocated, the lock corresponding to that counter entry can be released.

[0031] In the fourth embodiment, there may be a single counter queue for storing counts of generated records, a unified record-index queue for storing indices of all records and counter entries, and each consumer node may have an index queue that stores pointers to entries in the unified record queue. For example, a producer may generate three records, each for a different consumer node. The counter entry for the producer node with a value of 3 can be added to the counter queue. All three records can be added to the record-index queue along with pointer values ​​identifying the producer's counter entry in the counter queue. Entries can be added to the index queue of each consumer node pointing to the corresponding record in the unified record-index queue.

[0032] The number of slots in the counter queue can be equal to the number of producer nodes that can run simultaneously. Each slot in the counter queue is large enough to store the pre-configured maximum counter value.

[0033] The number of slots in the unified record-index queue can be equal to the maximum number of records that can be generated by one or more producer nodes currently in operation. In some embodiments, the number of slots in the queue can be doubled because the unified record-index queue stores index values, as discussed above. Each slot in the unified record-index queue is large enough to store the maximum record size generated by one or more producer nodes and a value identifying the counter queue entry corresponding to that record (e.g., a memory address pointer, index, offset, etc.).

[0034] The number of slots in the per-consumer node index queue can be equal to the maximum number of records a consumer node can receive from each producer multiplied by the maximum number of producer nodes that can be executed simultaneously. In some embodiments, the number of slots in the queue can be doubled because the consumer node index queue stores index values, as discussed above. Each slot in each per-consumer node index queue is large enough to store a value that identifies the corresponding uniform record-index queue entry (e.g., a memory address pointer, index, offset, etc.).

[0035] When a record has been consumed, both the unified record-index queue entry and the per-consumer node index queue entry can be marked as "ready to be deallocated," and the counter entry corresponding to that record can be decremented by one. Once the value of the counter entry reaches zero, the counter entry can be marked as "ready to be deallocated." Once the counter entry is deallocated, the lock corresponding to that counter entry can be released.

[0036] In any of the embodiments discussed above, more than one consumer node may depend on data in a particular record. In this case, the record may store a counter corresponding to the number of consumer nodes that need to access the data. As each consumer node accesses the data in the record, the counter value of the record may be decremented. Once the counter value of the record reaches zero, indicating that all consumer nodes that need the data in the record have used the data in the record, the record can be marked as "ready to be deallocated," as discussed above.

[0037] Compared with the prior art, the advantages of the disclosed embodiments include, but are not limited to, reduced memory storage requirements and more efficient lock management for storing records in queues in the node working graph.

[0038] Figure 1This is a block diagram of an example system 102 for an efficient working graph queue structure according to at least one embodiment. System 102 may include a memory 104 and one or more processors 122. The memory 104 may include read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (such as synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), etc.). The processor 122 may include one or more processing units, such as a central processing unit (CPU), a graphics processing unit (GPU), a data processing unit (DPU), a parallel processing unit, an accelerator, a physical processing unit (PPU), etc. The memory 104 and the processor 122 may be connected via a queue management processing circuitry system 120.

[0039] In some embodiments, the operations performed by the queue management processing circuitry system 120 may be performed by one or more processes executed by a processor (such as one of the processors in processor 122). In some embodiments, the operations performed by the queue management processing circuitry system 120 may be performed by one or more groups of circuits.

[0040] System 102 can be used to perform the work defined in the node graph. Each node can represent a unit of work to be executed by one or more processors 122 of system 102. Nodes can depend on other nodes, which can indicate the execution order and control the data flow between nodes. Producer nodes can produce data for use by consumer nodes. In some cases, data from producer nodes can be stored in queues associated with consumer nodes. The queues associated with each node can be stored in memory 104 by queue management processing circuitry system 120. Memory 104 may also include a record lock 118, which can be used to ensure that sufficient output space is available before a new record is generated for the queue.

[0041] In some embodiments, the work defined in the node graph is graph rendering work. In some embodiments, the work defined in the node graph is artificial intelligence and / or machine learning work, such as training an AI model and / or performing inference on the AI ​​model. (The remaining text appears to be incomplete and requires further context.) Figure 9A , Figure 9B , Figure 10 , Figure 11 and Figure 12 The training and use of artificial intelligence models are described in more detail.

[0042] As an example of a simple working graph, the first node (e.g., a producer node) may be executed by a first processor in processor 122 (or by a first thread of the processor). During execution, the first processor may acquire a record lock for each output record that the producer node will produce (e.g., via queue management processing circuitry system 120). In some embodiments, if not enough record locks are available, the first processor may idle until enough record locks are available. Memory may be allocated for each new record generated by the producer node, and the first processor may allow data to be stored in the new record.

[0043] The second node (e.g., a consumer node) can be executed by a second processor in processor 122 (or by a second thread of the first processor). Once all data has been stored in a queue entry (e.g., once data is ready to be consumed by the second node), the processor executing the second node can access the data stored in its queue (e.g., via queue management processing circuitry system 120). After the second node has finished using the data stored in the queue entry, the queue entry can be marked as "ready to be deallocated". Once the record (e.g., by queue management processing circuitry system 120) is deallocated, the lock acquired before the entry was written to the queue can be released.

[0044] The queue management processing circuitry system 120 can manage one or more queues associated with nodes in a node graph. In some embodiments, each consumer node has an associated record queue (e.g., record queue 106), such as... Figure 3 As described. In some embodiments, each consumer node has an associated index queue (e.g., index queue 108) that points to entries in a unified record queue (e.g., unified record queue 110), such as... Figure 4 As described. In some embodiments, each consumer node has an associated record-index queue (e.g., record-index queue 114) that points to entries in a counter queue (e.g., counter queue 112), such as... Figure 5 As described. In some embodiments, each consumer node has an index queue (e.g., index queue 108) that points to entries in a unified record-index queue (e.g., unified record-index queue 116), which in turn points to entries in a counter queue (e.g., counter queue 112), such as... Figure 6 As described.

[0045] The following section will discuss in more detail the number of slots in each queue and the size of each slot in the queue.

[0046] Although all queues are shown in memory 104, some embodiments may include a subset of the queues shown, and may not include all queues. For example, in conjunction with... Figure 3 In embodiments similar to the depicted examples, the record queue may be the only queue used. In the context of... Figure 4 In embodiments similar to the depicted examples, index queues and unified record queues can be used without other queues, and so on. In some embodiments, all queues are used. In some embodiments, the first set of nodes in the node graph can be used with... Figure 5 The queue is similar to the queue depicted in the diagram, and the second set of nodes in the node graph can be used with... Figure 6 The queue is similar to the queue described in the text.

[0047] The queue management processing circuitry system 120 can also manage locks associated with the queue (e.g., record locks, reference counters). In some embodiments, a lock can be acquired for each entry being added to the consumer queue. In some embodiments, a single lock can be acquired for a particular producer node, regardless of the number of entries being produced.

[0048] Figure 2 This is a block diagram of an example node and queue working graph 200 according to at least one embodiment. Working graph 200 may include node A 202, node B 204, node C 206, node D 208, and node E 210. Node A 202 may be a producer node that provides records for consumption by node B 204 and / or node C 206. For example, node A 202 may generate a first record and store it in input queue 212 for consumption by node B 204. Node A 202 may also generate a second record and store it in input queue 214 for consumption by node C 206. Node A 202 may be located at a first level of the node graph (e.g., level 1 220). When generating records to be stored in input queues (e.g., input queue 212, input queue 214), all nodes at level 1 220 may share the same lock set.

[0049] Nodes B 204 and C 206 can be consumer nodes that use data stored in records created by node A 202. For example, node B 204 can read one or more entries from input queue 212. Node C 206 can read one or more entries from input queue 214. As entries are consumed and deallocated from the queue, the locks acquired by node A 202 for writing these entries can be released (e.g., by node B 204 or node C 206).

[0050] Nodes B 204 and C 206 can also be producer nodes that provide records for consumption by nodes D 208 and / or E 210. For example, node B 204 can generate a first record and store it in input queue 216 for consumption by node D 208. Node B 204 can generate a second record and store it in input queue 218 for consumption by node E 210. Node C 206 can generate a third record and store it in input queue 218 for consumption by node E 210.

[0051] Nodes B 204 and C 206 can be located at the second level of the node graph (e.g., level 2 222). When generating records to be stored in the input queues (e.g., input queue 216, input queue 218), all nodes at level 2 222 can share the same set of locks. For example, level 2 222 may have 5 locks available. Node B 204 can acquire two locks before generating its output record (e.g., one lock for the record to be stored in input queue 216 and one lock for the record to be stored in input queue 218). After node B 204 acquires these locks, there may be 3 more locks. Node C 206 can acquire one lock before generating its output record (e.g., one lock for the record to be stored in input queue 218). After node C 206 acquires these locks, there may be 2 more locks.

[0052] As another example, if only two locks are available at level 222, node B 204 may have already acquired both locks before generating its record. In this case, node C 206 may need to wait until at least one of the records generated by node B 204 has been consumed and deallocated to release the corresponding lock. Once the lock is released, it can be acquired by node C 206, which can then generate its output record.

[0053] Nodes D 208 and E 210 may be located at the third level of the node graph (e.g., level 3 224) and may be consumer nodes using data stored in records created by node B 204 and / or node C 206. For example, node D 208 may read one or more entries from input queue 216. Node E 210 may read one or more entries from input queue 218. As entries are consumed and deallocated from the queue, locks acquired by node B 204 and / or node C 206 for writing these entries may be released (e.g., by node D 208 or node E 210).

[0054] Figure 3This is an example block diagram of a set of consumer nodes 300 having associated record queues at a specific level of a node working graph according to at least one embodiment. The set of consumer nodes 300 may include nodes 0 302, 1 304, and a node array 310 that may include five nodes (e.g., a node array, index 0 306 to node array 4 308). Each node may have a configured maximum number of records it can receive from the producer nodes (e.g., “maximum record”) and a configured maximum amount of data that may be included in each record (e.g., “record size”). For example, node 0 302 may be able to receive up to one record of size 10 (e.g., “record size” equals 10) (e.g., “maximum record” equals 1). Although the examples depicted in this specification do not include units, it should be understood that the record size can be represented using any suitable memory unit (e.g., bit, byte, kilobyte, megabyte, etc.).

[0055] Node 1 304 can also be configured to receive up to one record, but its record size can be up to 30. Nodes included in node array 310 (e.g., node array, indexes 0 306 to 4 308) can share some configurations. For example, node array 310 as a whole can receive up to five records (e.g., "maximum records" equals 5), where each individual node receives up to one record (e.g., "maximum records per node" = 1). The size of all records provided to node array 310 can be up to 20.

[0056] Each node in the consumer node set 300 may have an associated record queue. For example, record queue 312 may be associated with node 0 302, record queue 314 may be associated with node 1 304, record queue 316 may be associated with node array, index 0 306, and record queue 318 may be associated with node array, index 4 308.

[0057] Each record queue may include one or more records generated by the producer node at a level in the node working graph that is just above the consumer node's level. For example, a producer node may have generated a record 320 of size 10 and stored it in record queue 312. A producer node may have generated a record 322 of size 30 and stored it in record queue 314. A producer node may have generated a record 324 of size 20 and stored it in record queue 316, and may have generated a record 326 of size 20 and stored it in record queue 318.

[0058] Before the producer node begins execution (e.g., before it starts generating records 320, 322, 324, and 326), the producer node may have already acquired four locks, one lock for each record to be generated. In some embodiments, the producer node may have already generated a number of records equal to the maximum number that each node can receive. For example, since node 0 302 can receive up to one record, node 1 304 can receive up to one record, and node array 310 can receive up to five records, the producer node may have already generated seven records (e.g., the "maximum record sum"). The producer node may have acquired seven locks, generated four records, and then immediately released the three unused additional locks.

[0059] Record queue slot 328 can indicate the number of slots in each record queue. In some embodiments, this number of slots can be equal to the "maximum sum of records" (e.g., 7 in this example) across all consumer nodes at a specific level of the node working graph. Therefore, the number of slots in each record queue can be the same for all consumer nodes in the consumer node set 300. Specifically, record queues associated with nodes in node array 310 can have the same number of slots.

[0060] In some embodiments, multiple producers can be executed concurrently, and the number of slots in each record queue can be adjusted to take into account records from these other producers. For example, if N producers are executing concurrently, the record queue slot 328 can be equal to the "maximum sum of records" multiplied by N.

[0061] After a specific record is accessed (e.g., “consumed”) by a consumer node, that specific record can be marked as “ready to be deallocated.” Once the record is actually deallocated, the lock acquired by the producer node to write the record to the queue can be released (e.g., by the consumer node). In some embodiments, multiple consumer nodes can access a specific record in the queue. In this case, the record can store a counter (not shown) initialized to the number of consumer nodes that will access the data. As each consumer node accesses data in the record, the counter value for that record can be decremented. Once the counter value reaches zero, the record can be marked as “ready to be deallocated.”

[0062] Figure 4This is an example block diagram of a consumer node set 400 having associated index queues pointing to a unified record queue at a specific level of a node working graph, according to at least one embodiment. The consumer node set 400 may include node 0 402, node 1 404, and node array 410, which may include five nodes (e.g., node array, index 0 406 to node array, index 4 408). Node 0 402 may be configured to receive up to one record of size 10. Node 1 404 may be configured to receive up to one record of size 30. Node array 410 as a whole may receive up to five records (e.g., "maximum record" equals 5). Nodes in node array 410 (e.g., node array, index 0 406 to node array, index 4 408) may all be configured to receive up to one record of size 20 (e.g., "maximum record per node" equals 1).

[0063] Each node in the consumer node set 400 may have an associated index queue. For example, index queue 422 may be associated with node 0 402, index queue 424 may be associated with node 1 404, index queue 426 may be associated with the node array, index 0 406, and index queue 428 may be associated with the node array, index 4 408.

[0064] Each index queue may include one or more index records that point to records generated by the producer node at a level in the node working graph just before the consumer node. All generated records can be stored in a unified record queue 412. For example, a producer node may have already generated record 414, stored record 414 in unified record queue 412, and stored the index 432 pointing to record 414 in index queue 424. Because record 414 is used for node 1 404, the size of record 414 can be up to 30.

[0065] The producer node may have already generated record 416, stored record 416 in unified record queue 412, and stored the index 430 pointing to record 416 in index queue 422. Because record 416 is used for node 0402, the size of record 416 can be up to 10, and its slots in unified record queue 412 may not be filled.

[0066] The producer node may have already generated record 418, stored record 418 in unified record queue 412, and stored index 436 pointing to record 418 in index queue 428. Because record 418 is used for the node array and index 4408, the size of record 418 can be up to 20, and its slots in unified record queue 412 may not be filled.

[0067] The producer node may have already generated record 420, stored record 420 in unified record queue 412, and stored the index 434 pointing to record 420 in index queue 426. Because record 420 is used for the node array and index 0 406, the size of record 420 can be up to 20, and its slots in unified record queue 412 may not be filled.

[0068] Before the producer node begins execution (e.g., before it starts generating records 414, 416, 418, 420, etc.), it may have already acquired four locks, one lock for each record to be generated. In some embodiments, as discussed above, the producer node may have already generated the "maximum record sum" record (e.g., seven records). The producer may have already acquired the "maximum record sum" (e.g., seven) locks, generated four records, and then immediately released the three unused additional locks.

[0069] Each index in the index queue may include a value indicating which record in the unified record queue 412 should be accessed. In some embodiments, the index value is a memory address pointer. In some embodiments, the index value is an index within the unified record queue 412. In some embodiments, the index value is an offset value indicating a position within the unified record queue 412.

[0070] Uniform record queue slot 438 can indicate the number of slots in uniform record queue 412, which in some embodiments may be equal to the "maximum sum of records" (e.g., 7 in this example). In some embodiments, multiple producers may be executed concurrently, and the number of slots in uniform record queue 412 may be adjusted to take records from these other producers into account. For example, if N producers are executing concurrently, uniform record queue slot 438 may be equal to the "maximum sum of records" multiplied by N.

[0071] The size of each slot in the unified record queue 412 can be equal to the maximum record size of all nodes in the consumer node set 400. For example, the maximum record size among 10, 30, and 20 is 30, so each slot in the unified record queue 412 can store a record of size 30, even if the record is eventually smaller than 30.

[0072] Index queue slot 440 can indicate the number of slots in each index queue, and in some embodiments, this number of slots can also be equal to the "maximum sum of records" of all consumer nodes at a specific level of the node working graph (e.g., 7 in this example). Therefore, the number of slots in each index queue can be the same for all consumer nodes in the consumer node set 400. Specifically, the index queues associated with nodes in the node array 410 can have the same number of slots.

[0073] If N producers are executing concurrently, then index queue slot 440 can be equal to the "maximum sum of records" multiplied by N. Each slot in each index queue is large enough to store an index value pointing to a record in the unified record queue 412.

[0074] Consumer nodes can access index values ​​stored in the index queue associated with them and use these values ​​to locate the corresponding record in the unified record queue 412. After a particular record is accessed (e.g., “consumed”) by a consumer node, both the index queue entry and the unified record queue entry can be marked as “ready to be deallocated”.

[0075] When a unified record queue is deallocated, the lock corresponding to the unified record queue entry can be released; however, if the corresponding index queue entry has not yet been deallocated, there may not be enough space for a new index queue entry. In some embodiments, the lock is only released when both the unified record queue entry and the corresponding index queue entry have been deallocated. However, this may cause delays and increase waiting time. In some embodiments, the number of slots in each index queue can be doubled to allow the lock to be released when a unified record queue entry is deallocated, while still ensuring that there is enough space in each index queue for a new index queue entry. Therefore, in some embodiments, index queue slot 440 may be equal to the "maximum sum of records" multiplied by N multiplied by 2.

[0076] As discussed earlier, multiple consumer nodes can access a specific record, and this record can track how many more consumer nodes need to access it. Once all consumer nodes have accessed the record, it can be marked as "ready to be deallocated".

[0077] Figure 5This is an example block diagram of a consumer node set 500 having a corresponding record-index queue pointing to a counter queue at a specific level of a node working graph, according to at least one embodiment. The consumer node set 500 may include node 0502, node 1 504, and node array 510, which may include five nodes (e.g., node array, index 0506 to node array, index 4 508). Node 0 502 may be configured to receive up to one record of size up to 10. Node 1 504 may be configured to receive up to one record of size up to 30. All nodes in node array 510 (e.g., node array, index 0 506 to node array, index 4 508) may be configured to receive up to one record of size up to 20.

[0078] When a record is generated, the producer node can acquire one lock for an entry in counter queue 520. A counter queue entry (e.g., counter 522) can have a counter value equal to the number of records generated (or to be generated) by the producer node. All records generated by the producer node can have an index value pointing to a counter queue entry.

[0079] Each node in the consumer node set 500 may have an associated record-index queue. For example, record-index queue 512 may be associated with node 0 502, record-index queue 514 may be associated with node 1 504, record-index queue 516 may be associated with node array, index 0 506, and record-index queue 518 may be associated with node array, index 4 508.

[0080] Each record-index queue entry may include a record generated by the producer node at a level in the node working graph that is just above the level of the consumer node, and an index pointing to a counter entry in counter queue 520. For example, the producer may have already generated record 526 and stored it in record-index queue 512, which has an index pointing to counter 522 in counter queue 520. Because record 526 is used for node 0 502, the size of record 526 can be up to 10.

[0081] The producer node may have already generated record 528 and stored it in the record-index queue 514, which has an index pointing to counter 522 in the counter queue 520. Because record 528 is used for node 1 504, the size of record 528 can be up to 30.

[0082] The producer node may have already generated record 530 and stored it in record-index queue 516, which has an index pointing to counter 522 in counter queue 520. Because record 530 is used for the node array and indexed 0 506, the size of record 530 can be up to 20.

[0083] The producer node may have already generated record 532 and stored it in record-index queue 518, which has an index pointing to counter 522 in counter queue 520. Because record 532 is used for the node array and index 4 508, the size of record 532 can be up to 20.

[0084] The index value in each record-index queue entry may include a value indicating which counter entry in counter queue 520 should be accessed (e.g., decremented). In some embodiments, the index value is a memory address pointer. In some embodiments, the index value is an index within counter queue 520. In some embodiments, the index value is an offset value indicating a location within counter queue 520.

[0085] Before the producer node begins execution (e.g., before the producer node starts generating records 526, 528, 530, 532, etc.), the producer node may have already acquired a lock for counter 522 in counter queue 520. Because the producer node has generated 4 records, counter 522 may have already been initialized with a counter value of 4.

[0086] Consumer nodes can access records stored in the record-index queue corresponding to the consumer node. After a particular record is accessed by a consumer node (e.g., "consumed"), the record-index queue entry can be marked as "ready to be deallocated," and the counter value pointed to by the record-index queue can be decremented by 1. Once the counter value in the counter queue entry reaches zero (indicating that all records generated by the producer node have been consumed), the counter queue entry can be marked as "ready to be deallocated." Once the counter queue entry is deallocated, the lock corresponding to the counter queue entry can be released.

[0087] Counter queue slot 524 can indicate the number of slots in counter queue 520, which in some embodiments may be equal to the number of producer nodes that can be executed simultaneously (e.g., "N"). Therefore, each producer node can have one entry in counter queue 520 at a time.

[0088] Each slot in counter queue 520 is large enough to store a predetermined maximum counter value, which can represent the maximum number of records a single producer node can generate at one time. For example, if the maximum number of records a single producer node can generate at one time is 64, then each slot in counter queue 520 is large enough to store a counter value of 64, such as at least 6 bits of data.

[0089] Record-index queue slot 534 may indicate the number of slots in each record-index queue, which in some embodiments may be equal to the maximum number of records that the consumer node corresponding to the record-index queue can receive (e.g., "maximum records per node"). The "maximum records per node" value for a node outside the node array may be equal to the node's "maximum records" value. The "maximum records per node" value for a node within the node array may differ from the "maximum records" value of the node array as a whole. In some embodiments, the "maximum records per node" value for a node within the node array may differ from the "maximum records per node" value for other nodes within the same node array.

[0090] For example, node 0 502 has a "maximum records" value (and a "maximum records per node" value) of 1, and may have 1 slot in its corresponding record-index queue 512. If another node has a "maximum records" value of 2, then that other node may have twice the number of slots in its corresponding record-index queue. Therefore, the number of slots in each record-index queue can be different for each consumer node in the consumer node set 500. Specifically, the record-index queues associated with nodes in the node array 510 can have different numbers of slots.

[0091] For example, in some embodiments, the "maximum records" value of the node array can be 10, and the node array can include 3 nodes. The "maximum records per node" value of the first node in the node array can be 2, while the "maximum records per node" value of the second and third nodes in the node array can be 4. The number of slots in the record-index queue associated with the second and third nodes in the node array can be twice the number of slots in the record-index queue associated with the first node in the node array.

[0092] If N producers are executing simultaneously, then record-index queue slot 534 can be equal to the "maximum record" of the consumer node corresponding to the queue multiplied by N.

[0093] In some embodiments, to avoid the increased latency and waiting time discussed above with reference to other index queues, the number of slots in each record-index queue can be doubled so that the lock can be released when a counter queue entry is deallocated, while still ensuring that there is sufficient space in each record-index queue for new record-index entries. Therefore, in some embodiments, the record-index queue slot 534 can be equal to the "maximum record" multiplied by N multiplied by 2.

[0094] As discussed earlier, multiple consumer nodes can access a specific record, and this record can track how many more consumer nodes need to access it. Once all consumer nodes have accessed the record, it can be marked as "ready to be deallocated," and the counter pointed to by the record can be decremented.

[0095] Figure 6 This is an example block diagram of a set of consumer nodes 600 having a related index queue pointing to a unified record-index queue at a specific level of a node working graph, according to at least one embodiment. The unified record-index queue points to a counter queue. The set of consumer nodes 600 may include nodes 0 602, 1 604, and a node array 610, which may include five nodes (e.g., a node array, indexes 0 606 to 608). Node 0 602 may be configured to receive up to one record of size up to 10. Node 1 604 may be configured to receive up to one record of size up to 30. All nodes in the node array 610 (e.g., the node array, indexes 0 606 to 608) may be configured to receive up to one record of size up to 20.

[0096] When a record is generated, the producer node can acquire one lock for an entry in counter queue 640. A counter queue entry (e.g., counter 642) can have a counter value equal to the number of records generated (or to be generated) by the producer node. All records generated by the producer node can be stored in unified record-index queue 630 and can have index values ​​pointing to counter queue entries. For each record generated by the producer node, an index queue entry pointing to the record in the unified record-index queue can be added to the consumer node's index queue.

[0097] Each node in the consumer node set 600 may have an associated index queue. For example, index queue 612 may be associated with node 0 602, index queue 614 may be associated with node 1 604, index queue 616 may be associated with the node array, index 0 606, and index queue 618 may be associated with the node array, index 4 608.

[0098] Each index queue entry may include an index value pointing to a record in the unified record-index queue 630 generated by the producer node at a level in the node working graph just before the consumer node. For example, the producer may have already generated record 632, stored record 632 in the unified record-index queue 630 (which has an index pointing to counter 642), and stored index 626 pointing to record 632 in index queue 618. Because record 632 is intended for the node array, index 4608, the size of record 632 can be up to 20, and its slots in the unified record-index queue 630 may not be filled.

[0099] The producer node may have already generated record 634, stored record 634 in unified record-index queue 630 (which has an index pointing to counter 642), and stored index 620 pointing to record 634 in index queue 612. Because record 634 is intended for node 0 602, the size of record 634 can be up to 10, and its slots in unified record-index queue 630 may not be filled.

[0100] The producer node may have already generated record 636, stored record 636 in the unified record-index queue 630 (which has an index pointing to counter 642), and stored the index 622 pointing to record 636 in the index queue 614. Because record 636 is intended for node 1 604, the size of record 636 can be up to 30, filling its slots in the unified record-index queue 630.

[0101] The producer node may have already generated record 638, stored record 638 in the unified record-index queue 630 (which has an index pointing to counter 642), and stored index 624 pointing to record 638 in index queue 616. Because record 638 is intended for the node array, index 0 606, the size of record 638 can be up to 20, and its slots in the unified record-index queue 630 may not be filled.

[0102] The index value in each index queue entry may include a value indicating which uniform record-index entry in the uniform record-index queue 630 should be accessed (e.g., consumed). In some embodiments, the index value is a memory address pointer. In some embodiments, the index value is an index within the uniform record-index queue 630. In some embodiments, the index value is an offset value indicating a location within the uniform record-index queue 630.

[0103] The index value in each uniform record-indexed queue entry may include a value indicating which counter queue entry in counter queue 640 should be accessed (e.g., decremented). In some embodiments, the index value is a memory address pointer. In some embodiments, the index value is an index within counter queue 640. In some embodiments, the index value is an offset value indicating a location within counter queue 640.

[0104] Before the producer node begins execution (e.g., before the producer node starts generating records 632, 634, 636, 638, etc.), the producer node may have already acquired a lock for counter 642 in counter queue 640. Because the producer node has generated 4 records, counter 642 may have already been initialized with a counter value of 4.

[0105] Consumer nodes can access records in the unified record-index queue 630 using index values ​​stored in the index queue. After a particular record is accessed by a consumer node (e.g., "consumed"), the index queue entry can be marked as "ready to be deallocated". The unified record-index queue entry can also be marked as "ready to be deallocated". The counter value pointed to by the unified record-index queue entry can be decremented by one. Once the counter value in the counter queue entry reaches zero (indicating that all records generated by the producer node have been consumed), the counter queue entry can be marked as "ready to be deallocated". Once the counter queue entry is deallocated, the lock corresponding to the counter queue entry can be released.

[0106] Index queue slot 628 can indicate the number of slots in each index queue, which in some embodiments may be equal to the maximum number of records that the consumer node corresponding to the record-index queue can receive (e.g., "maximum records per node"). The "maximum records per node" value for a node outside the node array may be equal to the node's "maximum records" value. The "maximum records per node" value for a node within the node array may differ from the "maximum records" value of the node array as a whole. In some embodiments, the "maximum records per node" value for a node within the node array may differ from the "maximum records per node" value for other nodes within the same node array.

[0107] For example, node 0 602 has a "maximum records" value (and a "maximum records per node" value) of 1, and may have 1 slot in its corresponding index queue 612. If another node has a "maximum records" value of 2, then that other node may have twice as many slots in its corresponding index queue. Therefore, the number of slots in each index queue can be different for each consumer node in the consumer node set 600. Specifically, the index queues associated with nodes in the node array 610 can have different numbers of slots.

[0108] For example, in some embodiments, the "maximum records" value of the node array can be 10, and the node array can include 3 nodes. The "maximum records per node" value of the first node in the node array can be 2, while the "maximum records per node" value of the second and third nodes in the node array can be 4. The number of slots in the index queue associated with the second and third nodes in the node array can be twice the number of slots in the index queue associated with the first node in the node array.

[0109] If N producer nodes are executing simultaneously, then index queue slot 628 can be equal to the "maximum record" of the consumer node corresponding to the queue multiplied by N.

[0110] In some embodiments, to avoid the increased latency and waiting time discussed above with reference to other index queues, the number of slots in each index queue can be doubled so that the lock can be released when a counter queue entry is deallocated, while still ensuring that there is enough space in each index queue for a new index queue entry. Therefore, in some embodiments, index queue slot 628 can be equal to "maximum record" multiplied by N multiplied by 2.

[0111] Each slot in each index queue is large enough to store an indication of which entry in the uniform record-index queue 630 corresponds to the index value of a particular consumer node.

[0112] The number of slots in the unified record-index queue 630 can be equal to the "maximum sum of records" discussed earlier. If N producer nodes are executing simultaneously, the number of slots in the unified record-index queue 630 can be equal to the "maximum sum of records" multiplied by N. In some embodiments, the number of slots in the unified record-index queue 630 can be equal to the "maximum sum of records" multiplied by N multiplied by 2 to avoid the increased latency and waiting time discussed above with other index queues.

[0113] The size of each slot in the unified record-index queue 630 can be equal to the maximum record size of all nodes in the consumer node set 600 plus the amount of space required to represent the index value pointing to the entry in the counter queue 640. For example, the maximum record size among 10, 30, and 20 is 30, so each slot in the unified record-index queue 630 can store a record of size 30 (even if the record ends up being less than 30) plus an index value.

[0114] The number of slots in counter queue 640 can be equal to the number of producer nodes that can execute simultaneously (e.g., "N"). Therefore, each producer node can have one entry in counter queue 640 at a time.

[0115] Each slot in counter queue 640 is large enough to store a predetermined maximum counter value, which can represent the maximum number of records a single producer node can generate at one time. For example, if the maximum number of records a single producer node can generate at one time is 64, then each slot in counter queue 640 is large enough to store the counter value 64, such as at least 6 bits of data.

[0116] As discussed earlier, multiple consumer nodes can access a specific record, and this record can track how many more consumer nodes need to access it. Once all consumer nodes have accessed the record, it can be marked as "ready to be deallocated," and the counter pointed to by the record can be decremented.

[0117] Figure 7 This is a flowchart of an example method 700 for an efficient working graph queue structure according to at least one embodiment.

[0118] Method 700 can be executed using one or more processing units (e.g., CPU, GPU, accelerator, physical processing unit (PPU), data processing unit (DPU), etc.), which may include one or more memory devices (or communicate with them). In at least one embodiment, method 700 can be executed using one or more processing devices. In at least one embodiment, method 700 can use... Figure 1 The method 700 is executed by the processing unit of system 102. In at least one embodiment, the method 700 can be performed by... Figure 1The queue management processing circuitry system 120 executes the method. In at least one embodiment, the processing unit executing method 700 can execute instructions stored on a non-transitory computer-readable storage medium. In at least one embodiment, method 700 can be executed using multiple processing threads (e.g., CPU threads and / or GPU threads), individual threads executing one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, the processing threads implementing method 700 can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 700 can execute asynchronously with respect to each other. Figure 7 Compared to the order shown, the various operations of method 700 can be performed in different orders. Some operations of method 700 can be performed concurrently with other operations. In at least one embodiment, Figure 7 One or more operations shown may not always be performed.

[0119] At block 702, the processing unit performing method 700 may acquire a first record lock. The first record lock may be a reference count or another value indicating the number of available output record spaces. At block 704, the processing unit may generate one or more records for consumption by one or more consumer processes. In some embodiments, if more than one record is generated, more than one record lock may be acquired.

[0120] At block 706, the processing unit may store one or more records in one or more queues. A first queue may be associated with a first consumer process. At least a first value associated with the first record among the one or more records may be stored in the first queue. In some embodiments, the first value is the data of the first record. In some embodiments, the first value is an index value pointing to an entry in another queue (e.g., a unified record queue, a unified record-index queue) where the first record is stored. In some embodiments, the first value includes the data of the first record and an index value pointing to an entry in a counter queue.

[0121] In some embodiments, one or more queues include a first queue associated with a first consumer process, a counter queue, and a unified record-index queue. The first queue associated with the first consumer process may be a first index queue. Storing one or more records in one or more queues may include: storing counter values ​​in a counter queue corresponding to the counts of one or more records; storing the first record in a unified record-index queue; and storing the index of the first record in a first index queue. The index of the first record may be a first value associated with the first record. Storing one or more records in one or more queues may further include: storing the counter index associated with the counter value in the counter queue, in conjunction with storing the first record in the unified record-index queue.

[0122] In some embodiments, the processing unit may also decrement the counter value in the counter queue in response to receiving a first signal from the first consumer process. In some embodiments, the first record lock corresponds to the counter value in the counter queue, and releasing the first lock may be performed in response to the counter value being equal to the lock release value. In some embodiments, the lock release value may be zero.

[0123] In some embodiments, the counter value can be incremented instead of decremented. For example, the counter value can be initialized to zero, incremented as records are consumed, and can have a lock release value equal to the count of one or more records.

[0124] In some embodiments, the length of the first queue (e.g., the first index queue) can be equal to twice the maximum (max) active entry value of the first queue. For example, if a particular consumer node can receive 3 records from a producer node, and there can be N producer nodes that are active simultaneously, then the maximum active entry value can be equal to 3 multiplied by N. Therefore, the length of the first queue can be equal to 3 multiplied by N multiplied by 2.

[0125] In some embodiments, the length of the unified record-index queue can be equal to twice the maximum number of active entries in the unified record-index queue. For example, if a particular producer node can output 7 records and there can be N producer nodes that are active simultaneously, then the maximum number of active entries in the unified record-index queue can be equal to 7 multiplied by N. Therefore, the length of the unified record-index queue can be equal to 7 multiplied by N multiplied by 2.

[0126] At block 708, the processing unit may remove the first value associated with the first record from the first queue in response to receiving a first signal from the first consumer process. The first signal from the first consumer process may indicate that the first record has been consumed and the first value associated with the first record is ready to be deallocated.

[0127] At box 710, the processing unit can release the first record lock.

[0128] Figure 8 This is a block diagram illustrating an exemplary computer system according to at least one embodiment of the present disclosure. Computer system 800 may correspond to... Figure 1 System 102 is described. Computer system 800 can operate as a server or endpoint machine in an endpoint server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. This machine can be a television, personal computer (PC), tablet computer, set-top box (STB), personal digital assistant (PDA), mobile phone, network device, server, network router, switch, or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify the actions the machine should take. Furthermore, although only one machine is shown in the figure, the term "machine" should also be understood to include any set of machines that, individually or jointly, execute a set (or more) of instructions to perform one or more methods discussed herein.

[0129] Example computer system 800 includes a processing device (processor) 802, main memory 804 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM), double data rate (DDRSDRAM) or DRAM (RDRAM), etc.), static memory 806 (e.g., flash memory, static random access memory (SRAM), etc.) and data storage device 816, which communicate with each other via bus 828.

[0130] Processor (processing device) 802 represents one or more general-purpose processing devices, such as microprocessors, central processing units, etc., and may include processing logic 822. More specifically, processor 802 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor that implements other instruction sets or combinations of instruction sets. Processor 802 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. Processor 802 is configured to execute instructions 826 (e.g., for generating threat indicator alerts) to perform the operations described herein.

[0131] The computer system 800 may also include a network interface device 808. The computer system 800 may also include a video display unit 810 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an input device 812 (e.g., a keyboard, alphanumeric keypad, motion-sensing input device, touchscreen), a cursor control device 814 (e.g., a mouse), and a signal generation device 818 (e.g., a speaker). In some embodiments, the computer system 800 may not include the video display unit 810, the input device 812, and / or the cursor control device 814 (e.g., a headless configuration).

[0132] Data storage device 816 may include a non-transitory machine-readable storage medium 824 (also referred to as a computer-readable storage medium) storing one or more sets of instructions 826 (e.g., for efficient work graph queue structures) embodying one or more methods or functions described herein. The instructions 826 may also reside wholly or at least partially in main memory 804 and / or processor 802 during execution by computer system 800, which also constitute machine-readable storage media. These instructions may also be sent or received on network 820 via network interface device 808.

[0133] In one embodiment, instruction 826 includes instructions for an efficient working graph queue structure. While computer-readable storage medium 824 (machine-readable storage medium) is shown as a single medium in the exemplary embodiment, the terms "computer-readable storage medium" and "machine-readable storage medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more sets of instructions. The terms "computer-readable storage medium" and "machine-readable storage medium" should also be understood to include any medium capable of storing, encoding, or carrying a set of machine-executable instructions and enabling a machine to perform one or more methods as described in this disclosure. Therefore, the terms "computer-readable storage medium" and "machine-readable storage medium" should be understood to include, but are not limited to, solid-state memories, optical media, and magnetic media.

[0134] Reasoning and training logic

[0135] Figure 9A Inference and / or training logic 915 is shown for performing inference and / or training operations associated with one or more embodiments.

[0136] In at least one embodiment, the inference and / or training logic 915 may include, but is not limited to, code and / or data storage 901 for storing forward and / or output weights and / or input / output data, and / or other parameters of neurons or layers of a neural network trained for and / or used for inference in one or more embodiments. In at least one embodiment, the training logic 915 may include graph code or other software for controlling timing and / or sequence (or coupled to code and / or data storage 901 for storing graph code or other software for controlling timing and / or sequence), wherein weight and / or other parameter information is loaded to configure the processing unit, eight logic units, integer and / or floating-point units (collectively referred to as an arithmetic logic unit (ALU) or simple circuit). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network corresponding to the code. In at least one embodiment, code and / or data storage 901 is stored in the weight parameters and / or input / output data of each layer of a neural network trained or used in conjunction with one or more embodiments during the forward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data storage 901 may be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0137] In at least one embodiment, any portion of the code and / or data storage 901 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 901 may be a cache memory, dynamic random-addressable memory (“DRAM”), static random-addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 901 is internal or external to the processor, for example, or composed of DRAM, SRAM, flash memory, or some other storage type, may depend on the available on-chip or off-chip storage space, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0138] In at least one embodiment, the inference and / or training logic 915 may include, but is not limited to, code and / or data storage 905 to store backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, during training and / or inference using one or more embodiments, the code and / or data storage 905 stores weight parameters and / or input / output data of each layer of a neural network trained or used in one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 915 may include (or coupled to) code and / or data storage 905 for storing graph code or other software for controlling timing and / or order, wherein weight and / or other parameter information is loaded to configure a processing unit, including logic units comprising integer and / or floating-point units (collectively referred to as an arithmetic logic unit (ALU)).

[0139] In at least one embodiment, code (such as graph code) causes the architecture of the neural network corresponding to that code to load weights or other parameter information into the processor ALU. In at least one embodiment, any portion of the code and / or data storage 905 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 905 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 905 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice between the code and / or data storage 905 being internal or external to the processor, for example, whether it consists of DRAM, SRAM, flash memory, or some other type of storage, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the data batch size used in the inference and / or training of the neural network, or some combination of these factors.

[0140] In at least one embodiment, code and / or code and / or data storage 901 and code and / or data storage 905 may be separate storage structures. In at least one embodiment, code and / or data storage 901 and code and / or data storage 905 may be combined storage structures. In at least one embodiment, code and / or data storage 901 and code and / or data storage 905 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data storage 901 and code and / or data storage 905 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0141] In at least one embodiment, the inference and / or training logic 915 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 910 (including integer and / or floating-point units) for performing logical and / or mathematical operations at least in part based on or instructed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from layers or neurons within a neural network) stored in activation storage 920, which are functions of input / output and / or weight parameter data stored in code and / or data storage 901 and / or code and / or data storage 905. In at least one embodiment, activation is activated in response to execution instructions or other code, linear algebraic and / or matrix-based mathematical generation performed by ALU 910, and the activation is stored in activation storage 920, wherein weight values ​​stored in code and / or data storage 905 and / or code and / or data storage 901 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, and any or all of these can be stored in code and / or data storage 905 or code and / or code and / or data storage 901 or another on-chip or off-chip memory.

[0142] In at least one embodiment, one or more processors or other hardware logic devices or circuits include one or more ALUs 910, while in another embodiment, one or more ALUs 910 may be located outside the processor or other hardware logic device or the circuitry using them (e.g., a coprocessor). In at least one embodiment, one or more ALUs 910 may be included within an execution unit of a processor, or otherwise included in a group of ALUs accessible by the execution unit of the processor, which may be within the same processor or distributed among different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, code and / or data storage 901, code and / or data storage 905, and activation storage 920 may share a processor or other hardware logic device or circuitry, while in another embodiment, they may be located in different processors or other hardware logic devices or circuitry, or in some combination of the same and different processors or other hardware logic devices or circuitry. In at least one embodiment, any portion of activation storage 920 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be retrieved and / or processed using the processor’s fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0143] In at least one embodiment, the active memory 920 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 920 may be wholly or partially located inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory 920 is internal to or external to the processor may depend on the available on-chip or off-chip storage, the latency requirements for training and / or inference functions, the batch size of data used in inference and / or training the neural network, or some combination of these factors. For example, it may include DRAM, SRAM, flash memory, or some other memory type.

[0144] In at least one embodiment, Figure 9A The inference and / or training logic 915 shown can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 9AThe inference and / or training logic 915 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware (such as field programmable gate array (“FPGA”)).

[0145] Figure 9B An inference and / or training logic 915 according to at least one embodiment is illustrated. In at least one embodiment, the inference and / or training logic 915 may include, but is not limited to, hardware logic, wherein computational resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 9B The inference and / or training logic 915 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 9B The inference and / or training logic 915 shown can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., field-programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 915 includes, but is not limited to, code and / or data storage 901 and code and / or data storage 905, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 9B In at least one embodiment shown, each of code and / or data storage 901 and code and / or data storage 905 is associated with dedicated computing resources (e.g., computing hardware 902 and computing hardware 906). In at least one embodiment, each of computing hardware 902 and computing hardware 906 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) only on the information stored in code and / or data storage 901 and code and / or data storage 905, respectively, and the results of the function execution are stored in activation storage 920.

[0146] In at least one embodiment, each of the code and / or data storage 901 and 905 and the corresponding computing hardware 902 and 906 corresponds to a different layer of the neural network, such that activations obtained from one storage / computation pair 901 / 902 of the code and / or data storage 901 and computing hardware 902 provide input as input to the next storage / computation pair 905 / 906 of the code and / or data storage 905 and computing hardware 906, in order to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / computation pair 901 / 902 and 905 / 906 may correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic 915 after or in parallel with the storage / computation pairs 901 / 902 and 905 / 906.

[0147] Neural network training and deployment

[0148] Figure 10 Training and deployment of a deep neural network according to at least one embodiment are illustrated. In at least one embodiment, an untrained neural network 1006 is trained using a training dataset 1002. In at least one embodiment, the training framework 1004 is the PyTorch framework, while in other embodiments, the training framework 1004 is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training frameworks. In at least one embodiment, the training framework 1004 trains the untrained neural network 1006 and enables it to be trained using the processing resources described herein to generate a trained neural network 1008. In at least one embodiment, the weights may be randomly selected or pre-trained using a deep belief network. In at least one embodiment, training may be performed in a supervised, partially supervised, or unsupervised manner.

[0149] In at least one embodiment, supervised learning is used to train an untrained neural network 1006, wherein the training dataset 1002 includes inputs paired with desired outputs for input, or wherein the training dataset 1002 includes inputs with known outputs and the neural network 1006 is manually graded output. In at least one embodiment, the untrained neural network 1006 is trained in a supervised manner, and inputs from the training dataset 1002 are processed, and the resulting outputs are compared with a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through the untrained neural network 1006. In at least one embodiment, a training framework 1004 adjusts the weights controlling the untrained neural network 1006. In at least one embodiment, the training framework 1004 includes tools for monitoring the degree to which the untrained neural network 1006 converges to a model (e.g., a trained neural network 1008) adapted to generate the correct answer (e.g., result 1014) based on input data (e.g., a new dataset 1012). In at least one embodiment, the training framework 1004 repeatedly trains the untrained neural network 1006 while adjusting the weights to improve the output of the untrained neural network 1006 using a loss function and tuning algorithm (e.g., stochastic gradient descent). In at least one embodiment, the training framework 1004 trains the untrained neural network 1006 until the untrained neural network 1006 reaches the desired accuracy. In at least one embodiment, the trained neural network 1008 can then be deployed to implement any number of machine learning operations.

[0150] In at least one embodiment, unsupervised learning is used to train an untrained neural network 1006, wherein the untrained neural network 1006 attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised learning training dataset 1002 will include input data without any associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 1006 can learn groupings within the training dataset 1002 and can determine how each input relates to the untrained dataset 1002. In at least one embodiment, unsupervised training can be used to generate a self-organizing graph in the trained neural network 1008, which is capable of performing operations useful for reducing the dimensionality of the new dataset 1012. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows the identification of data points in the new dataset 1012 that deviate from the normal patterns of the new dataset 1012.

[0151] In at least one embodiment, semi-supervised learning can be used, a technique in which the training dataset 1002 includes a mixture of labeled and unlabeled data. In at least one embodiment, the training framework 1004 can be used to perform incremental learning, for example, through transfer learning techniques. In at least one embodiment, incremental learning enables the trained neural network 1008 to adapt to the new dataset 1012 without forgetting the knowledge injected into the trained neural network 1008 during initial training.

[0152] Reference Figure 11 , Figure 11 This is an example data flow diagram of process 1100 for generating and deploying a processing and inference pipeline according to at least one embodiment. In at least one embodiment, process 1100 may be deployed to perform game name recognition analysis and inference on user feedback data at one or more facilities 1102, such as a data center.

[0153] In at least one embodiment, process 1100 may be executed within training system 1104 and / or deployment system 1106. In at least one embodiment, training system 1104 may be used to train, deploy, and implement machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use with deployment system 1106. In at least one embodiment, deployment system 1106 may be configured to offload processing and computing resources in a distributed computing environment to reduce infrastructure requirements of facility 1102. In at least one embodiment, deployment system 1106 may provide a pipeline platform for selecting, customizing, and implementing virtual instruments for use with computing devices at facility 1102. In at least one embodiment, virtual instruments may include software-defined applications for performing one or more processing operations on feedback data. In at least one embodiment, one or more applications in the pipeline may use or invoke services of deployment system 1106 (e.g., inference, visualization, computation, AI, etc.) during application execution.

[0154] In at least one embodiment, some applications used in the advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at facility 1102 using feedback data 1108 stored at facility 1102 (e.g., imaging data) or feedback data 1108 from another or more facilities, or a combination thereof. In at least one embodiment, training system 1104 may be used to provide applications, services, and / or other resources to generate a deployable machine learning model for the operation of deployment system 1106.

[0155] In at least one embodiment, the model registry 1124 may be supported by an object storage system that supports version control and object metadata. In at least one embodiment, it may be available from within a cloud platform via, for example, cloud storage (e.g., Figure 12 The system uses a cloud-compatible application programming interface (API) to access object storage. In at least one embodiment, machine learning models within the model registry 1124 can be uploaded, listed, modified, or deleted by the developer or partner of the system interacting with the API. In at least one embodiment, the API can provide access to methods that allow users with appropriate credentials to associate models with applications, enabling the models to be executed as part of the containerized instantiation of the application.

[0156] In at least one embodiment, training pipeline 1204 ( Figure 12 This can include situations where facility 1102 is training its own machine learning model or has an existing machine learning model that needs optimization or updating. In at least one embodiment, feedback data 1108 can be received from various channels (such as forums, web forms, etc.). In at least one embodiment, once feedback data 1108 is received, AI-assisted annotation 1110 can be used to help generate annotations corresponding to the feedback data 1108 for use as ground truth data for the machine learning model. In at least one embodiment, AI-assisted annotation 1110 can include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that can be trained to generate annotations corresponding to certain types of feedback data 1108 (e.g., from certain devices), and / or certain types of anomalies in the feedback data 1108. In at least one embodiment, AI-assisted annotation 1110 can then be used directly, or it can be adjusted or fine-tuned using annotation tools to generate ground truth data. In at least one embodiment, in some examples, labeled data 1112 can be used as ground truth data for training the machine learning model. In at least one embodiment, AI-assisted annotations 1110, labeled data 1112, or a combination thereof can be used to train a machine learning model (e.g., via...). Figure 11 and / or Figure 12 The model is trained on ground-based real-world data (1114). In at least one embodiment, the trained machine learning model may be referred to as output model 1116 and may be used by deployment system 1106 as described herein.

[0157] In at least one embodiment, training pipeline 1204 ( Figure 12This may include situations where facility 1102 requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 1106, but facility 1102 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for this purpose). In at least one embodiment, an existing machine learning model may be selected from model registry 1124. In at least one embodiment, model registry 1124 may include machine learning models trained to perform various inference tasks on imaging data. In at least one embodiment, the machine learning model in model registry 1124 may be trained on imaging data from a different facility (e.g., a remote facility) instead of facility 1102. In at least one embodiment, the machine learning model may have already been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a specific location (which may be in the form of feedback data 1108), training may be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of the imaging data off-site (e.g., in compliance with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model has been trained or partially trained at one location, a machine learning model can be added to model registry 1124. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be used in model registry 1124. In at least one embodiment, a machine learning model (referred to as output model 1116) can then be selected from model registry 1124 and used in deployment system 1106 to perform one or more processing tasks for one or more applications of the deployment system.

[0158] In at least one embodiment, training pipeline 1204 ( Figure 12This can be used in scenarios including facility 1102, which requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 1106, but facility 1102 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for this purpose). In at least one embodiment, the machine learning model selected from model registry 1124 may not be fine-tuned or optimized for the feedback data 1108 generated at facility 1102 due to population differences, genetic variations, robustness, anomalous diversity of training data, and / or other problems with the training data used to train the machine learning model. In at least one embodiment, AI-assisted annotation 1110 can be used to help generate annotations corresponding to the feedback data 1108 for use as ground-based data for retraining or updating the machine learning model. In at least one embodiment, labeled data 1112 can be used as ground-based data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 1114. In at least one embodiment, model training 1114 may include data (e.g., AI-assisted annotations 1110, labeled data 1112, or a combination thereof) that can be used as ground-based data to retrain or update the machine learning model.

[0159] In at least one embodiment, deployment system 1106 may include software 1118, service 1120, hardware 1122, and / or other components, features, and functions. In at least one embodiment, deployment system 1106 may include a software "stack" such that software 1118 can be built on top of service 1120 and can be used to perform some or all of the processing tasks, and service 1120 and software 1118 can be built on top of hardware 1122 and can be used to perform processing, storage, and / or other computing tasks of deployment system 1106.

[0160] In at least one embodiment, software 1118 may include any number of different containers, each of which may perform an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in an advanced processing and inference pipeline. In at least one embodiment, for each type of computing device, there may be any number of containers that may perform data processing tasks on feedback data 1108 (or other data types, such as those described herein). In at least one embodiment, in addition to the containers that receive and configure imaging data for use by each container and / or for use by facility 1102 after processing through the pipeline, the advanced processing and inference pipeline may also be defined based on the selection of different containers desired or required for processing feedback data 1108 (e.g., to convert output back to a usable data type for storage and display in facility 1102). In at least one embodiment, a combination of containers within software 1118 (e.g., which constitute a pipeline) may be referred to as a virtual instrument (as described in more detail herein), and the virtual instrument may utilize service 1120 and hardware 1122 to perform some or all of the processing tasks of the application instantiated in the container.

[0161] In at least one embodiment, data may be preprocessed as part of a data processing pipeline to prepare it for processing by one or more applications. In at least one embodiment, post-processing may be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare output data for the next application and / or to prepare output data for user transmission and / or use (e.g., as a response to an inference request). In at least one embodiment, the inference task may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include the output model 1116 of the training system 1104.

[0162] In at least one embodiment, the tasks of the data processing pipeline can be encapsulated in one or more containers, each container representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing a machine learning model. In at least one embodiment, a container or application can be published to a private (e.g., limited access) area of ​​a container registry (described in more detail herein), and trained or deployed models can be stored in model registry 1124 and associated with one or more applications. In at least one embodiment, an image of the application (e.g., a container image) can be used in the container registry, and once a user selects an image from the container registry for deployment in the pipeline, that image can be used to generate containers for instantiation of the application for use by the user's system.

[0163] In at least one embodiment, the developer can develop, publish, and store an application (e.g., as a container) for performing processing and / or inference on the provided data. In at least one embodiment, a software development kit (SDK) associated with the system can be used to perform development, publication, and / or storage (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application can be tested locally using the SDK (e.g., at a first facility, on data from a first facility), the SDK serving as a system (e.g.,...). Figure 12 System 1200 may support at least some services 1120. In at least one embodiment, once verified by system 1200 (e.g., for accuracy, etc.), the application becomes available in the container registry for users (e.g., hospitals, clinics, laboratories, healthcare providers, etc.) to select and / or implement one or more processing tasks on data at the user's facility (e.g., a second facility).

[0164] In at least one embodiment, the developer can then share the application or container over a network for the system (e.g., Figure 12 The system 1100 allows user access to and use of the data processing pipeline. In at least one embodiment, a completed and validated application or container may be stored in a container registry, and the associated machine learning model may be stored in a model registry 1124. In at least one embodiment, a requesting entity (which provides an inference or image processing request) may browse the container registry and / or model registry 1124 for applications, containers, datasets, machine learning models, etc., select the desired combination of elements to be included in the data processing pipeline, and submit a processing request. In at least one embodiment, the request may include input data necessary to execute the request, and / or may include a selection of the application and / or machine learning model to be executed when the request is processed. In at least one embodiment, the request may then be passed to one or more components of the deployment system 1106 (e.g., the cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 1106 may include referencing elements (e.g., applications, containers, models, etc.) selected from the container registry and / or model registry 1124. In at least one embodiment, once results are generated through the pipeline, the results may be returned to the user for reference (e.g., for viewing in a suite of viewing applications executed on a local machine, local workstation, or terminal).

[0165] In at least one embodiment, service 1120 may be utilized to assist in processing or executing applications or containers in the pipeline. In at least one embodiment, service 1120 may include computing services, collaborative content creation services, simulation services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, service 1120 may provide functionality common to one or more applications in software 1118, thus abstracting functionality into services that can be invoked or utilized by applications. In at least one embodiment, the functionality provided by service 1120 can operate dynamically and more efficiently, while also allowing applications to process data in parallel (e.g., using...). Figure 12 The parallel computing platform 1230 in the system can be scaled well. In at least one embodiment, it is not required that every application providing the same functionality as the shared service 1120 must have a corresponding instance of the service 1120, but rather the service 1120 can be shared between and among various applications. In at least one embodiment, as a non-limiting example, the service may include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included, which can provide the ability to train and / or retrain machine learning models.

[0166] In at least one embodiment, where service 1120 includes an AI service (e.g., an inference service), as part of application execution, one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) can be executed by invoking (e.g., as an API call) the inference service (e.g., an inference server) to execute one or more machine learning models or their processing. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, the application can invoke the inference service to execute the machine learning models for performing one or more processing operations associated with the segmentation task. In at least one embodiment, the software 1118 implementing the high-level processing and inference pipeline can be pipelined because each application can invoke the same inference service to execute one or more inference tasks.

[0167] In at least one embodiment, hardware 1122 may include a GPU, CPU, data processing unit (DPU), AI / deep learning system (e.g., an AI supercomputer, such as NVIDIA's DGX). TM(Supercomputer system), cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1122 may be used to provide efficient, specially built support for software 1118 and services 1120 in deployment system 1106. In at least one embodiment, GPU processing may be used to perform local processing (e.g., at facility 1102) within the AI / deep learning system, in the cloud system, and / or in other processing components of deployment system 1106 to improve the efficiency, accuracy, and performance of game name recognition.

[0168] In at least one embodiment, as a non-limiting example, regarding deep learning, machine learning and / or high-performance computing, simulation and visual computing, software 1118 and / or service 1120 may be optimized for GPU processing. In at least one embodiment, at least some of the computing environment in which system 1106 is deployed and / or training system 1104 can be in software with GPU optimization (e.g., NVIDIA DGX). TM The system's hardware and software combination is executed in a data center or one or more supercomputers or high-performance computing systems. In at least one embodiment, as described herein, hardware 1122 may include any number of GPUs that can be invoked to perform data processing in parallel. In at least one embodiment, the cloud platform may also include GPU-optimized execution for deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, AI / deep learning supercomputers and / or GPU-optimized software (e.g., such as NVIDIA's DGX) may be used. TM The system provides a hardware abstraction and scaling platform to execute cloud platforms (e.g., NVIDIA's NGC). TM In at least one embodiment, the cloud platform can integrate application container cluster systems or coordination systems (e.g., KUBERNETES) across multiple GPUs to achieve seamless scaling and load balancing.

[0169] Figure 12 This is a system diagram of an example system 1200 for generating and deploying a deployment pipeline according to at least one embodiment. In at least one embodiment, system 1200 can be used to implement Figure 11 The process 1100 and / or other processes include advanced processing and inference pipelines. In at least one embodiment, system 1200 may include training system 1104 and deployment system 1106. In at least one embodiment, training system 1104 and deployment system 1106 may be implemented using software 1118, service 1120 and / or hardware 1122, as described herein.

[0170] In at least one embodiment, system 1200 (e.g., training system 1104 and / or deployment system 1106) may be implemented in a cloud computing environment (e.g., using cloud 1226). In at least one embodiment, system 1200 may be implemented locally (regarding facilities) or as a combination of cloud computing resources and local computing resources. In at least one embodiment, access to the API in cloud 1226 may be restricted to authorized users by establishing security measures or protocols. In at least one embodiment, the security protocol may include a network token, which may be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and may carry appropriate authorization. In at least one embodiment, the API of the virtual instrument (described herein) or other instances of system 1200 may be restricted to a set of public Internet Service Providers (ISPs) that have been audited or authorized for interaction.

[0171] In at least one embodiment, the various components of system 1200 may communicate with and among each other using any of a variety of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communication between facilities and components of system 1200 (e.g., for sending inference requests, for receiving the results of inference requests, etc.) may be transmitted via one or more data buses, wireless data protocols (e.g., Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0172] In at least one embodiment, similar to the description herein. Figure 11 As described, training system 1104 can execute training pipeline 1204. In at least one embodiment, where deployment system 1106 uses one or more machine learning models in deployment pipeline 1210, training pipeline 1204 can be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1206 (e.g., without retraining or updating). In at least one embodiment, as a result of training pipeline 1204, output model 1116 can be generated. In at least one embodiment, training pipeline 1204 can include any number of processing steps, AI-assisted annotation 1110, labeling or annotating feedback data 1108 to generate labeled data 1112, selecting a model from a model registry, model training 1114, training, retraining, or updating the model, and / or other processing steps. In at least one embodiment, DICOM adapter 1202a can be used to access DICOM data. In at least one embodiment, different training pipelines 1204 can be used for different machine learning models used by deployment system 1106. In at least one embodiment, similar to the... Figure 11 The training pipeline 1204 described in the first example can be used for the first machine learning model, similar to the one described above. Figure 11 The training pipeline 1204 described in the second example can be used for a second machine learning model, similar to that described above. Figure 11 The training pipeline 1204 of the third example described can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1104 can be used according to the requirements of each respective machine learning model. In at least one embodiment, one or more machine learning models may have already been trained and are ready for deployment, so the training system 1104 may not perform any processing on the machine learning models, and the machine learning models may be implemented by the deployment system 1106.

[0173] In at least one embodiment, one or more output models 1116 and / or pre-trained models 1206 may comprise any type of machine learning model. In at least one embodiment, and not limited thereto, the machine learning model used by system 1200 may include models using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recursion, perceptrons, long / short-term memory (LSTM), Bi-LSTM, Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0174] In at least one embodiment, training pipeline 1204 may include AI-assisted annotation. In at least one embodiment, labeled data 1112 (e.g., conventional annotation) can be generated using any number of techniques. In at least one embodiment, in some examples, labels or other annotations can be generated by drawing programs (e.g., annotation programs), computer-aided design (CAD) programs, tagging programs, another type of application suitable for generating annotations or labels for ground reality, and / or can be hand-drawn. In at least one embodiment, ground reality data can be synthetically generated (e.g., generated from computer models or renderings), realistically generated (e.g., designed and generated from real-world data), automatically generated by machines (e.g., extracting features from data using feature analysis and learning, and then generating labels), manually annotated (e.g., taggers or annotation experts, defining the placement of labels), and / or combinations thereof. In at least one embodiment, for each instance of feedback data 1108 (or other data types used by the machine learning model), there may be corresponding ground reality data generated by training system 1104. In at least one embodiment, AI-assisted annotation can be performed as part of deployment pipeline 1210; supplementing or replacing AI-assisted annotation included in training pipeline 1204. In at least one embodiment, system 1200 may include a multi-layer platform, which may include a software layer (e.g., software 1118) of a diagnostic application (or other application type) that can perform one or more medical imaging and diagnostic functions.

[0175] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or certified API that can invoke (e.g., call) an application or container from an external environment (e.g., facility 1102). In at least one embodiment, the application may then invoke or execute one or more services 1120 to perform computational, AI, or visualization tasks associated with their respective applications, and the software 1118 and / or services 1120 may utilize the hardware 1122 to perform processing tasks efficiently and effectively.

[0176] In at least one embodiment, deployment system 1106 may execute deployment pipeline 1210. In at least one embodiment, deployment pipeline 1210 may include any number of applications, which may be sequential, non-sequential, or otherwise applied to feedback data (and / or other data types) – including AI-assisted annotations, as described above. In at least one embodiment, as described herein, deployment pipeline 1210 for an individual device may be referred to as a virtual instrument for the device. In at least one embodiment, for a single device, there may be more than one deployment pipeline 1210, depending on the desired information from the data generated from the device.

[0177] In at least one embodiment, the applications that can be used to deploy pipeline 1210 may include any application that can perform processing tasks on feedback data or other data from the device. In at least one embodiment, since various applications can share common image operations, in some embodiments, data augmentation libraries (e.g., as one of services 1120) may be used to accelerate these operations. In at least one embodiment, to avoid bottlenecks in traditional processing methods that rely on CPU processing, parallel computing platform 1230 may be used for GPU acceleration of these processing tasks.

[0178] In at least one embodiment, deployment system 1106 may include user interface 1214 (e.g., graphical user interface, web interface, etc.) which may be used to select applications to be included in deployment pipeline 1210, deploy applications, modify or change applications or their parameters or configurations, use and interact with deployment pipeline 1210 during setup and / or deployment, and / or otherwise interact with deployment system 1106. In at least one embodiment, although not shown with respect to training system 1104, UI 1214 (or different user interfaces) may be used to select models to be used in deployment system 1106, to select models to be trained or retrained in training system 1104, and / or to otherwise interact with training system 1104.

[0179] In at least one embodiment, in addition to the application coordination system 1228, a pipeline manager 1212 may be used to manage interactions between applications or containers deployed through the pipeline 1210 and services 1120 and / or hardware 1122. In at least one embodiment, the pipeline manager 1212 may be configured to facilitate interactions from application to application, from application to service 1120, and / or from application or service to hardware 1122. In at least one embodiment, although shown as included in software 1118, this is not intended to be limiting, and in some examples, the pipeline manager 1212 may be included within service 1120. In at least one embodiment, the application coordination system 1228 (e.g., Kubernetes, DOCKER, etc.) may include a container coordination system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline 1210 (e.g., rebuilding applications, splitting applications, etc.) with individual containers, each application can execute in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0180] In at least one embodiment, each application and / or container (or its image) can be developed, modified, and deployed independently (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer). This allows focus on the tasks of a single application and / or container without being hindered by the tasks of other applications or containers. In at least one embodiment, the pipeline manager 1212 and the application coordination system 1228 can facilitate communication and collaboration between different containers or applications. In at least one embodiment, the application coordination system 1228 and / or the pipeline manager 1212 can facilitate communication and resource sharing between and within each application or container, provided that the expected inputs and / or outputs of each container or application are known to the system (e.g., based on the construction of the application or container). In at least one embodiment, since one or more applications or containers in the deployment pipeline 1210 can share the same services and resources, the application coordination system 1228 can coordinate, load balance, and determine the sharing of services or resources between and within the various applications or containers. In at least one embodiment, the scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. Therefore, in at least one embodiment, taking into account the system's needs and availability, the scheduler can allocate resources to different applications and distribute resources between and among applications. In some examples, the scheduler (and / or other components of the application coordination system 1228) can determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as Quality of Service (QoS), the urgency of data output (e.g., to determine whether to perform real-time processing or delayed processing), etc.

[0181] In at least one embodiment, service 1120, utilized and shared by applications or containers in deployment system 1106, may include computing service 1216, collaborative content creation service 1217, AI service 1218, simulation service 1219, visualization service 1220, and / or other service types. In at least one embodiment, an application may invoke (e.g., execute) one or more services 1120 to perform processing operations for the application. In at least one embodiment, an application may utilize computing service 1216 to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more computing services 1216 may be utilized to perform parallel processing (e.g., using parallel computing platform 1230) to process data substantially simultaneously through one or more applications and / or one or more tasks of a single application. In at least one embodiment, parallel computing platform 1230 (e.g., NVIDIA's...) General-purpose computing can be implemented on a GPU (GPGPU) (e.g., GPU / graphics 1222). In at least one embodiment, the software layer of the parallel computing platform 1230 can provide access to the GPU's virtual instruction set and parallel computing elements to execute a computing kernel. In at least one embodiment, the parallel computing platform 1230 may include memory, and in some embodiments, memory may be shared between and within multiple containers, and / or between and within different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls can be generated for multiple containers and / or multiple processes within containers to use the same data from a shared memory segment of the parallel computing platform 1230 (e.g., where multiple different stages of one or more applications are processing the same information). In at least one embodiment, instead of copying data and moving it to different locations in memory (e.g., read / write operations), the same data in the same location in memory can be used for any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, this information about the new location of the data can be stored and shared between various applications because the data as a result of processing is used to generate new data. In at least one embodiment, the location of the data, and the location of the updated or modified data, can be part of the definition of how the payload in the container is understood.

[0182] In at least one embodiment, AI service 1218 may be used to perform an inference service for executing a machine learning model associated with the application (e.g., a task to perform one or more processing tasks of the application). In at least one embodiment, AI service 1218 may utilize AI system 1224 to execute a machine learning model (e.g., a neural network such as a CNN) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, the application deploying pipeline 1210 may use one or more output models 1116 from training system 1104 and / or other models from the application to perform inference on imaging data (e.g., DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.). For example, DICOM adapter 1202b may be used to access DICOM data. In at least one embodiment, two or more examples of inference using application coordination system 1228 (e.g., a scheduler) may be available. In at least one embodiment, a first category may include a high-priority / low-latency path that can implement a higher service level protocol, such as for performing inference on urgent requests in emergency situations or for radiologists during diagnostic procedures. In at least one embodiment, the second category may include standard priority paths, which can be used for requests that may not be urgent or for situations where analysis can be performed at a later time. In at least one embodiment, the application coordination system 1228 may allocate resources (e.g., services 1120 and / or hardware 1122) based on priority paths for different inference tasks of the AI ​​service 1218.

[0183] In at least one embodiment, shared memory may be installed into AI service 1218 in system 1200. In at least one embodiment, shared memory may operate as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a set of API instances of deployment system 1106 may receive the request and may select one or more instances (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request may be fed into a database, and if not already in the cache, a machine learning model may be located from model registry 1124. A verification step may ensure that an appropriate machine learning model is loaded into the cache (e.g., shared memory), and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not already running or there are not enough instances of the application, a scheduler (e.g., the scheduler of pipeline manager 1212) may be used to start the application referenced in the request. In at least one embodiment, if an inference server has not yet been started to execute the model, an inference server may be started. In at least one embodiment, any number of inference servers may be started for each model. In at least one embodiment, in a pull model that clusters inference servers, the model can be cached whenever load balancing is favorable. In at least one embodiment, the inference servers can be statically loaded into the corresponding distributed servers.

[0184] In at least one embodiment, an inference server running in a container can be used to perform inference. In at least one embodiment, an instance of the inference server can be associated with a model (and optionally multiple versions of the model). In at least one embodiment, if an instance of the inference server does not exist when a request to perform inference on the model is received, a new instance can be loaded. In at least one embodiment, when the inference server is started, a model can be passed to the inference server, allowing the same container to be used to serve different models, as long as the inference server runs as different instances.

[0185] In at least one embodiment, during application execution, an inference request for a given application can be received, and a container (e.g., an instance of a hosted inference server) can be loaded (if not already loaded), and a launcher can be invoked. In at least one embodiment, preprocessing logic within the container can (e.g., using a CPU and / or GPU) load, decode, and / or perform any additional preprocessing on the incoming data. In at least one embodiment, once the data is ready for inference, the container can infer the data as needed. In at least one embodiment, this can include a single inference call for an image (e.g., a hand X-ray) or can request inference for hundreds of images (e.g., a chest CT scan). In at least one embodiment, the application can summarize the results before completion, which may include, but is not limited to, a single confidence score, pixel-level segmentation, voxel-level segmentation, generating visualizations, or generating text to summarize the results. In at least one embodiment, different priorities can be assigned to different models or applications. For example, some models may have a real-time (turnaround time less than 1 minute) priority, while other models may have a lower priority (e.g., turnaround time less than 10 minutes). In at least one embodiment, model execution time can be measured from the requesting agency or entity, and may include cooperative network traversal time and inference service execution time.

[0186] In at least one embodiment, the transfer of requests between service 1120 and the inference application can be hidden behind a software development kit (SDK) and robust transfer can be provided via queues. In at least one embodiment, requests are placed in queues via an API for individual application / tenant ID combinations, and the SDK pulls requests from the queues and provides them to the application. In at least one embodiment, the name of the queue can be provided in the environment where the SDK picks up requests. In at least one embodiment, asynchronous communication via queues may be useful because it allows any instance of the application to pick up work as it becomes available. In at least one embodiment, results can be sent back via queues to ensure no data loss. In at least one embodiment, queues can also provide the ability to partition work, as the highest priority work can go into a queue connected to a majority of instances of the application, while the lowest priority work can go into a queue connected to a single instance that processes tasks in the order they are received. In at least one embodiment, the application can run on a GPU-accelerated instance generated in cloud 1226, and the inference service can perform inference on the GPU.

[0187] In at least one embodiment, visualization service 1220 can be used to generate visualizations for viewing the output of application and / or deployment pipeline 1210. In at least one embodiment, visualization service 1220 can utilize GPU / graphics 1222 to generate visualizations. In at least one embodiment, visualization service 1220 can implement rendering effects such as ray tracing or other light transport simulation techniques to generate higher quality visualizations. In at least one embodiment, visualizations can include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slicing, virtual reality display, augmented reality display, etc. In at least one embodiment, a virtualized environment can be used to generate virtual interactive displays or environments (e.g., virtual environments) for system users (e.g., doctors, nurses, radiologists, etc.) to interact with. In at least one embodiment, visualization service 1220 can include an internal visualizer, cinematic and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0188] In at least one embodiment, hardware 1122 may include a GPU / graphics 1222, an AI system 1224, a cloud 1226, and / or any other hardware for executing training system 1104 and / or deployment system 1106. In at least one embodiment, the GPU / graphics 1222 (e.g., NVIDIA's...) The GPU / graphics 1222 may include any number of GPUs that can be used to perform processing tasks of any feature or function of computing service 1216, collaborative content creation service 1217, AI service 1218, simulation service 1219, visualization service 1220, other services, and / or software 1118. For example, for AI service 1218, GPU / graphics 1222 may be used to perform preprocessing on imaging data (or other data types used by machine learning models), postprocessing on the output of machine learning models, and / or to perform inference (e.g., to execute machine learning models). In at least one embodiment, other components of cloud 1226, AI system 1224, and / or system 1200 may use GPU / graphics 1222. In at least one embodiment, cloud 1226 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1224 may use GPUs, and one or more AI systems 1224 may be used to execute cloud 1226 (or tasks that are at least part of deep learning or inference). Similarly, although hardware 1122 is shown as a discrete component, this is not intended to be limiting, and any component of hardware 1122 can be combined with or utilized by any other component of hardware 1122.

[0189] In at least one embodiment, the AI ​​system 1224 may include a specially built computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, in addition to a CPU, RAM, memory, and / or other components, features, or functions, the AI ​​system 1224 (e.g., NVIDIA's DGX) TM It may also include software (e.g., a software stack) that can use multiple GPUs / graphics 1222 to perform GPU-optimized tasks. In at least one embodiment, one or more AI systems 1224 may be implemented in the cloud 1226 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1200.

[0190] In at least one embodiment, cloud 1226 may include GPU-accelerated infrastructure (e.g., NVIDIA's NGC). TM This can provide a GPU-optimized platform for performing processing tasks of system 1200. In at least one embodiment, cloud 1226 may include AI system 1224 for performing one or more AI-based tasks of system 1200 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1226 may be integrated with application coordination system 1228 utilizing multiple GPUs to achieve seamless scaling and load balancing between and within applications and services 1120. In at least one embodiment, as described herein, cloud 1226 may be responsible for performing at least some of the services 1120 of system 1200, including computing service 1216, AI service 1218, and / or visualization service 1220. In at least one embodiment, cloud 1226 may perform large and small batch inference (e.g., performing NVIDIA TensorRT). TM ), providing an accelerated parallel computing platform 1230 (e.g., NVIDIA's), The system 1228 executes an application coordination system (e.g., KUBERNETES), provides graphics rendering APIs and platforms (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality cinematic effects), and / or may provide other functionalities for the system 1200. In at least one embodiment, the parallel computing platform 1230 may include APIs.

[0191] In at least one embodiment, to protect patient confidentiality (e.g., in cases of off-site use of patient data or records), cloud 1226 may include a registry—e.g., a deep learning container registry. In at least one embodiment, the registry may store containers for instantiating applications that can perform preprocessing, post-processing, or other processing tasks on patient data. In at least one embodiment, cloud 1226 may receive data, including patient data and sensor data in containers, perform the requested processing only on the sensor data in those containers, and then forward the output and / or visualization to appropriate parties and / or devices (e.g., local medical devices for visualization or diagnosis), without extracting, storing, or otherwise accessing the patient data. In at least one embodiment, the confidentiality of patient data is maintained in accordance with HIPAA and / or other data regulations.

[0192] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0193] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar references, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (referring to a physical connection where not modified) should be interpreted as partially or wholly contained, attached to, or joined together, even with some intervention. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0194] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” means multiple items). In at least one embodiment, the number of items in the multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” or “at least based on” rather than “based on only”.

[0195] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is executed jointly on one or more processors via hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within transient signal transceivers. In at least one embodiment, code (e.g., executable code or source code) is stored on one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media lack all the code, but the multiple non-transitory computer-readable storage media collectively store all the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0196] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the operations of the processes described herein, either individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating in different ways, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0197] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not constitute a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0198] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference as if each reference were individually and specifically indicated to be incorporated herein by reference and the entire contents of which are described herein.

[0199] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0200] Unless otherwise expressly stated, in some embodiments, it will be understood that throughout the specification, terms such as “processing,” “computing,” “determining,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that process and / or convert data represented as physical quantities (e.g., electrons) in the registers and / or memory of the computing system into other data represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0201] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or intermittently, sequentially, or in parallel. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0202] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an application programming interface, or an inter-process communication mechanism.

[0203] While the description herein illustrates exemplary embodiments of the described technologies, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for descriptive purposes, various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0204] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A method comprising: obtaining a first record lock; generating one or more records for consumption by one or more consumer processes; storing the one or more records in one or more queues, wherein a first queue is associated with a first consumer process of the one or more consumer processes and at least a first value associated with a first record of the one or more records is stored in the first queue; in response to receiving a first signal from the first consumer process, unsetting the first value associated with the first record from the first queue; and releasing the first record lock.

2. The method of claim 1, wherein: the one or more queues comprise the first queue associated with the first consumer process and a counter queue; the first queue associated with the first consumer process is a record-index queue; and storing the one or more records in the one or more queues comprises: storing a counter value corresponding to a count of the one or more records in the counter queue; and storing the first record in the first queue, wherein the first record is the first value associated with the first record. storing, in conjunction with storing the first record in the first queue, a counter index associated with the counter value in the counter queue. in response to also receiving the first signal from the first consumer process, decrementing the counter value in the counter queue.

3. The method of claim 2, wherein storing the first record in the first queue further comprises:

5. The method of claim 2, wherein the first record lock corresponds to the counter value in the counter queue; and wherein releasing the first record lock is performed in response to the counter value being equal to a lock release value.

4. The method of claim 2, further comprising:

6. The method of claim 2, wherein a length of the first queue is at least twice a maximum active entry value of the first queue.

7. The method of claim 2, wherein the first consumer process is associated with an array of nodes, and wherein the length of the first queue is based on a maximum number of records that the first consumer process is capable of receiving and a maximum number of producer processes that are capable of being concurrently executed.

8. The method of claim 2, wherein a length of the counter queue is based on a maximum number of producer processes that are capable of being concurrently executed.

9. The method of claim 1, wherein: the one or more queues comprise the first queue associated with the first consumer process and a unified record queue; the first queue associated with the first consumer process is a first index queue; and storing the one or more records in the one or more queues comprises: storing the first record in the unified record queue; and storing an index of the first record in the first queue, wherein the index of the first record is the first value associated with the first record. ​ ​ ​ ​ 10. The method of claim 9, wherein the first record lock corresponds to a count of the one or more records; and wherein releasing the first record lock is performed in response to the one or more records being de-queued from the unified record queue.

11. The method of claim 9, wherein the first queue associated with the first consumer process is a first record queue.

12. The method of claim 9, wherein a length of the first queue is at least twice a maximum active entry value of the first queue.

13. The method of claim 9, wherein the first consumer process is associated with an array of nodes, and wherein a length of at least one of the first queue or the unified record queue is based on a sum of a maximum number of records that a subset of the one or more consumer processes can receive and a maximum number of producer processes that can be executed concurrently.

14. A system comprising: a memory that stores one or more queues; and processing circuitry coupled to the memory, the processing circuitry to: obtain a first record lock; receive one or more records for consumption by one or more consumer processes; store the one or more records in the one or more queues, wherein a first queue is associated with a first consumer process of the one or more consumer processes and at least a first value associated with a first record of the one or more records is stored in the first queue; in response to receiving a first signal from the first consumer process, de-queue the first value associated with the first record from the first queue; and release the first record lock.

15. The system of claim 14, wherein: the one or more queues include the first queue associated with the first consumer process and a counter queue; the first queue associated with the first consumer process is a record-index queue; and to store the one or more records in the one or more queues, the processing circuitry is to: store a counter value corresponding to a count of the one or more records in the counter queue; and store the first record in the first queue, wherein the first record is the first value associated with the first record.

16. The system of claim 15, wherein to store the first record in the first queue, the processing circuitry is further to store, in conjunction with storing the first record in the first queue, a counter index associated with the counter value in the counter queue.

17. The system of claim 15, wherein the processing circuitry is further to decrement the counter value in the counter queue in response to receiving the first signal from the first consumer process. ​ ​ 18. The system of claim 15, wherein the first record lock corresponds to the counter value in the counter queue; and wherein releasing the first record lock is performed in response to the counter value being equal to a lock release value.

19. The system of claim 15, wherein a length of the first queue is at least twice a maximum active entry value of the first queue.

20. The system of claim 14, wherein: the one or more queues include the first queue associated with the first consumer process and a unified record queue; the first queue associated with the first consumer process is a first index queue; and to store the one or more records in the one or more queues, the processing circuitry is to: store the first record in the unified record queue; and store an index of the first record in the first queue, wherein the index of the first record is the first value associated with the first record.

21. The system of claim 20, wherein the first record lock corresponds to a count of the one or more records; and wherein releasing the first record lock is performed in response to the one or more records being de-queued from the unified record queue.

22. The system of claim 20, wherein the first queue associated with the first consumer process is a first record queue.

23. A system comprising: a first processor; a second processor to generate one or more records for access by the first processor; and processing circuitry coupled to the first processor and the second processor, the processing circuitry to: obtain a first record lock; store the one or more records in one or more queues, wherein a first queue is associated with a first consumer process executed by the first processor, and at least a first value associated with a first record of the one or more records is stored in the first queue; in response to receiving a first signal from the first consumer process, de-queue the first value associated with the first record from the first queue; and release the first record lock.

24. The system of claim 23, wherein: the one or more queues include the first queue associated with the first consumer process and a counter queue; the first queue associated with the first consumer process is a record-index queue; and to store the one or more records in the one or more queues, the processing circuitry is to: store a counter value corresponding to a count of the one or more records in the counter queue; and store the first record in the first queue, wherein the first record is the first value associated with the first record.

Images