In-switch embedding bag pooling
The computing system addresses the memory bandwidth issues in neural network-based recommendation systems by having a switch perform reduction operations on embedding rows, thereby reducing power consumption and enhancing performance.
WO2025071703A1 Publication Date: 2025-04-03ADVANCED MICRO DEVICES INC
1 Cites 2 Cited by
Patent Information
- Application Number
- PCT/US2024/033415
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-09-29
- Filing Date
- 2024-06-11
- Publication Date
- 2025-04-03
AI Technical Summary
Technical Problem
The large number of embedding tables and their sizes in neural network-based recommendation systems lead to increased memory bandwidth requirements, reducing system performance and increasing power consumption.
Method used
A computing system with a switch that performs reduction operations on embedding rows stored in local memories, reducing the need for processors to handle these operations and thereby decreasing memory bandwidth requirements and power consumption.
Benefits of technology
By offloading reduction operations from processors to a switch, the system reduces memory bandwidth usage, lowers power consumption, and enhances overall performance by allowing processors to focus on other tasks.
✦ Generated by Eureka AI based on patent content.
Abstract
An apparatus and method for reducing the memory bandwidth of executing machine learning models. A computing system includes two or more processing nodes, each including at least one or more processors and a corresponding local memory. Switch circuitry communicates with at least the local memories and a system memory of the computing system. The switch includes multiple direct memory access (DMA) interfaces. Each of one or more processing nodes stores multiple embedding rows of embedding tables. A processor of the processing node identifies two or more embedding rows as source operands of a reduction operation. The switch executes memory access requests to retrieve data of the two or more embedding rows from the corresponding local memory, and generates a result by performing the reduction operation. The switch sends the result to the local memory.
Need to check novelty before this filing date? Find Prior Art