System, processing device and method of operation of a processing device for running a pre-treatment
By introducing a base address register and DMA technology between the central processing unit and the processing device, the processing device independently performs preprocessing operations and data transmission, solving the problems of data processing latency and low energy efficiency in multimodal artificial intelligence systems, and realizing efficient data pipeline construction and performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SK HYNIX INC
- Filing Date
- 2025-10-23
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies suffer from long data processing latency and low energy efficiency when dealing with big data, especially in multimodal artificial intelligence systems, where preprocessing operations often lead to increased complexity in the data pipeline and performance degradation.
By introducing a base address register and direct memory access (DMA) between the central processing unit and the processing device, the processing device can independently perform preprocessing operations and data transfers, reducing the intervention of the central processing unit and enabling efficient construction of data pipelines and improved energy efficiency.
It shortens data processing latency, simplifies the data pipeline, improves the overall performance and energy efficiency of the system, and reduces unnecessary data transmission overhead.
Smart Images
Figure CN122152736A_ABST
Abstract
Description
[0001] Cross-reference to related applications
[0002] This application claims priority to Korean Patent Application No. 10-2024-0177554, filed on December 3, 2024, and Korean Patent Application No. 10-2025-0055112, filed on April 28, 2025, which are incorporated herein by reference in their entirety. Technical Field
[0003] Embodiments of this disclosure relate to a system for performing preprocessing, a processing apparatus, and a method of operating the processing apparatus. Background Technology
[0004] With the advancement of artificial intelligence technology, people are actively researching systems that use artificial intelligence to process various types of data.
[0005] In particular, the growth potential of multimodal AI systems for processing large amounts of data such as video, audio, and images has attracted considerable attention in recent years, and research in this area is actively underway. Multimodal AI systems require the transformation of large amounts of data into formats that AI can process, and the demand for technologies that efficiently perform such transformations is also increasing. Summary of the Invention
[0006] The embodiments of this disclosure aim to provide a system, processing apparatus, and method of operating the processing apparatus, which can efficiently perform operations of preprocessing large data and transmitting preprocessed data, thereby shortening data processing latency, facilitating the construction of data pipelines, and improving energy efficiency.
[0007] The purposes of the embodiments disclosed herein are not limited to those described herein, and other unmentioned purposes will become apparent to those skilled in the art from the following description.
[0008] In one embodiment, a system may include: a central processing unit (CPU); a first processing device including a first processor, a first base address register, and a first memory; and a second processing device including a second processor, a second base address register, and a second memory. The CPU may store preprocessing information in the first base address register, which requests the first processing device to perform preprocessing operations on raw data stored in the first memory. The first processing device may read raw data from the first memory, perform preprocessing operations on the raw data using the preprocessing information to generate result data, and may transfer the result data to the second processing device without intervention from the CPU.
[0009] In one embodiment, a processing device may include: a processor; a base address register; and a memory. The processor can read raw data from the memory, read preprocessing information from the base address register, use the preprocessing information to perform preprocessing operations on the raw data to generate result data, and can directly transmit the result data to a target device outside the processing device.
[0010] In an embodiment, a method of operating a processing device may include: reading raw data from a memory, reading preprocessing information from a base address register; performing preprocessing operations on the raw data using the preprocessing information to generate result data; and directly transmitting the result data to an external device.
[0011] According to embodiments of this disclosure, a system, processing apparatus, and method of operating the processing apparatus can be provided, which can efficiently perform operations of preprocessing big data and transmitting preprocessed data, thereby shortening data processing latency, facilitating the construction of data pipelines, and improving energy efficiency.
[0012] The effects of this disclosure are not limited to the foregoing objectives, and other objectives will be apparent to those skilled in the art from the following detailed description. Attached Figure Description
[0013] This disclosure will be more fully understood through the following detailed description and accompanying drawings, which are provided for illustrative purposes only and are not intended to limit the scope of this disclosure.
[0014] Figure 1 It is a schematic configuration diagram of the system based on this disclosure.
[0015] Figure 2 This is a diagram illustrating the operation of storing preprocessed information in a first processing device by a central processing unit according to the present disclosure.
[0016] Figure 3 This is a diagram illustrating the operation of generating result data according to the first processing apparatus of this disclosure.
[0017] Figure 4 This is a diagram illustrating the operation of storing mapping information in a first processing device and a second processing device according to the central processing unit of this disclosure.
[0018] Figure 5 This is a diagram illustrating an example of mapping information according to this disclosure.
[0019] Figure 6 This is a diagram illustrating the operation of transmitting result data from the first processing device to the second processing device according to the present disclosure.
[0020] Figure 7This is a diagram illustrating a method of operating a processing apparatus according to an embodiment of the present disclosure. Detailed Implementation
[0021] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. Throughout this specification, references to "embodiment," "another embodiment," etc., do not necessarily refer to only one embodiment, and different references to any such phrases are not necessarily limited to the same embodiment. When used herein, the term "embodiment" does not necessarily refer to all embodiments.
[0022] Various embodiments of the present disclosure are described in more detail below with reference to the accompanying drawings. However, the present disclosure may be implemented in different forms and variations and should not be construed as limited to the embodiments set forth herein. Rather, the described embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the contents of the disclosure to those skilled in the art to which this disclosure pertains. Throughout the disclosure, the same reference numerals are used throughout the various figures and embodiments to refer to the same parts.
[0023] The methods, processes, and / or operations described herein can be executed by code or instructions that run on a computer, processor, controller, or other signal processing device. The computer, processor, controller, or other signal processing device can be those devices described herein, or elements other than those described herein. Because the algorithms underlying the methods (or the operations of the computer, processor, controller, or other signal processing device) are described in detail, the code or instructions for operations of embodiments of the methods can convert a computer, processor, controller, or other signal processing device into a dedicated processor for performing the methods herein.
[0024] When implemented at least in part as software, controllers, processors, devices, modules, units, multiplexers, logic, interfaces, decoders, drivers, generators, and other signal generation and signal processing features may include, for example, memory or other storage devices for storing code or instructions that will be executed by, for example, a computer, processor, microprocessor, controller, or other signal processing device.
[0025] Figure 1 It is a schematic configuration diagram of system 100 according to this disclosure.
[0026] Reference Figure 1 The system 100 may include a central processing unit 110, a first processing device 120, and a second processing device 130.
[0027] The central processing unit 110 can control the first processor device 120 and the second processor device 130. The central processing unit 110 can perform predefined logical calculations to control the first processor device 120 and the second processor device 130. The central processing unit 110 can generate commands and signals for controlling the first processor device 120 and the second processor device 130.
[0028] The central processing unit 110 may additionally include internal memory for storing data required to perform the above-described logical calculations.
[0029] The first processing device 120 may include a first processor PU1, a first base address register BAR1, and a first memory MEM1.
[0030] The first memory MEM1 can be a volatile or non-volatile memory (e.g., SRAM, DRAM, or NAND) capable of storing data.
[0031] The first base address register BAR1 is a register that stores memory mapping information between the first processing device 120 and the system 100. Using the memory mapping information stored in the first base address register BAR1, the first processing device 120 can directly transfer data to other devices included in the system 100 (e.g., the second processing device 130) without the intervention of the central processing unit 110 (i.e., without the central processing unit taking any action or participating). The device receiving data from the first processing device 120 can also be referred to as the target device. Specifically, the first processing device 120 can transfer data to other devices included in the system 100 (e.g., the second processing device 130) via direct memory access (DMA).
[0032] The first base address register BAR1 can be the base address register defined in the High Speed Peripheral Component Interconnect (PCIe) standard.
[0033] The first processor PU1 can perform calculations on data stored in the first memory MEM1. For example, the calculations performed by the first processor PU1 can be defined in firmware, which is stored in the first memory MEM1. As another example, information related to the calculations to be performed by the first processor PU1 can be received from the central processing unit 110.
[0034] The first processing device 120 can be implemented in various ways. For example, the first processing device 120 can be a computing storage device capable of storing data and performing calculations on the stored data.
[0035] The second processing device 130 may include a second processor PU2, a second base address register BAR2, and a second memory MEM2.
[0036] The second processor PU2 can perform computations (e.g., graphics computations or tensor computations) on the data stored in the second memory MEM2. The second processor PU2 can be implemented by a graphics processing unit (GPU), an accelerated processing unit (APU), a digital signal processor (DSP), a neural processing unit (NPU), a tensor processing unit (TPU), a hardware accelerator, or a machine learning accelerator capable of memory mapping.
[0037] The second base address register BAR2 is a register that stores memory mapping information between the second processing device 130 and the system 100. Through the memory mapping information stored in the second base address register BAR2, the second processing device 130 can directly receive data from other devices included in the system 100 (e.g., the first processing device 120) without the intervention of the central processing unit 110 (i.e., without the central processing unit taking any action or participating). Specifically, the second processing device 130 can receive data from other devices included in the system 100 (e.g., the first processing device 120) via direct memory access (DMA).
[0038] Similar to the first base address register BAR1, the second base address register BAR2 can be a base address register defined in the High-Speed Peripheral Component Interconnect (PCIe) standard.
[0039] The second memory MEM2 can store data required for the second processor PU2 to perform calculations. The second memory MEM2 can also store data received from other devices included in the system 100 (e.g., the first processing device 120). For example, the second memory MEM2 can be a volatile memory (e.g., SRAM).
[0040] The second processing device 130 can be implemented in various ways. For example, the second processing device 130 can be a processing device (e.g., a graphics processing device) capable of processing multiple calculations (e.g., graphics calculations) in parallel.
[0041] In embodiments of this disclosure, the central processing unit 110, the first processing device 120, and the second processing device 130 can communicate with each other via a preset interface (e.g., PCIe). For this purpose, the system 100 may additionally include a communication device (e.g., a PCIe switch) connected to the central processing unit 110, the first processing device 120, and the second processing device 130.
[0042] The specific operation of the central processing unit 110, the first processing device 120, and the second processing device 130 will be described below.
[0043] Figure 2This is a diagram illustrating the operation of the central processing unit 110 storing preprocessed information PREP_INFO in the first processing device 120 according to the present disclosure.
[0044] Reference Figure 2 The central processing unit 110 can store the preprocessing information PREP_INFO in the first base address register BAR1. That is, by storing the memory mapping information for DMA and the preprocessing information PREP_INFO for preprocessing operations together in the first base address register BAR1, the central processing unit 110 enables the first processing device 120 to perform preprocessing operations.
[0045] The preprocessing information PREP_INFO is a request from the first processing device 120 to perform preprocessing operations on the raw data ORIG_DATA stored in the first memory MEM1.
[0046] In embodiments of this disclosure, the preprocessing operation performed on the raw data ORIG_DATA refers to the operation of converting the raw data ORIG_DATA from its original format to a different format (e.g., vector) so that the raw data ORIG_DATA is suitable for a given computation (e.g., artificial intelligence computation).
[0047] For example, when the raw data ORIG_DATA is encoded in a video format (e.g., H264 or mpeg), the preprocessing operation on the raw data ORIG_DATA could be to decode the raw data ORIG_DATA and convert it into a preset format.
[0048] The central processing unit 110 can process metadata for the raw data ORIG_DATA to generate preprocessed information PREP_INFO, and then store the preprocessed information PREP_INFO in the first base address register BAR1.
[0049] The raw data ORIG_DATA and the preprocessed information PREP_INFO can be implemented in a variety of ways.
[0050] For example, the raw data ORIG_DATA can be unstructured data. Unstructured data is data without a predefined structure, and therefore is unstructured. In the embodiments of this disclosure, the type of the raw data ORIG_DATA can be video, image, sound, audio, etc.
[0051] Due to the characteristics of unstructured data, artificial intelligence models that use unstructured data need to preprocess the unstructured data.
[0052] For example, the preprocessing information PREP_INFO can be semi-structured data. Semi-structured data is data that includes target data and structural information about the target data (e.g., tags or labels). In embodiments of this disclosure, the preprocessing information PREP_INFO may include the location of the raw data ORIG_DATA and metadata (e.g., resolution, bitrate, and frames per second) for the raw data ORIG_DATA.
[0053] Therefore, by using the structural information included in the semi-structured data, the first processing device 120 can check the preprocessing operation indicated by the preprocessing information PREP_INFO.
[0054] Figure 3 This is a diagram illustrating the operation of generating result data RES_DATA according to the first processing device 120 of this disclosure.
[0055] Reference Figure 3 The first processor PU1 can detect that the central processing unit 110 has stored the preprocessing information PREP_INFO in the first base address register BAR1. The first processor PU1 can perform preprocessing operations based on the detected preprocessing information PREP_INFO. Through the preprocessing information PREP_INFO, the first processor PU1 can obtain the permission from the central processing unit 110 to preprocess the raw data ORIG_DATA.
[0056] For example, the central processing unit 110 can store the preprocessed information PREP_INFO in a preset target field in the first base address register BAR1. When the target field of the first base address register BAR1 is found to have a preset value, the first processor PU1 determines that the preprocessed information PREP_INFO sent from the central processing unit 110 is stored in the first base address register BAR1.
[0057] In another example, when the size of the data written by the central processing unit 110 to the first base address register BAR1 is found to have a preset threshold, the first processor PU1 determines that the preprocessing information PREP_INFO is stored in the first base address register BAR1.
[0058] In another example, after storing the preprocessed information PREP_INFO into the first base address register BAR1, the central processing unit 110 is designed to send a command or signal to the first processor PU1 indicating that the preprocessed information PREP_INFO has been stored in the first base address register BAR1. Upon receiving the command or signal, the first processor PU1 determines that the preprocessed information PREP_INFO has been stored in the first base address register BAR1.
[0059] Based on the preprocessing information PREP_INFO stored in the first base address register BAR1, the first processor PU1 can perform calculations on the preprocessed raw data ORIG_DATA to generate the result data RES_DATA.
[0060] The resulting data RES_DATA can be implemented in several ways. For example, the resulting data RES_DATA can be structured data. Structured data is data stored according to a predetermined structure and format.
[0061] The first processing device 120 can transmit the result data RES_DATA to the second processing device 130.
[0062] As described above, according to this disclosure, the operation of preprocessing the raw data ORIG_DATA can be performed in the first processing device 120 without involving the central processing unit 110 (i.e., without the central processing unit taking any action or participating).
[0063] In contrast, in traditional data processing systems, preprocessing operations are performed by the central processing unit (CPU), which can lead to bottlenecks. For example, a bottleneck may occur when the raw data ORIG_DATA stored in the first processing unit is transferred to the CPU.
[0064] In another example, a bottleneck problem may further arise when the central processing unit performs preprocessing operations and then transmits the results of the preprocessing operations back to the first processing unit.
[0065] Therefore, when using traditional data processing systems, the data pipeline becomes complex during the preprocessing of the raw data ORIG_DATA and the transmission of the preprocessed results. Furthermore, unnecessary data transmission overhead occurs because a central processing unit needs to maintain the entire data pipeline. As a result, system performance deteriorates significantly.
[0066] According to embodiments of this disclosure, the first processing device 120 performs (i) a preprocessing operation on the raw data ORIG_DATA and (ii) a transmission operation on the resulting data RES_DATA to the second processing device 130. In other words, the preprocessing operation on the raw data ORIG_DATA and the transmission operation on the resulting data RES_DATA to the second processing device 130 are performed without the intervention of the central processing unit 110 (i.e., without the central processing unit taking any action or participating). Since the participation of the central processing unit 110 is minimized or absent during the preprocessing operation and the transmission operation of the resulting data RES_DATA, data transmission overhead can be reduced, and as a result, the overall performance of the system 100 is improved.
[0067] Figure 4 This is a diagram illustrating the operation of the central processing unit 110 of this disclosure storing mapping information MAP_INFO in the first processing device 120 and the second processing device 130.
[0068] Reference Figure 4 The central processing unit 110 can store the mapping information MAP_INFO in the first base address register BAR1 of the first processing device 120 and the second base address register BAR2 of the second processing device 130.
[0069] The mapping information MAP_INFO can indicate the mapping between the memory space corresponding to the first processing device 120 and the memory space corresponding to the second processing device 130. Therefore, through the mapping information MAP_INFO, the memory mapping between the first processing device 120 and the second processing device 130 can be realized, and data transmission and reception can be realized through direct memory access (DMA).
[0070] In this structure, the central processing unit 110 does not participate in the data transmission and reception process between the first processing device 120 and the second processing device 130.
[0071] Figure 5 This is a diagram illustrating an example of the mapping information MAP_INFO according to this disclosure.
[0072] The mapping information MAP_INFO can indicate that portions A to (A+k) of the memory space of the first processing device 120 are mapped to portions B to (B+k) of the memory space of the second processing device 130.
[0073] Therefore, by writing data to portions A to (A+k), the first processing device 120 can transfer data to the second processing device 130 via DMA.
[0074] Figure 6 This is a diagram illustrating the operation of the first processing device 120 transmitting result data RES_DATA to the second processing device 130 according to the present disclosure.
[0075] Reference Figure 6 The first processing unit 120 can use the mapping information MAP_INFO to transfer the resulting data RES_DATA to the second processing unit 130 via direct memory access (DMA). Thus, the first processing unit 120 and the second processing unit 130 can be configured with a data pipeline without the intervention of the central processing unit 110 (i.e., without the central processing unit taking any action or participating).
[0076] The second processing device 130 can store the result data RES_DATA in the second memory MEM2. The second processor PU2 of the second processing device 130 can use the result data RES_DATA stored in the second memory MEM2 to perform predefined calculations.
[0077] Figure 7 This is a diagram illustrating a method 700 of an operation processing apparatus according to an embodiment of the present disclosure.
[0078] The processing device may include a processor, a base address register, and memory.
[0079] The method 700 of operating the processing device may include step S710: reading preprocessing information from a base address register, the preprocessing information requesting preprocessing operations on raw data stored in memory.
[0080] The method 700 of operating the processing device may include step S720: preprocessing the raw data based on preprocessing information to generate result data.
[0081] For example, the raw data can be unstructured, while the resulting data can be structured.
[0082] For example, the preprocessed information can be semi-structured data.
[0083] The method 700 of operating the processing device may include step S730: transmitting the generated result data to the target device.
[0084] Step S730 can utilize the mapping information stored in the base address register to transfer the result data to the target device by performing a direct memory access (DMA) operation. The mapping information can indicate the mapping between the memory space corresponding to the processing device and the memory space of the target device.
[0085] The processing device may be as described above. Figure 1 The first processing device 120 is described. The processor may be a first processor PU1, the base address register may be a first base address register BAR1, and the memory may be a first memory MEM1.
[0086] The target device may be as described above. Figure 1 The second processing device 130 described.
[0087] Although exemplary embodiments of this disclosure have been described for illustrative purposes, those skilled in the art will understand that various modifications, additions, and substitutions may be made without departing from the scope and spirit of this disclosure. Therefore, the embodiments disclosed above and in the accompanying drawings should be considered descriptive only and not limiting of the scope of the technology. The technical scope of this disclosure is not limited to the embodiments and the accompanying drawings. The spirit and scope of this disclosure should be interpreted in conjunction with the claims and cover all equivalent solutions falling within the scope of the claims.
Claims
1. A system comprising: Central processing unit; The first processing device includes a first processor, a first base address register, and a first memory; as well as The second processing device includes a second processor, a second base address register, and a second memory. in, The central processing unit stores preprocessing information in the first base address register. This preprocessing information requests the first processing device to perform preprocessing operations on the raw data stored in the first memory. The first processing device reads the raw data from the first memory and performs preprocessing operations on the raw data using the preprocessing information to generate result data. The first processing device transmits the result data to the second processing device without the intervention of the central processing unit.
2. The system according to claim 1, wherein, The first processing device is a computing storage device capable of storing data and performing calculations on the stored data, and The second processing device is a processing device capable of processing multiple calculations in parallel.
3. The system according to claim 1, wherein, The original data is unstructured data. The resulting data is structured data.
4. The system according to claim 3, wherein, The preprocessed information is semi-structured data.
5. The system according to claim 1, wherein, When the preset target field of the first base address register has a preset value, the first processor determines that the preprocessing information is stored in the first base address register.
6. The system according to claim 1, wherein, The central processing unit stores mapping information in the first base address register and the second base address register. This mapping information indicates the mapping between the memory space corresponding to the first processing device and the memory space corresponding to the second processing device. The first processing device uses the mapping information to transfer the result data to the second processing device by performing a direct memory access operation, i.e., a DMA operation.
7. A processing apparatus, comprising: processor; Base address register; as well as memory, The processor reads raw data from the memory, reads preprocessing information from the base address register, performs preprocessing operations on the raw data using the preprocessing information to generate result data, and directly transmits the result data to a target device outside the processing device.
8. The processing apparatus according to claim 7, wherein, The original data is unstructured data, and The resulting data is structured data.
9. The processing apparatus according to claim 8, wherein, The preprocessed information is semi-structured data.
10. The processing apparatus according to claim 7, wherein, The mapping information is stored together in both the base address register and the target device. The processor uses the mapping information to transfer the result data to the target device by performing a direct memory access operation, i.e., a DMA operation.
11. A method of operating a processing device, comprising: The processing device includes a processor, a base address register, and a memory. Read raw data from the memory; Read preprocessing information from the base address register; Preprocessing operations are performed on the original data using the preprocessing information to generate result data, and The result data is transmitted directly to an external device.
12. The method of claim 11, wherein, The original data is unstructured data, and The resulting data is structured data.
13. The method according to claim 12, wherein, The preprocessed information is semi-structured data.
14. The method of claim 11, wherein, Transferring the result data to the external device includes: using the mapping information stored in the base address register, performing a direct memory access (DMA) operation to transfer the result data to the external device, and The mapping information indicates the mapping between the memory space corresponding to the processing device and the memory space corresponding to the external device.