Data processing method and system and computer readable program product
By processing image frames in blocks and remapping physical addresses in the system-on-chip, the problem of limited memory bandwidth is solved, and system performance is improved.
Patent Information
- Application Number
- CN202410969765.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
In vehicle driver/passenger monitoring systems, due to the limited on-chip system memory bandwidth, existing technologies cause severe memory bandwidth consumption when processing image data, which significantly impacts system performance.
By dividing the image frame into multiple data blocks and using line interrupt signals for storage and preprocessing, the memory requirements of external storage units are reduced. The block processing method includes storing the image frame in the first storage unit of the on-chip system, issuing a line interrupt signal, the first processor obtaining the data block from the first storage unit for preprocessing and storing it in the external storage unit, and the third processor performing physical address remapping to optimize memory usage.
This reduces the memory requirements of external storage units and the overall memory bandwidth consumption, thereby improving the operating performance of the on-chip system.
Smart Images

Figure CN118939189A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, system and computer-readable program product. Background Art
[0002] In the application of the driver / passenger monitoring system of the vehicle computer, after the image data is obtained from the camera and stored in the memory, when the data is passed to the processor as input for processing, the signal processor (DSP) generally needs to pre-process the image first. The usual practice is to store a complete frame of image data in the memory, and notify the central processing unit (CPU) after the storage action is completed. The CPU controls the signal processor to obtain data from the memory and perform the required pre-processing, and then save it to another memory. After the storage action is completed, the processor reads the data and processes it. This process consumes a lot of memory bandwidth. For the system on chip, due to the large number of devices involved, its memory bandwidth is very tight. If the above method is used to process the image, the occupation of memory bandwidth will seriously affect the operating performance of the system on chip. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a data processing method, system and computer-readable program product.
[0004] The embodiment of the present application adopts the following technical solution: a data processing method, applied to a system on chip, comprising:
[0005] Acquiring an image frame based on an acquisition unit of the system on chip, storing the image frame in a first storage unit of the system on chip, and during the storage process, issuing a line interrupt signal based on a plurality of data blocks of the image frame;
[0006] Responding to a line interrupt signal based on the first processor of the system on chip, obtaining a stored data block from the first storage unit;
[0007] Preprocessing the obtained data block based on the first processor, and storing the preprocessed data block in a second storage unit located outside the system on chip, wherein the preprocessed data block is adapted to the processing method of the second processor of the system on chip for the image frame;
[0008] Remapping the physical address of the first storage unit in response to a row interrupt signal based on the third processor of the system on chip to obtain a remapped page table, wherein the page table is used to represent a mapping relationship between a physical address and a virtual address;
[0009] The second processor obtains the image frame from the second storage unit.
[0010] In some embodiments, issuing a line interrupt signal based on a plurality of data blocks of the image frame comprises:
[0011] Dividing the image frame into a plurality of data blocks;
[0012] When the storage of each data block is completed, a line interrupt signal is issued;
[0013] The image frame is divided into a plurality of data blocks in at least one of the following ways:
[0014] Division based on the size of the image frame;
[0015] Structural division based on image frames;
[0016] Pixel parameter partitioning based on image frames.
[0017] In some embodiments, storing the image frame in the first storage unit of the system on chip includes:
[0018] Two adjacent data blocks are stored in a first storage area of the first storage unit and a second storage area of the first storage unit respectively, wherein the first storage area has a first physical address and the second storage area has a second physical address.
[0019] In some embodiments, the method further comprises:
[0020] The acquisition unit obtains the image frame from a target device outside the system on chip, wherein the acquisition unit has a first page table;
[0021] The first processor obtains stored data blocks from the first storage unit one by one, wherein the first processor has a second page table.
[0022] In some embodiments, based on the third processor of the system on chip responding to the row interrupt signal, remapping the physical address of the first storage unit to obtain a remapped page table includes:
[0023] The third processor sends a remapping signal in response to the line interrupt signal;
[0024] The input-output memory management unit of the system on chip remaps the first physical address of the first storage area and the second physical address of the second storage area to the first page table of the acquisition unit in response to the remapping signal to obtain a remapped first remapping page table, wherein the first page table has a plurality of first virtual addresses;
[0025] The input-output memory management unit of the on-chip system remaps the first physical address of the first storage area and the second physical address of the second storage area to the second page table of the first processor in response to the remapping signal to obtain a remapped second remapping page table, wherein the second page table has multiple second virtual addresses.
[0026] In some embodiments, after obtaining the stored data block from the first storage unit, the method further includes:
[0027] Based on the row interruption state of the acquisition unit, a first flag is set for the acquisition unit, wherein the first flag indicates that the acquisition of the stored data block from the first storage unit has been completed;
[0028] Determining a read position and a size of a storage space of the first storage unit based on a sequence number of the row interrupt and a number of data blocks;
[0029] In the case that the line break of the current image frame is the first line break, the destination address of the image frame is used as the write address, and the processing type corresponding to the image frame is set to the pre-processing type;
[0030] Determining a write position and a size of a storage space of the second storage unit based on the write address and the sequence number of the row interrupt;
[0031] A second flag is set for the acquisition unit, wherein the second flag indicates that the processing of the acquired stored data block has been completed.
[0032] In some embodiments, before remapping the physical address of the first storage unit, the method further includes:
[0033] Based on the setting of the first flag bit, the third processor clears the line interrupt state and clears the first flag bit;
[0034] Based on the setting of the second flag bit, the third processor clears the second flag bit;
[0035] When the line interrupt is the last line interrupt of the current image frame, the address of the next image frame is written into its destination address;
[0036] When the line interrupt is the first line interrupt of the current image frame, a corresponding operation signal is sent to the second processor.
[0037] In some embodiments, the method further comprises:
[0038] Based on the remapped page table, receiving the remaining data blocks in the image frame, or,
[0039] Receive the next image frame.
[0040] The embodiment of the present application further provides a data processing system, including: a system on chip and a second storage unit, wherein the system on chip includes:
[0041] an acquisition unit configured to acquire an image frame based on the acquisition unit of the system on chip, store the image frame in a first storage unit of the system on chip, and during the storage process, send a line interrupt signal based on a plurality of data blocks of the image frame;
[0042] a first storage unit configured to store image frames from the acquisition unit;
[0043] a first processor configured to obtain a stored data block from the first storage unit in response to a line interrupt signal;
[0044] Preprocessing the obtained data blocks, and storing the preprocessed data blocks in a second storage unit located outside the system on chip, wherein the preprocessed data blocks are adapted to the processing method of the second processor of the system on chip for the image frame;
[0045] a third processor configured to remap the physical address of the first storage unit in response to a line interrupt signal to obtain a remapped page table, wherein the page table is used to represent a mapping relationship between a physical address and a virtual address;
[0046] A second processor is configured to obtain the image frame from the second storage unit.
[0047] An embodiment of the present application also provides a computer-readable program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the data processing method described in any of the above embodiments are implemented.
[0048] The beneficial effects of the embodiments of the present application are:
[0049] When multiple data blocks of the obtained image frame are stored in the first storage unit of the on-chip system, a line interrupt signal is issued based on the data block, and the first processor obtains the stored data block from the first storage unit and preprocesses it, that is, the image frame is processed in blocks instead of preprocessing the entire stored image frame. Compared with the traditional processing method, the memory demand and occupancy of the second storage unit are reduced, the overall memory bandwidth consumption of the on-chip system is reduced, and the operating performance of the on-chip system is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0051] Figure 1 This is a flow chart of the data processing method in an embodiment of the present application;
[0052] Figure 2 This is a structural block diagram of the system on chip, the second memory and the camera in the embodiment of the present application;
[0053] Figure 3 For the embodiment of this application Figure 1 A flowchart of one embodiment of step S10;
[0054] Figure 4 For the embodiment of this application Figure 3 A flowchart of one embodiment of step S11;
[0055] Figure 5 For the embodiment of this application Figure 1 A flowchart of one embodiment of step S40;
[0056] Figure 6 A relationship diagram of the physical addresses corresponding to the first storage area and the second storage area in the first memory and the first page table in an embodiment of the present application;
[0057] Figure 7 A relationship diagram of the physical addresses corresponding to the first storage area and the second storage area in the first memory and the first page table when remapping is performed in an embodiment of the present application;
[0058] Figure 8 For the embodiment of this application Figure 1 A flowchart of one embodiment of step S20;
[0059] Fig. 9 This is a structural block diagram of a register in an embodiment of the present application;
[0060] Fig.10 A block diagram of a flag bit, a destination address, and a processing type in an embodiment of the present application;
[0061] Fig.11 For the embodiment of this application Figure 1 A flowchart of another embodiment of step S40;
[0062] Fig.12 This is a structural block diagram of a processing device according to an embodiment of the present application. DETAILED DESCRIPTION
[0063] Various aspects and features of the present application are described herein with reference to the accompanying drawings.
[0064] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.
[0065] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.
[0066] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.
[0067] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.
[0068] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.
[0069] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.
[0070] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.
[0071] First, the application scenarios and technical terms involved in the embodiments of the present application are briefly introduced.
[0072] The driver monitoring system (DMS) or the occupant monitoring system (OMS) plays an important role in modern cars, providing drivers and passengers with a safer and more comfortable driving experience through advanced technical means. Taking the DMS system as an example, the DMS system collects images of the driver's face, eyes and other key areas in real time through cameras installed in the vehicle's cab (usually infrared cameras or high-definition cameras). These cameras can capture the driver's subtle expressions, movements and physiological characteristics, such as blinking frequency, yawning, and line of sight. The qualitative expression features are quantified, and facial feature points and feature indicators are extracted as the basis for judgment. Combined with multiple experimental data, the system can summarize the recognition method based on these parameters, and input the obtained state data for recognition and judgment. According to the judgment results, the system will issue different levels of warnings for different states of the driver, such as mild fatigue, severe fatigue, etc. The warning methods can include flashing red light, steering wheel vibration, sound reminders, etc., to remind the driver to pay attention to the current state. For example, in autonomous driving, the driver needs to take over the vehicle under certain conditions. The DMS system can monitor the driver's attention status and ensure that the driver can take over the vehicle in time when necessary.
[0073] SOC (System on Chip) refers to the integration of multiple hardware components on a single chip, such as CPU, memory controller, graphics processing unit (GPU), etc., to realize a complete computing system; SOC can contain a wafer (Die) or multiple identical or different wafers.
[0074] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.
[0075] In order to solve the problems in the background technology, the embodiment of the present application provides a data processing method, which is applied to the system on chip. Figure 1 and Figure 2 , the data processing method comprises the following steps:
[0076] S10, acquiring an image frame based on the acquisition unit of the system on chip, storing the image frame in a first storage unit of the system on chip, and during the storage process, issuing a line interrupt signal based on a plurality of data blocks of the image frame.
[0077] For example, taking the DMS system as an example, the camera installed in the vehicle cab continuously captures images of the driver to form image frames. The image frames captured by the camera are acquired from the camera by the acquisition unit of the system on chip. The image frames may include data of key areas such as the driver's face and eyes. The camera can capture the driver's subtle expressions, movements and physiological characteristics, such as blinking frequency, yawning, and line of sight.
[0078] The acquisition unit of the system on chip may be a camera serial interface (CSI). Specifically, the CSI is a serial interface between the camera and the SOC, which is used to transmit the image data captured by the camera. That is, a control signal is sent through the controller of the CSI of the system on chip to acquire the image frame from the above-mentioned camera through the CSI, and the obtained image frame is stored in the first storage unit of the system on chip, wherein the first storage unit may be an on-chip static random access memory (IRAM). In the process of storing the image frame in the IRAM, multiple line interrupts may be triggered. For example, a line interrupt signal may be issued based on multiple data blocks of the image frame, wherein each issuance of the line interrupt signal represents that a data block in the image frame has been stored in the IRAM; the line interrupt signal may be issued by the controller of the CSI. Based on the line interrupt signal, multiple processors on the system on chip may share the line interrupt signal and make corresponding feedback operations based on the line interrupt signal.
[0079] A line interrupt indicates the end of the specified number of lines of data transmission. For example, when the camera is capturing images and sending data, the CSI controller will send a line interrupt signal every time the specified number of lines of pixel data transmission is completed. The main function of the line interrupt signal is to notify the receiving end that the specified number of lines of data transmission has been completed. For example, the CSI controller can be configured to trigger an interrupt after writing a specified number of lines, such as dividing a frame of image into 4 parts, each part has a certain number of lines.
[0080] S20: Obtaining a stored data block from the first storage unit based on the first processor of the system on chip responding to a row interrupt signal.
[0081] Exemplarily, the system on chip includes a first processor, which may be a digital signal processor (DSP). The digital signal processor may receive a line interrupt signal sent by a controller of the CSI, and the digital signal processor may respond based on the received line interrupt signal. For example, the digital signal processor may respond to the line interrupt signal by sending an acquisition signal to a first storage unit of the system on chip, and then obtain a data block that has been stored from the first storage unit of the system on chip based on the acquisition signal.
[0082] S30, preprocessing the obtained data block based on the first processor, and storing the preprocessed data block in a second storage unit located outside the system on chip, wherein the preprocessed data block is adapted to the processing method of the second processor of the system on chip for the image frame.
[0083] Exemplarily, after the first processor obtains a data block of an image frame from the first storage unit, the data block can be preprocessed, wherein the preprocessing may include but is not limited to operations such as improving image quality, adjusting image size, extracting specific parts of the image, or changing the viewing angle of the image on the data block, so that the preprocessed data block can improve image quality, improve image processing efficiency and accuracy, and make the image more suitable for specific application scenarios, etc.
[0084] After the preprocessing of the data block is completed, the preprocessed data block is written to the destination address, i.e., the second storage unit, and the processed data block is stored in the second storage unit located outside the system on chip. The second storage unit can be an external memory for storing a large amount of data, for example, the second storage unit can be a dynamic random access memory (DRAM). That is, the preprocessed data block is sent to the DRAM located outside the system on chip, and the DRAM stores each preprocessed data block.
[0085] Memory bandwidth generally refers to the amount of data read and written by DRAM per unit time. Because various components on the system on chip may use DRAM at the same time, when the throughput reaches the upper limit, there will be a large delay in accessing DRAM, and the entire system will slow down. In this application, the data blocks obtained by dividing the image frame through the CSI controller are first written into the IRAM of the system on chip through the camera serial interface, and then the DSP reads the data blocks from the IRAM of the system on chip, and writes the data blocks to the second storage unit (DRAM) outside the system on chip after the first processor completes the processing of the data blocks. Compared with the traditional method of writing the image frame directly to DRAM, then reading the image frame from DRAM to process it, and then writing the processed image frame to DRAM. The present application can reduce the number of reads and writes to DRAM, that is, it reduces the reading and writing of DRAM by the entire system per unit time, and improves the operating performance of the system on chip.
[0086] When multiple data blocks corresponding to an image frame are preprocessed by the first processor, the second processor of the system on chip can obtain the image frame from the second storage unit. Therefore, the preprocessed data blocks are adapted to the processing method of the image frame by the second processor of the system on chip. For example, when outputting to the second processor for model inference (such as DMS: deep learning model), one model input resolution requirement is 640×480, and the other model input resolution requirement is 1920×1080, and the resolution size of the camera image frame is 1920×1080, then the data block can be processed in two ways, one of which is to reduce the processing, and the other is to keep the original size, and send two frames of data to the second processor at the same time. In addition, for example, if the original image has black edges, the black edges can be cropped. In addition, if the image format needs to be converted at the same time, for example, converting the yuv format to the bgr format.
[0087] For some time-consuming preprocessing, compared with the method of preprocessing after obtaining a complete image frame, the present application obtains the data block of the image frame through the first processor and preprocesses the data block, which can reduce time consumption and improve preprocessing efficiency.
[0088] S40, based on the third processor of the system on chip responding to the row interrupt signal, remap the physical address of the first storage unit to obtain a remapped page table, where the page table is used to represent the mapping relationship between the physical address and the virtual address.
[0089] Exemplarily, the third processor of the system on chip and the first processor of the system on chip can be connected to the same interrupt controller. When a line interrupt signal is issued, the third processor of the system on chip and the first processor of the system on chip will receive the line interrupt signal, that is, both can share the line interrupt signal issued by the CSI controller. For example, the external interrupts of the third processor and the DSP connected to the CSI controller can be configured as level triggers to achieve sharing. The third processor of the system on chip and the first processor of the system on chip can have their own interrupt processing functions for responding to the line interrupt signal and performing corresponding operations. Among them, the third processor can be but is not limited to a central processing unit (CPU). When a line interrupt is issued, the third processor can receive the line interrupt signal and can make corresponding feedback operations in response to the received line interrupt signal.
[0090] For example, the third processor may send a control signal to remap the physical address of the first storage unit to obtain a remapped first remapped page table, so that the continuous virtual address to be written in the next row interrupt is mapped to the physical address of the on-chip static random access memory actually used, and a control signal may be sent to the second processor according to the processing completion status, so that the second processor can perform the next processing operation.
[0091] S50: The second processor obtains the image frame from the second storage unit.
[0092] Exemplarily, after receiving the control signal sent by the third processor, the second processor can obtain the image frame from the second storage unit based on the received control signal. Here, the second processor can obtain the complete image frame from the second memory outside the system on chip and perform corresponding processing on the image frame. Here, each data block stored in the second memory is obtained by the second processor again in the form of an image frame.
[0093] In the embodiment of the present application, when storing multiple data blocks of the obtained image frame into the first storage unit of the on-chip system, a line interrupt signal is issued based on the data block, and the first processor obtains the stored data block from the first storage unit and pre-processes the data block, that is, the image frame is processed in blocks instead of pre-processing the entire stored image frame. Compared with the traditional processing method, the memory demand and occupancy of the second storage unit are reduced, the overall memory bandwidth consumption of the on-chip system is reduced, and the operating performance of the on-chip system is improved.
[0094] In some embodiments, in combination Figure 3 , based on the plurality of data blocks of the image frame, issuing a line interrupt signal, comprising:
[0095] S11, dividing the image frame into a plurality of data blocks.
[0096] For example, an image frame is 20×20 pixels, and the image frame can be divided into 4 data blocks (5×20), that is, each data block includes 5 rows. Continuing with the above embodiment, the controller of the CSI can sequentially store each data block in the first storage unit of the system on chip.
[0097] S12, when the storage of each data block is completed, a row interrupt signal is issued.
[0098] Exemplarily, when the first 5×20 data block is sent to the first storage unit, the CSI controller sends a line interrupt signal; when the second 5×20 pixel data block is sent to the first storage unit, the CSI controller sends another line interrupt signal; and so on, until all data blocks of the image frame are stored.
[0099] The image frame is divided into a plurality of data blocks in at least one of the following ways:
[0100] Divide based on the size of the image frame. For example, if the size of the image frame is 1920×1080, it can be divided into 4 data blocks, and the number of rows in each data block is 270. The control register of the CSI is configured with the number of rows. The CSI controller will trigger the interrupt of the DSP and CPU after writing the specified number of rows. According to the calculated number of blocks, the image frame is evenly divided into multiple rectangular blocks. Combined with the above embodiment, the number of rows in each data block is 270. Each block contains a fixed number of pixels, and the blocks do not overlap.
[0101] The number of rows of each data block is allocated according to the size of IRAM that can be occupied. The smaller the number of rows, the less IRAM is required. The corresponding number of row interrupts is also more, and the more row interrupts, the greater the consumption of the system to process the row interrupts. Therefore, the size of the data block divided by the image frame can be reasonably determined based on the operating performance of the system on chip. The number of blocks needs to balance the size of the IRAM occupied and the consumption of the row interrupt processing.
[0102] In some embodiments, storing the image frame in the first storage unit of the system on chip includes:
[0103] Two adjacent data blocks are stored in a first storage area of the first storage unit and a second storage area of the first storage unit respectively, wherein the first storage area has a first physical address and the second storage area has a second physical address.
[0104] For example, the first physical address of the first storage area is 0×100000, and the second physical address of the second storage area is 0×200000. Each data block has a write destination address before being stored in the first storage unit. For example, if a data block has a write destination address of 0×100000 before being stored in the first storage unit, the data block will be written to the first storage area of the first storage unit; if the data block has a write destination address of 0×200000 before being stored in the first storage unit, the data block will be written to the second storage area of the first storage unit. After the CSI stores a data block in the first storage area of the first storage unit, the next data block is stored in the second storage area of the first storage unit, which does not affect the first processor's acquisition of the data blocks stored in the first storage area.
[0105] Specifically, each image frame of the acquisition unit is divided into four data blocks, requiring two storage areas, and each block only needs to be large enough to accommodate one data block of the CSI.
[0106] In some embodiments, in combination Figure 4 , the method further comprises:
[0107] S111, the acquisition unit obtains the image frame from a target device outside the system on chip, wherein the acquisition unit has a first page table.
[0108] Exemplarily, in combination with the above embodiment, the target device outside the system on chip is a camera installed in the vehicle cab, and the camera can capture image frames about the driver of the vehicle. The acquisition unit can be a CSI, and the controller of the CSI can control the CSI to obtain the image frames about the driver of the vehicle captured by the camera from the camera. The first page table of the CSI can be configured to have a first virtual address, a second virtual address, a third virtual address and a fourth virtual address, wherein the first virtual address, the second virtual address, the third virtual address and the fourth virtual address correspond to 0×000000, 0×100000, 0×200000 and 0×300000, respectively. It should be noted that the virtual address 0×000000 can be arbitrary, as long as the four virtual addresses are continuous virtual addresses. The first virtual address will be configured as the image frame write address in the register of the CSI controller.
[0109] S112: The first processor obtains stored data blocks one by one from the first storage unit, wherein the first processor has a second page table.
[0110] Exemplarily, in combination with the above embodiment, the first processor obtains a stored data block from the first storage unit each time, and the data block may be a data block located in the first storage area of the first storage unit, or a data block located in the second storage area of the first storage unit. The first processor may obtain two data blocks from the first storage area of the first storage unit in succession, or may obtain one data block from the first storage area and then obtain another data block from the second storage area. This is only used as an example and does not constitute a limitation on the scope of protection of the claims.
[0111] In some embodiments, in combination Figure 5 , based on the third processor of the system on chip responding to the row interrupt signal, remapping the physical address of the first storage unit to obtain a remapped first remapping page table, including:
[0112] S41, the third processor sends a remapping signal in response to the line interrupt signal. The remapping signal is used to indicate that the physical address of the first storage unit is remapped. The third processor sends a remapping signal each time it receives the line interrupt signal.
[0113] S42, the input / output memory management unit of the system-on-chip responds to the remapping signal, remaps the first physical address of the first storage area and the second physical address of the second storage area to the first page table of the acquisition unit, and writes the mapping relationship between the obtained physical address and the virtual address into the page table used by the acquisition unit. Among them, the first page table has multiple first virtual addresses. Of course, it should be noted that when remapping is performed here, each physical address is only mapped to one virtual address at the same time, and the corresponding different line interrupts will cancel and remap the virtual address configured in the previous line interrupt, so as to achieve the effect that the virtual address is continuous when the device accesses by virtual address, and the actual physical address is just the two addresses in the first page table that are continuously remapped in a cycle. In this embodiment, for simple explanation, two fixed physical addresses are used. In actual use, the physical address used in each line interrupt can be any address allocated in the IRAM.
[0114] Exemplarily, in combination with the above embodiment, the acquisition unit is CSI, and the system on chip is configured with an input-output memory management unit (IOMMU). In response to the received remapping signal, the IOMMU remaps the first physical address of the first storage area and the second physical address of the second storage area to the first page table of the CSI, and writes the obtained mapping relationship between the physical address and the virtual address into the page table used by the CSI. Continuing with the above embodiment, specifically, see Figure 6 When the CSI and DSP drivers of the system on chip are initialized, first press Figure 6 The mapping relationship shown configures the page table of the IOMMU. The CSI image data write address remains at 0x00000 (as mentioned above, the address here is only an example and can be any starting address). When the line interrupt signal is issued, the third processor controls the IOMMU to rewrite the page table in sequence and map the physical address of the IRAM to the device virtual address.
[0115] As one example, the virtual addresses of the first page table may be 0×000000, 0×100000, 0×200000 and 0×300000. In combination with 7, taking an image frame corresponding to four line interrupt signals as an example, in response to a remapping signal, the IOMMU configures the CSI table. Similarly, in response to the next remapping signal, the IOMMU can reconfigure the CSI page table. By analogy, the four virtual addresses in the first page table are remapped cyclically in turn.
[0116] The number and size of IRAM in a system on chip are limited. This application uses IOMMU to minimize the IRAM requirements of the system on chip. After four line interrupts, that is, after a frame ends, the CSI loop continues to write from 0×00000.
[0117] After the corresponding page table is modified and refreshed, the work on the third processor side is completed.
[0118] S43, the input and output memory management unit of the on-chip system remaps the first physical address of the first storage area and the second physical address of the second storage area to the second page table of the first processor in response to the remapping signal, to obtain a remapped second remapping page table, wherein the second page table has multiple second virtual addresses.
[0119] Exemplarily, in combination with the above embodiment, the first processor is a DSP. Similar to the acquisition unit, the first processor has a second page table, which is also configured with four second virtual addresses, and the second virtual address can be the same as the first virtual address of the acquisition unit. After receiving the remapping signal, the input-output memory management unit responds to the remapping signal and remaps the first physical address of the first storage area and the second physical address of the second storage area to the second page table of the first processor (IOMMU allocates an independent page table for the DSP), and obtains the remapped second remapped page table. The two devices, the first processor and the acquisition unit, are configured with their own independent page tables and are unrelated to each other. For example, when remapping, the real physical address of IRAM is 0×2000, 0×2000 in the page table of DSP is mapped to the virtual address 0×3000, and the page table of CSI is mapped to the virtual address 0×3000. Then when the DSP accesses 0×3000, it actually accesses the physical address 0×2000 of IRAM, and when the CSI accesses 0×3000, it actually accesses the physical address 0×2000 of IRAM.
[0120] The specific remapping process is basically the same as the remapping process of the first page table, and will not be repeated here.
[0121] When the CSI controller issues a line interrupt, the third processor maps the IRAM address that has been written to the next IOVA (IO virtual address) address waiting to be written by the CSI, writes this mapping relationship into the first page table of the CSI, and writes the same mapping relationship into the second page table of the DSP, thus realizing the function of cyclically writing 4 data blocks through the two storage areas of the IRAM.
[0122] In some embodiments, in combination Figure 8 After obtaining the stored data block from the first storage unit, the method further includes:
[0123] S21, based on the row interruption state of the acquisition unit, setting a first flag bit for the acquisition unit, wherein the first flag bit indicates that the acquisition of the stored data block from the first storage unit has been completed.
[0124] Exemplarily, in combination with the above embodiment, the acquisition unit is CSI, and the CSI controller has a register, specifically, Fig. 9 As shown, the register may include an interrupt status register, an interrupt control register, a row number configuration register and a write address register.
[0125] like Fig.10 As shown, when the CSI controller sends a line interrupt signal, the first processor obtains the stored data block from the first storage unit. When the first processor completes the acquisition of the data block, it completes the reading of the register and sets the CSI to the first flag to notify the third processor that the interrupt status can be cleared. Here, it should be noted that Fig.10 The data structure can actually be placed at a pre-agreed fixed address, for example, an address of the internal SRAM (static random access memory) of the DSP itself. Among them, the first flag bit can be a synchronization flag bit, which indicates that the stored data block has been obtained from the first storage unit. The synchronization flag bit is used to prevent interrupt loss. It is set by the DSP after completing the reading of the CSI status / configuration register. When the CPU clears the CSI interrupt status, it needs to wait for the flag bit to be set, and clear this flag bit after clearing the CSI interrupt status. The initial value of the synchronization flag bit can be 0.
[0126] S22, determining a reading position and a size of a storage space of the first storage unit based on the sequence number of the row interrupt and the number of data blocks.
[0127] Exemplarily, if the data blocks are sequentially stored in the first storage area and the second storage area of the first storage unit, since the order of the data blocks is directly related to the sequence number issued by the line interrupt, the reading position and the size of the storage space of the first storage unit can be determined based on the sequence number of the line interrupt and the number of data blocks. For example, if the data block is the second data block corresponding to the second line interrupt signal, it can be stored in the second storage area of the first storage unit. The size of an image frame acquired by the camera is determined, and when the number of data blocks into which the image frame is divided is determined, the size of the data block is determined, and a storage space of appropriate size can be allocated to the data block in the first storage unit according to the size of the data block, and then the size of the first storage unit is determined so that it has enough space to store the image frame obtained by the CSI from the camera.
[0128] S23, when the line break of the current image frame is the first line break, taking the destination address of the image frame as the write address, and setting the processing type corresponding to the image frame to the pre-processing type.
[0129] Exemplarily, when the line interrupt of the current image frame is the first one, the write address can be determined according to the destination address of the image frame, and the preprocessing type can be set according to the processing type corresponding to the current image frame so that the preprocessed data block can adapt to the processing type required by the image frame.
[0130] S24, determining the write position and the size of the storage space of the second storage unit based on the write address and the sequence number of the row interrupt.
[0131] Exemplarily, according to the write address, the specific location of the data block divided into the image frame is determined to be sent to the second storage unit after preprocessing. According to the need of line interruption, the sequence number and size of the data block are determined, and according to the size of the data block, a storage space of an appropriate size is allocated to it in the second storage unit, that is, the size of the storage space of the second storage unit is determined.
[0132] S25, setting a second flag bit for the acquisition unit, wherein the second flag bit indicates that the processing of the acquired stored data block has been completed.
[0133] Exemplarily, in combination with the above embodiment, the DSP obtains the virtual address stored in the register from the write address register of the CSI, and then obtains the actual physical address, wherein the mapping relationship between the physical address and the virtual address used by the DSP and the CSI is the same. In this embodiment, it can be 0×00000, and the write destination address and the required processing type of the image frame are updated at the first line interruption of each image frame. After processing the data block corresponding to this line interruption, the corresponding second storage unit address is written, and the second flag bit is set to notify the third processor to perform the corresponding operation.
[0134] The second flag may be a done flag, which indicates that the processing of the acquired stored data block has been completed. The done flag is set by the DSP (digital signal processor) after the processing is completed, and the third processor reads and clears the done flag in the CSI interrupt processing. If the done flag is not set, it means that the previous part of the DSP has not been processed, and the error handling process can be entered. The initial value of the done flag may be 1.
[0135] The destination address and processing type of the image frame can be written by the third processor, where the destination address is used to transfer the memory destination address that the DSP needs to write; the processing type can be used to specify the processing operation that the DSP needs to perform, such as resize / rotate (zoom, rotation), etc.
[0136] In some embodiments, in combination Fig.11 , before remapping the physical address of the first storage unit, the method further includes:
[0137] S44: Based on the setting of the first flag bit, the third processor clears the line interrupt status and clears the first flag bit.
[0138] Exemplarily, continuing with the above embodiment, after receiving the CSI line interrupt, the third processor first waits for the DSP to set the first flag (synchronization flag). After the DSP sets it, the interrupt status and the first flag can be safely cleared to facilitate setting the CSI again in response to the next line interrupt signal.
[0139] S45: Based on the setting of the second flag bit, the third processor clears the second flag bit.
[0140] Exemplarily, the third processor reads the second flag bit (done flag bit) to determine whether the data block to be processed in the last interrupt has been processed by the DSP.
[0141] S46, when the line interruption is the last line interruption of the current image frame, writing the address of the next image frame into its destination address.
[0142] Exemplarily, when this line interruption is the last line interruption of the current image frame, it means that the current image frame has been completely written, and the destination address in the image frame can be replaced with the address of the next image frame to facilitate the storage of the next image frame.
[0143] S47: When the line interrupt is the first line interrupt of the current image frame, send a corresponding operation signal to the second processor.
[0144] Exemplarily, when this line interrupt is the first line interrupt of the current frame, it means that the DSP has completed processing the last data block of the previous image frame, and can notify the second processor (for example, the second processor is a neural network processor (NPU)) to read the preprocessed previous image frame.
[0145] In some embodiments, the method further comprises:
[0146] Based on the remapped page table, receiving the remaining data blocks in the image frame, or,
[0147] Receive the next image frame.
[0148] Exemplarily, if there are still unstored data blocks in the current image frame, the remaining data blocks in the image frame can be received based on the remapped page table. If all data blocks in the current image frame have been stored, the next image frame can be received based on the remapped page table.
[0149] The embodiment of the present application further provides a data processing system, comprising: a system on chip and a second storage unit, wherein the second storage unit is located outside the system on chip.
[0150] The system-on-chip includes:
[0151] The acquisition unit is configured to acquire an image frame based on the acquisition unit of the system on chip, store the image frame in the first storage unit of the system on chip, and issue a line interrupt signal based on multiple data blocks of the image frame during the storage process.
[0152] For example, taking the DMS system as an example, the camera installed in the vehicle cab continuously captures images of the driver to form image frames. The image frames captured by the camera are acquired from the above-mentioned camera by the acquisition unit of the system on chip. The image frames may include data of key areas such as the driver's face and eyes. The camera can capture the driver's subtle expressions, movements and physiological characteristics, such as blinking frequency, yawning, and line of sight.
[0153] The acquisition unit may be a camera serial interface (CSI). Specifically, the CSI is a serial interface between the camera and the SOC, which is used to transmit image data captured by the camera. That is, the controller of the CSI of the system on chip controls the CSI to acquire image frames from the above-mentioned camera, and stores the acquired image frames in the first storage unit of the system on chip. In the process of storing the image frames in the first storage unit, multiple line interrupts may be triggered. For example, a line interrupt signal may be issued based on multiple data blocks of the image frame, wherein each issuance of the line interrupt signal represents that a data block in the image frame has been stored in the first storage unit; the line interrupt signal may be issued by the controller of the CSI. Based on the line interrupt signal, multiple processors on the system on chip may share the line interrupt signal, and perform corresponding feedback operations based on the line interrupt signal.
[0154] A line interrupt indicates the end of the specified number of lines of data transmission. For example, when the camera is capturing images and sending data, the CSI controller will send a line interrupt signal every time the specified number of lines of pixel data transmission is completed. The main function of the line interrupt signal is to notify the receiving end that the data transmission of the specified number of lines has been completed. For example, the CSI can be configured to trigger an interrupt after writing a specified number of lines, such as dividing a frame of image into 4 parts, each of which has a certain number of lines.
[0155] A first storage unit is configured to store the image frames from the acquisition unit.
[0156] Exemplarily, the first storage unit may be an on-chip static random access memory (IRAM).
[0157] The first processor is configured to obtain a stored data block from the first storage unit in response to a line interrupt signal.
[0158] Exemplarily, the first processor may be a digital signal processor (DSP). The digital signal processor may receive a line interrupt signal sent by the CSI, and the digital signal processor may respond based on the received line interrupt signal. For example, the digital signal processor may respond to the line interrupt signal by sending an acquisition signal to the first storage unit of the system on chip, and then obtain the data block that has been stored from the first storage unit of the system on chip based on the acquisition signal.
[0159] The first processor also preprocesses the obtained data blocks and stores the preprocessed data blocks in a second storage unit located outside the system on chip, wherein the preprocessed data blocks are adapted to the processing method of the second processor of the system on chip for the image frame.
[0160] Exemplarily, after the first processor obtains a data block of an image frame from the first storage unit, the data block can be preprocessed, wherein the preprocessing may include but is not limited to operations such as improving image quality, adjusting image size, extracting specific parts of the image, or changing the viewing angle of the image on the data block, so that the preprocessed data block can improve image quality, improve image processing efficiency and accuracy, and make the image more suitable for specific application scenarios, etc.
[0161] After the preprocessing of the data block is completed, the preprocessed data block is written to the destination address, i.e., the second storage unit. The second storage unit may be an external memory for storing a large amount of data, for example, the second storage unit may be a dynamic random access memory (DRAM). That is, the preprocessed data block is sent to the DRAM located outside the system on chip, and the DRAM stores each preprocessed data block.
[0162] Memory bandwidth generally refers to the amount of data read and written by DRAM per unit time. Because various components on the system on chip may use DRAM at the same time, when the throughput reaches the upper limit, there will be a large delay in accessing DRAM, and the entire system will slow down. In the present application, the data blocks obtained by dividing the image frame are first written into the IRAM of the system on chip through CSI, and then the DSP reads the data blocks from the IRAM of the system on chip, and writes the data blocks to the second storage unit (DRAM) outside the system on chip after the first processor completes the processing of the data blocks. Compared with the traditional method of writing the image frame directly to DRAM, then reading the image frame from DRAM to process it, and then writing the processed image frame to DRAM. The present application can reduce the number of reads and writes to DRAM, that is, it reduces the reading and writing of DRAM by the entire system per unit time, and improves the operating performance of the system on chip.
[0163] When multiple data blocks corresponding to an image frame have been preprocessed by the first processor, the second processor of the system-on-chip can obtain the image frame from the second storage unit. Therefore, the preprocessed data blocks are adapted to the processing method of the image frame by the second processor of the system-on-chip. For example, when outputting to the second processor for model inference (such as DMS: deep learning model), one model input resolution requirement is 640×480, and the other model input resolution requirement is 1920×1080, and the resolution size of the camera's image frame is 1920×1080, so two processing can be performed on the data block, one of which is to reduce the processing and the other is to keep the original size. In addition, for example, if the original image has black edges, the black edges can be cropped. In addition, if the image format needs to be converted at the same time, for example, converting the yuv format to the bgr format.
[0164] For some time-consuming preprocessing, compared with the method of preprocessing after obtaining a complete image frame, the present application obtains the data block of the image frame through the first processor and preprocesses the data block, which can reduce time consumption and improve preprocessing efficiency.
[0165] The third processor is configured to remap the physical address of the first storage unit in response to the row interrupt signal to obtain a remapped page table, wherein the page table is used to represent a mapping relationship between the physical address and the virtual address.
[0166] Exemplarily, the third processor may be a central processing unit. The third processor and the first processor of the system on chip may both be connected to the same interrupt controller. When a line interrupt signal is issued, the third processor of the system on chip and the first processor of the system on chip will both receive the line interrupt signal, that is, both can share the line interrupt signal issued by the controller of the CSI. For example, the external interrupts of the third processor and the DSP connected to the CSI can be configured as level triggers to achieve sharing. The third processor of the system on chip and the first processor of the system on chip can have their own interrupt processing functions for responding to the line interrupt signal and performing corresponding operations. When the line interrupt is issued, the third processor can receive the line interrupt signal and can perform corresponding feedback operations in response to the received line interrupt signal.
[0167] For example, the third processor may send a control signal to remap the physical address of the first storage unit to obtain a remapped page table, so that the continuous virtual address to be written in the next row interrupt is mapped to the physical address of the on-chip static random access memory actually used, and may send a control signal to the second processor according to the processing completion status, so that the second processor can perform the next processing operation.
[0168] A second processor is configured to obtain the image frame from the second storage unit.
[0169] Exemplarily, after receiving the control signal sent by the third processor, the second processor can obtain the image frame from the second storage unit based on the received control signal. Here, the second processor can obtain the complete image frame from the second memory outside the system on chip and perform corresponding processing on the image frame. Here, each data block stored in the second memory is obtained by the second processor again in the form of an image frame. The second processor can be a neural network processor.
[0170] An embodiment of the present application also provides a computer-readable program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the data processing method described in any of the above embodiments are implemented.
[0171] In various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0172] Those skilled in the art will appreciate that the various illustrative logical blocks (ILBs) and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0173] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0176] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.
[0177] Multiple embodiments of the present application are described in detail above, but the present application is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concept of the present application, and these variations and modifications should all fall within the scope of protection required by the present application.
Claims
1. A data processing method, applied to a system on chip, characterized in that: include: Acquiring an image frame based on an acquisition unit of the system on chip, storing the image frame in a first storage unit of the system on chip, and during the storage process, issuing a line interrupt signal based on a plurality of data blocks of the image frame; Responding to a line interrupt signal based on the first processor of the system on chip, obtaining a stored data block from the first storage unit; Preprocessing the obtained data block based on the first processor, and storing the preprocessed data block in a second storage unit located outside the system on chip, wherein the preprocessed data block is adapted to the processing method of the second processor of the system on chip for the image frame; Remapping the physical address of the first storage unit in response to a row interrupt signal based on the third processor of the system on chip to obtain a remapped page table, wherein the page table is used to represent a mapping relationship between a physical address and a virtual address; The second processor obtains the image frame from the second storage unit.
2. The data processing method according to claim 1, characterized in that: Based on the plurality of data blocks of the image frame, a line interrupt signal is issued, comprising: Dividing the image frame into a plurality of data blocks; When the storage of each data block is completed, a line interrupt signal is issued; The image frame is divided into a plurality of data blocks in at least one of the following ways: Division based on the size of the image frame; Structural division based on image frames; Pixel parameter partitioning based on image frames.
3. The data processing method according to claim 2, characterized in that: Storing the image frame in a first storage unit of the system on chip includes: Two adjacent data blocks are stored in a first storage area of the first storage unit and a second storage area of the first storage unit respectively, wherein the first storage area has a first physical address and the second storage area has a second physical address.
4. The data processing method according to claim 3, characterized in that: The method further comprises: The acquisition unit obtains the image frame from a target device outside the system on chip, wherein the acquisition unit has a first page table; The first processor obtains stored data blocks from the first storage unit one by one, wherein the first processor has a second page table.
5. The data processing method according to claim 4, characterized in that: Remapping the physical address of the first storage unit based on the third processor of the system on chip in response to the row interrupt signal to obtain a remapped page table, including: The third processor sends a remapping signal in response to the line interrupt signal; The input-output memory management unit of the system on chip remaps the first physical address of the first storage area and the second physical address of the second storage area to the first page table of the acquisition unit in response to the remapping signal to obtain a remapped first remapping page table, wherein the first page table has a plurality of first virtual addresses; The input-output memory management unit of the on-chip system remaps the first physical address of the first storage area and the second physical address of the second storage area to the second page table of the first processor in response to the remapping signal to obtain a remapped second remapping page table, wherein the second page table has multiple second virtual addresses.
6. The data processing method according to claim 4, characterized in that: After obtaining the stored data block from the first storage unit, the method further includes: Based on the row interruption state of the acquisition unit, a first flag is set for the acquisition unit, wherein the first flag indicates that the acquisition of the stored data block from the first storage unit has been completed; Determining a read position and a size of a storage space of the first storage unit based on a sequence number of the row interrupt and a number of data blocks; In the case that the line break of the current image frame is the first line break, the destination address of the image frame is used as the write address, and the processing type corresponding to the image frame is set to the pre-processing type; Determining a write position and a size of a storage space of the second storage unit based on the write address and the sequence number of the row interrupt; A second flag is set for the acquisition unit, wherein the second flag indicates that the processing of the acquired stored data block has been completed.
7. The data processing method according to claim 6, characterized in that: Before remapping the physical address of the first storage unit, the method further includes: Based on the setting of the first flag bit, the third processor clears the line interrupt state and clears the first flag bit; Based on the setting of the second flag bit, the third processor clears the second flag bit; When the line interrupt is the last line interrupt of the current image frame, the address of the next image frame is written into its destination address; When the line interrupt is the first line interrupt of the current image frame, a corresponding operation signal is sent to the second processor.
8. The data processing method according to claim 1, characterized in that: The method further comprises: Based on the remapped page table, receiving the remaining data blocks in the image frame, or, Receive the next image frame.
9. A data processing system, characterized in that: include: A system on chip and a second storage unit, wherein the system on chip comprises: an acquisition unit configured to acquire an image frame based on the acquisition unit of the system on chip, store the image frame in a first storage unit of the system on chip, and during the storage process, send a line interrupt signal based on a plurality of data blocks of the image frame; a first storage unit configured to store image frames from the acquisition unit; a first processor configured to obtain a stored data block from the first storage unit in response to a line interrupt signal; Preprocessing the obtained data blocks, and storing the preprocessed data blocks in a second storage unit located outside the system on chip, wherein the preprocessed data blocks are adapted to the processing method of the second processor of the system on chip for the image frame; a third processor configured to remap the physical address of the first storage unit in response to a line interrupt signal to obtain a remapped page table, wherein the page table is used to represent a mapping relationship between a physical address and a virtual address; A second processor is configured to obtain the image frame from the second storage unit.
10. A computer readable program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.
