Chip and image scaling transformation method based on chip
By using small-capacity SRAM on the chip to cache image content and perform parallel calculations, the problem of slow image scaling and transformation speed caused by DDR memory is solved, and efficient image scaling and transformation processing is achieved.
Patent Information
- Application Number
- PCT/CN2023/139025
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2023-12-15
- Publication Date
- 2024-11-14
AI Technical Summary
In computer systems based on CPU and DDR, the calculation speed of image scaling transformation is slow and time-consuming, mainly due to the long read and write access time of the DDR memory to random addresses.
Design a chip to cache the image content stored in off-chip memory using small capacity static random memory (SRAM), and calculate the scalable transformed pixel values through up/down sampling to make up/down sampling calculations The process and the image caching process are carried out in parallel to form a multi-stage pipeline.
It significantly improves the calculation speed of image scaling transformation and reduces chip cost.
Smart Images

Figure CN2023139025_14112024_PF_FP_ABST
Abstract
Description
A chip and a chip-based image scaling method
[0001] This application claims priority to the Chinese patent application filed with the Patent Office of China on May 11, 2023, with application number 202310533429.6, entitled “A chip and a chip-based image scaling method”, the entire contents of which are incorporated herein by reference. Technical Field
[0002] The present application relates to the field of image technology, and in particular to a chip and a method for image scaling and transformation. Background Art
[0003] In image recognition, it is often necessary to zoom in and out. For example, in face detection and recognition, for a face detected from an image, it is necessary to extract a partial image and zoom it to a standard size before feeding it into the face recognizer for recognition.
[0004] Given the resolutions of the original image and the target image, scaling is performed by upsampling or downsampling the pixels in the original image to obtain the pixel values in the target image. This calculation process is usually implemented in software.
[0005] In computer systems based on central processing units (CPUs) and double-data-rate synchronous dynamic random access memory (DDR SDRAM), directly implementing scaling through software calculations is slow and time-consuming. This is due to the characteristics of DDR memory. As a dynamic memory, read and write accesses to random addresses in DDR take significantly longer than accesses to sequential addresses. Technical issues
[0006] In view of this, the embodiments of the present application provide a chip and a chip-based image scaling method, which can solve one or more technical problems in related technologies. Technical Solutions
[0007] In the first aspect, an embodiment of the present application provides a chip, a first input cache and a second input cache, which are used to respectively obtain K lines of data of the original image from the off-chip memory according to the data scheduling requirements until all the data of the original image are obtained and then stop obtaining; wherein, the first input cache and the second input cache are executed once as per the data scheduling requirements as one cycle; a first horizontal scaling module is used to perform horizontal up / down sampling on the K lines of data of the original image stored in the first input cache to obtain K lines of data of the target image and cache them into the first intermediate cache; a second horizontal scaling module is used to perform horizontal up / down sampling on the K lines of data of the original image stored in the second input cache to obtain K lines of data of the target image and cache them into the second intermediate cache cache; a first vertical scaling module, used to perform vertical up / down sampling on K lines of target image data stored in the first intermediate cache and the second intermediate cache in the current cycle to obtain M lines of target image data that meet the target resolution and cache them into the first output cache; a second vertical scaling module, used to perform vertical up / down sampling on K lines of target image data of a partial area stored in the first intermediate cache and the second intermediate cache in the next cycle to obtain N lines of target image data that meet the target resolution and cache them into the second output cache; a first input cache and the second output cache, used to transfer data of a partial area of the target image to a specified address; wherein K, M, and N are natural numbers, and their values can be equal or unequal.
[0008] In the second aspect, an embodiment of the present application provides a chip-based image scaling method, which is applied to the chip as described in the first aspect, and the method includes: triggering the first input cache and the second input cache to read K lines of data of the original image from the off-chip memory according to data scheduling requirements, and writing them into the first input cache and the second input cache; actuating the first horizontal scaling module and the second horizontal scaling module to perform horizontal up / down sampling on the original image data obtained by the first input cache and the second input cache in the current cycle to obtain K lines of target image data that meet the target horizontal resolution, and cache them into the first intermediate cache and the second intermediate cache respectively, and at the same time triggering the first input cache and the second input cache to read K lines of original image data according to data scheduling requirements; judging whether it is necessary to perform vertical scaling on several lines of original image data obtained in the current cycle, and if necessary, actuating the first vertical scaling module to read the data cached in the first intermediate cache and the second intermediate cache in the current cycle to perform vertical up / down sampling to obtain the target image data that meets the target vertical resolution. M rows of target image data are obtained and cached in the first output cache; when all the target image partial area data obtained in the current cycle are cached in the first output cache, the first horizontal scaling module and the second horizontal scaling module are activated to perform horizontal up / down sampling on the original image K rows of data obtained from the first input cache and the second input cache in the next cycle to obtain K rows of target image data that meet the target horizontal resolution, and cache them in the first intermediate cache and the second intermediate cache respectively; it is determined whether it is necessary to perform vertical scaling on several rows of original image data obtained in the next cycle. If necessary, the first vertical scaling module is activated to read the data cached in the first intermediate cache and the second intermediate cache in the next cycle to perform vertical up / down sampling to obtain N rows of target image data that meet the target vertical resolution and cache them in the second output cache; when all the target image partial area data obtained in the current cycle are cached in the second output cache, the above steps are repeated to perform a new round of upscaling transformation on the remaining image data of the original image until a complete target image that meets the target resolution is obtained.
[0009] In a third aspect, an embodiment of the present application provides an electronic device, comprising a chip as described in any embodiment of the first aspect.
[0010] In a fourth aspect, an embodiment of the present application provides a depth camera, comprising the above-mentioned chip, which is used to perform scaling transformation processing on an image captured by the depth camera to obtain an image that meets the target resolution.
[0011] In a fifth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any embodiment of the second aspect is implemented.
[0012] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device implements the method described in any embodiment of the second aspect. Beneficial effects
[0013] The embodiment of the present application caches the image content stored in the off-chip memory through several small-capacity on-chip memories, and calculates the pixel values after the scaling transformation through the up / down sampling method, so that the up / down sampling calculation process and the image caching process are performed in parallel to form a multi-stage pipeline, thereby greatly improving the calculation speed of the image scaling transformation. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0015] FIG1 is a simplified schematic diagram of a chip architecture provided by an embodiment of the present application;
[0016] FIG2 is a detailed schematic diagram of a chip architecture provided by an embodiment of the present application;
[0017] FIG3 is a flow chart of a chip-based image scaling method provided in an embodiment of the present application. Best Mode for Carrying Out the Invention
[0018] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.
[0019] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0020] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.
[0021] In addition, in the description of this application, "a plurality of" means two or more. The terms "first" and "second" are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0022] It should also be understood that, unless otherwise expressly specified or limited, the term "connection" should be understood broadly. For example, it can mean fixed connection, detachable connection, or integration; it can mean direct connection, indirect connection through an intermediate medium, internal communication between two elements, or interaction between two elements. Those skilled in the art will understand the specific meanings of the above terms in this application based on specific circumstances.
[0023] Currently, in CPU- and DDR-based computer systems, the biggest problem with directly implementing scaling through software calculations is slow speed and time consumption. This is due to the characteristics of DDR memory. As a dynamic memory, reading and writing to random addresses in DDR takes significantly longer than accessing consecutive addresses. To increase speed, the best approach is to read consecutive addresses from DDR, cache them in static random access memory (SRAM), and then perform random access from SRAM. However, large SRAM capacities are very expensive.
[0024] Therefore, embodiments of the present application provide a method and chip for implementing image scaling on a chip. The chip developed based on this application can cache local image content stored in off-chip memory using several small-capacity SRAMs, and calculate the scaled pixel values using up / down sampling. The up / down sampling calculation process is performed in parallel with the image caching process, and the intermediate calculation results are also cached in SRAM, forming a multi-stage pipeline, thereby greatly improving the calculation speed of image scaling.
[0025] In order to better illustrate the technical solution of the present application, the technical solution of the present application is described and explained in detail with reference to some specific parameters. It should be understood that these parameters are only preferred parameters for the convenience of explanation and cannot be interpreted as specific limitations of the present application.
[0026] FIG1 is a schematic diagram of the architecture of a chip for image scaling provided by the present application. To better illustrate the chip of this embodiment, FIG1 shows an off-chip memory coupled to the chip. Specifically, the chip includes an input buffer 10, a first scaling module 11, an intermediate buffer 12, a second scaling module 13, and an output buffer 14. The input buffer 10 is configured to retrieve partial data of an original image from the off-chip memory and stop retrieving data until all the data of the original image has been retrieved. The first scaling module 11 is configured to perform horizontal / vertical scaling on historical partial data of the original image while the input buffer 10 is retrieving current partial data of the original image and obtain horizontal / vertical scaling results of a partial region of the target image. The intermediate buffer 12 is configured to cache the horizontal / vertical scaling results of the partial region of the target image. The second scaling module 13 is configured to retrieve historical horizontal / vertical scaling results of the partial region of the target image and perform vertical / horizontal scaling on the historical horizontal / vertical scaling results of the partial region of the target image to obtain a partial region of the target image that meets a target resolution. The output buffer 14 is configured to store and output the partial region of the target image.
[0027] In one embodiment, the input buffer 10, the intermediate buffer 12, and the output buffer 14 in the present application include static random access memories.
[0028] In one embodiment, the raw image data is pre-stored in an off-chip memory. Preferably, the off-chip memory may be a DDR memory. In other embodiments, the off-chip memory may be replaced by a non-limiting alternative, such as a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), or a pseudo static random access memory (PSRAM).
[0029] FIG2 is a schematic diagram of the detailed architecture of the chip provided in the present application. The chip at least includes a first input cache 101 and a second input cache 102, a first horizontal scaling module 110 and a second horizontal scaling module 111, a first intermediate cache 120 and a second intermediate cache module 121, a first vertical scaling module 130 and a first vertical scaling module 131, and a first output cache 140 and a second output cache 141; wherein, the first input cache 101 and the second input cache 102 are used to obtain K lines of data of the original image from the off-chip memory according to the data scheduling requirements until all the data of the original image is obtained and then stop obtaining; wherein, the first input cache 101 and the second input cache 102 execute one cycle according to the data scheduling requirements; the first horizontal scaling module 110 is used to perform horizontal up / down sampling on the K lines of data of the original image stored in the first input cache 101 to obtain K lines of data of the target image and cache them in the first intermediate cache 120; the second horizontal scaling module 111 is used to perform horizontal up / down sampling on the second The input buffer 102 performs horizontal up / down sampling on K lines of original image data stored in the first and second intermediate buffers 120, 121 to obtain K lines of target image data, which are then cached in the second intermediate buffer 121. The first vertical scaling module 130 is configured to perform vertical up / down sampling on the K lines of target image data stored in the first and second intermediate buffers 120, 121 in the current cycle to obtain M lines of target image data that meet the target resolution, which are then cached in the first output buffer 140. The second vertical scaling module 131 is configured to perform vertical up / down sampling on the K lines of target image data stored in the first and second intermediate buffers 120, 121 in the next cycle to obtain N lines of target image data that meet the target resolution, which are then cached in the second output buffer 141. The target image data that meet the target resolution is then transmitted to a designated address via the first input buffer 140 and the second output buffer 141. K, M, and N are natural numbers that may or may not be equal. It should be noted that K, M, and N are preferably 1, enabling the caching of partial image content stored off-chip using several small-capacity caches, thereby reducing chip costs.
[0030] In one embodiment, the first horizontal scaling module 110 and the second horizontal scaling module 111 can directly read the original image data from the first input cache 101 and the second input cache 102 respectively, and the two are independent of each other. Therefore, the first horizontal scaling module 110 and the second horizontal scaling module 111 can work in parallel; but the first vertical scaling module 130 needs to obtain the data stored in the current cycle of the first intermediate cache 120 and the second intermediate cache 121 for processing, and the second vertical scaling module 131 needs to obtain the data stored in the next cycle of the first intermediate cache 120 and the second intermediate cache 121 for processing. It can be seen that the first vertical scaling module 130 and the second vertical scaling module 131 share the first intermediate cache 120 and the second intermediate cache 121, and they process the data stored in the first intermediate cache 120 and the second intermediate cache 121 in different cycles. Therefore, the first vertical scaling module 130 and the second vertical scaling module 131 work in serial.
[0031] In one embodiment, the chip also includes a scheduling module 15, which includes a read request submodule and a receive request submodule. The read request submodule is used to send a read data request to the off-chip memory according to the data scheduling requirements and / or the cache space of the first input cache 101 and the second input cache 102, so as to read K lines of data of the original image stored in the off-chip memory in units of lines; the receive request submodule is used to receive K lines of data of the original image sent by the off-chip memory in response to the read data request, and write the received data into the first input cache 101 and the second input cache 101.
[0032] In one embodiment, the chip includes a parameter register for storing parameters for up / down sampling of the horizontal scaling module and the vertical scaling module. Specifically, before the chip reads the original image data from the off-chip memory, it is necessary to first obtain the parameters for up / down sampling of the horizontal scaling module and the vertical scaling module and store them in the parameter memory, so that the subsequent horizontal scaling module and the vertical scaling module can use the parameters to determine the sub-pixel position of each pixel in the target image in the original image, thereby ensuring the accuracy of the image scaling transformation. It should be noted that the parameters for up / down sampling of the horizontal scaling module and the vertical scaling module can be obtained through the host computer or the off-chip CPU processor and directly reflected in the design of the chip as design parameters (such as directly storing the obtained parameters in the chip during chip design), rather than being specified by the host computer or the off-chip CPU processor.
[0033] Preferably, the parameters for up / down sampling of the horizontal scaling module and the vertical scaling module include horizontal sub-pixel parameters and vertical sub-pixel parameters, respectively, wherein the horizontal sub-pixel parameters are used to specify that the distance between adjacent pixels in the horizontal direction of the original image is refined into several parts, and the vertical sub-pixel parameters are used to specify that the distance between adjacent pixels in the vertical direction of the original image is refined into several parts. Furthermore, the parameters for up / down sampling of the horizontal scaling module and the vertical scaling module also include horizontal integer step size and horizontal sub-pixel step size, vertical integer step size and vertical sub-pixel step size. According to the above step size, it can be obtained which row / column in the original image is the next valid row / column required for up / down sampling, and the sub-pixel position of the target image row / column after up / down sampling in the original image, so that invalid rows / columns are skipped during horizontal scaling or vertical scaling, and only valid rows / columns are up / down sampled, thereby improving the efficiency of image scaling transformation.
[0034] In one embodiment, a method for obtaining a horizontal integer step size and a horizontal sub-pixel step size includes: determining a sub-pixel distance HD between adjacent pixels in a row of an original image based on the precision required for the target image after up / down sampling. Assuming that the original image has A rows and the target image has B rows: the horizontal integer step size is the integer portion of the total number of rows A in the original image divided by the total number of rows B in the target image, and the horizontal sub-pixel step size is the product of the decimal portion of the total number of rows A in the original image divided by the total number of rows B in the target image and the sub-pixel distance HD between adjacent pixels in a row of the original image. In another embodiment, a method for obtaining a vertical integer step size and a vertical sub-pixel step size includes: determining a sub-pixel distance VD between adjacent pixels in a column of the original image based on the precision required for the target image after up / down sampling. Assuming that the original image has C columns and the target image has D columns: the vertical integer step size is the integer portion of the total number of columns C in the original image divided by the total number of columns D in the target image, and the vertical sub-pixel step size is the product of the decimal portion of the total number of columns C in the original image divided by the total number of columns D in the target image and the sub-pixel distance VD between adjacent pixels in a column of the original image.
[0035] Taking each row in the target image as an example, the corresponding row in the original image for the target row is determined starting from the first row of pixels in the target image, and the row data corresponding to the target row in the original image is retrieved from the off-chip memory. The first row in the target image corresponds to the first row in the original image. Based on the number of rows in the target image, the number of rows in the original image, and the up / down sampling parameters of the horizontal and vertical scaling modules, the sub-pixel position of each remaining row of pixels in the target image is calculated in the original image. If the sub-pixel position is between two rows in the original image, then the target row in the target image corresponds to two rows of data in the original image. If the sub-pixel position is on an integer row in the original image, then the target row in the target image corresponds to a row of data at an integer row position in the original image.
[0036] In one embodiment, before the chip obtains the corresponding row data of the current target row of the target image in the original image from the off-chip memory, it is necessary to calculate the sub-pixel positions of the current target row and the next target row in the original image, and then determine the corresponding rows of the current target row and the next target row in the original image.
[0037] Specifically, if the current target row and the next target row have one or two identical corresponding rows in the original image, then after obtaining the corresponding row data of the current target row in the original image, the chip will no longer repeatedly obtain the same data for the next target row. Instead, it will directly use the data stored in the corresponding input buffer for subsequent calculations and transmission, further improving the efficiency of image scaling transformation. Among them, the current target row and the next target row have one or two identical corresponding rows in the original image, including the following situations:
[0038] In the first case, the sub-pixel positions of the current target row and the next target row in the original image are located between the same two rows of pixels. In this case, the current target row and the next target row have two identical corresponding rows in the original image.
[0039] In the second case, the current target row and the next target row both have two corresponding rows in the original image, and the corresponding row below the current target row in the original image coincides with the corresponding row above the next target row in the original image. In this case, the current target row and the next target row have only one identical corresponding row in the original image.
[0040] In the third case, the sub-pixel position of the current target row in the original image is on an integer row, while the sub-pixel position of the next target row in the original image is between two rows of pixels, and the sub-pixel distance between the current target row and the next target row in the original image is less than the sub-pixel distance between two adjacent rows of pixels in the original image. In this case, the current target row and the next target row have only one common corresponding row in the original image.
[0041] In the fourth case, the sub-pixel position of the current target row in the original image is between two rows of pixels, while the sub-pixel position of the next target row in the original image is on an integer row, and the sub-pixel distance between the current target row and the next target row in the original image is less than the sub-pixel distance between two adjacent rows of pixels in the original image. In this case, the current target row and the next target row have only one identical corresponding row in the original image.
[0042] Furthermore, to ensure the accuracy of image scaling, before the output buffer is fully populated with the current target row of the target image, if a certain group of input buffers is in an idle state and a corresponding row of the next target row in the original image has not yet been acquired, then the corresponding row data of the next target row in the original image that has not yet been acquired is read from the off-chip memory and stored in a group of input buffers in an idle state. However, the data in this group of input buffers cannot be scaled horizontally immediately. Instead, the data in this group of input buffers cannot be scaled horizontally and stored in a corresponding group of intermediate buffers until the current target row is fully stored in the corresponding output buffer and the next target row becomes the new current target row. The idle state of a certain group of input buffers includes the following two situations:
[0043] In the first case, the sub-pixel position of the current target row in the original image is located on an integer row. In this case, the current target row has only one corresponding row in the original image. Therefore, only one row of data needs to be obtained and stored in one set of input buffers, and the other set of input buffers is idle.
[0044] In the second case, a corresponding row of the current target row in the original image has been stored in a certain group of input buffers and has been used up, that is, a row of original image data in this group of input buffers has been read and horizontally scaled and stored in the corresponding intermediate buffer; at this time, the used input buffer group is in an idle state.
[0045] In one embodiment, the chip further includes multiple flag registers for coordinating and controlling data transfer between the input buffer, the intermediate buffer, and the output buffer, thereby enabling the up / down sampling calculation process to proceed in parallel with the image caching process and forming a multi-stage pipeline. Specifically, the first input buffer 101 and the second input buffer 102 each correspond to a set of status flag registers for indicating whether the first input buffer 101 and the second input buffer 102 have been filled with data from the off-chip memory or whether they are idle. When the status flag register corresponding to the first input buffer 101 or the second input buffer 102 is high, it indicates that the first input buffer 101 or the second input buffer 102 has been filled with data from the off-chip memory. When the register corresponding to the status flag register corresponding to the first input buffer 101 or the second input buffer 102 is low, it indicates that the data in the first input buffer 101 or the second input buffer 102 has been used up and has entered the next-level module (such as the first horizontal scaling module or the second horizontal scaling module), and the first input buffer 101 or the second input buffer 102 is idle, thereby implementing a ping-pong operation. Furthermore, the chip includes a data identification register for indicating whether the row data of the target image is located on an integer row of the original image, so as to ensure that the first input buffer 101 or the second input buffer 102 reads the row data of the target image corresponding to the row in the original image.
[0046] The first intermediate buffer 120 and the second intermediate buffer 121 each have corresponding status flag registers, which are used to indicate whether image data destined for the first output buffer 140 or the second output buffer 141 has been filled. Specifically, assuming that the two sets of status flag registers corresponding to the first intermediate buffer 120 are S1 and S2, when status flag register S1 is at a high level, it indicates that the first intermediate buffer 120 has been filled with image data destined for the first output buffer 140, and the result of vertical upsampling / downsampling of the buffered data in the first intermediate buffer 120 must be written to the first output buffer 140; when status flag register S2 is at a high level, it indicates that the first intermediate buffer 120 has been filled with image data destined for the second output buffer 141, and the result of vertical upsampling / downsampling of the buffered data in the first intermediate buffer 120 must be written to the second output buffer 141. Similarly, the status flag register corresponding to the second intermediate buffer 121 coordinates the data flow passing through the second intermediate buffer 121, which will not be further described here. Furthermore, the chip includes a position identification register for indicating the upper and lower relative positions of a plurality of rows of target image data stored in the first intermediate buffer 120 and the second intermediate buffer 121 , respectively.
[0047] The first output buffer 140 and the second output buffer 141 respectively correspond to a set of status flag registers for marking whether they are filled with data or whether they are idle. Specifically, when the status flag register corresponding to the first output buffer 140 is at a high level, it indicates that the first output buffer 140 is in an idle state and can be filled with M rows of target image data that meet the target resolution after vertical up / down sampling from the first vertical scaling module 130. When the M rows of target image data from the first vertical scaling module 130 are all filled into the first output buffer 140, the status flag register corresponding to the first output buffer 140 flips from a high level to a low level to indicate that the first output buffer 140 has been filled. When the status flag register corresponding to the second output buffer 141 is at a high level, it indicates that the second output buffer 141 is in an idle state and can be filled with N rows of target image data that meet the target resolution after vertical up / down sampling from the second vertical scaling module 131. When the N rows of target image data from the second vertical scaling module 131 are all filled into the second output buffer 141, the status flag register corresponding to the second output buffer 141 flips from a high level to a low level to indicate that the second output buffer 141 has been filled, thereby ensuring that the first output buffer 140 and the second output buffer 141 obtain the target image data in turn.
[0048] Furthermore, the chip includes an output flag register for identifying whether data is currently being read from the first output buffer 140 or the second output buffer 141 and output to a specified address. Specifically, when the output flag register is at a high level, it indicates that data is currently being read from the first output buffer 140 and output to the specified address; when the output flag register is at a low level, it indicates that data is currently being read from the second output buffer 141 and output to the specified address.
[0049] In one embodiment, the chip further includes a priority flag register for indicating whether the image data being scaled during the current cycle of the chip is preferentially filled into the first output buffer 140 or the second output buffer 141. Specifically, when the priority flag register is at a high level, data acquired by the first input buffer 101 and the second input buffer 102 during the current cycle is filled into the first output buffer 140; when the priority flag register is at a low level, data acquired by the first input buffer 101 and the second input buffer 102 during the current cycle is filled into the second output buffer 141.
[0050] In some embodiments, during the process of transmitting the target image obtained after the scaling transformation to the off-chip memory, it cannot be ensured that the off-chip memory is available at any time. Therefore, a backup data pool coupled to the off-chip memory is also provided in the chip; when the off-chip memory is unavailable, the backup data pool is used to cache the target image data read from the first output cache 140 and the second output cache 141; when the off-chip memory is available, the data is read from the backup data pool and provided to the off-chip memory.
[0051] Specifically, the backup data pool is a non-open register within the chip, used to temporarily store a portion of target image pixels read from the first output buffer 140 or the second output buffer 141, to be sent to the off-chip memory. Since it takes a certain period of time for the output identification register to read data from the first output buffer 140 or the second output buffer 141, to ensure continuous data output to the off-chip memory, the backup data pool in this application can store a number of data that is greater than or equal to the number of clock cycles required for the output identification register to read a single data item from the first output buffer 140 or the second output buffer 141. This ensures that data is always available for transmission to the off-chip memory, effectively improving the efficiency and stability of data transmission.
[0052] In one embodiment, the output flag register outputs a row of pixels of the previous target row stored in the first output buffer 140 or the second output buffer 141 to the off-chip memory, including: when the off-chip memory is unavailable, stopping initiating a read operation to the corresponding output buffer, queuing the previously initiated historical read operations on the pipeline, and sequentially storing the data read from the corresponding output buffer by the historical read operations into the data backup pool; when the off-chip memory is available, initiating a read operation to the corresponding output buffer, and processing according to the following three situations respectively:
[0053] Case 1: If there is data in the data backup pool, but there is no data at the end of the read operation pipeline of the corresponding output cache, the earliest stored data is taken from the data backup pool and output to the off-chip memory.
[0054] Case 2: If there is no data in the backup data pool, then the end of the read operation pipeline of the corresponding output cache must have read data. In this case, the data at the end of the read operation pipeline can be directly output to the off-chip memory.
[0055] Case 3: If there is data in the standby data pool and at the end of the read operation pipeline of the corresponding output cache, the earliest data stored in the standby data pool is output to the off-chip memory, and the data at the end of the read operation pipeline is stored in the standby data pool.
[0056] Based on the above chip, this application provides an image scaling method. For details not provided in detail, please refer to the above embodiments. Specifically, the method includes:
[0057] S10: triggering the first input cache and the second input cache to read K lines of data of the original image from the off-chip memory according to the data scheduling requirement, and writing the data into the first input cache and the second input cache;
[0058] S20: activating the first horizontal scaling module and the second horizontal scaling module to perform horizontal up / down sampling on the original image data obtained by the first input buffer and the second input buffer in the current cycle to obtain K lines of target image data that meet the target horizontal resolution, and buffering them into the first intermediate buffer and the second intermediate buffer respectively. At the same time, triggering the first input buffer and the second input buffer to read the K lines of original image data according to data scheduling requirements;
[0059] S30: Determine whether it is necessary to perform vertical scaling on a plurality of rows of data of the original image acquired in the current cycle. If necessary, activate the first vertical scaling module to read the data cached in the first intermediate buffer and the second intermediate buffer in the current cycle, perform vertical upsampling / downsampling on the data, obtain M rows of data of the target image meeting the target vertical resolution, and cache the data into the first output buffer.
[0060] S40: When all the target image partial area data obtained in the current cycle is cached in the first output buffer, the first horizontal scaling module and the second horizontal scaling module are activated to perform horizontal up / down sampling on K lines of original image data obtained from the first input buffer and the second input buffer in the next cycle to obtain K lines of target image data that meet the target horizontal resolution, and cache them in the first intermediate buffer and the second intermediate buffer respectively;
[0061] S50: Determine whether it is necessary to perform vertical scaling on a plurality of rows of data of the original image acquired in the next cycle. If necessary, activate the first vertical scaling module to read the data cached in the first intermediate buffer and the second intermediate buffer for the next cycle, perform vertical upsampling / downsampling on the data, obtain N rows of data of the target image meeting the target vertical resolution, and cache the data into the second output buffer.
[0062] S60: When all the target image partial area data obtained in the current cycle is cached into the second output cache, repeat the above steps S10 to S50 to perform a new round of stretching transformation on the remaining image data of the original image until a complete target image meeting the target resolution is obtained.
[0063] In one embodiment, determining whether vertical scaling is required for a plurality of rows of original image data acquired in the current cycle or the next cycle further includes: if the determination result is that vertical scaling is not required for K rows of original image data acquired in the current cycle or the next cycle, directly writing the K rows of target image data that meet the target horizontal resolution, which are buffered in the first intermediate buffer and the second intermediate buffer, into the first output buffer and the second output buffer, respectively, for transmission to the designated address. The number of pixels in the K rows of the original image stored in the first intermediate buffer and the second intermediate buffer, respectively, is the same as the number of pixels in the K rows of the target image.
[0064] It should be noted that in the above process, since the original image first passes through the horizontal scaling module and then the vertical scaling module, the input cache reads the original data in units of rows when reading the data of the original image; if the original image first passes through the vertical scaling module and then the horizontal scaling module, the input cache reads the original data in units of columns when reading the data of the original image. The image scaling transformation process that first passes through the vertical scaling module and then the horizontal scaling module is similar to the above process. It is sufficient to exchange the module positions, which will not be repeated here.
[0065] As described above, an embodiment of the present application provides a chip that can cache local image content stored in DDR through several dedicated small-capacity SRAMs, and calculate the pixel values after scaling transformation through up / down sampling methods, so that the up / down sampling calculation process and the image caching process are carried out in parallel and form a multi-stage pipeline, thereby greatly improving the calculation speed of image scaling transformation.
[0066] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0067] An embodiment of the present application further provides an electronic device, comprising the chip as described in any of the aforementioned embodiments.
[0068] One embodiment of the present application further provides a camera including the aforementioned chip, which is configured to perform scaling processing on images captured by the camera to obtain images that meet a target resolution. The camera can be a color camera, a black and white camera, a grayscale camera, a structured light camera, a time-of-flight camera, a binocular camera, or a multi-camera camera.
[0069] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the aforementioned method embodiments can be implemented.
[0070] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps of any of the aforementioned method embodiments.
[0071] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.
[0072] Computer programs include computer program code, which may be in source code, object code, executable files, or some intermediate form. Computer-readable media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), RAM, electric carrier signals, telecommunications signals, and software distribution media. It should be noted that the content of computer-readable media may be appropriately expanded or reduced based on the requirements of legislation and patent practice within a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media do not include electric carrier signals or telecommunications signals.
[0073] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.
Claims
1. A chip for image scaling, the chip being coupled to an off-chip memory, It is characterized in that The chip at least includes: a first input cache and a second input cache, a first horizontal scaling module and a second horizontal scaling module, a first intermediate cache and a second intermediate cache module, a first vertical scaling module and a first vertical scaling module, and a first output cache and a second output cache, wherein: The first input buffer and the second input buffer are used to respectively obtain K lines of data of the original image from the off-chip memory according to data scheduling requirements until all data of the original image is obtained and then stop obtaining; wherein, one execution of the first input buffer and the second input buffer according to the data scheduling requirements is one cycle; The first horizontal scaling module is used to perform horizontal up / down sampling on the K lines of data of the original image stored in the first input buffer to obtain K lines of data of the target image and cache them into the first intermediate buffer; The second horizontal scaling module is used to perform horizontal up / down sampling on the K lines of data of the original image stored in the second input buffer to obtain K lines of data of the target image and cache them into the second intermediate buffer; The first vertical scaling module is used to perform vertical up / down sampling on the K lines of data of the target image stored in the first intermediate cache and the second intermediate cache in the current cycle to obtain M lines of data of the target image that meet the target resolution and cache them into the first output cache; The second vertical scaling module is used to perform vertical up / down sampling on K lines of data of the partial area of the target image stored in the first intermediate cache and the second intermediate cache in the next cycle to obtain N lines of data of the target image meeting the target resolution and cache them into the second output cache; The first input buffer and the second output buffer are used to transfer data of a partial area of the target image to a designated address; wherein K, M, and N are natural numbers, and the values of the three may be equal or unequal.
2. The chip according to claim 1, It is characterized in that The chip further includes a scheduling module, the scheduling module including a read request submodule and a receive request submodule, the read request submodule is used to send a read data request to the off-chip memory according to data scheduling requirements and / or the cache space of the first input cache and the second input cache; The receiving request submodule is used to receive the original image data sent by the off-chip memory in response to the read data request, and write the received data into the first input buffer and the second input buffer.
3. The chip according to claim 1, It is characterized in that The chip also includes parameter registers for storing parameters for up / down sampling of the horizontal scaling module and the vertical scaling module. The parameters can be obtained through a host computer or an off-chip processor and directly reflected in the design of the chip as design parameters.
4. The chip according to claim 3, It is characterized in that The parameters include horizontal sub-pixel parameters and vertical sub-pixel parameters, wherein the horizontal sub-pixel parameters are used to specify that the distance between horizontally adjacent pixels of the original image is refined into several parts, and the vertical sub-pixel parameters are used to specify that the distance between vertically adjacent pixels of the original image is refined into several parts.
5. The chip according to claim 3, It is characterized in that The parameters also include horizontal integer step size and horizontal sub-pixel step size, vertical integer step size and vertical sub-pixel step size. According to the step size, it can be obtained which row / column of the original image the next valid row / column required for up / down sampling is and the sub-pixel position of the target image row / column in the original image after up / down sampling.
6. The chip according to claim 5, It is characterized in that Before the chip obtains the corresponding row data of the current target row of the target image in the original image from the off-chip memory, it is necessary to calculate the sub-pixel positions of the current target row and the next target row in the original image, and then determine the corresponding rows of the current target row and the next target row in the original image; if the current target row and the next target row have one or two identical corresponding rows in the original image, then after obtaining the corresponding row data of the current target row in the original image, the chip no longer repeatedly obtains the same data for the next target row, but directly uses the data stored in the corresponding input buffer for subsequent calculations and transmissions.
7. The chip according to claim 5, It is characterized in that Before the corresponding output cache has not completely stored the current target row of the target image, if a group of input caches is in an idle state and a corresponding row of the next target row in the original image has not been acquired, then the corresponding row data of the next target row in the original image that has not been acquired will continue to be read from the off-chip memory and stored in a group of input caches that are in an idle state. However, the data in this group of input caches must wait until the current target row is completely stored in the corresponding output cache and the next target row becomes the new current target row before it can be horizontally scaled and stored in a corresponding group of intermediate caches.
8. The chip according to claim 1, It is characterized in that The chip includes a plurality of identification registers for coordinating and controlling the data flow of the input cache, the intermediate cache and the output cache, so that the up / down sampling calculation process and the image cache process are performed in parallel and form a multi-stage pipeline.
9. The chip according to claim 1, It is characterized in that The chip also includes a backup data pool, which is used to temporarily store partial pixels of a target image read from the first output buffer or the second output buffer to be sent to the off-chip memory; the number of data stored in the backup data pool is greater than or equal to the number of clock cycles required to read one data from the first output buffer or the second output buffer.
10. A chip-based image scaling method. It is characterized in that Applied to the chip according to claims 1 to 9, the method comprises: Reading K lines of data of the original image from the off-chip memory according to data scheduling requirements, and writing the data into the first input buffer and the second input buffer; Actuating the first horizontal scaling module and the second horizontal scaling module to perform horizontal up / down sampling on the original image data obtained by the first input buffer and the second input buffer in the current cycle to obtain K lines of target image data that meet the target horizontal resolution, and cache them into the first intermediate buffer and the second intermediate buffer respectively, and at the same time triggering the first input buffer and the second input buffer to read the K lines of original image data according to the data scheduling requirement; wherein the first input buffer and the second input buffer are executed once according to the data scheduling requirement as one cycle; Determine whether it is necessary to perform vertical scaling on a plurality of rows of data of the original image acquired in the current cycle, and if necessary, actuate the first vertical scaling module to read the data cached in the current cycle of the first intermediate cache and the second intermediate cache to perform vertical up / down sampling to obtain M rows of data of the target image meeting the target vertical resolution and cache them into the first output cache; When all the data of the partial area of the target image obtained in the current cycle are cached in the first output cache, the first horizontal scaling module and the second horizontal scaling module are activated to perform horizontal up / down sampling on the K lines of data of the original image obtained by the first input cache and the second input cache in the next cycle to obtain the K lines of data of the target image that meet the target horizontal resolution, and cache them in the first intermediate cache and the second intermediate cache respectively; Determine whether it is necessary to perform vertical scaling on a plurality of rows of data of the original image acquired in the next cycle, and if necessary, actuate the first vertical scaling module to read the data cached in the first intermediate cache and the second intermediate cache in the next cycle to perform vertical up / down sampling to obtain N rows of data of the target image meeting the target vertical resolution and cache them into the second output cache; When all the data of the partial area of the target image obtained in the current cycle are cached in the second output cache, the above steps are repeated to perform a new round of stretching transformation on the remaining image data of the original image until a complete target image that meets the target resolution is obtained.
11. A camera, It is characterized in that The chip comprises the chip as claimed in any one of claims 1 to 9, wherein the chip is used to perform scaling transformation processing on the image captured by the camera to obtain an image that meets the target resolution.
Citation Information
Patent Citations
Image zooming method and device, computer readable storage medium and terminal equipment
CN114418844A
Image zooming method, processing equipment and storage medium
CN115471404A
Image reduction method and device, terminal and storage medium
CN115689879A
Chip and image scaling transformation method based on chip
CN116664378A
Efficient scaling of image data
US20070041662A1
Cited By
IP core for remote sensing image preprocessing
CN119938597A
Image processing method and device, processor and computer equipment
CN120953053A
