Mobile device and image processing method

Abstract

The invention relates to a mobile device and an image processing method. An image processing circuit is electrically connected to two buses. The processor transmits an instruction and a plurality of coefficients of the inverse matrix to the image processing circuit through the second bus, wherein the instruction is used for instructing to execute a conversion program on the input image. The image processing circuit calculates the coordinates of the input pixels of the input image according to the inverse matrix and the coordinates of the output pixels, thereby reading the input pixels from the memory through the first bus. The image processing circuit also writes the output pixel into the memory through the first bus.

Description

Technical Field

[0001] This disclosure relates to image processing methods applicable to mobile devices. Background Technology

[0002] Mobile devices (such as smartphones, smartwatches, and smart bands) typically have display panels. To present the user interface, the image needs to undergo processing, such as perspective transformation and affine transformation. These transformations consume computational resources, thus affecting power consumption, which is especially critical in mobile devices with limited battery capacity. In some existing technologies, the graphics processing unit performs these procedures, but this consumes a significant amount of power. Summary of the Invention

[0003] Embodiments of this disclosure provide a mobile device including a processor, a first bus, a memory, a bridge, a second bus, a display controller, and an image processing circuit. The first bus is electrically connected to the processor. The memory is electrically connected to the first bus. A first terminal of the bridge is electrically connected to the first bus. The second bus is electrically connected to a second terminal of the bridge, wherein the bandwidth of the second bus is lower than the bandwidth of the first bus. The display controller is electrically connected to the second bus. The image processing circuit is electrically connected to the first bus and the second bus. The processor transmits instructions and multiple coefficients of the inverse matrix of a transformation matrix to the image processing circuit via the first bus, the bridge, and the second bus. The instructions instruct the image processing circuit to perform a transformation procedure on an input image. Upon receiving the instructions, the image processing circuit calculates the coordinates of an input pixel of the input image based on the coefficients and the coordinates of the output pixel, thereby reading the input pixel from the memory via the first bus. The image processing circuit writes the output pixel to the memory via the first bus.

[0004] In some embodiments, for the coordinates of the output pixel, the number of input pixels is 4, and the image processing circuit is also used to perform bilinear interpolation on the input pixels to calculate the equivalent input pixels.

[0005] In some embodiments, after the image processing circuit generates an output image, the next output image is automatically generated through a contiguous mode, an auto-reload mode, or a linked list mode.

[0006] In some embodiments, the conversion process includes perspective conversion or affine conversion, and the image processing circuit multiplies the coordinates of the output pixel by an inverse matrix and then divides them by a preset value to obtain the coordinates of the input pixel.

[0007] In some embodiments, the number of input images is greater than one, and the image processing circuit is used to perform a mixing procedure on multiple input images.

[0008] In some embodiments, the first bus is an Advanced eXtensible Interface (AXI), and the second bus is an Advanced Peripheral Bus (APB).

[0009] In some embodiments, the image processing circuit includes the following components: a first port connected to a first bus; a second port connected to a second bus; a global register connected to the second port; a write layer register connected to the second port; and a write circuit connected to the write layer register and the first port. The image processing circuit also includes a plurality of layer processors, each layer processor including: a read layer register connected to the second port; a coordinate calculation circuit for calculating the coordinates of input pixels; a read circuit for controlling a plurality of direct memory access channels; an input buffer connected to the first port for storing input pixels; and a calculation circuit connected to the input buffer. The image processing circuit also includes: an interpolation circuit connected to the calculation circuit; a mixing circuit connected to the interpolation circuit; a pixel format converter connected to the mixing circuit; and an output buffer connected to the first port.

[0010] In some embodiments, the number of input images is greater than one, and the mixing circuit includes a plurality of mixers, each of which is used to accept two input images and perform a mixing procedure.

[0011] In some embodiments, the input buffer and the output buffer have a first-in, first-out (FIFO) mechanism.

[0012] In some embodiments, the processor is also configured to transmit pause instructions, resume instructions, or abort instructions to the image processing circuit via a first bus, a bridge, and a second bus.

[0013] In some embodiments, the mobile device further includes an external device electrically connected to the image processing circuitry. The image processing circuitry communicates with the external device based on a direct memory access handshake.

[0014] From another perspective, embodiments of this disclosure propose an image processing method applicable to an image processing circuit in a mobile device. The image processing method includes: receiving instructions from a processor and multiple coefficients of the inverse matrix of a transformation matrix via a second bus; after receiving the instructions, calculating the coordinates of input pixels of an input image based on the coefficients and the coordinates of output pixels, thereby reading the input pixels from a memory via a first bus; and writing the output pixels to a memory via the first bus.

[0015] Furthermore, the number of the at least one input pixel is 4, and the image processing method further includes: performing a bilinear interpolation on the at least one input pixel to calculate an equivalent input pixel.

[0016] Furthermore, the image processing method further includes: after generating an output image, automatically generating the next output image through a continuous mode, an automatic reload mode, or a linked list mode.

[0017] Furthermore, the conversion process includes a perspective conversion or an affine conversion, and the image processing method further includes: multiplying the coordinates of the output pixel by the inverse matrix and then dividing by a preset value to obtain the coordinates of the at least one input pixel.

[0018] Furthermore, if the number of at least one input image is greater than 1, the image processing method further includes performing a mixing procedure on the at least one input image.

[0019] Furthermore, the first bus is an advanced expandable interface, and the second bus is an advanced peripheral bus.

[0020] Furthermore, the image processing method further includes receiving a pause command, a resume command, or a stop command from the processor via the second bus. Attached Figure Description

[0021] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings.

[0022] Figure 1 This is a schematic diagram of the system architecture of a mobile device according to one embodiment.

[0023] Figure 2 This is a schematic diagram illustrating the transformed coordinates and bilinear interpolation according to one embodiment.

[0024] Figure 3 This is a schematic diagram illustrating a mixing process according to one embodiment.

[0025] Figure 4 This is a schematic diagram of the internal architecture of an image processing circuit 170 according to one embodiment.

[0026] Figure 5 This is a flowchart illustrating an image processing method according to an embodiment.

[0027] Symbol Explanation

[0028] 100: Mobile devices

[0029] 110: Processor

[0030] 120: First bus

[0031] 130: Bridge

[0032] 140: Second bus

[0033] 150: Asynchronous bridge (non-synchronous bridge)

[0034] 160: Display controller

[0035] 170: Image processing circuit

[0036] 181-183: Memory

[0037] 190: External devices

[0038] 210: Input Image

[0039] 220: Output Image

[0040] 301-304: Layers

[0041] 310, 320, 330: Mixers

[0042] 311, 321: Output

[0043] 312, 322, 332: Multiplexers

[0044] 401: Port 1

[0045] 402: Second Port

[0046] 403: Global Register

[0047] 404: Write to layer register

[0048] 405: Write circuit

[0049] 406: Coordinate Generator

[0050] 407: Mixed Circuits

[0051] 408: Pixel Format Converter

[0052] 409: Output buffer

[0053] 410: Layer Processor

[0054] 411: Reading the layer register

[0055] 412: Reading circuit

[0056] 413: Coordinate Calculation Circuit

[0057] 414: Input buffer

[0058] 415: Calculation Circuits

[0059] 416: Interpolation circuit

[0060] 417: Signal

[0061] 501-503: Steps Detailed Implementation

[0062] The terms "first," "second," etc., used in this document do not specifically refer to order or sequence, but are merely used to distinguish elements or operations described using the same technical terminology. In this disclosure, when describing two elements electrically connected to each other, other elements, such as bridges, resistors, switches, etc., may be placed between these two elements.

[0063] Figure 1 This is a schematic diagram illustrating the system architecture of a mobile device according to one embodiment. Please refer to... Figure 1 The mobile device 100 is, for example, a smartwatch, a smart bracelet, a mobile phone, or other mobile devices. The mobile device 100 includes a processor 110, a first bus 120, a bridge 130, a second bus 140, an asynchronous bridge 150, a display controller 160, an image processing circuit 170, and memories 181-183.

[0064] Processor 110, such as a central processing unit (CPU) or microprocessor, is electrically connected to the first bus 120. Memory 181-183 is also electrically connected to the first bus 120. In this embodiment, memory 181-183 are of different types; for example, memory 181 is static random access memory (SRAM), memory 182 is flash memory, and memory 183 is pseudo RAM, but this disclosure is not limited thereto. Memory 181-183 stores input images that are yet to be processed, and processed images can also be stored in memory 181-183.

[0065] A first bus 120 is electrically connected to a first terminal of bridge 130, and a second bus 140 is electrically connected to a second terminal of bridge 130. Bridge 130 is used to transfer data from the first terminal (or the second terminal) to the second terminal (or the first terminal). In this embodiment, the bandwidth of the second bus 140 is lower than that of the first bus 120. For example, the first bus 120 may be an Advanced eXtensible Interface (AXI), and the second bus 140 may be an Advanced Peripheral Bus (APB), but this disclosure is not limited thereto. An asynchronous bridge 150 is electrically connected to the second bus 140, and a display controller 160 is electrically connected to the asynchronous bridge 150. The display controller 160 may also be electrically connected to a display (not shown). Because the clock or timing (e.g., frame rate) of the display controller 160 differs from the clock on the second bus 140, the asynchronous bridge 150 is required for data transmission.

[0066] The image processing circuit 170 is electrically connected to the first bus 120 and the second bus 140 to perform a conversion program on the input images stored in the memories 181-183. This conversion program may include perspective transformation or affine transformation. Specifically, the conversion program can be expressed as the following mathematical formula 1.

[0067] [Mathematical Expression 1]

[0068] M×X=Y

[0069] Where X represents the coordinates of a pixel in the input image, including three values: x-coordinate, y-coordinate, and homogeneous coordinate. M is the transformation matrix, with a size of 3×3. Y represents the coordinates of the corresponding pixel in the output image, also including x-coordinate, y-coordinate, and homogeneous coordinate. By setting the coefficients in the transformation matrix, perspective transformation and affine transformation can be achieved. In this embodiment, the coordinates of the input pixel are derived from the coordinates of the output pixel, as shown in the following mathematical formula 2.

[0070] [Mathematical Expression 2]

[0071] X = M -1 ×Y

[0072] Where M -1 The inverse matrix representing the transformation matrix is ​​also 3×3 in size, and its coefficients are provided by processor 110. Processor 110 will sequentially transmit instructions and the inverse matrix M through the first bus 120, bridge 130, and second bus 140.-1 The coefficients are given to the image processing circuit 170, where the instruction is used to instruct the conversion procedure to be performed on the input image. After receiving the instruction, the image processing circuit 170 uses the inverse matrix M to... -1 The coefficients and the Y coordinates of the output pixels are used to calculate the X coordinates of the input pixels in the input image. After obtaining the X coordinates, the image processing circuit 170 reads the input pixels from the memory 181-183 via the first bus 120. After completing the conversion process, the image processing circuit 170 writes these output pixels (forming the output image) into the memory 181-183 via the first bus 120.

[0073] Due to the inverse matrix M -1 The coefficients may be floating-point numbers. To reduce the computational load on the image processing circuit 170, the processor 110 can first multiply these coefficients by a preset value (e.g., "65536", but this disclosure is not limited to this), thereby making all coefficients represent integers. Therefore, after the image processing circuit 170 multiplies the coordinates of the output pixel by the inverse matrix, it needs to divide by the preset value to obtain the coordinates of the input pixel. In some embodiments, any complement can also be used to represent negative numbers.

[0074] In some embodiments, the image processing circuit 170 may be a circuit module integrated into a System on a Chip (SOC). Compared to the prior art where the graphics processing unit (GPU) performs these conversion processes, the image processing circuit 170 has advantages such as low cost and small size. For example, the prior art uses the memory in the graphics processing unit to temporarily store input images, but this requires a large amount of memory space and may even lead to insufficient memory. In contrast, in this embodiment, the image processing circuit 170 does not need such a large amount of memory space to store the entire (or multiple) input images. Instead, it first deduces the coordinates of the input pixels from the coordinates of the output pixels and then reads the required input pixels from the memory 181-183. This approach can save memory costs. In addition, the image processing circuit 170 is electrically connected to a first bus 120 and a second bus 140 with different bandwidths. The second bus 140 with lower bandwidth is used to transmit instructions, while the first bus 120 with higher bandwidth is used to transmit images. This allocation can improve overall efficiency.

[0075] In some embodiments, the image processing circuit 170 is also electrically connected to an external device 190. The image processing circuit 170 communicates with the external device 190 based on a Direct Memory Access (DMA) handshake, thereby reading input images or writing output images to the external device 190. The external device 190 is, for example, a memory card, a portable hard drive, or any storage device. In DMA handshake mode, each data transfer requires confirmation of readiness from the external device 190. This approach better synchronizes data transfer with the status of the external device 190, and also provides better data transfer efficiency, especially in scenarios involving images or videos.

[0076] Figure 2 This is a schematic diagram illustrating the transformed coordinates and bilinear interpolation according to one embodiment. Please refer to... Figure 2 Here, the input image 210 needs to be transformed to produce the output image 220. In this example, the coordinates of the output pixels are (x... r ,y r After the above calculations, the coordinates of the corresponding input pixels may not be integers. Therefore, the four closest input pixels will be found, with coordinates (x, y, y) respectively. s ,y s ), (x s+1 ,y s ), (x s ,y s+1 ), (x s+1 ,y s+1 By performing bilinear interpolation on these four input pixels, we can obtain the grayscale value of the equivalent input pixel, which will then be used in the above conversion process.

[0077] The image processing circuit 170 also supports alpha masking, allowing preprocessing of the alpha level for each input primitive. For example, for the grayscale value A of an input pixel... m A can be performed m ×A c The calculation of / 255, where A c Transparency is also used as a weight in the blending process to weight the grayscale values ​​of two input images. Figure 3 This is a schematic diagram illustrating the mixing process according to one embodiment. Please refer to it. Figure 3The image processing circuit 170 includes multiple mixers 310, 320, and 330. There are four input images, represented as layers 301 to 304. These four layers 301 to 304 can first undergo the aforementioned conversion process before being mixed. The first input of mixer 310 is layer 301, and the second input is layer 302. The output 311 of mixer 310 serves as the input of mixer 320, and another input of mixer 320 is layer 303. The output 321 of mixer 320 serves as the input of mixer 330, and another input of mixer 330 is layer 304. Of the two layers input to the mixer, one is treated as the foreground and the other as the background. The operation of a single mixer can be represented by the following mathematical formula 3.

[0078] [Mathematical Expression 3]

[0079]

[0080]

[0081] Where C FG This represents the color components of the foreground, such as grayscale values ​​for red, green, or blue. C BG This represents the color components of the background, such as grayscale values ​​for red, green, or blue. A FG Indicates the transparency of the foreground. A BG The transparency of the background. R A represents the color components output by the mixer. R It is the transparency of the mixer output.

[0082] Furthermore, multiplexers 312, 322, and 332 are used to determine whether to use the value "0" for mixing. For example, multiplexer 312 can select the value "0" as the input to mixer 310, thus the grayscale value C in mathematical formula 3... BG It will be set to 0.

[0083] In some embodiments, the source of mixers 310, 320, and 330 can also be a solid color image. For example, processor 110 may specify a solid color grayscale value, where each pixel in the solid color image has a solid color grayscale value. Assuming layer 302 is a solid color image, mixer 310 will mix the input image (layer 301) with this solid color image. In such an example, image processing circuit 170 does not need to read the solid color image from memories 181-183; it only needs to retrieve the grayscale value C from the aforementioned mathematical formula 3. FG Or grayscale value C BGReplacing the image with a solid color grayscale value is equivalent to performing a mixing process on the input image and the solid color image. In other embodiments, the image processing circuit 170 may also have more mixers, which is not limited to this disclosure.

[0084] The image processing circuit 170 can use red, green, and blue (RGB) format for input and output, and also supports various different RGB modes, such as RGB565, RGB888, etc., or it can also support A8 format. In other embodiments, the image processing circuit 170 can also use other color formats, such as Yuv. The image processing circuit 170 can also perform format conversion on the input or output image. The image processing circuit 170 can also support color keying, which can change a set grayscale value (also called a key) in the input image to a preset grayscale value (e.g., 0), that is, all pixels that match the set grayscale value will be changed to 0. In some embodiments, the color key can also be a range, that is, grayscale values ​​within a certain range will be changed to the preset grayscale value.

[0085] The image processing circuit 170 also provides a multi-frame mechanism, which can employ a contiguous mode, an auto-reload mode, or a linked list mode. After the image processing circuit 170 generates an output image, it can automatically process (e.g., generate) the next output image using one of these modes. For example, in the linked list mode, the processor 110 sends a list to the image processing circuit 170. This list records multiple nodes, each containing the start point of the frame and a pointer to the next node. Those skilled in the art will understand the above modes, and will not elaborate further here. In this way, the processor 110 does not need to issue corresponding instructions for each frame; the image processing circuit 170 automatically executes image processing procedures for multiple frames.

[0086] The image processing circuit 170 can support secure and non-secure access. The image processing circuit 170 can also support suspending, resuming, or aborting during processing. That is, the processor 110 can also transmit suspend, resume, and abort commands to the image processing circuit 170 through the first bus 120, bridge 130, and second bus 140.

[0087] Figure 4 This is a schematic diagram illustrating the internal architecture of the image processing circuit 170 according to one embodiment. Please refer to... Figure 1 and Figure 4The image processing circuit 170 includes a first port 401, a second port 402, a global register 403, a write layer register 404, a write circuit 405, a coordinate generator 406, a mixing circuit 407, a pixel format converter 408, an output buffer 409, and multiple layer processors 410. Each layer processor 410 includes a read layer register 411, a read circuit 412, a coordinate calculation circuit 413, an input buffer 414, a calculation circuit 415, and an interpolation circuit 416.

[0088] The first port 401 is connected to the first bus 120, and the second port 402 is connected to the second bus 140. For example, the first port 401 conforms to the AXI bus specification, and the second port 402 conforms to the APB bus specification. It is worth noting that... Figure 4 The positions of the first bus 120 and the second bus 140 are different. Figure 1 The position is in the middle, but it does not affect the following explanation.

[0089] Global register 403 is connected to second port 402 to store global information. For example, the global information includes suspend, resume, or abort instructions issued by processor 110. The global information may also indicate which of the following modes—continuous mode, automatic reload mode, and linked list mode—to execute. In some embodiments, image processing circuitry 170 is also configured to receive / transmit a signal 417 representing an interrupt, for example, an interrupt may be issued after processing an image.

[0090] The write layer register 404 is connected to the second port 402 to store various information about the output image, such as the size and position of the output image. The write circuit 405 is connected to the write layer register 404 and the first port 401 to convert the position information of the output image into a direct memory access channel, and then write the output pixels (output image) to the memory 181-183 through the first port 401 and the first bus 120.

[0091] The layer read register 411 is connected to the second port 402 to store various information about the input image, such as the size and position of the input image, the starting position of the layer when blending, the transparency, the set grayscale value, the solid color grayscale value, and the coefficients of the inverse matrix. In some embodiments, each layer has its own corresponding inverse matrix coefficients. The coordinate generator 406 is used to generate the coordinates of the output pixels, for example, generating coordinates from left to right and then from top to bottom in the output image. The coordinate calculation circuit 413 is used to calculate the coordinates of the required input pixels based on the coordinates of the output pixels and the coefficients of the inverse matrix. This step has been described in detail above and will not be repeated here. The read circuit 412 is connected to the layer read register 411 to convert the position information of the input pixels into a direct memory access channel, and then reads the input pixels from the memory 181-183 through the first port 401 and the first bus 120.

[0092] Input buffer 414 is connected to the first port 401, and the input pixels read by reading circuit 412 are stored in input buffer 414. Calculation circuit 415 is connected to input buffer 414 and is used to perform pixel format conversion, color keying, etc. Interpolation circuit 416 is connected to calculation circuit 415 and is used to perform bilinear interpolation; the relevant operations have been explained above.

[0093] Each layer processor 410 processes a single layer, and the interpolation circuit 416 in these layer processors 410 is connected to the mixing circuit 407. The mixing circuit 407 includes multiple mixers for performing a mixing procedure, the operation of which has been described above. For example, each of the multiple mixers accepts two of the input images and performs the mixing procedure. A pixel format converter 408 is connected to the mixing circuit 407 to perform pixel format conversion, and the converted output pixels are stored in the output buffer 409. The output buffer 409 is connected to the first port 401, and the output pixels are written to the corresponding memories 181-183 under the control of the write circuit 405. In some embodiments, the input buffer 414 and the output buffer 409 have a First In First Out (FIFO) mechanism.

[0094] Figure 5 This is a flowchart illustrating an image processing method according to an embodiment, which is executed by an image processing circuit 170. In step 501, instructions and multiple coefficients of the inverse matrix of the transformation matrix are received from the processor via a second bus. In step 502, the coordinates of the input pixels of the input image are calculated based on these coefficients and the coordinates of the output pixels, thereby reading the input pixels from the memory via a first bus. In step 503, the output pixels are written to the memory via the first bus. Figure 5Each step has been explained in detail above and will not be repeated here. It is worth noting that... Figure 5 Each step can be implemented as multiple program codes or circuits, but this invention is not limited thereto. Furthermore, Figure 5 The method can be used in conjunction with the above embodiments or alone; in other words, Figure 5 Other steps can also be added between the various steps.

[0095] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes and modifications without departing from the concept and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A mobile device, comprising: One processor; A first bus is electrically connected to the processor; A memory, electrically connected to the first bus; A bridge, wherein a first terminal of the bridge is electrically connected to the first bus; A second bus is electrically connected to a second terminal of the bridge, wherein a bandwidth of the second bus is lower than a bandwidth of the first bus; A display controller, electrically connected to the second bus; and An image processing circuit is electrically connected to the first bus and the second bus. The processor transmits an instruction and multiple coefficients of an inverse matrix of a transformation matrix to the image processing circuit via the first bus, the bridge, and the second bus. The instruction instructs the execution of a transformation procedure on at least one input image. Upon receiving the instruction, the image processing circuit calculates the coordinates of at least one input pixel of the at least one input image based on the plurality of coefficients and the coordinates of an output pixel, thereby reading the at least one input pixel from the memory via the first bus. The image processing circuit is used to write the output pixel to the memory via the first bus.

2. The mobile device of claim 1, wherein the number of the at least one input pixel is 4 for the coordinate of the output pixel, and the image processing circuit is further configured to perform a bilinear interpolation on the at least one input pixel to calculate an equivalent input pixel.

3. The mobile device as claimed in claim 1, wherein after the image processing circuit generates an output image, the next output image is automatically generated through a continuous mode, an automatic reload mode, or a linked list mode.

4. The mobile device as claimed in claim 1, wherein the conversion process includes a perspective conversion or an affine conversion, and the image processing circuit is used to multiply the coordinates of the output pixel by the inverse matrix and then divide by a preset value to obtain the coordinates of the at least one input pixel.

5. The mobile device of claim 1, wherein the number of the at least one input image is greater than 1, and the image processing circuit is configured to perform a mixing procedure on the at least one input image.

6. The mobile device of claim 1, wherein the first bus is an advanced expandable interface and the second bus is an advanced peripheral bus.

7. The mobile device of claim 1, wherein the image processing circuit comprises: A first port is connected to the first bus; A second port is connected to the second bus; A global temporary register is connected to this second port; Write to the layer register and connect it to the second port; A write circuit is connected to the write layer register and the first port; Multiple layer processors, each of which includes: A layer buffer is read and connected to the second port; A coordinate calculation circuit for calculating the coordinates of the at least one input pixel; A read circuit is used to control multiple direct memory access channels; An input buffer, connected to the first port, is used to store the at least one input pixel; A computing circuit is connected to the input buffer; and An interpolation circuit is connected to the computing circuit; A hybrid circuit is connected to the interpolation circuit; A pixel format converter is connected to the mixing circuit; and An output buffer is connected to this first port.

8. The mobile device of claim 7, wherein the number of the at least one input image is greater than 1, and the mixing circuit includes a plurality of mixers, each of the plurality of mixers being configured to receive two of the at least one input image and perform a mixing procedure.

9. The mobile device of claim 7, wherein the input buffer and the output buffer have a first-in-first-out (FIFO) mechanism.

10. The mobile device of claim 1, wherein the processor is further configured to transmit a pause instruction, a resume instruction, or a stop instruction to the image processing circuit via the first bus, the bridge, and the second bus.

Images