System and method of image compression

Abstract

An image compression system includes a memory with a frame buffer, a temporal noise reduction circuit, a compression circuit, and a controller. The temporal noise reduction circuit is configured to receive frames from an image sensor and mix these frames with a decompressed reference frame from the frame buffer to form a temporal noise reduction frame. The compression circuit is configured to receive the temporal noise reduction frame from the temporal noise reduction circuit and compress the temporal noise reduction frame into a compressed reference frame using a ratio in the frame buffer. The compression circuit is further configured to decompress the compressed reference frame in the frame buffer into a decompressed reference frame using this ratio. The controller is configured to adaptively increase the ratio according to the number of programs to ensure that the memory has sufficient storage space to execute the programs.

Description

[Technical Field]

[0001] The present invention relates to a compression system in a camera module, and more particularly to an image compression system. [Previous Technology]

[0002] High screen-to-body ratio has always been a requirement for consumer devices such as laptops and mobile phones. Given this requirement, camera modules in consumer devices need to be smaller, especially the Y-size, while maintaining low cost. Image sensor pixel sizes are continuously shrinking to 1.12μm or even smaller, therefore image signal processors need to properly handle sensor noise. Temporal noise reduction (TNR) is an effective method for handling high sensor noise, typically achieved by cascading temporal noise reduction and spatial noise reduction (2DNR).

[0003] The camera module includes an image signal processor (ISP) for processing images, which can employ methods such as temporal noise reduction, spatial noise reduction, black level subtraction (BLS), automatic white balance (AWB), and color correction. The camera module may also include artificial intelligence (AI) detection for rear-view applications, thus limiting the size and cost of the camera module. This means that various operations / programs will share limited static random-access memory (SRAM) in a serial or parallel manner. When two or more memory-intensive programs execute in parallel and cannot utilize dynamic random-access memory (DRAM), the SRAM embedded in the camera module will not be able to support the parallel operation of these programs.

[0004] A new method or system is needed to make more efficient use of limited embedded static random access memory to support parallel operation of programs. [Summary of the Invention]

[0005] The purpose of this invention is to provide an image compression system and method that can still support the parallel execution of programs even when the memory space of the camera module is limited.

[0006] One aspect of the present invention provides an image compression system, comprising a memory, a temporal noise reduction (TNR) circuit, a compression circuit, and a controller. The memory includes a frame buffer. The TNR circuit is configured to receive frames from an image sensor and mix the frames with decompressed reference frames from the frame buffer to form a TNR frame. The compression circuit is coupled to the TNR circuit and configured to receive the TNR frames from the TNR circuit, compress the TNR frames into a compressed reference frame using a ratio, and store the compressed reference frame in the frame buffer. The compression circuit is further configured to decompress the compressed reference frame in the frame buffer into a decompressed reference frame using a ratio. The controller is configured to adaptively adjust the ratio when at least two programs are executed in parallel.

[0007] In some embodiments, the time-denoising circuit is further configured to perform weighted calculations on the pixel data of the frame and the pixel data of the decompressed reference frame to obtain a time-denoising frame.

[0008] In some embodiments, the image compression system further includes a spatial noise reduction (2DNR) circuit configured to receive temporal noise reduction frames from a temporal noise reduction circuit and perform weighted calculations to process the temporal noise reduction frames into spatial noise reduction frames.

[0009] In some embodiments, the spatial noise reduction circuit is further configured to perform weighted calculations on the pixel data of the temporal noise reduction frame and the surrounding pixel data to obtain the spatial noise reduction frame.

[0010] In some embodiments, the time noise reduction circuit is further configured to determine whether each pixel in the frame is a stationary pixel or a moving pixel in order to obtain a pixel state result.

[0011] In some embodiments, the temporal noise reduction circuit and the spatial noise reduction circuit are both weighted calculations based on the pixel state results.

[0012] In some embodiments, the controller uses a first noise reduction (NR) setting to perform noise reduction in response to a decrease in the ratio, and the controller uses a second noise reduction setting to perform noise reduction in response to an increase in the ratio, wherein the first noise reduction setting consists of a first temporal noise reduction setting and a first spatial noise reduction setting, and wherein the second noise reduction setting consists of a second temporal noise reduction setting and a second spatial noise reduction setting.

[0013] In some embodiments, the controller is further configured to increase the ratio in response to the need to execute these at least two programs in order to free up a portion of the memory.

[0014] In some embodiments, the at least two programs include at least one video program and at least one computer vision / neural network (CV / NN) program, wherein the controller is further configured to utilize a portion of the memory to execute the computer vision / neural network programs.

[0015] Another aspect of the present invention is to provide an image compression method, comprising: receiving a frame from an image sensor; mixing the frame with a decompression reference frame from a frame buffer into a time-denoised frame through a time-denoised circuit; adjusting a ratio through a controller when at least two programs are executed in parallel; compressing the time-denoised frame into a compressed reference frame through a compression circuit according to a ratio determined by the controller; and decompressing the compressed reference frame into a decompressed reference frame through a compression circuit according to a ratio determined by the controller, wherein the controller increases the ratio in response to the need to execute at least two programs, and decreases the ratio in response to the completion of the at least two programs.

Implementation Method

[0017] Embodiments of the present invention will be discussed in detail below. It will be understood that the embodiments provide many applicable concepts that can be implemented in a wide variety of specific situations. The embodiments discussed and disclosed are for illustrative purposes only and are not intended to limit the scope of the invention.

[0018] FIG1 is a functional block diagram of an image compression system 100 according to an embodiment of the present invention. The image compression system 100 includes a memory 110 with a frame buffer (not shown in FIG1), a temporal noise reduction (TNR) circuit 120, a spatial noise reduction (2DNR) circuit 130, a compression circuit 140 coupled to the temporal noise reduction circuit 120, a controller 150, a first scaler 160, a second scaler 170, a bus 180, and a neural network processing unit (NPU) 190. The memory 110 may be random access memory (RAM), static random access memory (SRAM), flash memory, solid state drive (SSD), or other similar components or combinations thereof, but is not limited thereto. The processor included in controller 150 may be a central processing unit (CPU), graphics processing unit (GPU), microcontroller unit (MCU), microprocessor, system-on-chip (SoC), digital signal processor (DSP), application-specific integrated circuit (ASIC), programmable logic controller (PLC), or a combination of the above, but is not limited thereto. Bus 180 may be an advanced extensible interface (AXI) bus or an advanced high performance bus (AHB), but is not limited thereto. Neural network processing unit 190 is coupled to memory 110 and second scaler 170 via bus 180.

[0019] The image compression system 100 shown in Figure 1 is mainly used in image signal processor (ISP) chips, digital signal processor (DSP) chips, or artificial intelligence (AI) processor chips embedded in camera modules of electronic devices (such as laptops). In some embodiments, the image signal processor chip (or DSP chip, AI processor chip) can also be used in external universal serial bus (USB) network cameras. It should be noted that the memory 110, temporal noise reduction circuit 120, spatial noise reduction circuit 130, compression circuit 140, controller 150, first scaler 160, second scaler 170, bus 180, and neural network processing unit 190 are all located inside the image signal processor chip (or DSP chip, AI processor chip). The image compression system 100 communicates with the host system H of the electronic device through interface I, as shown in Figure 1.

[0020] In some embodiments, the image sensor may be a visible light sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, other light sensing elements, other light sensing devices, or combinations of the above elements, but is not limited thereto. It should be noted that one or more image sensors may be provided depending on the application.

[0021] After the frame is captured by the image sensor, the time-denoising circuit 120 is configured to receive the frame from the image sensor and mix the frame with the decompressed reference frame from the frame buffer to form a time-denoising frame.

[0022] Specifically, the temporal noise reduction circuit 120 is configured to mix this frame with the decompressed reference frame using its internal mixing device. It should be noted that the mixing device performs a weighted calculation on the pixel data of this frame and the pixel data of the decompressed reference frame to mix the frame with the decompressed reference frame, thereby obtaining the temporal noise reduction frame.

[0023] Although temporal noise reduction is an effective method for processing high noise in image sensors, it requires a considerable amount of memory space in memory 110 as a frame buffer to store the processed frames. Therefore, to reduce the cost of memory 110, a compression / decompression process can be employed. In this way, the frames processed by the temporal noise reduction circuit 120 can be compressed to reduce the frame size and then written to a smaller storage space.

[0024] Compression circuit 140 is coupled to time-denoising circuit 120. To reduce the cost of memory 110, after time-denoising circuit 120 obtains a time-denoising frame, compression circuit 140 is configured to receive the time-denoising frame from time-denoising circuit 120 and compress the time-denoising frame into a compressed reference frame using a ratio. The compressed reference frame is then written into a frame buffer. Since the size of the compressed reference frame is smaller than the time-denoising frame, the size of memory 110 can be reduced to lower chip costs. Furthermore, compression circuit 140 is further configured to decompress the compressed reference frame in the frame buffer according to a ratio, using it as the decompressed reference frame for the next frame following this frame.

[0025] The time-based noise reduction circuit 120 is further configured to determine whether each pixel within the frame is a stationary pixel or a moving pixel, thereby obtaining a pixel state result. In one embodiment of the present invention, the pixel state result includes weights for stationary pixels and moving pixels.

[0026] After obtaining the pixel state result, the controller 150 processes the stationary / moving pixels in the image using the temporal noise reduction circuit 120 and the spatial noise reduction (2DNR) circuit 130 based on the pixel state result. That is, the controller 150 processes the stationary / moving pixels in the frame using the temporal noise reduction circuit 120 and the spatial noise reduction circuit 130 according to the weight of the stationary / moving pixels. Specifically, the image compression system 100 further includes a spatial noise reduction circuit 130 coupled to the temporal noise reduction circuit 120, which is configured to receive the temporal noise reduction frame from the temporal noise reduction circuit 120 and process the stationary / moving pixels in the temporal noise reduction frame into a spatial noise reduction frame. Further, the spatial noise reduction circuit 130 is configured to perform weighted calculations on the center pixel data and surrounding pixel data of the stationary / moving pixels in the temporal noise reduction frame to obtain the spatial noise reduction frame.

[0027] It should be noted that the controller 150 uses both the temporal noise reduction circuit 120 and the spatial noise reduction circuit 130 to process each pixel in the frame (including the input frame and the temporal noise reduction frame). In one embodiment of the present invention, since the temporal noise reduction circuit 120 has a better noise reduction effect on stationary pixels, when the temporal noise reduction circuit 120 determines that a pixel in the frame is a stationary pixel, the controller 150 can process this stationary pixel by increasing the weight of the decompressed reference frame (while decreasing the weight of the input frame) to obtain the temporal noise reduction frame.

[0028] In another embodiment of the present invention, since the spatial noise reduction circuit 130 has a better noise reduction effect on moving pixels, when the temporal noise reduction circuit 120 determines that the pixel in the frame is a moving pixel, the controller 150 can process the moving pixel by increasing the weight of the surrounding pixel data of the temporal noise reduction frame (while reducing the weight of the center pixel data of the temporal noise reduction frame) to obtain the spatial noise reduction frame.

[0029] When the camera module has limited memory space, the image signal processor chip may not have enough memory space to execute computer vision / neural network programs in parallel and output video for user viewing simultaneously. Therefore, the controller 150 in the image compression system 100 is configured to adaptively adjust the ratio to ensure that the memory 110 has sufficient storage space for executing computer vision / neural network programs in parallel and outputting video for user viewing simultaneously. Specifically, when at least two programs need to be executed (e.g., at least one video program and at least one computer vision / neural network program), the controller 150 will increase the ratio to free up some memory 110.

[0030] Since adjusting the ratio may change the resolution of the output video, in order to make the output video smoother and ensure the user's viewing experience, it is necessary to adopt appropriate noise reduction (NR) settings.

[0031] In one embodiment of the present invention, the controller 150 performs noise reduction using a first noise reduction setting in response to a decrease in the ratio, wherein the first noise reduction setting consists of a first temporal noise reduction setting and a first spatial noise reduction setting. The controller 150 increases the temporal noise reduction intensity in response to a decrease in the ratio. Specifically, the temporal noise reduction circuit 120 uses more decompression reference frames than the input frame to mix the input frame and the decompression reference frames into a temporal noise reduction frame.

[0032] In another embodiment of the present invention, the controller 150 uses a second noise reduction setting for noise reduction in response to an increase in the ratio, wherein the second noise reduction setting consists of a second temporal noise reduction setting and a second spatial noise reduction setting. The controller 150 increases the spatial noise reduction intensity in response to an increase in the ratio. Specifically, when performing weighted calculations, the spatial noise reduction circuit 130 utilizes more of the second pixels surrounding the first pixel in the temporal noise reduction frame than the first pixel, processing the temporal noise reduction frame into a spatial noise reduction frame.

[0033] It should be noted that in the above two embodiments, the time noise reduction intensity of the first noise reduction setting is greater than the time noise reduction intensity of the second time noise reduction setting, and the spatial noise reduction intensity of the first spatial noise reduction setting is less than the spatial noise reduction intensity of the second spatial noise reduction setting.

[0034] The image compression system 100 applied to the image signal processor chip can operate in two modes: video mode and video + computer vision / neural network mode. Figure 2 is a schematic diagram of the parallel execution of video programs and computer vision / neural network programs according to an embodiment of the present invention. The present invention proposes a dynamic ratio image compression method using time-denoising. When the image compression system 100 is in video mode, only video programs are executed. When video programs and computer vision / neural network programs need to be executed simultaneously, the image compression system 100 switches from video mode to video + computer vision / neural network mode (also shown as video + computer vision mode in Figure 2), and can increase the ratio through the controller 150 to free up some memory space for parallel execution of computer vision / neural network programs. It should be noted that when the image compression system 100 is in video + computer vision / neural network mode, the process of capturing frames from the image sensor and outputting the video to the host system (e.g., the host system H shown in Figure 1) continues. Once the computer vision / neural network program has finished executing, the controller 150 reduces the ratio to allow the time-de-noise circuit 120 to use more memory space, and the quality of the output video can be restored to normal.

[0035] It should be noted that the image compression system 100 series intermittently switches to video + computer vision / neural network mode, rather than continuously being in video + computer vision / neural network mode. Therefore, the compression ratio will increase / decrease in response to the intermittent execution of the computer vision / neural network program.

[0036] It should be noted that the switching operation between the video mode and the video + computer vision / neural network mode of the image compression system 100 is progressive. In one embodiment of the present invention, referring to the first two frames of FIG2, in the video mode, the compression ratio (W_CR#1) and decompression ratio (R_CR#1) of the first noise reduction setting (denoised as noise reduction setting #1 in FIG2) are, for example, 3.33. Then, referring to the third frame in FIG2, when the image compression system 100 switches from the video mode to the video + computer vision / neural network mode (represented as video + computer vision mode in FIG2), the controller 150 increases the compression ratio from 3.33 to 5 (for example, W_CR#2 is 5). Then, referring to the fourth frame in FIG2, the decompression ratio also increases from 3.33 to 5 (for example, R_CR#2 is 5), and a portion of the memory 110 is released to store small-sized frames f for parallel execution of computer vision / neural network programs. Simultaneously, in response to the increase in the ratio, the controller 150 changes the first noise reduction setting to the second noise reduction setting (represented as noise reduction setting #2 in Figure 2). That is, the controller 150 uses the second noise reduction setting to reduce noise in the output video frames. Next, referring to the fifth frame in Figure 2, after the computer vision / neural network program has finished executing, the controller 150 reduces the compression ratio from 5 to 3.33. Then, referring to the sixth frame in Figure 2, the controller 150 also reduces the decompression ratio from 5 to 3.33. It should be noted that the switching operation after the seventh frame in Figure 2 (switching between video mode and video + computer vision / neural network mode) can be referenced from the switching operation of the first to sixth frames.

[0037] The image compression system 100 further includes a first scaler 160 and a second scaler 170, both of which are coupled to the spatial noise reduction circuit 130. The first scaler 160 is configured to adjust the resolution of the video output to the host system H accordingly. In one embodiment of the present invention, the user can set the desired resolution (e.g., 5MP, 1080p, 720p, etc.) in the host system H of the camera application (APP), and the first scaler 160 of the image compression system 100 in the camera module then adjusts the resolution of the video output to the host system H accordingly. In addition, the second scaler 170 is configured to output a small frame f as shown in FIG2 for executing computer vision / neural network programs. It should be noted that when the small frame f of the computer vision / neural network program is output through the second scaler 170, the video output by the first scaler 160 for the user to view is still being executed in parallel.

[0038] After adaptively adjusting the compression / decompression ratio, the controller 150 then uses the freed portion of memory 110 to execute computer vision / neural network programs. In one embodiment of the invention, the controller 150 instructs the neural network processing unit 190 to receive small frames f from the second scaler 170 via bus 180 and execute the required computer vision / neural network programs. In another embodiment of the invention, the second scaler 170 first stores the small frames f in memory 110, and the neural network processing unit 190 then accesses the small frames f stored in memory 110 to execute computer vision / neural network programs. After the neural network processing unit 190 has executed each computer vision / neural network program, the controller 150 is further configured to reduce the ratio, and the image compression system 100 switches from video + computer vision / neural network mode to video mode.

[0039] Figure 3 is a flowchart of an image compression method 300 according to an embodiment of the present invention. The image compression method 300 can be used in the system shown in Figure 1, which includes memory, temporal noise reduction circuit, spatial noise reduction circuit, compression circuit, controller, first scaler, second scaler, bus, and neural network processing unit (e.g., the image compression system 100 shown in Figure 1) or other similar systems. As shown in Figure 3, the image compression method 300 includes steps S310 to S350, and the implementation of each step will be described below with reference to Figures 1 to 3.

[0040] Step S310: Receive a frame from the image sensor. For an explanation of step S310, please refer to the operation of each element in the image compression system 100 shown in Figure 1, for example, and will not be repeated here.

[0041] Step S320: The frame and the decompressed reference frame from the frame buffer are mixed into a time-denoised frame by the time-denoising circuit. The explanation of step S320 can be found in the operation of each component in the image compression system 100 shown in Figure 1, for example, and will not be repeated here.

[0042] Step S330: When at least two programs are executed in parallel, the ratio is adjusted by the controller. For an explanation of step S330, please refer to the operation of each component in the image compression system 100 shown in Figure 1, for example, and will not be repeated here.

[0043] Step S340: The time-denoising frame is compressed into a compressed reference frame by the compression circuit according to the ratio determined by the controller. The explanation of step S340 can be found in the operation of each component in the image compression system 100 shown in Figure 1, for example, and will not be repeated here.

[0044] Step S350: The compression reference frame is decompressed into a decompressed reference frame according to the ratio determined by the controller through the compression circuit. For an explanation of step S350, please refer to the operation of each component in the image compression system 100 shown in Figure 1, for example, and will not be repeated here.

[0045] In summary, the image compression system and method of the present invention provide parallel execution of programs (such as video programs, computer vision / neural network programs) when the memory space of the camera module is limited, and adopt appropriate noise reduction settings to reduce noise in response to the increase / decrease of the compression ratio, so as to ensure the user's viewing experience.

[0046] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the scope of the present invention. Anyone skilled in the art can make various changes, substitutions and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be defined by the appended claims. [Simplified Explanation of the Diagram]

[0016] To make the above and other objects, features, advantages and embodiments of the present invention more apparent and understandable, the accompanying drawings are described as follows: Figure 1 is a functional block diagram of an image compression system according to an embodiment of the present invention; Figure 2 is a schematic diagram of a parallel execution of a video program and a computer vision / neural network program according to an embodiment of the present invention; and Figure 3 is a flowchart of an image compression method according to an embodiment of the present invention. [Biomaterial Storage]

[0048] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. An image compression system, comprising: a memory including a frame buffer; a temporal noise reduction (TNR) circuit configured to receive a frame from an image sensor and perform a weighted calculation on pixel data of the frame and pixel data of a decompressed reference frame from the frame buffer to mix the frame and the decompressed reference frame into a temporal noise reduction frame; a compression circuit coupled to the TNR circuit, configured to receive the TNR frame from the TNR circuit, compress the TNR frame into a compressed reference frame using a ratio, and store the compressed reference frame in the frame buffer, the compression circuit further configured to decompress the compressed reference frame in the frame buffer into the decompressed reference frame using the ratio; a controller configured to adaptively adjust the ratio when at least two programs are executed in parallel; and a spatial noise reduction circuit. A spatial noise reduction (SDR) circuit is configured to receive the temporal noise reduction frame from the temporal noise reduction circuit and perform weighted calculations on the pixel data of the temporal noise reduction frame and the surrounding pixel data to obtain a spatial noise reduction frame.

2. The image compression system as claimed in claim 1, wherein the time-denoising circuit is further configured to determine whether each pixel in the frame is a stationary pixel or a moving pixel in order to obtain a pixel state result.

3. The image compression system as described in claim 2, wherein both the temporal noise reduction circuit and the spatial noise reduction circuit perform weighted calculations based on the pixel state results.

4. The image compression system as claimed in claim 3, wherein the controller performs noise reduction using a first noise reduction (NR) setting in response to a decrease in the ratio, and wherein the controller performs noise reduction using a second noise reduction setting in response to an increase in the ratio, wherein the first noise reduction setting consists of a first temporal noise reduction setting and a first spatial noise reduction setting, and wherein the second noise reduction setting consists of a second temporal noise reduction setting and a second spatial noise reduction setting.

5. The image compression system as claimed in claim 1, wherein the controller is further configured to increase the ratio in response to the need to execute the at least two programs in order to free up a portion of the memory.

6. The image compression system as claimed in claim 5, wherein the at least two programs include at least one video program and at least one computer vision / neural network (CV / NN) program, and wherein the controller is further configured to utilize a portion of the memory to execute the at least one computer vision / neural network program.

7. An image compression method, comprising: receiving a frame from an image sensor; performing a weighted calculation on pixel data of the frame and pixel data of a decompressed reference frame from a frame buffer via a temporal noise reduction circuit to mix the frame and the decompressed reference frame into a temporal noise reduction frame; adjusting a ratio via a controller while at least two programs are executed in parallel; compressing the temporal noise reduction frame into a compressed reference frame via a compression circuit according to the ratio determined by the controller; decompressing the compressed reference frame into the decompressed reference frame via the compression circuit according to the ratio determined by the controller; and receiving the temporal noise reduction frame from the temporal noise reduction circuit and performing a weighted calculation on pixel data of the temporal noise reduction frame and surrounding pixel data via a spatial noise reduction circuit to obtain a spatial noise reduction frame, wherein the controller increases the ratio in response to the need to execute the at least two programs, and wherein the controller decreases the ratio in response to the completion of the at least two programs.

8. The image compression method as described in claim 7, wherein the temporal noise reduction circuit is further configured to determine whether each pixel in the frame is a stationary pixel or a moving pixel, so as to obtain a pixel state result.

9. The image compression method as described in claim 8, wherein both the temporal noise reduction circuit and the spatial noise reduction circuit perform weighted calculations based on the pixel state results.

10. The image compression method as claimed in claim 9, wherein the controller performs noise reduction using a first noise reduction setting in response to a decrease in the ratio, and wherein the controller performs noise reduction using a second noise reduction setting in response to an increase in the ratio, wherein the first noise reduction setting comprises a first temporal noise reduction setting and a first spatial noise reduction setting, and wherein the second noise reduction setting comprises a second temporal noise reduction setting and a second spatial noise reduction setting.

11. The image compression method as described in claim 7, wherein the controller is further configured to increase the ratio in response to the need to execute the at least two programs in order to free up a portion of the memory.

12. The image compression method as claimed in claim 11, wherein the at least two programs include at least one video program and at least one computer vision / neural network program, and wherein the controller is further configured to use a portion of the memory to execute the at least one computer vision / neural network program.

Images