Image processing method and image processing apparatus with object detection

Abstract

The invention provides an image processing method and an image processing device, and the method comprises the steps: receiving an input image which comprises a plurality of blocks in an RGB color space, calculating a difference value between a first pixel in a first block in the blocks and a plurality of adjacent pixels of the first pixel in the first block in each RGB channel, the method comprises the steps of determining a first color channel with a maximum difference value of pixel values of a first block in RGB channels, processing the first block according to the maximum difference value of the pixel values of the first block in the first color channel so as to generate a processed first block, and generating an output image comprising the processed first block.

Description

Technical Field

[0001] This application relates to an image processing technique with object detection. Background Technology

[0002] Due to memory and bandwidth limitations, motion estimation and motion compensation are primarily based on intensity values ​​and rely solely on the brightness or luminance channel (often referred to as the "Y channel"). However, these methods tend to be ineffective when dealing with objects that have a vibrant, saturated color appearance and can lead to edge discontinuities and loss of detail. Summary of the Invention

[0003] Based on the above, this application provides an image processing method and an image processing apparatus.

[0004] In the embodiments of this application, the above-described image processing method includes receiving an input image comprising multiple blocks in an RGB color space, calculating the difference in pixel values ​​in each RGB channel between a first pixel in a first block and multiple neighboring pixels of the first pixel in the first block, determining a first color channel in the RGB channels where the pixel values ​​of the first block have the largest difference, processing the first block according to the largest difference in pixel values ​​of the first block in the first color channel to generate a processed first block, and generating an output image including the processed first block.

[0005] In embodiments of this application, the image processing apparatus described above includes memory and a processor. The processor is configured to receive an input image comprising multiple blocks in an RGB color space, calculate the difference in pixel values ​​in each RGB channel between a first pixel in a first block and multiple neighboring pixels of the first pixel in the first block, determine a first color channel in the RGB channels where the pixel values ​​of the first block have the largest difference, process the first block according to the largest difference in pixel values ​​of the first block in the first color channel to generate a processed first block, and generate an output image including the processed first block. Attached Figure Description

[0006] Figure 1 A schematic diagram of an image processing apparatus according to an embodiment of the present application is shown.

[0007] Figure 2 A flowchart illustrating an image processing method according to an embodiment of this application is shown.

[0008] Figure 3 A first block of an input image according to an embodiment of this application is illustrated.

[0009] Figure 4The diagram illustrates the preset relationship between the maximum difference in pixel values ​​and the gain value within a block according to an embodiment of this application.

[0010] Figure 5 Images of the image processing method according to an embodiment of this application are shown at various stages.

[0011] Figure 6 A scenario illustrating an image processing method according to an embodiment of this application is shown. Detailed Implementation

[0012] Some embodiments of this application will now be described more fully below with reference to the accompanying drawings, which illustrate some, but not all, of the embodiments of this application. In fact, various embodiments of this application may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to enable this application to meet applicable legal requirements. The same reference numerals refer to the same components throughout.

[0013] Figure 1 A schematic diagram of an image processing apparatus according to an embodiment of this application is shown. All components and configurations of the apparatus will be described in detail below. Figure 1 The functionality of the component will be explained later. Figure 2 A more detailed explanation follows.

[0014] Please refer to Figure 1 The image processing apparatus 100 includes at least a memory 110 and a processor 120. The image processing apparatus 100 can be an electronic system or a computer system. The memory 110 can be various forms of random-access memory (RAM), such as dynamic random-access memory (DRAM). The processor 120 can be one or more of a Northbridge, Southbridge, field-programmable gate array (FPGA), programmable logic device (PLD), application-specific integrated circuit (ASIC), other similar devices, or combinations thereof. The processor 120 can also be a central processing unit (CPU), a programmable general-purpose or special-purpose microprocessor, a digital signal processor (DSP), a graphics processing unit (GPU), other similar devices, or combinations thereof.

[0015] Figure 2 A flowchart illustrating an image processing method according to an embodiment of this application is shown, wherein... Figure 2 The steps can be made by Figure 1 The image processing device 100 is implemented.

[0016] Please refer to the following at the same time Figure 2 as well as Figure 1 The processor 120 of the image processing apparatus 100 receives an input image comprising multiple blocks in the RGB color space (step S202). In one scenario, the input image may be a frame from a game stream or video stream. In another scenario, if the game stream or video stream was originally encoded in another color space (e.g., YUV / YCbCr color space), the input image may be the result of color space conversion of the original image in the stream. It should be noted that the input image will be divided into multiple blocks (e.g., each 3×3 pixel block) and each block will be processed one by one. For ease of explanation, only one block of the input image (hereinafter referred to as the "first block") will be presented below, while other blocks can be derived in a similar manner.

[0017] Next, the processor 120 calculates the differences in pixel values ​​in each RGB channel between the first pixel in the first block and its multiple neighboring pixels in the first block (step S204), and determines the first color channel in the RGB channels where the pixel values ​​of the first block have the largest difference (step S206). From another perspective, the pixels in the first block exhibit significant differences, thus different objects can be more easily identified in the first color channel than in other color channels.

[0018] In one embodiment, the difference in pixel values ​​across the RGB channels between a first pixel in the first block and its multiple neighboring pixels can be calculated as the sum of absolute differences (SAD). For example, Figure 3 A first block of an input image according to an embodiment of this application is illustrated.

[0019] Please refer to the following at the same time Figure 3 as well as Figure 1 The first block 310 includes nine pixels (P00, P01, P02, P10, P11, P12, P20, P21, and P22) arranged in a 3×3 grid, where each pixel has a pixel value in each of the RGB channels. Assuming the center pixel P11 is the first pixel, the processor 120 can calculate the sum of the absolute differences in pixel values ​​in each RGB channel between the center pixel and each of its neighboring pixels using the following formula:

[0020]

[0021] Here, R11, G11, and B11 represent the pixel values ​​of the center pixel P11 in the RGB channels. Rij, Gij, and Bij represent the pixel values ​​of the neighboring pixels of the center pixel P11 in the RGB channels.

[0022] Next, the processor 120 can determine the first color channel with the largest difference in pixel values ​​of the first block according to the following formula:

[0023] MaxDiff=max(R_diff,G_diff,B_diff)

[0024] For example, if MaxDiff is R_diff, then the first color channel will be the R channel.

[0025] Please return Figure 2 The processor 120 processes the first block based on the maximum difference in pixel values ​​of the first block in the first color channel to produce a processed first block (step S208), and generates an output image including the processed first block (step S210). In one embodiment, the pixel values ​​of the first block in the first color channel can be enhanced using a gain value. This gain value can be determined based on the maximum difference in pixel values ​​of the first block according to a default relationship stored in the memory 110.

[0026] For example, Figure 4 The diagram illustrates a preset relationship between the maximum difference in pixel values ​​within a block and a gain value according to an embodiment of this application. Here, when the maximum difference in pixel values ​​within a block is greater than a preset threshold th0, it will be proportional to a gain value between gain0 (e.g., 0) and gain1 (e.g., 16) 410.

[0027] After determining the gain value of the first block, the processor 120 processes the first block based on the gain value and the pixel values ​​of the first color channel. Specifically, the processor 120 obtains the intensity values ​​of the first pixel and its neighboring pixels in the first block according to the following formula, and generates a processed first block based on the maximum difference between the intensity value, the gain value, and the pixel values ​​of the first block:

[0028]

[0029] Here, Yin represents the intensity of the pixels in the first block (e.g., the pixel value of the Y channel), C represents the maximum difference in pixel values ​​in the first block, C_coef represents the gain value, and Yout represents the pixels in the first block that have been processed.

[0030] To better understand, Figure 5 Images of the image processing method according to an embodiment of this application are shown at various stages.

[0031] Please refer to Figure 5 The input image 510 is an image from a game stream, including a sky background 512, a heart icon 514 representing health, and a weapon crosshair icon 516. The intermediate image 520 is the image where the maximum difference in pixel values ​​across the RGB channels for each block is identified. The maximum difference in pixel values ​​for each block in the input image 510 will be at the edges of the heart icon 514 and weapon crosshair icon 516 in the R channel, and will be represented as heart icon 524 and weapon crosshair icon 526 in the intermediate image 520. Non-edge pixels in the input image 510 will be rendered as white (pixel value 0) in the intermediate image 520, such as the sky background 522 and the inner portions of heart icon 524 and weapon crosshair icon 526. Next, the output image 530 with the processed blocks will include enhanced heart icon 534 and weapon crosshair icon 536, which will create a significant contrast with the sky background 532.

[0032] Figure 6 The illustration depicts a scenario of an image processing method according to an embodiment of this application, wherein... Figure 6 The situation can be derived from Figure 1 The image processing device 100 is implemented.

[0033] Please refer to the following at the same time Figure 6 as well as Figure 1 The processor 120 receives a raw image IMG in YUV / YCbCr color space from a video stream or game stream, and performs a color space conversion 610 on the raw image IMG to generate an input image IMG' in RGB color space. Then, the processor 120 will... Figure 2 Steps S202 to S210 perform image enhancement 620 on the input image IMG' to produce an output image IPME (also known as "input image for motion estimation"). In applications, the processor 120 can perform motion estimation 630A or object detection 630B on the output image IPME. This architecture significantly improves the performance of motion estimation and object detection, especially for objects with vibrant, saturated colors.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized in that, include: Receive an input image comprising multiple blocks in the RGB color space, wherein the multiple blocks include a first block; Calculate the difference in pixel value in each RGB channel between the first pixel in the first block and multiple neighboring pixels of the first pixel in the first block; Determine the first color channel in the first block of the RGB channels that has the largest difference in pixel values; Based on the maximum difference in pixel values ​​of the first block in the first color channel, the first block is processed to produce a processed first block; as well as An output image is generated that includes the processed first block.

2. The method according to claim 1, characterized in that, Before receiving the input image that includes the block in the RGB color space, the method further includes: Receive the original image in another color space; and The original image is converted to a different color space to produce the input image in the RGB color space.

3. The method according to claim 1, characterized in that, The difference in pixel value in each of the RGB channels between the first pixel in the first block and the neighboring pixels of the first pixel in the first block is the sum of the absolute differences in pixel value in each of the RGB channels between the first pixel and each of the neighboring pixels.

4. The method according to claim 1, characterized in that, The first pixel is the center pixel of the first block.

5. The method according to claim 1, characterized in that, The step of processing the first block to generate the processed first block based on the maximum difference in pixel values ​​of the first block in the first color channel includes: Based on the maximum difference in the pixel values ​​of the first block, a gain value is determined; and Based on the gain value and the pixel value of the first color channel, the first block is processed to generate the processed first block.

6. The method according to claim 5, characterized in that, The step of processing the first block based on the gain value and the pixel value of the first color channel to generate the processed first block includes: Obtain the intensity values ​​of the first pixel and its neighboring pixels in the first block; and The processed first block is generated based on the maximum difference between the intensity value, the gain value, and the pixel value of the first block.

7. The method according to claim 5, characterized in that, The gain value is directly proportional to the maximum difference between the pixel values.

8. The method according to claim 1, characterized in that, Also includes: Motion estimation is performed on the output image including the processed first block.

9. The method according to claim 1, characterized in that, Also includes: Object detection is performed on the output image including the processed first block.

10. An image processing apparatus, characterized in that, include: Memory; as well as Processor, used for: Receive an input image comprising multiple blocks in the RGB color space, wherein the multiple blocks include a first block; Calculate the difference in pixel value in each RGB channel between the first pixel in the first block and multiple neighboring pixels of the first pixel in the first block; Determine the first color channel in the first block of the RGB channels that has the largest difference in pixel values; Based on the maximum difference in pixel values ​​of the first block in the first color channel, the first block is processed to produce a processed first block; as well as An output image is generated that includes the first block after the processing.

Images