Image encoding device, method, and program

Abstract

To improve the coding efficiency of an image, particularly color difference components.SOLUTION: An image encoder of the present invention comprises: a determination unit that determines the characteristics of color difference components in an input frame; a first coding unit that recursively divides a block image formed of luminance components of the input frame into blocks according to a predetermined algorithm to code the block image; a second coding unit that, with reference to a result of block division performed by the first coding unit, divides a block image formed of color difference components of the input frame to code the block image; a third coding unit that divides the block image formed of the color difference components of the input frame into blocks of a preset size to code the block image; a selection unit that, based on a determination made by the determination unit, selects any one of the second and third coding units for the block image formed of the color difference components; and an output unit that outputs coding data of the luminance components obtained by the first coding unit and coding data of the color difference components obtained by the coding unit selected by the selection unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image encoding technique. [Background technology]

[0002] In recent years, the amount of data handled by video devices has been steadily increasing, with the advent of 4K and 8K resolutions. As a result, progress is being made on standardizing VVC (Versatile Video Coding) / MPEG-I Part 3, a next-generation video encoding method that can compress video even more efficiently than H.265 / HEVC.

[0003] One of the technologies being considered for adoption as a new coding tool in the VVC standard is a block division method known as the Chroma Separate Tree (CST) or Dual Tree Structure (DT).

[0004] In the previous HEVC standard, the block division of the coding unit (CTU: Coding Tree Unit) only had the same block division structure for luminance component pixel samples and chrominance component pixel samples, but the VVC standard now makes it possible to select independent coding tree structures for luminance component samples and chrominance component samples in intra slices that have undergone intra-screen predictive coding.

[0005] In general, natural images have less pixel value variation in chrominance components than in luminance components, and therefore have higher spatial redundancy. Therefore, if we can use the CST method described above to assign a larger block size to chrominance components and reduce the number of levels in the tree structure, we can reduce the amount of coded data for redundant segmentation flags and syntax information indicating the number of levels, which is expected to improve coding efficiency.

[0006] Patent Document 1 discloses a technology in which an encoding device has means for independently dividing a luminance component and a chrominance component into blocks and encoding them, and the encoding device has means for determining the block division shape of the chrominance component based on the block division shape of the encoded luminance component, and determines the block division shape of the chrominance component so that the block boundaries of the chrominance component are the same as the block boundaries of the corresponding luminance component. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-152851 Summary of the Invention [Problem to be solved by the invention]

[0008] However, as described in Patent Document 1, in an encoder that performs block division of luminance and chrominance components independently using the CST, when determining the block division shape, calculations are made for both the case where the luminance and chrominance components are divided using the same block shape and tree structure, and the case where each component is divided independently, and the generated code amount of both or the difference with the predicted image is calculated. absolute Since the final decision is made after a cost comparison such as the sum of values ​​(SAD) is performed, there is a risk that the calculation load will increase.

[0009] In view of the above-mentioned problems, the present invention aims to provide a technique for improving the efficiency of image encoding without increasing the computational load of the encoding process. [Means for solving the problem]

[0010] In order to solve this problem, for example, an image coding device of the present invention has the following arrangement: An image encoding device that compresses and encodes an input frame obtained by an imaging means in units of pixel blocks of a predetermined size, a determining means for determining characteristics of a color difference component in the input frame; a first encoding means for recursively dividing a block image formed by luminance components of the input frame into blocks according to a predetermined algorithm and encoding the divided blocks; a second encoding means for dividing and encoding a block image formed by color difference components of the input frame by referring to the block division result by the first encoding means; a third encoding means for dividing a block image formed by color difference components of the input frame into blocks of a predetermined size and encoding the divided blocks; Based on the determination by the determining means, the block image of the color difference component is As a means of encoding a selection means for selecting either the second or third encoding means; an output means for outputting the coded data of the luminance component obtained by the first coding means and the coded data of the color difference component obtained by the coding means selected by the selection means; 、 the determining means determines whether or not the image has been captured in a shooting mode in which changes in color difference components are suppressed, among a plurality of preset shooting modes; The selection means selects the third encoding means when the determination means determines that the imaging mode is one in which changes in color difference components are suppressed, and selects the second encoding means when the imaging mode is any other mode. do. [Effects of the Invention]

[0011] According to the present invention, it is possible to improve the coding efficiency of the color difference component. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a system configuration diagram of an encoding device according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of an encoding unit according to the first embodiment. [Figure 3] 4 is a flowchart showing the procedure of an encoding process according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of application of block division according to the first embodiment. [Figure 5] 10 is a flowchart showing the procedure of an encoding process according to the second embodiment. [Figure 6] FIG. 11 is a block diagram showing the configuration of an encoding unit according to a third embodiment. [Figure 7] 10 is a flowchart showing the procedure of an encoding process according to a third embodiment. [Figure 8] 10 is a flowchart showing the procedure of an encoding process according to a fourth embodiment. [Figure 9] 13 is a flowchart showing the procedure of an encoding process according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] [First embodiment] Fig. 1 shows a system configuration diagram of a video encoding device to which this embodiment is applied, Fig. 2 shows a block configuration diagram of the video encoding unit 102 in Fig. 1, and Fig. 3 is a flowchart showing the processing procedure thereof.

[0015] <Overall system configuration> The video encoding device comprises an imaging unit 100, an image processing unit 101, an image encoding unit 102, a recording unit 103, a memory bus 104, a frame memory 105, a CPU 106, a flash memory 107, a CPU bus 108, and an operation unit 109.

[0016] The imaging unit 100 includes a camera unit such as a lens and a CCD, an optical unit, and an imaging sensor that converts optical signals captured through the lens into electrical signals. Three types of filters, red, blue, and green, are repeatedly arranged on the imaging surface of the imaging sensor. In this embodiment, this arrangement is a Bayer array. Therefore, the captured image data output by the imaging unit 100 is image data in the Bayer array. The imaging unit 100 outputs the image data in the Bayer array (digital RAW image data) obtained by capturing an image to a frame memory 105 such as a large-capacity DRAM via a memory bus 104. It is assumed that the imaging unit 100 captures images at a frame rate of, for example, 30 frames per second.

[0017] The image processing unit 101 performs de-Bayer processing (demosaic processing) on ​​the RAW image data stored in the frame memory 105, converting the data into signals consisting of luminance and color difference, and then performs so-called development processing, such as removing noise contained in each signal, correcting optical distortion, optimizing the image, etc. Then, the image processing unit 101 outputs the developed image data again to the frame memory 105 for compression encoding.

[0018] The image encoding unit 102 reads the developed image data from the frame memory 105, performs video compression using inter-frame prediction and entropy encoding to take advantage of the redundancy of the image data, and generates and outputs an encoded bitstream. The image encoding unit 102 in this embodiment is characterized by adopting the VVC (Versatile Video Codec) method as the video compression method.

[0019] The recording unit 103 converts the coded bitstream generated by the image coding block 102 into a predetermined container format, such as MP4 or MOV, that is compatible with various PC applications for playback and editing, and stores the converted bitstream in a non-volatile storage medium, such as a USB, SD card, or hard disk. Record Record on medium 150.

[0020] The memory bus 104 is a data bus that connects the imaging unit 100, image processing unit 101, image encoding unit 102, recording unit 103, and frame memory 105, and is used for high-speed transfer of image data, encoded data, and various parameter data.

[0021] The bus transfer method is ISA 、 Standards such as PCI-Express, AXI, etc. Na Bus Standards By adopting However, a unique bus system may be adopted, and the system is not particularly important in this embodiment.

[0022] The frame memory 105 stores original image data used in the encoding process of the image encoding unit 102, reference image data for inter-frame prediction, an encoded bit stream, and the like.

[0023] The CPU 106 is a controller that performs hardware control, such as starting and stopping the imaging unit 100, image processing unit 101, image encoding unit 102, and recording unit 103 that constitute the video encoding device in this embodiment, and interrupt notification, via the CPU bus.

[0024] The flash memory 107 is a non-volatile memory that stores programs executed by the CPU 106, parameters, etc., and is accessed by the CPU 106 in a fetch operation.

[0025] The CPU bus 108 is a control bus that connects the CPU 106 to each peripheral, and is a standard bus similar to the memory bus 104. Something like that A bus standard method may be used, or if there is sufficient processing margin, a low-speed serial method such as I2C may be used; there are no restrictions on the method.

[0026] The operation unit 109 is made up of various buttons, switches, a touch panel, etc., and notifies the CPU 106 of instructions from the user.

[0027] The above has described the configuration of the video encoding device in the first embodiment. Next, the internal configuration of the image encoding unit 102, which is a feature of this embodiment, and the processing content thereof will be described.

[0028] <Image Encoding Unit> The image encoding unit 102 includes a block division unit 200, an intra prediction unit 201, an inter prediction unit 202, an intra / inter determination unit 203, a predicted image generation unit 204, a transformation unit 205, a quantization unit 206, an entropy encoding unit 207, a code amount control unit 208, an inverse quantization unit 209, an inverse transformation unit 210, and a loop filter 211.

[0029] The block division unit 200 reads image data in units of coding target blocks (CTUs) from the frame image data (developed image data) stored in the frame memory 105, and further sets optimal prediction block sizes for each of the luminance and chrominance components of this image data.

[0030] Although FIG. 2 shows two frame memories 105, this is for convenience's sake, and it should be understood that the address space of the frame memory 105 in FIG. 1 is shown as being divided.

[0031] The CTU is divided into coding units (CUs) using a quadtree structure, and these units are used for the prediction process of the subsequent intra prediction or inter prediction.

[0032] The intra prediction unit 201 calculates the correlation between a block image to be coded, which has been divided by the block division unit 200, and multiple intra prediction images generated from reference pixel data around the block. The intra prediction unit 201 then selects the intra prediction method with the highest correlation and notifies the intra / inter determination unit 202 of the result.

[0033] The inter-prediction unit 202 similarly receives the divided block images divided by the block division unit 200 and the image data of the encoded frame in the frame memory 105 as reference images, and performs motion detection such as pattern matching between pixel data on a block-by-block basis to calculate a motion vector.

[0034] The intra / inter determining unit 203 selects and determines the prediction method for encoding the target block based on the output results from the intra predicting unit 201 and the inter predicting unit 202 .

[0035] As a specific selection method, the intra / inter determination unit 203 compares the difference between the image block to be coded and the intra-predicted image calculated by the intra prediction unit 201, and the difference between the image block to be coded and the inter-predicted image generated from the reference image indicated by the motion vector derived by the inter prediction unit 202. Then, the intra / inter determination unit 203 selects the one with the smaller difference.

[0036] Alternatively, the intra prediction unit 201 and the inter prediction unit 202 may obtain a prediction error between the determined predicted image and the image to be coded. Then, the intra / inter determination unit 203 may obtain the prediction errors from the intra prediction unit 201 and the inter prediction unit 202, and then compare and evaluate them to select a prediction method.

[0037] In either case, the intra / inter determining unit 203 determines the prediction mode with the smaller difference value as the encoding prediction mode and outputs it to the predicted image generating unit 204 .

[0038] The predicted image generating unit 204 generates a predicted image according to the prediction mode selected by the intra / inter determining unit 203. The predicted image generating unit 204 then outputs the generated predicted image to a subtractor in the previous stage of the converting unit 205, and converts the input image and The inverse transform unit 210 calculates a difference image between the measured image and the inverse transform unit 210. The difference image is output to an adder at the subsequent stage of the inverse transform unit 210 to generate a locally decoded image.

[0039] The conversion unit 205 converts the difference pixel data in units of blocks into the spatial frequency domain by spatial resolution conversion.

[0040] The quantization unit 206 calculates quantization coefficients based on the target code amount, and performs quantization processing on the coefficient data converted into the spatial frequency domain by the conversion unit 205. The quantization unit 206 outputs the quantized coefficient data to both the entropy coding unit 207 for entropy coding and the inverse quantization unit 209 for calculating a reference image and a predicted image.

[0041] The entropy coding unit 207 compresses the quantized coefficient data input from the quantization unit 206, or the vector values ​​used in motion prediction in the case of inter-prediction, by entropy coding that utilizes the bias in the occurrence probability of bit data, such as CABAC (Context Adaptive Arithmetic Coding), adds parameters required for decoding (header information such as SPS and PPS), shapes the data into a predetermined data format, and outputs the data to the frame memory 105. The recording unit 103 converts the coded data stored in the frame memory 105 into a predetermined container format and then saves it as a file on a recording medium. The entropy coding unit 207 also outputs the code amount obtained when coding is performed on a block-by-block basis to the code amount control unit 208.

[0042] The code amount control unit 208 accumulates the code amount per block supplied from the entropy coding unit 207 and calculates the code amount of encoded data per picture (frame). Then, the code amount control unit 208 calculates a target code amount per picture to be coded next based on the bit rate and buffer model, and then determines a quantization parameter for the next picture and performs feedback control to set it in the quantization unit 206. The inverse quantization unit 209 calculates coefficient data by multiplying the coefficient data quantized by the quantization unit 206 by the quantization coefficient again.

[0043] The inverse transform unit 210 performs the inverse transform of the transform unit 205 on the coefficient data output from the inverse quantization unit 209, generates differential pixel data in units of blocks, and outputs the differential pixel data to an adder at the subsequent stage.

[0044] The loop filter 211 performs a filter process on the image data obtained by adding the image data output from the inverse transform unit 210 and the predicted image to reduce coding distortion that occurs at block boundaries, and then outputs the result to the frame memory 105 as a locally decoded image. vinegar do.

[0045] The entropy coding unit 207 and code amount control unit 208 described above normally perform control in units of pictures for one screen, and other processing blocks perform control in units of predetermined rectangular pixel blocks.

[0046] In particular, in the VVC format to which the present invention is applied, quantization is performed in CU (Coding Unit) units, motion vector search for intra prediction and inter prediction is performed in PU (Prediction Unit) units, and transformation is performed in TU (Transform Unit) units.

[0047] The above is the configuration and operation of the image encoding unit 102 in this embodiment.

[0048] <Processing flow> Next, the processing flow of the block dividing unit 200 in the above-described device configuration, which is a feature of this embodiment, will be described with reference to the flowchart in Fig. 3. This processing flow is triggered by a user operation on the operation unit 109, and is executed for each frame when the device enters a recording state for captured video.

[0049] First, in S301, the block division unit 200 acquires the shooting mode set by the user's operation and setting from the operation unit 109 of the video encoding device.

[0050] The shooting modes are combinations of various setting parameters such as aperture, white balance, shutter speed, color, and blur level for the image capture unit 100, which are pre-defined and optimally set. Of course, other modes may be used, and the type is not particularly important. Next, in S302, the block division unit 200 determines whether block division processing has been completed for all CTU blocks of the image data of the frame to be coded. If block division has been completed for all CTU blocks (TRUE in S302), it is determined that the coding processing for the frame to be coded has been completed, and this flow ends.

[0051] On the other hand, if the block division unit 200 determines that there is a CTU that has not been divided into blocks, judged If (if S302 is FALSE), that is, if there is an unencoded CTU and If so, the process proceeds to S303. In S303, the block division unit 200 reads the original image data of the target CTU in the input image stored in the frame memory 105.

[0052] In this embodiment, the size of the CTU block is not particularly limited, but when expressed as the number of horizontal pixels x the number of vertical pixels (or the number of vertical lines), it is, for example, 128 x 128 pixels or 64 x 64 pixels, and the block dividing unit 200 reads this rectangular block image in units. Note that the image data format to be read is assumed to be a format consisting of multiple color components, such as luminance components and chrominance components, such as the YCbCr format or YUV format. Furthermore, in the encoding process in block units, which will be described later, block pixel data consisting of only luminance component pixels and block pixel data consisting of only chrominance component pixels are used. of Processing is performed on each of the collected block pixel data.

[0053] Hereinafter, the block division process performed by the block division unit 200 on the CTU block image data will be described.

[0054] In this embodiment, the division process by the block division unit 200 first performs block division of the luminance component, and then performs block division of the chrominance component. S304 is a step for determining whether or not to perform block division of the chrominance component in order to manage the processing order. If the target data for current CTU block division is a luminance component block, the block division unit 200 sets the determination result of S304 to FALSE and proceeds to S306. In S306, the block division unit 200 performs normal block division processing on the CTU of interest.

[0055] In the normal block division process of this embodiment, the block division unit 200 divides the rectangular block image into sub-block images of a quadtree, for example, and derives a variance value, which is an index indicating the variation in pixel values ​​within each sub-block region. If the variance values ​​of all sub-blocks are equal to or less than a predetermined threshold, the block division unit 200 adopts the block size before division for the CTU block. On the other hand, if the variance value of any sub-block is greater than the threshold, the sub-block is further divided into four smaller sub-block images, and each sub-block is similarly recursively divided based on the variance value. However, the size of the smallest divided sub-block is limited to a minimum size specified by the standard, such as 8x8 pixels, or a predetermined size larger than that size.

[0056] The algorithm for block division processing based on the variance value and threshold judgment for each sub-block region, described above as the normal block division processing, is just one example, and block division can also be performed using other methods, so it does not particularly limit the present invention.

[0057] When the block division unit 200 completes the block division of the luminance component data in S306, it changes the target of block division to blocks of chrominance component data, and returns the process to S304.

[0058] As a result of the above, the block to be divided is a block of color difference components, so the block division unit 200 determines the result of the determination in S304 as TRUE, and the process proceeds to S305. In S305, the block division unit 200 determines whether the input image to be coded is in a mode in which there is little change in pixel values ​​of color difference components, based on the shooting mode information acquired in S301.

[0059] In this embodiment, the monochrome mode or sepia mode is set as a mode in which there is little change in the color difference components (changes in the color difference components are suppressed), and when the above two modes are set, it is possible to determine before the encoding process that the input image is image data in which there is no or little color difference components.

[0060] When the above mode is set (TRUE in S305), applying the CST method, which divides the CTU block into blocks independently of the luminance component and the chrominance component, improves compression efficiency. Therefore, in S307, the block dividing unit 200 determines that the chrominance component block image will be divided into blocks of a predetermined size rather than dividing into blocks of a small size using a recursive hierarchical quadtree (S307).

[0061] The specified size for dividing the color difference component blocks is, for example, the size of the CTU itself, or up to the block size obtained by dividing into one layer of quadtree. It is particularly preferable if, assuming that there is no change in pixel value within the block, DC prediction, skip vectors, etc. are not generated in the subsequent prediction process, and if the result is equivalent to that, then the result is almost no prediction difference or coding cost.

[0062] On the other hand, if a mode value other than the above is set (FALSE in S305), the block dividing unit 200 proceeds to S308. In S308, the block dividing unit 200 divides the chrominance component blocks based on the block division information of the luminance component blocks processed in S306, in the same way as in the conventional HEVC standard method.

[0063] When S307 or S308 is executed for the block division of the chrominance components, the series of encoding processes described with reference to Fig. 2 are performed on that block. Therefore, the block division unit 200 returns the process to S302 for the division process and encoding of the next CTU block, and repeats this flow until the block division and encoding processes are completed for all CTU blocks. The block division process and encoding process in this embodiment has been described above.

[0064] <Block division example> An example of block division when the image encoding unit 102, particularly the block division unit 200, performs block division processing according to the above-described flow will be described.

[0065] Figure 4(a) and (b) show the CTU block size of 128 × 128 pixels. difference The cases shown are for a 4:2:0 chroma format with component subsampling.

[0066] 4(a) shows a division state 401 of the luminance component and a division state 402 of the chrominance component when the luminance component and the chrominance component are divided into the same blocks, as in the previous HEVC standard. When a mode in which there is a change in the chrominance component is selected as the shooting mode based on the flow described above in FIGS. 1 to 3, the block division is performed.

[0067] On the other hand, if the shooting mode selected is one that causes little change in the color difference components (monochrome mode or sepia mode), the luminance component undergoes conventional recursive block division in accordance with the structure of the subject, resulting in division state 403, as shown in Figure 4(b).However, since the color difference components have almost no pixel value gradient, they are not divided into blocks independently of the luminance component, or are encoded using a predetermined large block size, resulting in division state 404.

[0068] As a result, there is no need to transmit redundant syntax information such as a flag indicating that recursive block division has been performed for the entire CTU block, thereby improving coding efficiency.

[0069] [Second embodiment] In the first embodiment described above, when encoding an input image in a shooting mode where it is certain that there will be no pixel changes in the color difference components, such as monochrome or sepia mode, the CST method is applied to all CTU blocks of the input image, and the color difference components are divided into blocks of a predetermined block size independently of the luminance component.

[0070] In the second embodiment, instead of uniformly applying the CST method to all CTU blocks of the frame depending on the shooting mode set by the user, a different operation is performed in which costs are calculated and compared for both conventional block division and block division using the CST method on a block-by-block basis, and the more suitable block division is selected and coded.

[0071] The block division process of the block division unit 200 in the second embodiment will be described below with reference to the flowchart in Fig. 5. Note that the description of parts common to the first embodiment will be omitted as appropriate. Note that the device configuration realizing the second embodiment and the components and functions of the image encoding unit 102 are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0072] <Processing flow> In FIG. 5, steps S301 to S305 are the same as those in the first embodiment. In S305, when dividing the chrominance component into CTU blocks, the block dividing unit 200 determines whether a mode (monochrome mode or sepia mode) in which changes in the chrominance component are small (changes are suppressed) is selected as the shooting mode. If the block dividing unit 200 determines that a mode in which changes in the chrominance component are small is selected (if S305 is TRUE), the process proceeds to S307. In S307, the block dividing unit 200 performs predetermined block division on the chrominance component blocks, and the process proceeds to S308. In S308, the block dividing unit 200 refers to the division information obtained in S306 and performs the same block division as for the luminance component on the chrominance component block of interest. Then, in S401, the block dividing unit 200 performs block division. split The unit 200 performs cost calculations for the subsequent prediction block, taking into account the amount of code for syntax information including the division flag and the number of layers, for both the case where the CST method is applied and the case where conventional block division is performed, and selects the division method with the higher coding efficiency. Typically, coding is performed for each of the two block divisions, and a division method that generates the least amount of code is adopted.

[0073] The above is the flow of the image division process and image encoding process in the second embodiment.

[0074] [Third embodiment] In the first and second embodiments described above, when the mode value of the input image to be encoded has little change in chrominance components based on the shooting mode previously set by the user before shooting and recording, the input image is encoded by applying the CST method, which divides the input image into blocks of luminance components and chrominance components independently.

[0075] The third embodiment is characterized in that the image processing unit 101 constituting the encoding device described above is configured to perform image analysis of input image data in addition to development processing and notify the image encoding unit 102 of the analysis results, and further performs different operations. Hereinafter, the third embodiment will be described with reference to Figures 6 and 7. Note that in the description of the third embodiment, descriptions of parts common to the first and second embodiments will be omitted as appropriate.

[0076] <Block diagram> 6 is a block diagram of the image encoding unit 102 and its peripherals in the third embodiment. The image encoding unit 102 in the third embodiment has substantially the same configuration as the image encoding unit 102 in the first embodiment. A characteristic feature of the third embodiment is that, as shown in FIG. 6, a block division unit 200 transmits and receives information to and from the image processing unit 101 via an interface such as a CPU bus.

[0077] 1 and image output to the frame memory 105, the image processing unit 101 in the third embodiment further has a function of analyzing image data characteristics of the output image and outputting the analysis results. The analysis of image data characteristics involves, for example, acquiring a pixel histogram in the object image data and deriving the range of maximum and minimum values ​​of the histogram for each color component.

[0078] As an example, the image processing unit 101 obtains a pixel histogram of the developed frame image, and calculates the difference between the maximum and minimum values ​​of the color difference components Cr and Cb for pixel values ​​whose frequency is equal to or greater than a predetermined number. For ease of understanding, the maximum and minimum values ​​of the color difference Cr are expressed as Cr_max and Cr_min, and the maximum and minimum values ​​of the color difference Cb are expressed as Cb_max and Cb_min.

[0079] When the following expressions (1) and (2) are satisfied, the image processing unit 101 determines that the corresponding frame has little change in color difference. Cr_max - Cr_min ≦ Th_Cr …(1) Cb_max - Cb_min ≦ Th_Cb …(2) Here, Th_Cr and Th_Cb are preset positive thresholds. Although the determination is made on a frame-by-frame basis here, the determination may be made on a CTU-by-CTU basis.

[0080] Based on the analysis results of the image processing unit 101, the block division unit 200 can determine before the encoding process that the input image has little change in color difference components, thereby enabling more precise control than applying the CST method uniformly to all CTU blocks in the shooting mode.

[0081] In this embodiment, the characteristic analysis by acquiring a histogram is an example, and the image processing unit 101 is not particularly limited as long as it can quantitatively derive an equivalent analysis. As another analysis method, if the location of the in-focus area in the subject image acquired by the imaging unit 100 is known, the out-of-focus area outside that area will be blurred, although this will depend on the depth of field of the optical unit. It is possible to indirectly estimate that the blurred image is an area with a small gradient or change in pixel values.

[0082] In the third embodiment, the analysis results executed by the image processing unit 101 described above are transmitted and received between the image processing unit 101 and the image encoding unit 102 either for the entire input image frame or for each image area of ​​the CTU block executed by the block division unit 200. If the CPU bus for the analysis results for all CTU blocks is used, bus traffic may become complicated, so a method may be used in which the results are compiled as bitmap information that can be identified on a bit-by-bit basis, with blocks with large color changes being set to a bit "1" and blocks with small changes being set to a bit "0", and then the frame memory 105 is accessed in bursts. The other block functions are the same as in the first embodiment, so a description thereof will be omitted.

[0083] <Processing flow> The division process of the block division unit, which is a feature of the third embodiment, will be described with reference to the flowchart of Fig. 7. Note that a description of the processing steps that overlap with the first embodiment will be omitted.

[0084] First, in S701, the block division unit 200 acquires the image analysis result for the frame to be coded from the image processing unit 101. The subsequent steps from S302 to S304 are the same as those in the first embodiment.

[0085] In S702, when the sequence for dividing the chrominance components into CTU blocks is started, the block dividing unit 200 determines whether the input image has little change in the chrominance components by referring to the analysis result acquired in S701. If the block dividing unit 200 determines that the image has little change in the chrominance components based on the analysis result (TRUE in S702), the process proceeds to S307, and if the block dividing unit 200 determines that the image has much change in the chrominance components (FALSE in S702), the process proceeds to S308.

[0086] In S307, the block division unit 200 applies the CST method to the CTU block and divides it into blocks of a predetermined size. 0 divides the chrominance component blocks based on the block division information of the luminance component blocks processed in S306, in the same manner as in the conventional HEVC standard method.

[0087] After S307 or S308 is executed for the block division of the color difference components, the series of encoding process steps described with reference to Fig. 2 are executed for the blocks. After this encoding, the block division unit 200 returns to S302 and repeats this flow until the block division and encoding process is completed for all CTU blocks of the input image.

[0088] The block division processing and image encoding processing flow in the third embodiment has been described above.

[0089] [Fourth embodiment] In the above-described third embodiment, a method was shown in which, based on the analysis results of the input image in the image processing unit 101, when there is little pixel change in the chrominance nebula, the chrominance components are divided into blocks of a predetermined block size independently of the luminance component blocks.

[0090] In the fourth embodiment, rather than uniformly applying the CST method to all CTU blocks of an image frame in accordance with the analysis results for that frame, a different operation is implemented in which costs are calculated and compared for both conventional block division and block division using the CST method on a block-by-block basis, and the more suitable block division is selected and coded.

[0091] The fourth embodiment will be described below with reference to the flowchart in Fig. 8. In the description of the fourth embodiment, parts common to the first to third embodiments will be omitted as appropriate. Furthermore, the system configuration realizing the fourth embodiment and the components and functions of the image encoding unit 102 are the same as those of the third embodiment, and therefore will not be described here.

[0092] <Processing flow> In this fourth embodiment, the image processing unit 101 calculates a histogram for each CTU block in the frame to be encoded, and determines for all CTUs whether pixels with a predetermined frequency or higher satisfy the above-mentioned equations (1) and (2), and supplies the determination result (information indicating whether or not there is a change in the color difference component) to the block division unit 200.

[0093] 8 are the same as those in the third embodiment. However, in S701 of Fig. 8, the block division unit 200 acquires the result of analyzing whether or not there is a change in the color difference components of all CTU blocks of the frame to be coded by the image processing unit 101, and 2 The difference is that the determination is made for each CTU to be coded based on the acquired information.

[0094] If the block division unit 200 determines in S702 that the CTU of interest for the chrominance component has little change in the chrominance component, it applies the CST method to the chrominance component block and performs predetermined block division. Furthermore, in S308, the block division unit 200 refers to the division information performed in S306 and performs the same block division as the conventional luminance component. Then, in S401, the block division unit 200 calculates the cost of the predicted block, taking into account the code amount of syntax information including the division flag and the number of layers, for both the case where the CST method is applied and the case where the conventional block division is performed, and selects the division method with the higher coding efficiency. This concludes the image coding process flow in the fourth embodiment.

[0095] [Fifth embodiment] In the third and fourth embodiments described above, when there is little change in the chrominance components of the input image data to be coded based on the analysis results of the image characteristics of the image processing unit 101, coding is performed by applying the CST method, which independently divides the luminance component and the chrominance component into blocks.

[0096] In contrast to this, in the fifth embodiment ,above In the embodiment of the present invention described above, identification information indicating whether or not block division using the CST method has been applied is not only embedded in the compressed encoded stream for the purpose of determination on the decoder side as defined by the standard, but also defined and stored in an even higher file container layer as a further different operation.

[0097] The fifth embodiment will be described below with reference to the flowchart in Fig. 9. In the description of the fifth embodiment, parts common to the first to fourth embodiments will be omitted as appropriate. Furthermore, the device configuration realizing the fifth embodiment and the components and functions of the image encoding unit 102 are assumed to be the same as those of the fourth embodiment, and descriptions thereof will be omitted.

[0098] <Processing flow> In FIG. 9 showing the features of the fifth embodiment, the steps other than S901 and S902 are the same as those in the fourth embodiment, and therefore a description thereof will be omitted.

[0099] When block division of all CTU blocks is completed and encoding processing of all CTUs is completed (if S302 is TRUE), the block division unit 200 proceeds to S901. In S901, the block division unit 200 determines whether block division using the CST method, which divides the encoded frame image into blocks of luminance components and chrominance components independently and performs encoding processing, has been performed. If it is determined that the CST method has been applied (if S901 is TRUE), the block division unit 200 notifies the CPU 106 of this fact. The CPU 106 controls the image encoding unit 102 to record identification information indicating that the CST method has been applied in a parameter field (e.g., udta atom) defined as user metadata in the container structure of MP4 data when the recording unit 103 records the encoded data of the encoded image of the frame on the recording medium 150 such as an SD card.

[0100] By storing the above identification information as a container file, even if the playback device or decoder device does not support the decoding process of the CST method newly adopted in the VVC standard, it becomes possible to determine whether playback compatibility is supported before actually decoding the encoded stream.

[0101] The image encoding process flow in the fifth embodiment has been described above.

[0102] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0103] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0104] 100...imaging unit, 101...image processing unit, 102...image encoding unit, 103...recording unit, 104...memory bus, 105...frame memory, 106...CPU, 107...flash memory, 108...CPU bus, 200...block division unit, 201...intra prediction unit, 202...inter prediction unit, 203...intra / inter determination unit, 204...prediction image generation unit, 205...conversion unit, 206...quantization unit, 207...entropy encoding unit, 208...code amount control unit, 209...inverse quantization unit, 210...inverse transformation unit, 211...loop filter

Claims

1. An image encoding device that compresses and encodes an input frame obtained by an imaging means in units of pixel blocks of a predetermined size, comprising: a determining means for determining characteristics of a color difference component in the input frame; a first encoding means for recursively dividing a block image formed by luminance components of the input frame into blocks in accordance with a predetermined algorithm and encoding the divided blocks; a second encoding means for dividing and encoding a block image formed by color difference components of the input frame by referring to a result of the block division by the first encoding means; a third encoding means for dividing a block image formed by color difference components of the input frame into blocks of a predetermined size and encoding the divided blocks; a selection means for selecting either the second or third encoding means as a means for encoding a block image formed of color difference components based on the determination of the determination means; an output means for outputting the coded data of the luminance component obtained by the first coding means and the coded data of the color difference component obtained by the coding means selected by the selection means; Equipped with the determining means determines whether or not the image has been captured in a shooting mode in which changes in color difference components are suppressed, among a plurality of preset shooting modes; The selection means selects the third encoding means when the determination means determines that the imaging mode is one in which changes in color difference components are suppressed, and selects the second encoding means when the imaging mode is any other mode. An image encoding device comprising:

2. An image encoding device that compresses and encodes an input frame obtained by an imaging means in units of pixel blocks of a predetermined size, comprising: a determining means for determining characteristics of a color difference component in the input frame; a first encoding means for recursively dividing a block image formed by luminance components of the input frame into blocks in accordance with a predetermined algorithm and encoding the divided blocks; a second encoding means for dividing and encoding a block image formed by color difference components of the input frame by referring to a result of the block division by the first encoding means; a third encoding means for dividing a block image formed by color difference components of the input frame into blocks of a predetermined size and encoding the divided blocks; a selection means for selecting either the second or third encoding means as a means for encoding a block image formed of color difference components based on the determination of the determination means; an output means for outputting the coded data of the luminance component obtained by the first coding means and the coded data of the color difference component obtained by the coding means selected by the selection means; Equipped with the determining means determines whether or not the image has been captured in a shooting mode in which changes in color difference components are suppressed, among a plurality of preset shooting modes; The selection means when the determining means determines that the imaging mode is one in which changes in color difference components are suppressed, calculating the costs of the second encoding means and the third encoding means, and selecting one with higher encoding efficiency based on the costs; When the determining means determines that the photographing mode is one in which changes in color difference components are not suppressed, the second encoding means is selected. An image encoding device comprising:

3. An image encoding device that compresses and encodes an input frame obtained by an imaging means in units of pixel blocks of a predetermined size, comprising: a determining means for determining characteristics of a color difference component in the input frame; a first encoding means for recursively dividing a block image formed by luminance components of the input frame into blocks in accordance with a predetermined algorithm and encoding the divided blocks; a second encoding means for dividing and encoding a block image formed by color difference components of the input frame by referring to a result of the block division by the first encoding means; a third encoding means for dividing a block image formed by color difference components of the input frame into blocks of a predetermined size and encoding the divided blocks; a selection means for selecting either the second or third encoding means as a means for encoding a block image formed of color difference components based on the determination of the determination means; an output means for outputting the coded data of the luminance component obtained by the first coding means and the coded data of the color difference component obtained by the coding means selected by the selection means; Equipped with the determining means obtains a histogram of pixels in the input frame, determines a difference between a maximum value and a minimum value of a color difference component for pixel values ​​in the histogram that are equal to or greater than a predetermined frequency, and determines whether the difference is equal to or less than a predetermined threshold value; The selection means selects the third encoding means when the determination means determines that the difference is equal to or less than a threshold, and selects the second encoding means when the determination means determines that the difference is not equal to or less than a threshold. An image encoding device comprising:

4. An image encoding device that compresses and encodes an input frame obtained by an imaging means in units of pixel blocks of a predetermined size, comprising: a determining means for determining characteristics of a color difference component in the input frame; a first encoding means for recursively dividing a block image formed by luminance components of the input frame into blocks in accordance with a predetermined algorithm and encoding the divided blocks; a second encoding means for dividing and encoding a block image formed by color difference components of the input frame by referring to a result of the block division by the first encoding means; a third encoding means for dividing a block image formed by color difference components of the input frame into blocks of a predetermined size and encoding the divided blocks; a selection means for selecting either the second or third encoding means as a means for encoding a block image formed of color difference components based on the determination of the determination means; an output means for outputting the coded data of the luminance component obtained by the first coding means and the coded data of the color difference component obtained by the coding means selected by the selection means; Equipped with the determining means obtains a histogram for each pixel block in the input frame, and determines whether or not a difference between a maximum value and a minimum value of a color difference component is equal to or smaller than a predetermined threshold value for pixel values ​​that are equal to or greater than a predetermined frequency in the histogram; The selection means when the determining means determines that the difference in the pixel block of interest is equal to or less than a threshold, calculating costs for the second encoding means and the third encoding means, and selecting one with higher encoding efficiency based on the costs; If the determining means determines that the difference in the pixel block of interest is not equal to or less than a threshold value, the second encoding means is selected. An image encoding device comprising:

5. 5. The image encoding device according to claim 1, wherein the output means further outputs information indicating whether the encoded data for the input frame includes data encoded by the third encoding means.

6. 1. A control method for an image encoding device that compresses and encodes an input frame obtained by an imaging means in units of pixel blocks of a predetermined size, comprising: determining characteristics of the chrominance components in the input frame; a first encoding step of recursively dividing a block image formed by luminance components of the input frame into blocks according to a predetermined algorithm and encoding the divided blocks; a second encoding step of dividing and encoding a block image formed by color difference components of the input frame by referring to the block division result of the first encoding step; a third encoding step of dividing a block image formed by the color difference components of the input frame into blocks of a predetermined size and encoding the divided blocks; a selection step of selecting either the second or third encoding step as a step of encoding a block image composed of color difference components based on the determination of the determination step; an output step of outputting the coded data of the luminance component obtained in the first coding step and the coded data of the color difference component obtained in the coding step selected in the selection step; Equipped with In the determination step, it is determined whether or not the image has been captured in a photography mode in which changes in color difference components are suppressed, among a plurality of photography modes set in advance; In the selection step, if it is determined in the determination step that the imaging mode is one in which changes in color difference components are suppressed, the third encoding step is selected, and if it is another imaging mode, the second encoding step is selected.

2. A control method for an image encoding device comprising:

7. A computer program that, when read and executed by a computer, causes the computer to execute each step of the method for controlling an image encoding device according to claim 6.

Images