Image processing device, control method of them, and program

Abstract

To code a data amount when recording a VR image with a high compressibility ratio while suppressing an image quality deterioration of an image of a display object.SOLUTION: An image processing device that acquires an image imaged by first and second optical systems for imaging a VR image by an imaging sensor, and codes them, includes: a quantization part that quantizes the image data obtained by the imaging sensor in a block of a predetermined size; a quantization control part that determines a quantization parameter used by the quantization part, and sets it to the quantization part; and a coding part that codes a data after the quantization, obtained by the quantization part. Here, the quantization control part includes: a calculation part that calculates a position and a size of two circumferential images obtained by being imaged by two optical systems contained in the VR image; and a determination part that determines the quantization parameter of a quantization objective block on the basis of a position of the quantization object block, and a position and a size of the two circumferential images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image encoding technique. [Background technology]

[0002] There is a known technology that uses two optical systems to acquire images with a wide viewing angle and parallax, and then maps and displays them on a virtual sphere to display a three-dimensional virtual reality (VR) image. A twin-lens VR camera for capturing these VR images is equipped with two optical systems facing the same direction. Then, in one capture, two subject images with parallax are formed side-by-side on one sensor. The image captured by the sensor is then recorded as a VR image.

[0003] As a related technique, Patent Document 1 discloses a method in which two subject images with parallax are arranged side by side and recorded via one sensor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-46260 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology disclosed in the above-mentioned Patent Document 1 does not disclose a specific method for compressing and encoding VR images. In order to comfortably watch VR images and VR videos, it is necessary to record them at a high resolution and a high frame rate, which causes a problem of a large amount of recorded data.

[0006] The present invention has been made in consideration of such problems, and aims to provide a technology for encoding the amount of data when recording VR images at a high compression rate while suppressing deterioration of the image quality of the image to be displayed. [Means for solving the problem]

[0007] In order to solve this problem, for example, an image processing device according to the present invention has the following arrangement. An image processing device that acquires an image formed by a first and second optical system for capturing a VR image from an imaging sensor and encodes the image, quantization means for quantizing image data obtained by the image sensor in units of blocks of a predetermined size; a quantization control means for determining a quantization parameter used by the quantization means and setting the quantization parameter in the quantization means; encoding means for encoding the quantized data obtained by the quantization means; The quantization control means A calculation means for calculating the positions and sizes of two circular images formed by the first and second optical systems and included in the VR image; The method further comprises a determination means for determining a quantization parameter of the block to be quantized based on the position of the block to be quantized and the positions and sizes of the two circular images. Effect of the Invention

[0008] According to the present invention, it is possible to encode the amount of data when recording a VR image at a high compression rate while suppressing deterioration in image quality of the image to be displayed. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to a first embodiment. [Diagram 2] FIG. 1 is a diagram for explaining a VR image. [Diagram 3] FIG. 1 is a diagram for explaining a Bayer array. [Figure 4] FIG. 1 is a diagram for explaining a wavelet transform. [Diagram 5] FIG. 2 is a block diagram showing the configuration of a quantization control unit according to the first embodiment. [Figure 6]1A and 1B are diagrams for explaining the spatial positional relationship between pixels and coefficients after wavelet transformation, and the quantization control unit. [Figure 7] FIG. 2 is a diagram for explaining various parameters that configure a VR image. [Figure 8] 5A to 5C are diagrams showing examples of setting quantization parameter correction amounts in the first embodiment. [Figure 9] 5 is a flowchart showing a procedure for determining a quantization parameter correction amount by a control unit according to the first embodiment. [Figure 10] 5 is a diagram for explaining a calculation unit of feature information by a quantization control unit according to the first embodiment. [Figure 11] 5 is a diagram showing an example of setting a quantization parameter correction amount by a quantization control unit according to the first embodiment. [Figure 12] 5 is a flowchart showing a procedure for determining a quantization parameter correction amount by a quantization control unit according to the first embodiment. [Figure 13] 5 is a flowchart showing a quantization parameter output procedure by a quantization parameter correction unit according to the first embodiment. [Figure 14] 10 is a flowchart showing a procedure for determining a quantization parameter correction amount by a control unit according to a second embodiment. [Figure 15] FIG. 11 is a block diagram showing the configuration of an image processing apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [First embodiment] The configuration of an image processing device 100 in the first embodiment is shown in Fig. 1. The image processing device 100 includes a control unit 101, an imaging unit 102, a plane conversion unit 103, a frequency conversion unit 104, a quantization control unit 105, a quantization unit 106, an entropy coding unit 107, a recording medium 108, and an operation unit 109.

[0012] The control unit 101 is composed of a CPU, a ROM that stores programs executed by the CPU, and a RAM that the CPU uses as a work area. In addition to the programs, the ROM also stores lens information, which is information specific to the lens, as described below. If the lens is detachable, the lens information may be stored in a memory (ROM) provided in the lens.

[0013] The imaging unit 102 has a twin-lens optical system capable of capturing images with parallax between a left image and a right image, and an imaging sensor. This twin lens is an interchangeable VR180 lens that captures VR images capable of binocular stereoscopic vision. The VR180 lens forms a right image acquired through a right-eye optical system and a left image acquired through a left-eye optical system having parallax from the right image, side by side, on one imaging sensor. The image processing device 100 in the embodiment encodes and records the RAW format imaging data obtained through the imaging sensor. Hereinafter, an image including the left and right images formed on one imaging sensor is called a VR image.

[0014] FIG. 2 shows an example of a VR image captured by the imaging unit 102. The VR image 200 is a circular fisheye image (also simply called a circular image), and includes a right image 201 acquired by a right-eye optical system and a left image 202 acquired by a left-eye optical system. The right image 201 and the left image 202 each have a larger optical distortion as they approach the outer periphery of the image circle. A region 203 further outside the outer periphery of the image circle is a light-shielding region. In addition, a region 204 called an optical black region (hereinafter, OB region) that is used only for various correction processes such as development is added to the upper and left ends of the VR image 200. The OB region 204 is generally added to the edge of a captured image, and in this embodiment, it is described as being added to the left and upper ends as shown in the figure. By applying a geometric transformation process such as a perspective projection transformation process or an equirectangular transformation process to the VR image 200, it can be displayed as a natural image on an HMD (Head Mount Display), monitor, or the like. In the above-mentioned geometric deformation process, the light-shielded region 203 is not used as a display region, and the right image 201 and the left image 202 are used as a display region. When displayed on an HMD or the like, the outer peripheral regions of the right image 201 and the left image 202 are greatly stretched. Therefore, in the coding process of the VR image, the regions within the image circles of the right image 201 and the left image 202 are coded with a priority on reducing the amount of code (or coded with a relatively small number of codes assigned), and the outer peripheral regions within the image circles are coded with a priority on image quality (or coded with a larger number of codes assigned) in order to suppress deterioration of image quality due to stretching. Furthermore, it is important to increase coding efficiency by reducing the codes assigned to the light-shielded region 203, which is redundant data. For this purpose, it is necessary to separate the left image 201, the right image 202, the light-shielded region 203, and the OB region 204, and set an appropriate quantization parameter for each separated region.

[0015] Considering that the left and right images are circular fisheye images, it is possible to separate the left and right images 201, 202 from the light-shielded area 203 by using the central coordinates of the left image 201 and the right image 202 and the image circle diameter information stored as lens information. Also, as for the OB area 204, the sensor information stores how many pixels are assigned to which image edge, so if the number of pixels is known, it is possible to separate the OB area 204. Based on this, the configuration and operation of each part will be described.

[0016] The control unit 101 controls each processing unit constituting the image processing device 100, executes calculation processing, and the like.

[0017] Furthermore, the control unit 101 determines a compression rate in accordance with a shooting setting designated by the user via the operation unit 109, and outputs information about the compression rate to a quantization control unit 105, which will be described later.

[0018] Furthermore, the control unit 101 calculates the center coordinates of each of the right and left images in order to separate the right and left images and the light-shielded area in the VR image acquired from the imaging unit 102 described below. Since the center coordinates are subject to center shift caused by lens manufacturing errors, lens attachment / detachment, and tilt of the housing during imaging, it is necessary to calculate the coordinates taking this center shift into consideration.

[0019] The manufacturing error includes a correction coefficient for eliminating individual differences in optical performance from, for example, chart measurement results at the time of shipping from the factory, and is stored as lens information. The control unit 101 calculates the coordinate shift amount of the imaging point from this lens information. The lens information includes the center coordinates of the left and right images on the ideal imaging surface. The control unit 101 adds the center coordinates of the ideal left and right images acquired from the lens information to the above coordinate shift amount, and calculates the center coordinates of each of the right and left images taking into account the shift in center coordinates.

[0020] The mounting / removal error includes the coordinate shift amount of the left and right images when the lens is mounted and removed. In practice, when the lens is mounted and an image is captured, the control unit 101 performs pattern matching between the right and left images for the captured image, and calculates the coordinate shift amount in the horizontal, vertical, and rotational directions when the difference between the feature points of each image is smallest. The control unit 101 adds the above coordinate shift amount to the ideal center coordinates of the left and right images obtained from the lens information, and calculates the center coordinates of the right and left images taking into account the shift in center coordinates.

[0021] The tilt of the housing during imaging can be detected by, for example, a gyro sensor (not shown). The control unit 101 calculates the coordinate shift amounts in the horizontal, vertical, and rotational directions according to the detected tilt. Then, similar to the mounting / removal error, the control unit 101 adds the above coordinate shift amounts to the center coordinates of the ideal left and right images obtained from the lens information to calculate the center coordinates of the right and left images taking into account the shift in center coordinates.

[0022] When the control unit 101 determines the central coordinates of the right and left images on the image sensor as described above, it determines the correction amount of the quantization parameter according to the distance from a quantization target block of a predetermined size, and outputs the amount to the quantization control unit 105 described later. The method of determining the quantization parameter correction amount will be described later in detail.

[0023] The imaging unit 102 includes a lens optical system capable of optical zooming, including an optical lens, an aperture, focus control, and a lens driving unit, and an imaging sensor such as a CCD or CMOS sensor that converts optical information from the lens optical system into an electrical signal. The imaging unit 102 outputs RAW image data (VR image) obtained by converting an electrical signal obtained by the imaging sensor into a digital signal to the plane conversion unit 103. As described above, the imaging unit 102 has a twin-lens optical system, and forms a right image and a left image with different parallax on one sensor to output RAW image data. In addition, RGB color filters are regularly arranged in the imaging sensor of the imaging unit 102 in this embodiment. In this embodiment, the arrangement (arrangement) of the color filters is a Bayer array. The Bayer array refers to an array in which, for example, 2×2 pixels are composed of one R pixel, one B pixel, and two G pixels (G1, G2), and this 2×2 pixel pattern is repeated, as shown in FIG. 3.

[0024] The plane conversion unit 103 converts the RAW image data input from the imaging unit 102 into four independent plane data, each of which is composed of a single component. Then, the plane conversion unit 103 outputs the four generated plane data to the frequency conversion unit 104. Examples of conversion equations to plane data are shown in the following equations (1) to (4). The plane conversion unit 103 of the embodiment converts the RAW image data into a plane Y that approximately represents a luminance component and planes C0, C1, and C2 that represent the other three color difference components, in accordance with equations (1) to (4). Y=(R+G1+G2+B) / 4 ...(1) C0=RB ...(2) C1=(G0+G1) / 2-(R+B) / 2 ...(3) C2=G0-G1 ...(4) When the number of pixels in the horizontal direction of the RAW image data is W and the number of pixels in the vertical direction is H, the size of each of the planes Y, C0, C1, and C2 is W / 2×H / 2 pixels. The above conversion is an example, and the method of conversion to plane data is not limited to this, and other methods such as separating and outputting R, G1, G2, and B may be used. The plane conversion unit 103 supplies, for example, planes Y, C1, C2, and C3 to the frequency conversion unit 104 in this order.

[0025] The frequency transform unit 104 inputs each plane from the plane transform unit 103 and performs frequency transform on the planes. Then, the frequency transform unit 104 outputs transform coefficients generated by the frequency transform to the quantization control unit 105 and the quantization unit .

[0026] An example of frequency transformation using wavelet transform is shown in Fig. 4. Fig. 4 is a diagram of subband formation at decomposition level 3, where wavelet transform is performed three times in each of the vertical and horizontal directions. In Fig. 4, the leading numbers such as "1HL" indicate the decomposition level and also indicate which wavelet transform was used to obtain the subband. When wavelet transform is performed once, four subbands, LL, HL, LH, and HH, are generated. Then, wavelet transforms from the second time onwards are performed on the sub-subband LL obtained in the immediately preceding wavelet transform. Subbands LH, HL, and HH other than subband LL represent high frequency components.

[0027] As shown in FIG. 5, the quantization control unit 105 has a reference quantization parameter determination unit 105a, a feature information generation unit 105b, and a quantization parameter correction unit 105c, and determines the quantization parameter to be used in the quantization unit .

[0028] The reference quantization parameter determination unit 105a determines a quantization parameter to be applied to the entire screen according to the compression ratio input from the control unit 101. Then, the reference quantization parameter determination unit 105a outputs the determined quantization parameter to the quantization parameter correction unit 105c.

[0029] The frequency transform unit 104 of the embodiment performs wavelet transform on the RAW image data up to decomposition level 3. As shown in FIG. 6, ten subbands are generated from one plane: 1HL, 1LH, 1HH, 2HL, 2LH, 2HH, 3HL, 3LH, 3HH, and 3LL. In the embodiment, four planes are generated from one piece of RAW image data, so the number of subbands generated from one piece of RAW image data is 40. The frequency transform unit 104 then supplies the 40 generated subbands to the quantization unit 106. For the Y plane, the frequency transform unit 104 supplies subband 1LL to the quantization control unit 105 in addition to the generated subbands 1HL, 1LH, and 1HH. Although not shown in FIG. 4, the subband 1LL of the Y plane is obtained when the first wavelet transform is performed, and is also the target of the second wavelet transform.

[0030] The above reference quantization parameters have a relationship of one parameter for every 40 subbands. In the wavelet transform, subsampling is performed in the horizontal and vertical directions for each decomposition level. Therefore, the 1x1 coefficient in the subband of decomposition level 3, the 2x2 coefficient in the subband of decomposition level 2, the 4x4 coefficient in the subband of decomposition level 1, the 8x8 pixels of the plane data, and, taking into account plane separation, the 16x16 pixels of the RAW image data correspond to areas of the same spatial size.

[0031] Subband data and pixel data at the same spatial coordinates are aligned as a control unit of the quantization parameter so as not to impair controllability of image quality. In this embodiment, the control unit of the quantization parameter is 16 x 16 pixels in terms of RAW image data. Note that, for the value of the reference quantization parameter, statistical values ​​that guarantee image quality for each compression ratio are prepared in advance in a table, and the table to be referenced is switched depending on the compression ratio. Once the reference quantization parameter is determined, the control is simplified by using a mechanism in which the quantization parameter for each of the 40 subbands is uniquely determined.

[0032] The feature information generating unit 105b calculates feature information of each of lightness and complexity using each subband data of the Y plane input from the frequency transform unit 104, and generates a quantization parameter correction amount based on the calculated feature information. The feature information generating unit 105b then outputs the generated quantization parameter correction amount to the quantization parameter correction unit 105c. Details of the calculation method of the feature information and the method of determining the quantization parameter correction amount will be described later.

[0033] The quantization parameter correction unit 105c corrects the reference quantization parameter input from the control unit 105a by adding the first quantization parameter correction amount input from the control unit 101 and the second quantization parameter correction amount input from the feature information generation unit 105b to the reference quantization parameter input from the control unit 105a. Then, the quantization parameter correction unit 105c outputs the quantization parameter obtained by the correction to the quantization unit 106. Details of the procedure for outputting the quantization parameter by the quantization parameter correction unit 105c will be described later.

[0034] The quantization unit 106 quantizes the sub-band data input from the frequency conversion unit 104 using the quantization parameter input from the quantization control unit 105 , and outputs the quantized coefficients to the entropy coding unit 107 .

[0035] The entropy coding unit 107 compresses and codes the coefficients quantized by the quantization unit 106, and outputs the coded data to a recording medium 108. There is no particular limit to the type of compression coding, but it is assumed that the compression coding is performed using entropy coding such as Golomb coding.

[0036] The recording medium 108 is a recording medium formed of, for example, a non-volatile memory, and stores the encoded data output by the entropy encoding unit 107 as a file.

[0037] Various parameters related to the flow of determining the first quantization parameter correction amount are shown in Fig. 7. Fig. 7 is an overall configuration diagram of a VR image captured by the imaging unit 102, and description of contents overlapping with the description of Fig. 2 will be omitted. Note that the coordinate position of the upper left corner of the VR image is the origin (0,0), the horizontal axis shown in the figure is the X-axis, and the rightward direction is the positive direction. Similarly, the vertical axis is the Y-axis, and the downward direction is the positive direction.

[0038] Reference numeral 701 denotes the horizontal size of the OB region 204, and this size is designated as "a". Reference numeral 702 denotes the vertical size of the OB region 204, and this size is designated as "b". Reference numeral 703 denotes the image circle center of the right image acquired by the right eye optical system, and this coordinate is designated as (x1, y1). Reference numeral 704 denotes the image circle center of the left image acquired by the left eye optical system, and this coordinate is designated as (x2, y2). Reference numerals 705 and 707 denote the radii of the left and right image circles, and this length is designated as r.

[0039] Reference numeral 706 denotes the distance between the quantization target block (16×16 pixels) and the right-view circle center 703, and the length of the distance is defined as d_right. The quantization target block is a Bayer array of 16×16 pixels (=256 pixels). In the embodiment, the shortest distance between each pixel in the quantization target block and the left-view circle center 704 is defined as d_right.

[0040] Reference numeral 707 denotes the distance d_left between the block to be quantized (16×16 pixels) and the left image circle center 704. Like d_right, d_left is the shortest distance between the block to be quantized and the left image circle center 704. Reference numeral 708 denotes the pixel coordinate with the shortest distance from the left image circle center, and the coordinate is (x0, y0). Reference numeral 708 denotes the pixel coordinate having the minimum value of either d_right 706 or d_left 707. The parameters indicated by the above reference numerals 701 to 708 are used to classify the left image circle region, the light blocking region, and the OB region. Hereinafter, the above parameters will be used for explanation.

[0041] Next, a method for obtaining the first quantization parameter correction amount will be described with reference to the table in FIG. 8. As described above, in the coding process of a VR image, it is important to separate the image circle region, which is the display region, from the non-display region, which is the light-shielded region and the OB region, and to set different quantization parameters for each region. In addition, since the influence of image quality degradation due to geometric deformation differs even within the image circle region, it is also necessary to separate the inner circumference part and the outer circumference part within the image circle region. The coordinate conditions or the distance conditions from the center coordinates of the image circle for classifying the quantization target block shown in the first field of FIG. 8 into each region are shown below. OB area:x0 Inner circumference area: d≦dth Outer area: dth <d≦r Shading area:r <d In the above condition, "dth" is a distance threshold for classifying the inner peripheral area and the outer peripheral area.

[0042] Next, the correction amount of the first quantization parameter will be described. The second field quantization parameter correction amount in Fig. 8 indicates the correction amount with respect to the reference quantization parameter. The correction amount of the quantization parameter for each area is set so that the magnitude relationship satisfies the following relationship, assuming that the OB area 204 is qpcv0, the inner area is qpcv1, the outer area is qpcv2, and the light-shielded area 203 is qpcv3. qpcv2≦qpcv1≦qpcv0≦qpcv3 It should be noted that the larger the correction amount, the greater the degree of degradation in image quality and the higher the compression rate. Conversely, the smaller the correction amount, the less the degree of degradation in image quality and the lower the compression rate.

[0043] ​The quantization parameter correction amount of the OB region 204 is considered as a reference (=0). Since the inner and outer regions are display regions, their image quality importance is higher than that of the OB region 204. Therefore, negative values ​​are set so that qpcv1 and qpcv2 are relatively finer quantization than qpcv0. Even within the image circle, the outer region has a higher image quality importance than the inner region from the viewpoint of the above-mentioned geometric deformation processing. Therefore, qpcv2 is set to a negative value so that it is even finer quantization than qpcv1. On the other hand, the light-shielded region 203 is a non-display region like the OB region 204, but is not used in the above-mentioned development processing and is a redundant region, so its image quality importance is lower than that of the OB region 204. Therefore, positive values ​​are set so that qpcv3 is relatively coarser quantization than qpcv0. Since the light-shielded region is a redundant region, the first quantization parameter correction amount is set so that the generated code of qpcv3 is 0 (or the minimum).

[0044] Next, the process of determining the first quantization parameter correction amount performed by the control unit 101 will be described with reference to the flowchart in Fig. 9. In this embodiment, wavelet transformation is performed at decomposition level 3, so the size of the block to be quantized is 16 x 16 pixels in RAW image data. Therefore, the control unit 101 scans (moves) the quantization block in units of 16 pixels in the horizontal and vertical directions.

[0045] In S901, the control unit 101 acquires lens information, sensor information, and tilt information of the housing of the imaging unit 102. The lens information includes ideal lens center coordinates for the right and left images and correction coefficients related to lens manufacturing errors. The sensor information includes the horizontal and vertical sizes of the OB area.

[0046] In S902, the control unit 101 calculates the amount of deviation of the center coordinates using the lens information and tilt information of the housing acquired in S901. The control unit 101 also performs pattern matching between the left and right images, and calculates the amount of coordinate deviation when the difference between the feature points of each image is the smallest. The control unit 101 adds these deviation amounts to the ideal lens center coordinates included in the lens information, and calculates the center coordinates (x1, y1), (x2, y2) of the left and right image circles in the RAW image data, taking the center deviation into consideration.

[0047] In S903, the control unit 101 calculates a distance d that is the shortest between the center coordinates of the left or right image circle acquired in S902 and the coordinates of each pixel in the quantization target block of interest.

[0048] When calculating d, first, the distance between one pixel in the quantization target block of interest and the image circle center coordinates is calculated. This distance is defined as d[idx]. Here, idx is the index of the pixel in the quantization target block. In the case of the embodiment, the quantization target block is composed of 256 pixels of 16×16 pixels. Therefore, idx is assigned in the quantization target block in raster scan order. In other words, ids takes a value from 0 to 255. d[idx] is calculated according to the following formulas (5) to (7) using the coordinates (x, y) of the pixel in the quantization target block, the distance d_right from the right image center coordinates of the pixel in the quantization target block, and the distance d_left from the left image center coordinates. d_right=((x-x1) 2 +(y-y1) 2 ) 1 / 2 ...(5) d_left=((x-x2) 2 +(y-y2) 2 )) 1 / 2 ...(6) d[idx]=MIN(d_right,d_left) ...(7) Here, MIN() is a function that returns the minimum value of the given arguments.

[0049] The minimum value of d_right and d_left is substituted for d[idx]. The reason for substituting the minimum value will be explained here. If the block to be quantized is located within the left image circle, if the distance is calculated using the right image circle center coordinates, the block will be classified as a light-shielded region because the distance to the right image circle center coordinates is far. In this way, in order to properly classify the region, the distance must always be calculated using the image circle center coordinates that are close to the block to be quantized. Therefore, the minimum value of d_right and d_left is set as the distance d[idx]. If the values ​​of d_right and d_left are equal, either one may be set as the distance d[idx].

[0050] This is the flow for calculating the distance to one pixel in the block to be quantized, and the first pixel index d[0] is determined. The control unit 101 then repeats this series of steps for a total of 256 pixels (16x16), calculating up to d

[0255] . After the control unit 101 has finished calculating d[0] to d

[0255] , it calculates the distance d using the following equation (8). d=MIN(d[0],d[1],d[2],..., d

[0254] ,d

[0255] ) ...(8)

[0051] As shown in formula (8), by selecting the minimum value from the calculated distances to the 256 pixels, it is possible to calculate the distance d that is closest to either the left or right image circle center coordinate. The reason why the distance d is selected as the minimum of each distance in the quantization target block is to ensure that the quantization target block is classified as the image circle region when it straddles the image circle region and the light-shielding region. Since the image circle region is closer to the image circle center than the light-shielding region, if the minimum distance in the quantization target block is selected, it is automatically classified as the image circle region. In this way, it is possible to suppress the deterioration of image quality in the image circle region when the quantization target block straddles the image circle region and the light-shielding region. The pixel coordinates where the distance between the image circle center and the quantization target block is the minimum are (x0, y0).

[0052] In S904, the control unit 101 uses the information on the OB region acquired in S901 to determine whether the block to be quantized is in the OB region. Specifically, if either x0 is smaller than a or y0 is smaller than b is satisfied, the control unit 101 determines that the block to be quantized is in the OB region, and advances the process to S905. If the control unit 101 determines that the block to be quantized is outside the OB region, the control unit 101 advances the process to S906. Note that if some pixels in the block to be quantized are included in the OB region, the block to be quantized is determined to be in the OB region. This is to prevent the above-mentioned development process from being affected by performing coarse quantization.

[0053] In S905, the control unit 101 determines that the quantization target block of interest is in the OB region, and determines the first quantization parameter correction amount to be qpcv0.

[0054] In S906, the control unit 101 determines whether the quantization target block of interest is in the inner periphery region. Specifically, the control unit 101 compares the distance d with a preset threshold dth, and if it is determined that d is equal to or smaller than the threshold dth, the process proceeds to S907, otherwise the process proceeds to S908.

[0055] In S907, the control unit 101 determines that the quantization target block of interest is in the inner periphery area, and determines the first quantization parameter correction amount to be qpcv1.

[0056] In S908, the control unit 101 determines whether the quantization target block of interest is in the outer circumferential region by comparing the distance d with the distance r. If the control unit 101 determines that the distance d is equal to or smaller than the distance r, the control unit 101 advances the process to S909, and if not, the control unit 101 advances the process to S910.

[0057] In S909, the control unit 101 determines that the quantization target block of interest is in the outer circumferential area, and determines the first quantization parameter correction amount to be qpcv2.

[0058] In S910, the control unit 101 determines that the quantization target block of interest is in a light-shielded area, and determines the first quantization parameter correction amount to be qpcv3.

[0059] In S911, the control unit 101 determines whether the first quantization parameter correction amount has been determined for all the blocks to be quantized. If the control unit 101 determines that the first quantization parameter correction amount has been determined for all the blocks to be quantized, the control unit 101 completes this process. If the control unit 101 determines that there is a quantization block for which the first quantization parameter correction amount has not been determined, the control unit 101 returns the process to S903 and updates the position of the next block to be quantized. Then, the control unit 101 performs the above process with the quantization block after the position update as the quantization block of interest.

[0060] Next, a method of generating feature information by the feature information generating unit 105b for determining the second quantization parameter correction amount will be described. The feature information generating unit 105b performs light / dark judgment and complexity judgment for a predetermined wavelet coefficient unit (rectangular block of one or more coefficients) of the Y plane, calculates feature amounts for each, and generates a second quantization parameter correction amount according to the feature amounts. This point will be described in more detail below.

[0061] The brightness of light and dark represents the brightness level of a pixel. In the quantization process, if a relatively large quantization parameter is used for a dark area with a low brightness level, the coefficient after quantization is likely to become 0, and fine information is likely to be lost. Therefore, in order to reduce quantization distortion in the dark area, it is desirable to set the quantization parameter small. Also, in the bright area, it is desirable to quantize using a relatively large quantization parameter so that the code amount does not increase. The feature information generating unit 105b generates a correction amount for increasing or decreasing the quantization parameter according to the light and dark areas.

[0062] The complexity represents the height of the spatial frequency of the RAW data, and the lower the spatial frequency, the flatter the region, and the higher the spatial frequency, the more complex the region. In image compression, when information in a flat region is reduced, visual degradation is easily noticeable, and when information in a complex region is reduced, it is visually difficult to notice. Therefore, it is desirable to set a relatively small quantization parameter for a region including a flat region. Also, it is desirable to set a relatively large quantization parameter for a complex region where visual degradation is less noticeable, thereby improving the coding efficiency. The feature information generating unit 105b generates a correction amount for increasing or decreasing the quantization parameter according to the complexity.

[0063] In addition, it is preferable that the unit for performing the second quantization parameter correction amount and the unit for generating the feature information are the same control unit so as not to impair the controllability of the image quality. In this embodiment, the control unit is also aligned with 16×16 pixels, which is the calculation unit of the quantization parameter correction amount performed by the control unit 101, and the level 1 coefficient 4×4 coefficient of the wavelet decomposition process is set as the quantization block size and the generation unit of the quantization parameter correction amount and the feature information.

[0064] Here, a method for calculating the feature amounts of lightness and complexity will be described with reference to FIG.

[0065] The feature information generating unit 105b calculates the luminance feature amount dc as the average value of the 4×4 coefficient block 1001 of the subband 1LL representing the DC component of the Y plane data, according to the following equation (9). dc=(A1+A2+A3+...+A15+A16) / 16...(9) Note that subband 1LL here refers to the LL subband at decomposition level 1, not subband 3LL in FIG.

[0066] The coding efficiency of high frequency components is poor. In other words, the amount of high frequency components and complexity can be considered to have the same meaning. Therefore, the complexity feature ac is calculated using subbands 1HL, 1LH, and 1HH that represent the AC components (high frequency components) of the Y plane data. Specifically, the feature information generating unit 105b calculates the complexity feature ac according to the following equations (10) to (13) using a 4×4 coefficient block 1002 of the subband 1HL, a 4×4 coefficient block 1003 of the subband 1LH, and a 4×4 coefficient block 1004 of the subband 1HH, which are at the same spatial position. ac_1HL={ABS(B1)+ABS(B2)+ABS(B3)+...+ABS(B16)} / 16 ...(10) ac_1LH={ABS(C1)+ABS(C2)+ABS(C3)+...+ABS(C16)} / 16 ...(11) ac_1HH={ABS(D1)+ABS(D2)+ABS(D3)+...+ABS(D16)} / 16 ...(12) ac=MAX(ac_1HL,ac_1LH,ac_1HH) ...(13) Here, ABS() is a function that returns the absolute value of the argument, and MAX() is a function that returns the maximum value among the given arguments (average values ​​ac_1HL, ac_1LH, ac_1HH).

[0067] A method for determining the second quantization parameter correction amount performed by the feature information generating unit 105b will be described with reference to the feature classification table shown in FIG. 11. In FIG. 7, dcth is a threshold for determining brightness and darkness, and acth is a threshold for determining the complexity of flatness and complexity. Each threshold is a parameter that can be set arbitrarily according to the bit depth of the image, etc., and the features of the block to be quantized are classified according to the magnitude relationship between these two thresholds and dc and ac. qpcv4 to qpcv7 are the correction amounts of the second quantization parameter applied to each feature classified according to the magnitude relationship with each threshold. The magnitude relationship of the second quantization parameter correction amount for each feature of complexity is, for example, as follows: qpcv4≦qpcv5≦qpcv6≦qpcv7

[0068] The second quantization parameter correction amount is set to be the finest (smallest) in qpcv4 for dark and flat areas and the coarsest (largest) in qpcv7 for bright and complex areas, taking into account visual characteristics.

[0069] Next, the process of determining the second quantization parameter correction amount by the feature information generating unit 105b will be described with reference to the flowchart in FIG.

[0070] In S1201, the feature information generating unit 105b calculates the feature amounts dc and ac of the block to be quantized according to the above-mentioned equations (9) to (13).

[0071] In S1202, the feature information generating unit 105b performs a process of determining which is larger between the feature amount dc and the threshold value dcth, and between the feature amount ac and the threshold value acth. If the feature information generating unit 105b determines that dc is less than dcth and that ac is less than acth, the process proceeds to S1203, and if not, the process proceeds to S1204.

[0072] In S1203, since the block to be quantized is a dark / flat region, the feature information generating unit 105b determines the second quantization parameter correction amount to be qpcv4.

[0073] In S1204, the feature information generating unit 105b performs a process of determining which of the feature amount dc and the threshold value dcth, and which of the feature amount ac and the threshold value acth is larger or smaller. If the feature information generating unit 105b determines that the feature amount dc is equal to or larger than the threshold value dcth and that the feature amount ac is smaller than the threshold value acth, the process proceeds to S1205, otherwise the process proceeds to S1206.

[0074] In S1205, since the block to be quantized is a bright / flat region, the feature information generating unit 105b determines the second quantization parameter correction amount to be qpcv5.

[0075] In S1206, the feature information generating unit 105b performs a process of determining which is larger between the feature amount dc and the threshold value dcth, and between the feature amount ac and the threshold value acth. If the feature information generating unit 105b determines that the feature amount dc is less than the threshold value dcth and that the feature amount ac is equal to or greater than the threshold value acth, the process proceeds to S1207, otherwise the process proceeds to S1208.

[0076] In S1207, since the block to be quantized is a dark / complex region, the feature information generating unit 105b determines the second quantization parameter correction amount to be qpcv6.

[0077] In S1208, since the block to be quantized is a bright / complex region, the feature information generating unit 105b determines the second quantization parameter correction amount to be qpcv7.

[0078] In S1209, the feature information generating unit 105b judges whether the second quantization parameter correction amount of all the blocks to be quantized has been determined. If the second quantization parameter correction amount of all the blocks to be quantized has been determined, the process is completed. If not, the process returns to S1201, and a process for determining the second quantization correction parameter of the unprocessed blocks to be quantized is performed.

[0079] Finally, the quantization parameter output process performed by the quantization parameter correction unit 105c will be described with reference to the flowchart of FIG.

[0080] In S1301, the quantization parameter correction unit 105c obtains a reference quantization parameter from the control unit 105a.

[0081] Then, in S1302, the quantization parameter correction unit 105c adds the second quantization parameter correction amount supplied from the feature information generation unit 105b to the reference quantization parameter acquired in S1301.

[0082] In S1303, the quantization parameter correction unit 105c adds the first quantization parameter correction amount supplied from the control unit 101 to the addition result calculated in S1302. Then, the quantization parameter correction unit 105c outputs the addition result to the quantization unit 106 as a corrected quantization parameter.

[0083] The quantization unit 106 quantizes the coefficients in the subbands to be quantized in each plane using the corrected quantization parameters supplied from the quantization parameter correction unit 105c and the quantization step value determined by the type of the subband to be quantized. As described above, the types of subbands are 3LL, 3HL, 3LH, 3HH, 2HL, 2LH, 2HH, 1HL, 1LH, and 1HH. The magnitude relationship of the quantization step values ​​to be used is generally 3LL≦3HL, 3LH, 3HH≦2HL, 2LH, 2HH≦1HL, 1LH, and 1HH. It is only necessary to make a table of the relationship between the corrected quantization parameters and the types of subbands and the quantization step values ​​to be actually used.

[0084] Furthermore, in the addition processes in S1302 and S1303, clipping is performed at the upper or lower limit of the quantization parameter, making it possible to perform addition processes that prevent the quantization parameter from overflowing or underflowing.

[0085] As described above, it is possible to improve coding efficiency by classifying VR images based on image features based on coordinates and image features using brightness and complexity, and correcting the quantization parameters according to the image quality importance of each classified area.

[0086] In this embodiment, the quantization method is applied when a twin lens having two optical systems capable of photographing right and left images with parallax is attached. When a twin lens is not attached, quantization control using image features based on brightness and complexity is not performed, and only quantization control using image features based on brightness and complexity is performed.

[0087] In the present embodiment, the size of the block to be quantized is described as 16×16 pixels in RAW image data, but the block size is not limited to this. In the generation of feature information using wavelet coefficients, the size of the block to be quantized is described as decomposition level 1 subband 4×4 coefficients, but the block size is not limited to this. For the purpose of simplifying the explanation by unifying the quantization block size, it should be understood that the quantization block size is aligned to have the same coordinate relationship as 16×16 pixels and level 1 subband 4×4 coefficients.

[0088] In addition, when a quantization block extends beyond an image edge, processing may be performed only on pixels that do not fill the original block size, or the pixels that do not fill the block size may be interpolated using mirrored pixels and copied pixels.

[0089] In this embodiment, an example has been described in which image features based on coordinates are classified into four regions, namely, an inner region, an outer region, a light-shielded region, and an OB region. Also, an example has been described in which image features based on brightness and complexity are classified into a total of four regions, namely, two levels of light and dark, and two levels of flat and complex. However, the number of regions to be classified into features is not limited to this, and each feature may be further increased. In this case, a threshold value required for classification into multiple levels may be prepared, and classification may be performed according to whether the threshold value is less than the threshold value or more than the threshold value.

[0090] In this embodiment, the control unit 101 outputs the correction amount of the first quantization parameter to the quantization control unit 105c, but a classification map indicating which region the quantization target block belongs to may be output. In that case, the quantization parameter correction unit 105c determines the correction amount of the quantization parameter based on the classification map.

[0091] In the above embodiment, an example in which the decomposition level is 3 has been described, but the decomposition level is not particularly limited. When performing wavelet transform n times to perform wavelet transform up to decomposition level n, data consisting of coefficients at the same spatial position in subbands nLL, nHL, nLH, nHH, ..., 1HL, 1LH, 1HH becomes the block to be quantized.

[0092] [Second embodiment] The second embodiment 2 will be described. In the second embodiment, the method of dividing the area in the image circle and the method of determining the first quantization parameter correction amount are different from those in the first embodiment. In the first embodiment, only two areas, the inner area and the outer area, in the image circle are classified using the distance from the center coordinate and an arbitrary threshold value, and the first quantization parameter correction amount of a fixed pattern prepared in advance is set. Therefore, when dividing the area in the image circle finely, it is necessary to increase the threshold value and the first quantization parameter correction amount according to the number of areas, and there is a concern that the circuit scale will become bloated if a process of determining whether the center coordinate and the threshold value are large or small is also included. Therefore, in the second embodiment, a method will be described in which the first quantization parameter correction amount in the image circle is set more finely according to the distance from the center coordinate, and can be automatically determined without a process of determining whether the center coordinate and the threshold value are large or small.

[0093] The configuration of the image processing device of the second embodiment is shown in FIG. 15. The encoding process based on the H.264 standard by the image processing device 1500 of the second embodiment will be described below with reference to FIG. 15. In the first embodiment, the data of the Bayer array is separated into plane data, and frequency transform is performed on the plane data. In addition, since the wavelet transform is used in the frequency transform, the subband data is subsampled in the horizontal and vertical directions depending on the number of decomposition levels. As a result, the control unit of the quantization block is Bayer 16×16 pixels, plane 8×8 pixels, 4×4 coefficients in the subband of decomposition level 1, 2×2 coefficients in the subband of decomposition level 2, and 1×1 coefficients in the subband of decomposition level 1, and the block size is different in each processing step. In the H.264 standard of this embodiment, the encoding process unit is specified by the macroblock, so that the process is performed in units of macroblock size whether it is a pixel or a coefficient after orthogonal transform. In this embodiment, the unit of the encoding block and the block to be quantized is consistently unified to 16×16 pixels. Detailed description of processing units similar to those in the first embodiment will be omitted, and symbols such as parameters are also common to those in the first embodiment.

[0094] The imaging unit 1501 converts the VR 180 imaging data acquired through a twin-lens optical system from an RGB spatial representation to a YCbCr spatial representation, and stores the converted image in a target image memory 1502 .

[0095] The control unit 1503 corresponds to the control unit 101 in the first embodiment. The control unit 1503 outputs information on the compression ratio to the quantization control unit 1504 in response to an instruction input from a user via the operation unit 1520. The control unit 1503 also acquires lens information of the imaging unit 1501 and calculates the image formation point center coordinates of each of the left and right images. After the center coordinates of each of the left and right images are determined, the control unit 1503 determines a correction amount of the first quantization parameter in accordance with the distance from the quantization target block, and outputs the correction amount to the quantization control unit 1504 described later. A method for determining the first quantization parameter correction amount will be described later in detail.

[0096] The quantization control unit 1504 corrects the quantization parameter by adding the correction amount of the first quantization parameter to the reference quantization parameter input from the control unit 1503. Then, the quantization control unit 1504 outputs the corrected quantization parameter to the quantization unit 1508.

[0097] The prediction unit 1505 determines a predictive coding method for a coding target block, which is a coding unit, and generates a predicted image. The coding method includes intra-picture predictive coding, which performs coding only by predicting an image within a frame, and inter-picture predictive coding, which performs coding including predicting an image between frames.

[0098] When performing intra prediction coding, a block image to be coded, which is a coding unit, is read from a target image memory 1502. A prediction unit 1505 reads coded pixel data adjacent to the block to be coded (pixel data before being subjected to a deblocking filter 1512, which is output from an adder 1511, which will be described later) from a reference image memory 1513, performs block matching with the intra prediction images for the generated multiple prediction modes, and outputs the intra prediction image with the highest coding efficiency to a subtractor 1506.

[0099] On the other hand, when performing inter-frame prediction coding, a current block image to be coded, which is a coding unit, is read from current image memory 1502 and output to prediction unit 1505. Also, a previously coded reference image is read from reference image memory 1513 and output to prediction unit 104. Prediction unit 1505 performs motion search processing from the current block image to be coded and the reference image to detect a motion vector for the reference image. An inter-frame prediction image is generated using the detected motion vector and a reference image obtained from reference image memory 1513, and output to subtractor 1506.

[0100] The subtractor 1506 receives the predicted image generated by the prediction unit 1505 and the current block image to be coded read from the current image memory 1502, and generates a difference image of pixel values ​​between the block image to be coded and the predicted image. The subtractor 1506 then outputs the generated difference image to the orthogonal transformation unit 1507.

[0101] The orthogonal transform unit 1507 performs orthogonal transform on the difference image of the input pixel values, and outputs the transform coefficients to the quantization unit 1508.

[0102] The quantization unit 1508 uses the quantization parameter input from the quantization control unit 1504 to output the quantized transform coefficients to the entropy coding unit 1514 .

[0103] The transform coefficients quantized by the quantization unit 1508 are also input to the inverse quantization unit 1509. The inverse quantization unit 1509 inverse quantizes the input transform coefficients, and outputs the inverse quantized transform coefficients to the inverse orthogonal transformation unit 1510.

[0104] The inverse orthogonal transform unit 1510 performs inverse orthogonal transform processing on the dequantized coefficients input from the inverse quantization unit 1509. Then, the inverse orthogonal transform unit 1510 outputs the data after the transform processing to the adder 1511.

[0105] The adder 1511 adds the inverse orthogonally transformed data and the predicted image generated by the prediction unit 1505. The data after the addition becomes a decoded reconstructed image, is input to the reference image memory 1513, and is used to generate an intra-screen predicted image.

[0106] The decoded reconstructed image is subjected to a deblocking filter 1512 to reduce block distortion that occurs during encoding, and is stored in a reference image memory 1513 as a reference image to be used during inter-picture predictive encoding.

[0107] The entropy coding unit 1514 entropy codes the quantized variable coefficients input from the quantization unit 1508 .

[0108] The recording medium 1515 records the encoded data input from the entropy encoding unit 1514 .

[0109] Here, the process of determining the first quantization parameter correction amount performed by the control unit 1503 will be described with reference to the flowchart in Fig. 14. The difference from the first embodiment is that S1408 and S1409 are added. Detailed description of the same processing steps as in the first embodiment will be omitted, and the symbols of parameters, etc. are also common to the first embodiment.

[0110] In S1401, the control unit 1503 acquires lens information and sensor information of the imaging unit 1501.

[0111] In S1402, the control unit 1503 calculates the center coordinates of the left-right image circle taking into account the center deviation by using the lens information acquired in S1401.

[0112] In S1403, the control unit 1503 calculates the distance d using the center coordinates of the lateral view circle acquired in S1402 and the coordinates of a pixel in the quantization block. The pixel coordinates at which this distance d is calculated are (x0, y0).

[0113] In S1404, the control unit 1503 determines whether the block to be quantized is in the OB region using the information on the OB region acquired in S1401. If the control unit 1503 determines that x0 is smaller than a or that y0 is smaller than b, the process proceeds to S1405, and if not, the process proceeds to S1406.

[0114] In S1405, the control unit 1503 determines that the block to be quantized is within the OB region, and determines the quantization parameter correction amount to be qpcv0.

[0115] In S1406, the control unit 1503 determines whether the child block is in a light-shielded region. If the control unit 1503 determines that the distance d is greater than the radius r of the circular image, the process proceeds to S1407. If not, the control unit 1503 proceeds to S1408.

[0116] In S1407, the control unit 1503 determines that the block to be quantized is within a light-shielded area, and determines the quantization parameter correction amount to be qpcv3.

[0117] In S1408, the control unit 1503 acquires a quantization parameter setting lower limit qmin (>0) and a reference quantization parameter qbase (>0). Here, the quantization parameter setting lower limit qmin is the lower limit value of the quantization parameter according to the compression rate and is an arbitrary setting parameter. Also, as described above, qbase is the quantization parameter applied to the entire screen according to the compression rate.

[0118] In S1409, the control unit 1503 determines the first quantization parameter correction amount according to the following formula (14), assuming that the quantization target block is within the image circle. qpcv = d / r * (qmin - qbase)...(14) Here, "d / r" in the formula represents the ratio of the distance of how far the quantization target block is from the center coordinates. Since qmin < qbase, the first quantization parameter correction amount qpcb will have a value of 0 or a negative value.

[0119] When the quantization target block is at the center coordinates, d = 0, and the correction amount qpcv of the first quantization parameter becomes 0. Also, when the quantization target block is located on the image circle, d = r, and the first quantization parameter correction amount qpcv becomes the largest in the negative direction. By using such a calculation formula, it is possible to set a quantization parameter such that quantization becomes finer in the outer peripheral region within the image circle, and the distance threshold and the magnitude determination process with the threshold described in the first embodiment become unnecessary. Note that the calculation method of the quantization parameter correction amount in S1409 is not limited to formula 14, and other calculation formulas using the ratio of the distance may be used.

[0120] S1410 determines whether the quantization correction parameters of all quantization target blocks have been determined. If the quantization correction parameters of all quantization target blocks have been determined, the process is completed. Otherwise, the process returns to S1403, and thereafter, the process for determining the quantization correction parameters of the next quantization target block is performed.

[0121] As described above, by using a calculation formula using the ratio of the distance of the quantization target block from the center coordinates, it is possible to set a finer quantization parameter correction amount according to the distance for the area inside the image circle. In addition, since there are few setting parameters related to processing and no need for size judgment, it is possible to suppress the circuit scale.

[0122] In the second embodiment, the corrected quantization parameter is also clipped in the same manner as in the first embodiment, since the corrected quantization parameter is within the permitted range.

[0123] It is also possible to apply the technology to standardized encoding methods such as H.264.

[0124] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0125] The disclosure of this specification includes the following image processing device, its control method, and program. (Item 1) An image processing device that acquires an image formed by a first and second optical system for capturing a VR image from an imaging sensor and encodes the image, quantization means for quantizing image data obtained by the image sensor in units of blocks of a predetermined size; a quantization control means for determining a quantization parameter used by the quantization means and setting the quantization parameter in the quantization means; encoding means for encoding the quantized data obtained by the quantization means; The quantization control means A calculation means for calculating the positions and sizes of two circular images formed by the first and second optical systems and included in the VR image; A determination unit is provided for determining a quantization parameter of the block to be quantized based on the position of the block to be quantized and the positions and sizes of the two circular images. 13. An image processing device comprising: (Item 2) the determining means determines a correction amount ΔQ for correcting a reference quantization parameter Q based on a position of the quantization target block and center coordinates and radii representing positions and sizes of the left and right circular images; The correction amount ΔQ has ΔQ1, ΔQ2, and ΔQ3 having a magnitude relationship of ΔQ2≦ΔQ1≦ΔQ3, The determining means is If the position of the block to be quantized is in the left or right circular image and the distance from the center coordinate is equal to or less than the threshold value, ΔQ1, If the position of the quantization target block is in the left or right circular image and the distance from the center coordinate is equal to or greater than a threshold value, ΔQ2, When the position of the quantization target block is within the light-shielded area, ΔQ3 is determined as the correction amount ΔQ. 2. The image processing device according to item 1, (Item 3) the determining means determines a correction amount ΔQ for correcting a reference quantization parameter Q based on a distance d between a position of the block to be quantized and central coordinates of each of the left and right circular images and a radius r; The determining means, when a preset lower limit value of the quantization parameter is qmin and a reference quantization parameter is qbase, When the distance d is greater than the distance r from the center coordinate of the circular image to the outer periphery, ΔQ3; If the distance d is equal to or less than the distance r from the center coordinate of the circular image to the outer periphery, d / r×(qmin-qbase) is calculated. is determined as the correction amount ΔQ. 2. The image processing device according to item 1, (Item 4) The VR image includes an optical black area (OB area), When the position of the block to be quantized is within the OB region, the determining means determines ΔQ0 having a relationship of ΔQ1≦ΔQ0≦ΔQ3 as the correction amount ΔQ. 4. The image processing device according to item 2 or 3. (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the circumferential image is the outer periphery of an image circle formed by the first and second optical systems. (Item 6) The first and second optical systems are detachable lenses, The method further includes calculating a center coordinate and a radius of the left and right circular images by using information specific to the lens. 6. The image processing device according to any one of items 1 to 5, (Item 7) The imaging sensor is a sensor having a Bayer array filter, A conversion means for converting the image of the Bayer array obtained by the image sensor into four planes, namely, a plane Y representing a luminance component and planes C0, C1, and C2 representing color difference components; a frequency transform means for performing a wavelet transform on each plane obtained by the transform means a preset number of times to generate a subband at a decomposition level n; The data consisting of coefficients at the same spatial position in the subbands nLL, nHL, nLH, nHH, . . . , 1HL, 1LH, 1HH of the decomposition level n obtained by the frequency conversion means is set as the block to be quantized. 4. The image processing device according to item 2 or 3. (Item 8) A brightness / darkness determination means for determining whether the block to be quantized is a bright area or a dark area by comparing an average value of coefficients in the same space as the block to be quantized in the subband 1LL of the decomposition level 1 obtained by the frequency conversion means with a preset brightness / darkness determination threshold value; a complexity determination means for calculating average values ​​ac_1HL, ac_1LH, ac_1HH of coefficients in the same space as the block to be quantized in each of the subbands 1HL, 1LH, 1HH of the decomposition level 1 obtained by the frequency conversion means, and for determining whether the block to be quantized is complex or flat by comparing a maximum value of the calculated average values ​​ac_1HL, ac_1LH, ac_1HH with a preset complexity determination threshold value; a second determination means for determining a second correction amount Δ2Q for further correcting the reference quantization parameter Q of the quantization target block according to the results of the determination by the light / dark determination means and the determination by the complexity determination means, The second correction amount Δ2Q has ΔQ4, ΔQ5, ΔQ6, and ΔQ7 having a magnitude relationship of ΔQ4≦ΔQ5≦ΔQ6≦Q7, The second determination means includes: ΔQ4 if the block to be quantized is in a dark area and is flat; ΔQ5 if the block to be quantized is in a bright area and is flat; ΔQ6 if the block to be quantized is in a dark area and is complex; If the block to be quantized is in a bright area and is complex, ΔQ7; is determined as the second correction amount Δ2Q. 8. The image processing device according to item 7, (Item 9) A method for controlling an image processing device that acquires an image formed by a first and second optical system for capturing a VR image from an imaging sensor and encodes the image, comprising: a quantization step of quantizing the image data obtained by the image sensor into blocks of a predetermined size; a quantization control step of determining a quantization parameter to be used in the quantization step and setting the quantization parameter in the quantization step; An encoding step of encoding the quantized data obtained in the quantization step, The quantization control step includes: A calculation step of calculating positions and sizes of two circular images formed by the first and second optical systems and included in the VR image; A determination step of determining a quantization parameter of the block to be quantized based on the position of the block to be quantized and the positions and sizes of the two circular images. 23. A method for controlling an image processing apparatus comprising the steps of: (Item 10) A program that, when read and executed by a computer, causes the computer to execute each step of the method according to any one of items 1 to 8.

[0126] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0127] 101: control unit, 102: imaging unit, 103: plane conversion unit, 104: frequency conversion unit, 105: quantization control unit, 106: quantization unit, 107: entropy coding unit, 108: recording medium, 109: operation unit

Claims

1. An image processing device that acquires an image formed by a first and a second optical system for capturing a VR image from an imaging sensor and encodes the image, quantization means for quantizing image data obtained by the image sensor in units of blocks of a predetermined size; a quantization control means for determining a quantization parameter used by the quantization means and setting the quantization parameter in the quantization means; encoding means for encoding the quantized data obtained by the quantization means; The quantization control means a calculation means for calculating positions and sizes of two circular images formed by the first and second optical systems and included in the VR image; A determination unit is provided for determining a quantization parameter of the block to be quantized based on the position of the block to be quantized and the positions and sizes of the two circular images.

13. An image processing device comprising:

2. the determining means determines a correction amount ΔQ for correcting a reference quantization parameter Q based on a position of the block to be quantized and center coordinates and radii representing positions and sizes of each of the two circular images; The correction amount ΔQ has ΔQ1, ΔQ2, and ΔQ3 having a magnitude relationship of ΔQ2≦ΔQ1≦ΔQ3, The determining means is When the position of the block to be quantized is in the first or second circular image and the distance from the central coordinates is equal to or less than a threshold value, ΔQ1, When the position of the block to be quantized is in the first or second circular image and the distance from the central coordinates is equal to or greater than a threshold value, ΔQ2, When the position of the quantization target block is within the light-shielded area, ΔQ3 is determined as the correction amount ΔQ.

2. The image processing device according to claim 1,

3. the determining means determines a correction amount ΔQ for correcting a reference quantization parameter Q based on a distance d between a position of the quantization target block and central coordinates of each of the two circular images and a radius r; The determining means determines, when a preset lower limit value of a quantization parameter is qmin and a base quantization parameter is qbase, When the distance d is greater than the distance r from the center coordinate of the circular image to the outer periphery, ΔQ3; When the distance d is equal to or less than the distance r from the center coordinates of the circular image to the outer periphery, d / r×(qmin−qbase) is calculated as follows: is determined as the correction amount ΔQ.

2. The image processing device according to claim 1,

4. The VR image includes an optical black area (OB area), When the position of the block to be quantized is within the OB region, the determining means determines ΔQ0 having a relationship of ΔQ1≦ΔQ0≦ΔQ3 as the correction amount ΔQ.

4. The image processing device according to claim 2, wherein the first and second inputs are input to the image processing apparatus.

5. 2. The image processing apparatus according to claim 1, wherein the circumferential image is an outer periphery of an image circle formed by the first and second optical systems.

6. The first and second optical systems are detachable lenses, The method further includes calculating a center coordinate and a radius of the two circular images by using information specific to the lens.

2. The image processing device according to claim 1,

7. The imaging sensor is a sensor having a Bayer array filter, A conversion means for converting the image of the Bayer array obtained by the image sensor into four planes, namely, a plane Y representing a luminance component and planes C0, C1, and C2 representing color difference components; a frequency transform means for performing a wavelet transform on each plane obtained by the transform means a preset number of times to generate a subband at a decomposition level n; The data consisting of coefficients at the same spatial position in the subbands nLL, nHL, nLH, nHH, . . . , 1HL, 1LH, 1HH of the decomposition level n obtained by the frequency conversion means is set as the block to be quantized.

3. The image processing device according to claim 2.

8. a brightness / darkness determination means for determining whether the block to be quantized is a bright area or a dark area by comparing an average value of coefficients in the same space as the block to be quantized in the subband 1LL of the decomposition level 1 obtained by the frequency conversion means with a preset brightness / darkness determination threshold value; a complexity determination means for calculating average values ​​ac_1HL, ac_1LH, ac_1HH of coefficients in the same space as the block to be quantized in each of the subbands 1HL, 1LH, 1HH of the decomposition level 1 obtained by the frequency conversion means, and for determining whether the block to be quantized is complex or flat by comparing a maximum value of the calculated average values ​​ac_1HL, ac_1LH, ac_1HH with a preset complexity determination threshold value; a second determination means for determining a second correction amount Δ2Q for further correcting the reference quantization parameter Q of the block to be quantized according to the results of the determinations of the light and dark determination means and the complexity determination means, The second correction amount Δ2Q has ΔQ4, ΔQ5, ΔQ6, and ΔQ7 having a magnitude relationship of ΔQ4≦ΔQ5≦ΔQ6≦ΔQ7, The second determination means includes: ΔQ4 if the block to be quantized is in a dark area and is flat; ΔQ5 if the block to be quantized is in a bright area and is flat; ΔQ6 if the block to be quantized is in a dark area and is complex; If the block to be quantized is in a bright area and is complex, ΔQ7; is determined as the second correction amount Δ2Q.

8. The image processing device according to claim 7,

9. A method for controlling an image processing device that acquires, from an imaging sensor, an image formed by first and second optical systems for capturing a virtual reality (VR) image and encodes the image, comprising the steps of: a quantization step of quantizing the image data obtained by the image sensor into blocks of a predetermined size; a quantization control step of determining a quantization parameter to be used in the quantization step and setting the quantization parameter in the quantization step; An encoding step of encoding the quantized data obtained in the quantization step, The quantization control step includes: a calculation step of calculating positions and sizes of two circular images formed by the first and second optical systems and included in the VR image; A determination step of determining a quantization parameter of the block to be quantized based on the position of the block to be quantized and the positions and sizes of the two circular images.

23. A method for controlling an image processing apparatus comprising the steps of:

10. A program for causing a computer to execute each step of the method according to claim 9 when the program is read and executed by the computer.

Images