Image processing device, image processing method, and program

Abstract

To enhance the user's sense of immersion when viewing an image. [Solution] The system includes an image acquisition means (101) that acquires a wide-angle image whose field of view exceeds a predetermined value, a restriction means (101) that restricts the image region used for projection transformation from the wide-angle image to a restricted image region, and a transformation means (101) that projects the coordinate system of the image in the restricted image region of the wide-angle image to the coordinate system of the output image.

Description

Technical Field

[0001] The present disclosure relates to image conversion when obtaining a wide-angle image.

Background Art

[0002] Conventionally, as lenses capable of photographing a subject at a wide angle, for example, wide-angle lenses and fisheye lenses are known. In particular, since a fisheye lens has an angle of view of 180° or more, a wide-angle image can be obtained with a single lens. In recent years, with the evolution of the technology of head-mounted displays (HMDs), ultra-high-angle-of-view images with an angle of view exceeding 180° have been input into HMDs, and images cropped at the display angle of view in the front direction of the HMD can be displayed in real time by the HMD. For ultra-high-angle-of-view images input into an HMD, images obtained by converting an image taken with a fisheye lens into a 180° orthographic cylindrical image are often used. However, since a fisheye lens has a wide angle of view, a missing area where a part of the image is missing is likely to occur. Therefore, Patent Document 1 discloses a technique for compensating for a missing area by copying a part of the image in the corresponding area of the other circumferential fisheye image when a missing area exists in one of the left and right circumferential fisheye images.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, there is a possibility that false colors are generated at the corners of the image as image degradation caused by the imaging optical system. Therefore, the corners of the image displayed on the HMD may deteriorate. For this reason, the sense of immersion of the user viewing the image may be impaired.

Means for Solving the Problems

[0005] An image processing apparatus according to one aspect of the present disclosure is characterized by comprising: an image acquisition means for acquiring a wide-angle image whose field of view exceeds a predetermined value; a restriction means for restricting the image region used for projection transformation from the wide-angle image to a restricted image region; and a transformation means for projecting the coordinate system of the image in the restricted image region of the wide-angle image to the coordinate system of the image to be output. [Effects of the Invention]

[0006] This disclosure makes it possible to enhance the sense of immersion for users who view images. [Brief explanation of the drawing]

[0007] [Figure 1] This is a diagram illustrating the system configuration of an image processing device. [Figure 2] This is a block diagram illustrating the conversion from a fisheye image to an equirectangular image with a 180-degree field of view. [Figure 3] This is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180° field of view according to the first embodiment. [Figure 4] This is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180° field of view according to the second embodiment. [Figure 5] This is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180° field of view according to the third embodiment. [Figure 6] This is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180° field of view according to the fourth embodiment. [Figure 7] This is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180° field of view according to the fifth embodiment. [Figure 8] This figure shows examples of fisheye and equirectangular images. [Figure 9] This diagram illustrates an example of a user interface for specifying equirectangular transformation parameters. [Modes for carrying out the invention]

[0008] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. Note that the following embodiments are not limiting to the scope of this disclosure, and not all combinations of features described in the following embodiments are essential to the solutions of this disclosure. The same reference numerals are used for identical components.

[0009] (overview) In recent years, with the advancement of HMD (Head-Mounted Display) technology, ultra-high-angle images with a field of view exceeding 180° are now input to HMDs, and images cropped to the display field of view in the direction in front of the HMD can be displayed in real time by the HMD. This real-time display allows users to easily view so-called VR (Virtual Reality) images through the HMD. Furthermore, the ultra-high-angle images input to the HMD often use 180° equirectangular images obtained by projecting an image captured with a fisheye lens. Alternatively, the ultra-high-angle images input to the HMD may use a 360° equirectangular image generated by combining two images captured with two fisheye lenses positioned 180° apart. Alternatively, the ultra-high-angle images input to the HMD may use a stereo image generated by capturing images with two fisheye lenses positioned side by side facing the same direction.

[0010] Generally, HMDs can display a field of view of 90° to 120° at a time. Users can also view images with an even wider field of view by turning their heads from side to side. Also, because fisheye lenses have a wide field of view, vignetting is likely to occur. Vignetting refers to the black areas that appear at the corners of the screen. These black areas are caused by the lens hood or filter appearing as black in the image. In other words, because of the wide field of view, parts of the lens or camera body may appear in the image, resulting in vignetting areas (also called clipping areas) where a part of the image is missing. It is possible to eliminate these clipping areas by painting them with a predetermined color such as black.

[0011] Furthermore, there are two types of fisheye lenses: circular fisheye lenses and diagonal fisheye lenses. A circular fisheye lens is a lens that crops the image into a circular shape. Images captured using a circular fisheye lens have strong distortion at the four corners of the image, resulting in an overall distorted spherical image. A diagonal fisheye lens is a lens that has the widest field of view along the diagonal. Images captured using a diagonal fisheye lens are distorted spherically from the center outwards. In addition, there are two projection methods for fisheye lenses, such as equisolid angle projection and equidistance projection. Equisolid angle projection is suitable for capturing natural expanses, such as panoramic and landscape photographs. Equidistance projection is suitable for applications where precise angle measurement is required, such as astronomy and dome theaters. Other projection methods for fisheye lenses include orthogonal projection and stereoscopic projection. Orthogonal projection is suitable for eliminating perspective and displaying objects at an equal scale without distortion of their shape or dimensions. Stereoscopic projection offers a good balance between angle and area, making it suitable for panoramic photography and video production. Note that while a fisheye lens has a field of view of approximately 180° or more, a wide-angle lens has a field of view of approximately 60° or more.

[0012] When using an imaging optical system capable of acquiring wide-angle images exceeding a predetermined field of view, as described above, false colors may appear in the corners of the image, resulting in image degradation. Such false colors can impair user immersion. Additionally, missing areas may appear in the corners of the image. Even if these missing areas are filled with a predetermined color such as black, the user viewing the image will still see black areas at the edges of their field of view. In particular, when generating an equirectangular image where the image height of a fisheye lens is wider than the effective image height, and then filling the image area outside the effective image height with black in the equirectangular image, the following could occur: That is, when the user changes their viewing direction while viewing with an HMD, a black area suddenly appears at the edge of their field of view. This black area could impair user immersion.

[0013] Therefore, in the present disclosure, the image area used for projective transformation among the wide-angle images is limited to a restricted image area. Further, among the wide-angle images, the coordinate system of the image in the restricted image area is projected and transformed into the coordinate system of the output target image. On the other hand, among the wide-angle images, the coordinate system of the image in the out-of-restricted image area outside the restricted image area is not projected and transformed into the coordinate system of the output target image. According to such processing, since only the coordinate system of the image in the restricted image area is projected and transformed into the coordinate system of the output target image, false colors and missing areas do not occur. Therefore, it is possible to enhance the sense of immersion of the user who views the image. Hereinafter, the details of the present disclosure will be described.

[0014] (First Embodiment) An example of converting a fisheye image to an equirectangular image will be described below. The system configuration will be described below with reference to Figure 1. Figure 1 is a diagram illustrating the system configuration of an image processing device. The image processing system comprises a bus 100, a CPU 101, a non-volatile memory 102, a memory 103, a UI (user interface) device connection unit 104, a GPU 105, a general-purpose IF (interface) 106, and an NW (network) / IF 107. The CPU 101, non-volatile memory 102, memory 103, UI device connection unit 104, GPU 105, general-purpose IF 106, and NW / IF 107 are each connected via the bus 100. The bus 100 is a communication path for transmitting various data or control information. The CPU 101 controls the entire image processing system by appropriately executing various processes defined by software, applications, programs, etc. The CPU 101 can also perform various image processing by appropriately executing various processes defined by software, applications, programs, etc. Non-volatile memory 102 is a secondary storage area that stores software, applications, and programs. It also stores various data necessary for executing various processes defined by the software, applications, and programs. Memory 103 is a temporary storage area that constructs the execution environment for various processes defined by the software, applications, and programs. Specifically, when the image processing device starts up, memory 103 loads the software, applications, programs, and various data stored in non-volatile memory 102. Memory 103 also temporarily stores images acquired or generated by various processes. In other words, memory 103 functions as the work area of ​​the CPU 101. The UI device connection unit 104 functions as an interface for input / output devices such as a keyboard, mouse, and display. The GPU 105 performs image processing. The GPU 105 outputs the results of the image processing to the outside. The output destination of the GPU 105 is at least one of the HMD and the display. The general-purpose IF 106 functions as an interface for a capture device.For example, consider a use case where the capture device is connected between the imaging device and the general-purpose IF106. In this scenario, the camera image captured by the imaging device is supplied to the general-purpose IF106 via the capture device. This allows the image processing device to acquire the camera image captured by the imaging device. Alternatively, consider a use case where the capture device is connected between the video camera and the general-purpose IF106. In this scenario, the video signal captured by the video camera is supplied to the general-purpose IF106 via the capture device. This allows the image processing device to acquire the video signal captured by the video camera. The NW / IF107 is an interface that can input and output various data via a network. The communication medium through which the NW / IF107 connects to the network may be a wired medium or a wireless medium. Thus, in this embodiment, the image processing device may receive an image via any of the non-volatile memory 102, the general-purpose IF106, or the NW / IF107. In this embodiment, unless otherwise specified, it is assumed that the image stored in memory 103 consists of floating-point RGB values. Furthermore, when images stored in memory 103 are input / output as files or output externally, they are implicitly converted to a data format suitable for that purpose.

[0015] Furthermore, in this embodiment, we assume a use case where the image sensor has, for example, 4300 pixels vertically and 2400 pixels horizontally, and the captured image also has 4300 pixels vertically and 2400 pixels horizontally. In this use case, we assume that the fisheye lens is an equidistant projection circular fisheye lens, and that the effective image height of the lens that can guarantee image quality is 72°. Therefore, an effective image height of 72° is set within the range of the image circle. That is, an effective image height of 72° is set within the image circle in which optical performance can be ensured. Here, the shift of the center of the image circle of the fisheye lens will be explained using Figure 8. Figure 8 is a diagram showing an example of a fisheye image and an equirectangular image. Figure 8(a) is a diagram showing an example of a fisheye image captured using a fisheye lens. Figure 8(b) is a diagram showing an example of an equirectangular image obtained by projective transformation of the fisheye image in Figure 8(a). Figure 8(c) is a diagram showing an example of an equirectangular image obtained by restricting the image area subject to projective transformation according to predetermined conditions, compared to the equirectangular image in Figure 8(b). Figure 8(d) shows an example of an equirectangular image in which the image region subject to projection transformation is restricted by other conditions, compared to the equirectangular image in Figure 8(c).

[0016] Due to installation errors, manufacturing errors, etc. of the fish-eye lens and image sensor, the center of the image circle of the fish-eye lens may be misaligned. In this embodiment, it is assumed that the misalignment occurs within the range of ±1°. When the center of the image circle of the fish-eye lens is misaligned, an image in the image area exceeding the effective image height of the fish-eye lens will be captured by the image sensor. However, false colors are likely to occur in the image area exceeding the effective image height of the fish-eye lens due to aberrations and the like. Note that the misalignment of the center of the image circle of the fish-eye lens can be corrected by calculating calibration parameters through calibration and correcting the position of the fish-eye image. The calibration parameters in this embodiment are assumed to be external parameters, internal parameters, and distortion correction parameters of the imaging device. In this embodiment, an example of generating an orthographic cylindrical image from a fish-eye image by projective transformation is shown. During this projective transformation, the misalignment of the center of the image circle of the fish-eye lens can be corrected by generating a fish-eye image in which the calibration parameters are reflected in advance. Also, when an orthographic cylindrical image is generated from a fish-eye image by projective transformation, the misalignment of the center of the image circle of the fish-eye lens can be corrected by reflecting the calibration parameters. Hereinafter, the distance from the center of the image circle of the calibrated fish-eye lens is referred to as the radial distance, and the angle of the radial distance rotating around the optical axis of the fish-eye lens is referred to as the deflection angle.

[0017] Hereinafter, an example of generating and outputting an orthographic cylindrical image with a 180-degree angular field from a fish-eye image will be described. FIG. 2 is a block diagram for explaining the conversion from a fish-eye image to an orthographic cylindrical image with a 180-degree angular field. Each block in FIG. 2 may be either a circuit or a conceptual processing block processed by the CPU 101. FIG. 2(a) is a diagram for explaining the processing of projective transformation into an orthographic cylindrical image in the first embodiment. FIG. 2(b) is a diagram for explaining the processing of projective transformation into an orthographic cylindrical image in the second embodiment. FIG. 2(c) is a diagram for explaining the processing of projective transformation into an orthographic cylindrical image in the third embodiment. FIG. 2(b) and FIG. 2(c) are used for the explanations of the corresponding embodiments respectively.

[0018] Figure 3 is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180-degree field of view according to the first embodiment. The details of the processing performed by the image processing device will be explained below using Figures 2(a) and 3. The processing shown in Figure 3 is realized by the CPU 101 executing a program loaded into memory 103. The processing shown in Figure 3 is executed at the timing when the projection transformation processing is initiated. Note that some or all of the functions of the steps in Figure 3 may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates a step in the flowchart. Furthermore, the processing shown in Figure 3 can also be implemented as a cloud computing configuration where a single function is shared and processed collaboratively by multiple resources via the internet, as long as it realizes the functions of each block in Figure 2(a).

[0019] Hereafter, the processing of each block in Figure 2(a) will be explained in relation to the processing of each step in Figure 3. Furthermore, since the details of the processing are the same as the processing of each step in Figure 3, Figure 3 will be used to explain them in detail later. Each arrow in Figure 2 indicates the direction in which various data are transmitted between blocks.

[0020] Figure 2(a) shows the effective image height acquisition unit 200, the fisheye image acquisition unit 201, the coordinate acquisition unit 202, the first coordinate transformation unit 203, the limiting unit 204, the second coordinate transformation unit 205, the pixel value calculation unit 206, and the image output unit 207. The processing performed by the effective image height acquisition unit 200 corresponds to the processing in S301 of Figure 3. The processing performed by the fisheye image acquisition unit 201 corresponds to the processing in S302 of Figure 3. The processing performed by the coordinate acquisition unit 202 corresponds to the processing in S303 to S312 of Figure 3.

[0021] The coordinate acquisition unit 202 performs a process to acquire XY coordinates (x,y) for scanning the output image. In the processes S303 to SS312 in Figure 3, the coordinates (x,y) are calculated by scanning the pixels of interest in the output image. In this embodiment, the output image is an equirectangular image. Specifically, the values ​​specified by the XY coordinates (x,y) are calculated so that the topmost line of the equirectangular image is scanned to the left or right, and after scanning one line is completed, the line one level below is scanned again to the left or right. The first coordinate transformation unit 203 performs a process to transform the XY coordinates (x,y) into polar coordinates (φ,θ) with radius φ and angle θ (S305). That is, a projection transformation is performed from a Cartesian coordinate system to a polar coordinate system. The limiting unit 204 performs a process to limit the radius φ to within φ' (S306). The second coordinate transformation unit 205 performs a process to transform the restricted radial vector φ and angular deviation θ into fisheye XY coordinates (x',y') (S307). That is, a projection transformation is performed from the polar coordinate system to the Cartesian coordinate system. The pixel value calculation unit 206 performs a process to calculate the pixel values ​​of the output image specified by the XY coordinates (x,y) by referring to the pixel values ​​of the fisheye image specified by the fisheye XY coordinates (x',y'), and then performs a process to store these values ​​(S308). The image output unit 207 outputs the stored output image (S313).

[0022] The processing of each step in Figure 3 will be explained below. Unless otherwise specified, each step will be executed in the order of the arrows in Figure 2(a), however, the order may be changed if there is no dependence on the data input / output relationship of each step. It is assumed that the CPU 101 performs the execution of each step. The output image in the following explanation is an equirectangular image. In addition, a double loop executed in the processing of each step from S303 to S312 scans the pixels in the coordinate system of the output target and calculates the pixel values ​​of the output image. The details of the processing of each step will be explained below.

[0023] In S301, the CPU 101 obtains the effective image height threshold φ'. In this embodiment, it is assumed that the effective image height threshold φ' is obtained as the maximum image height at which false color does not occur. The maximum image height at which false color does not occur can be obtained from the design value, measured in advance, or detected from the input image. Specifically, in the optical design of a fisheye lens, the image quality guarantee range is predetermined. However, in reality, images exceeding the image height may be projected onto the image plane. For example, the optical axis center of the fisheye lens may be shifted during image processing. In this case, the image circle also shifts from its predetermined position. Therefore, images of the image region exceeding the effective image height of the fisheye lens may be captured by the image sensor. However, false color is likely to occur in the image region exceeding the effective image height of the fisheye lens. Factors causing false color include, for example, lateral chromatic aberration and axial chromatic aberration. Generally, the larger the image height, the larger the aberration. The larger the aberration, the more strongly color shift can be observed at the edges of the image. Chromatic aberration refers to the blurring of red (R) or blue (B) colors. For example, if pixels of a certain color extend beyond a predetermined number of pixels into a region of pixels of another color, it is assumed that false color occurs. According to this assumption, the maximum image height in which false color does not occur is the image height within which pixels of a certain color do not extend beyond a predetermined number of pixels. Also, for example, if only B exceeds 30% of RGB, this is also considered false color. In other words, a range is predetermined, and the maximum value that does not extend beyond that range becomes the effective image height. Note that the image height is determined by the radial distance of the radial vector representing the elements of the polar coordinate system when the position of each pixel on the image plane is expressed in polar coordinates, starting from the imaging center, which is the intersection point of the optical axis center of the fisheye lens and the image plane of the imaging optical system including the fisheye lens.

[0024] Alternatively, in this embodiment, the effective image height may be defined as the image height at which the shift of the wavelength representing green is within 1 pixel or less for specific wavelengths of light representing red (R) or blue (B). The wavelengths representing red (R), blue (B), and green (G) are, for example, 700.0 nm, 546 nm, and 436 nm, respectively, but these wavelengths may be adjusted as appropriate depending on the characteristics of the color filter of the image sensor. Furthermore, these wavelengths can be simulated using the lens shape, lens material, or a combination of lens shape and lens material. In this embodiment, the allowable shift value is set to, for example, 1 pixel, but is not limited to this. The higher the resolution of the image sensor, the higher the production cost of the lens, and the more difficult it is to produce false colors with a fisheye lens, the higher the production cost, so the allowable shift value may be set taking these costs into consideration.

[0025] Alternatively, the maximum image height at which false color generation is acceptable can also be determined by measurement. For example, the color shift, which can be considered the amount of false color generation, may be measured. Specifically, the color shift may be measured using a color filter having spectral characteristics that transmit only wavelengths of a predetermined width centered on the peak values ​​of each wavelength region (R, B, and G).

[0026] Alternatively, when determining the effective image height from design values ​​or measured values, the fact that aberration correction has been performed may also be taken into consideration. Even with aberration correction, color shift cannot be completely suppressed, so the effective image height may be set to the image height at which the remaining correction is acceptable.

[0027] In S302, CPU101 acquires a fisheye image. In S303, CPU101 initializes the variable y to 0. Variable y is the vertical coordinate (Y coordinate) of the output image. In S304, CPU101 initializes the variable x to 0. Variable x is the horizontal coordinate (X coordinate) of the output image. In S305, CPU101 converts the XY coordinates (x,y) of the output image to polar coordinates (φ,θ). That is, the coordinate system is projected from a Cartesian coordinate system to a polar coordinate system. The projective transformation is assumed to be performed by the following equation.

[0028]

number

[0029] w out ,h out The values ​​shown are the width and height of the output image, which is an equirectangular image. In equation (1), the center of the equirectangular image corresponds to φ=0, θ=0. The units of φ and θ are radians. Also, φ represents the radial movement and θ represents the angular displacement.

[0030] In S306, the CPU 101 limits the radial diameter φ to within the effective image height threshold φ'. The effective image height threshold φ' is set to a value less than or equal to the effective image height at which image quality can be guaranteed as a fisheye lens. In this embodiment, the effective image height threshold φ' is described as 70°. The limited value in this embodiment can be calculated under the following conditions.

[0031]

number

[0032] Hereinafter, φ is φ new This will be explained as a use case that has been replaced by [the other option].

[0033] In S307, CPU101 transforms from polar coordinates (φ,θ) to fisheye XY coordinates (x',y'). That is, the coordinate system is projectively transformed from a polar coordinate system to a Cartesian coordinate system. This is the inverse transformation of equation (1).

[0034]

number

[0035] k is the normalized value of the radial diameter φ of the image circle inscribed in the image sensor, and in this embodiment, the image circle with a 72° angle is inscribed in the image sensor. Here, k = 90 / 72.

[0036] In S308, the CPU 101 references the pixel values ​​of the fisheye image, which are specified by the fisheye XY coordinates (x',y'), calculates the pixel values ​​of the output image, which are specified by the XY coordinates (x,y), and stores the calculated pixel values. The pixel values ​​of the fisheye image, which are specified by the fisheye XY coordinates (x',y'), are real numbers. Therefore, the pixel values ​​of the four neighboring points, which are expressed in integer coordinates, are interpolated using the bilinear method (hereinafter referred to as bilinear interpolation). The interpolated pixel values ​​are successively stored in memory 103. When scanning all pixels is complete, the pixel values ​​of the output image are transferred from memory 103 to non-volatile memory 102, thereby becoming stored. Note that the interpolation process is not limited to bilinear interpolation. For example, it may also be interpolation using the bicubic method or the Lanczos method.

[0037] In S309, CPU101 determines whether x < output image width. If true, it proceeds to S311; otherwise, it proceeds to S310.

[0038] In S310, the CPU 101 determines whether y < output image height. If true, it proceeds to S312; otherwise, it proceeds to S313.

[0039] In S311, the CPU 101 increments x. Then, the process moves to S305. In S312, the CPU 101 increments y. Then, the process moves to S305. In S313, the CPU 101 outputs the stored output image. In this embodiment, the image may be output to the non-volatile memory 102 in file units.

[0040] By the way, if you leave the black areas outside the image circle of a circular fisheye image as they are, or fill the uncaptured areas with a predetermined color, such as black, to create an equirectangular image with a 180° field of view, the following occurs. That is, in the generated equirectangular image, as shown in Figure 8(b), the areas with high image heights will be black. Therefore, when viewing with an HMD with a display field of view of 100°, if the user turns their head 25° to the side, for example, a black area will appear at the edge of the field of view, impairing the sense of immersion. However, according to this embodiment, the radius movement is limited to within the threshold φ' of the effective image height (70°). Through this process, pixel values ​​with an image height of 70° or more are copied from pixel values ​​with an image height limited to less than 70° to generate an equirectangular image. The image thus limited will look like Figure 8(c).

[0041] (Example output image) (Use case i: End copy) Figure 8(c) shows an image generated by copying the edge of the image region with an effective height threshold φ' into an unrestricted image region, which is an image region specified by equation (2) where the image height is equal to or greater than the effective image height. That is, the CPU 101 generates an output image by copying the edge of the image region with an effective height threshold φ' into the unrestricted image region, and outputs and displays the generated output image. The output image at this time includes a partial image composed of the restricted image region and a partial image composed of the unrestricted image region surrounding the restricted image region. In the restricted image region, the image region used for projection transformation from the wide-angle image is restricted. Such a restricted image region is an image region specified by an image height smaller than the effective image height. The partial image composed of the unrestricted image region is a partial image generated by copying the image of the edge of the image region with an effective height threshold φ'.

[0042] Therefore, even in image regions with high image height, the image is not filled with black. As a result, even if the user turns their head 25° to the side, black will not appear at the edge of the user's field of view. Also, generally, the resolution at the edges of the human field of view tends to be low. Therefore, even if the correct image is not displayed at the edges, it does not significantly impair the user's sense of immersion. Thus, this embodiment is quite effective. Furthermore, when the image region outside the image circle is filled with black, jagged edges tend to occur at its boundary. However, in this embodiment, since there is no boundary with black in the first place, jagged edges are less likely to occur. Such jagged edges tend to occur especially when the interpolation accuracy during equirectangular transformation or the accuracy of coordinate transformation is reduced in order to speed up processing, but in this embodiment, the image region outside the image circle is not filled with black in the first place.

[0043] In other words, in this embodiment, a wide-angle image with a field of view exceeding a predetermined value is acquired. The image region used for projection transformation within the wide-angle image is restricted to a limited image region. The coordinate system of the image within the limited image region within the wide-angle image is projected to the coordinate system of the output image. With this process, the coordinate system of the image within the limited image region is projected to the coordinate system of the output image. Therefore, false colors and missing areas do not occur. Thus, it is possible to enhance the immersion of the user viewing the image. Furthermore, the coordinate system of the image outside the limited image region is not projected to the coordinate system of the output image. With this process, only the coordinate system of the image within the limited image region is projected to the coordinate system of the output image, so false colors and missing areas do not occur. In addition, the effective image height, which is the upper limit of the image height indicating the image height from the optical axis center of the imaging optical system used to capture the wide-angle image, is acquired as a threshold that can distinguish between the limited image region and the unlimited image region. With this process, it is possible to distinguish between the limited image region and the unlimited image region using the image height as a parameter. Furthermore, within the image region containing a wide-angle image, the image region identified by an image height smaller than the effective image height is defined as the restricted image region, while the image region identified by an image height equal to or greater than the effective image height is defined as the unrestricted image region. The unrestricted image region may include image regions where false color occurs.

[0044] Furthermore, in this embodiment, the effective image height threshold φ' is smaller than the effective image height of a fisheye lens that can guarantee image quality. Therefore, no false colors are introduced into the equirectangular image.

[0045] Furthermore, in S306 of this embodiment, the limiting method is set to the above formula (1), thereby limiting the moving diameter φ to within the threshold φ' of the effective image height, but the method is not limited to this. For example, it may be limited under the following conditions.

[0046]

number

[0047] (Example output image) (Use case ii: Fold-over copy at the edges) Figure 8(d) shows an image generated by copying the folded edge of the image region of the effective height threshold φ' into the unrestricted image region, as specified by equation (4). That is, the CPU 101 generates an output image by copying the folded edge of the image region of the effective height threshold φ' into the unrestricted image region, and outputs and displays the generated output image. The output image at this time includes a partial image composed of the restricted image region and a partial image composed of the unrestricted image region surrounding the restricted image region. In the restricted image region, the image region used for projection transformation from the wide-angle image is restricted. Such a restricted image region is an image region specified by an image height smaller than the effective height. The partial image composed of the unrestricted image region is a partial image generated by copying the folded image of the edge of the image region of the effective height threshold φ'. With an output image that includes such a partial image composed of the restricted image region, an equirectangular image in which the folded image spreads radially can be generated, making it possible to further enhance the user's sense of immersion.

[0048] In this embodiment, the image is described as a monocular fisheye image, but it is not limited to that. It may also be a stereo fisheye image or a 360° equirectangular image converted from two or more fisheye images.

[0049] Furthermore, although this embodiment describes the case where the effective image height of the fisheye lens is 72°, it is not limited to this. For example, it is also effective even if the effective image height of the fisheye lens is 90°. In this embodiment, if the amount of correction due to calibration is large, or if the amount of correction is large when correcting camera shake using the image, the following may occur if the radial movement is not limited. That is, it may become necessary to computationally refer to an area where the image is not captured, called vignetting. Such vignetted areas are often made black, but by using the process described in this embodiment, it is not necessary to make the vignetted areas black.

[0050] Furthermore, although this embodiment has been described as performing calibration, calibration is not necessarily required. However, in that case, it is not possible to set a single effective image height threshold φ'. Therefore, instead of processing in S306, the effective image height threshold φ' may be set for each deflection angle θ, as shown in the following equation.

[0051]

number

[0052] The function in equation (5) can be created in advance based on measured values ​​and lens design values. In this embodiment, the conversion from a fisheye image to an equirectangular image is described as being restricted by the radial diameter φ, but the output image may also be a fisheye image. In this case, if the size of the image circle is considered in terms of angles, the image circle of the fisheye image as the output image will be larger than the image circle of the fisheye image as the input image. The process involves copying pixels in the image region outside the effective image height threshold φ' of the fisheye image from pixels within the effective image height threshold φ', which restricts the radial diameter φ of the source pixels. Therefore, this embodiment is executable. Although the pixels in this image are spread radially, the projection method maintains equidistant projection, and even if it is projected and converted to an equirectangular image, an image similar to the equirectangular image converted by the process in Figure 3 can be produced.

[0053] Furthermore, although this embodiment describes the coordinate transformation using trigonometric functions as being calculated for each pixel of the equirectangular image, it is not limited to this. The calculation may be performed sparsely, and the values ​​may be linearly interpolated. That is, to speed up processing, the overall calculation may be sparse, and then the details may be calculated densely. In this case, the accuracy of the coordinate transformation will decrease, but as mentioned above, jagged edges will not occur at the boundaries, thus making this embodiment more effective.

[0054] (Second embodiment) Equirectangular transformation with shading will be explained using Figures 2(b) and 4. Figure 4 is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180° field of view according to the second embodiment. The details of the processing performed by the image processing device will be explained below using Figures 2(b) and 4. The processing shown in Figure 4 is realized by the CPU 101 executing the program loaded into memory 103. The processing shown in Figure 4 is executed at the timing when the projection transformation processing is started. Note that some or all of the functions of the steps in Figure 4 may be realized by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in the flowchart. Furthermore, the processing shown in Figure 4 can also be realized as a cloud computing configuration in which one function is shared and processed jointly by multiple resources via the internet, as long as it realizes the functions of each block in Figure 2(b). Furthermore, unless otherwise specified or changed, the description will be the same as the description of Figures 2(a) and 3 of the first embodiment.

[0055] Hereafter, the processing of each block in Figure 2(a) will be explained in relation to the processing of each step in Figure 4. Furthermore, since the details of the processing are the same as the processing of each step in Figure 4, Figure 4 will be used to describe them later.

[0056] In Figure 2(b), compared to Figure 2(a), a shading value calculation unit 208 and a pixel value calculation unit 2062 are added, and the pixel value calculation unit 206 is replaced by a pixel value calculation unit 2061. The processing performed by the shading value calculation unit 208 corresponds to the processing in S4055. The processing performed by the pixel value calculation unit 2062 corresponds to the processing in S4070. The processing performed by the pixel value calculation unit 2062 corresponds to the processing in S4075.

[0057] Hereafter, we will mainly explain the differences between the processing steps in Figure 4 and those in Figure 3. In S401, the CPU 101 calculates a shading value α, which decreases as the diameter increases. In this embodiment, the shading value α is calculated using the following formula.

[0058]

number

[0059] φ1 represents the shading start diameter where the pixel value begins to darken due to shading, and φ2 represents the shading end diameter where the pixel becomes black due to shading. At least φ2 is greater than the effective image height threshold φ'. In this embodiment, φ1 is set to the same value as the effective image height. It is preferable to set it to the same value as the effective image height in order to minimize degradation of areas where false colors do not occur. Processing is possible for any value of φ1 as long as the condition 0 < φ1 < φ2 is satisfied.

[0060] In S402, the CPU 101 references the pixel values ​​of the fisheye image, which are identified by the fisheye XY coordinates (x',y'), and calculates the pixel value v. In S403, the CPU 101 multiplies the pixel value v by the α value to calculate the pixel value of the output image, which is identified by the XY coordinates (x,y), and stores the calculated pixel value.

[0061] According to this embodiment, shading is applied to the image region at the boundary around the effective image height. In the first embodiment, the equirectangular image generated has an image that spreads radially. Therefore, when a user views the equirectangular image generated in the first embodiment on a flat-panel display, the user may experience discomfort. In this embodiment, this discomfort can be reduced by gradually darkening the image towards the outside. Also, when the user is viewing with an HMD, compared to the case where the boundary image region is suddenly made black, the discomfort of the user seeing black outside the user's field of view can be reduced. Furthermore, compared to the case where shading is only applied to a specific image region of the image, in this embodiment, shading is applied to the image region copied outside the effective image height threshold, so that the image within the effective image height is not degraded by shading and can be viewed effectively.

[0062] (Example output image) (Use case iii: Edge copy shading) In other words, as specified by equation (6), within the unrestricted image region, an image is generated between the shading start diameter φ1 and the shading end diameter φ2 in which the pixel value becomes darker as the diameter φ increases. That is, the CPU 101 generates an output image by copying the image in which the pixel value becomes darker as the diameter φ increases between the shading start diameter φ1 and the shading end diameter φ2 within the unrestricted image region, and outputs and displays the generated output image. The output image at this time includes a partial image composed of the restricted image region and a partial image composed of the unrestricted image region surrounding the restricted image region. In the restricted image region, the image region used for projection transformation from the wide-angle image is restricted. Such a restricted image region is an image region specified by an image height smaller than the effective image height. The partial image composed of the unrestricted image region is a partial image generated by copying the image in which the pixel value becomes darker as the diameter φ increases between the shading start diameter φ1 and the shading end diameter φ2. With an output image that includes a partial image composed of such an unrestricted image region, shading can be applied without reducing the effective field of view. Therefore, it is possible to give the user a greater sense of immersion. Here, the effective field of view will be explained. The field of view and image height are correlated via the fisheye lens, and in this embodiment, the field of view corresponding to the effective image height will be referred to as the effective field of view below.

[0063] (Example output image) (Use case iv: Edge mirror copy shading) Furthermore, the image to which the above shading is applied may be a partial image generated by copying the folded image at the end of the image region of the effective height threshold φ', where the unrestricted image region is located. The partial image generated by copying the folded image at the end of the image region of the effective height threshold φ' may be an image in which the pixel value becomes darker as the diameter φ increases between the shading start diameter φ1 and the shading end diameter φ2. In this case, the effective height threshold φ' and the shading start diameter φ1 are equal, and the shading end diameter φ2 is greater than the shading start diameter φ1. That is, it may be an output image of a combination of use case ii and use case iii. With an output image that includes a partial image configured in such an unrestricted image region, shading can be applied to an equirectangular image in which the folded image spreads radially, without reducing the effective field of view. Therefore, by preventing abrupt blacking at the effective field of view boundary, it is possible to smoothly attenuate to black. Also, shading can be avoided in the effective field of view. Therefore, it is possible to give the user a remarkable sense of immersion.

[0064] Furthermore, although the φ1 and φ2 in this embodiment were described as fixed values ​​determined in relation to the effective image height threshold φ' determined by the fisheye lens, this is not limited to this. The effective image height thresholds φ1 and φ2 may be specified by the user. An example of a UI (user interface) for specifying these parameters will be explained using Figure 9. Figure 9 is a diagram illustrating an example of a UI for specifying equirectangular transformation parameters. Figure 9 consists of a preview image and sliders for setting values. Figure 9(a) shows sliders corresponding to the effective image height threshold φ', the shading start diameter φ1, and the shading end diameter φ2, respectively. By adjusting each slider in Figure 9(a), φ', φ1, and φ2 can be increased or decreased, and a shaded equirectangular image can be generated in real time and displayed on the display. These images may also be output to an HMD. By adjusting each slider in Figure 9(a), the user can determine the parameters for equirectangular transformation while checking the degree of immersion.

[0065] (Third embodiment) Equirectangular transformation with smoothing will be explained using Figures 2(c) and 5. Figure 5 is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180° field of view according to the third embodiment. The details of the processing performed by the image processing device will be explained below using Figures 2(c) and 5. The processing shown in Figure 5 is realized by the CPU 101 executing the program loaded into memory 103. The processing shown in Figure 5 is executed at the timing when the projection transformation processing is started. Note that some or all of the functions of the steps in Figure 5 may be realized by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in the flowchart. Furthermore, the processing shown in Figure 5 can also be realized as a cloud computing configuration in which one function is shared and processed jointly by multiple resources via the internet, as long as it realizes the function of each block in Figure 2(c). Furthermore, unless otherwise specified or changed, the description will be the same as the description in Figures 2(a) and 3 of the first embodiment.

[0066] Hereafter, the processing of each block in Figure 2(c) will be explained in relation to the processing of each step in Figure 5. Furthermore, since the details of the processing are the same as those of each step in Figure 5, Figure 5 will be used to describe them later. In the following explanation, the smoothing process will be referred to as the LPF (low-pass filter) process.

[0067] Figure 2(c) shows the addition of a blend value calculation unit 209, a blend unit 210, an LPF processing unit 211, and an LPF pixel value calculation unit 2063 compared to Figure 2(a). Also, the pixel value calculation unit 206 is replaced by the pixel value calculation unit 2061 and the LPF pixel value calculation unit 2063. The processing performed by the blend value calculation unit 209 corresponds to the processing in S5012. The processing performed by the pixel value calculation unit 2061 corresponds to the processing in S5062. The processing performed by the LPF pixel value calculation unit 2063 corresponds to the processing in S5061. The processing performed by the LPF processing unit 211 corresponds to the processing in S5015. The processing performed by the blend unit 210 corresponds to the processing in S5075.

[0068] Hereafter, we will mainly explain the differences between the processing steps in Figure 5 and those in Figure 3. In S501, the CPU 101 applies an LPF to the fisheye image to generate an LPF fisheye image. In this embodiment, a Gaussian filter with σ=5 and radius 15 is applied. Furthermore, the CPU 101 refers to the pixel values ​​of the LPF fisheye image, which are identified by the fisheye XY coordinates (x',y'), and calculates the pixel value v LPF The CPU calculates the blend value α from the radial distance. α is a value that decreases as the radial distance increases, and is calculated using equation (6). φ1 is the radial distance at which blending begins, and φ2 is the radial distance at which the image switches to the LPF fisheye image. In S503, the CPU calculates the pixel value v by referring to the pixel value of the fisheye image specified by the fisheye XY coordinates (x',y'). In S504, the CPU calculates the pixel value v and the pixel value v using α as a weight. LPF The two are blended to calculate the pixel value of the output image, which is identified by the XY coordinates (x,y), and the calculated pixel value is stored. In this embodiment, the blended value v new The following formula shall be used to calculate it.

[0069]

number

[0070] In equation (7), the larger the radial movement (image height), the smaller α becomes, and the pixel value with a greater degree of smoothing can be calculated. In the first embodiment, as shown in Figures 8(c) and 8(d), the image is copied radially, so the larger the radial movement, the more unnatural the radial image may appear. According to this embodiment, the larger the radial movement, the larger the ratio of the LPF image blended, thereby obtaining a pseudo-smoothed, i.e., a low-contrast equirectangular image. Therefore, the aforementioned unnaturalness can be reduced. In this embodiment, the shading value α described in the second embodiment may also be used. In this case, the blend value α may be the same as the shading value α in the second embodiment, or it may be calculated individually. Also, as in the second embodiment, the blend value α may be set by the user via the UI to φ1 and φ2.

[0071] (Example output image) (Use case v: Edge copy shading LPF) In other words, as specified by equations (6) and (7), within the unrestricted image region, an image is generated between the radial φ1 where blending begins and the radial φ2 where the image switches to the LPF fisheye image, in which the ratio of the LPF image increases as the radial φ increases. That is, the CPU 101 generates the output image by copying the image within the unrestricted image region, between the radial φ1 where blending begins and the radial φ2 where the image switches to the LPF fisheye image, in which the ratio of the LPF image increases as the radial φ increases. The CPU 101 outputs and displays the generated output image. The output image at this time includes a partial image composed of the restricted image region and a partial image composed of the unrestricted image region surrounding the restricted image region. In the restricted image region, the image region used for projection transformation from the wide-angle image is restricted. Such a restricted image region is an image region specified by an image height smaller than the effective image height. The partial image formed in the unrestricted image region is a partial image generated by copying an image with a larger ratio of the LPF image between the radial movement φ1 where blending begins and the radial movement φ2 where it switches to the LPF fisheye image. The output image, which includes such a partial image formed in the unrestricted image region, yields the following effects: In the image region created by copying within the unrestricted image region, it is possible to generate pixel values ​​that gradually blur outward from the center of the image circle along the radial direction of the radial movement. Therefore, it is possible to make the roughness of the unrestricted image region less noticeable. Consequently, it is possible to give the user a particularly noticeable sense of immersion.

[0072] In this embodiment, the Gaussian filter is set to σ=5 and radius 15, but it is not limited to these settings. For example, it may be configured so that the user can set it using a UI. Also, in this embodiment, it was explained that an LPF is applied to the fisheye image used as the input image. However, a similar effect can be obtained by first creating an equirectangular image as the output image, applying an LPF to the equirectangular image, and then blending the LPF-applied equirectangular image with the un-LPF-applied equirectangular image using a blend value α determined according to the radius.

[0073] (Fourth embodiment) The equirectangular transformation, in which the degree of smoothing increases with increasing radius, will be explained using Figure 6. Figure 6 is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180-degree field of view according to the fourth embodiment. The details of the processing performed by the image processing device will be explained below using Figure 6. The processing shown in Figure 6 is realized by the CPU 101 executing a program loaded into memory 103. The processing shown in Figure 6 is executed at the timing when the projection transformation processing is started. Note that some or all of the functions of the steps in Figure 6 may be realized by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in the flowchart. Furthermore, the processing shown in Figure 6 can also be realized as a cloud computing configuration in which one function is shared and processed jointly by multiple resources via the internet. Unless otherwise specified or changed, the description will be the same as the description in Figure 3 of the first embodiment.

[0074] The process shown in Figure 6 differs from the process shown in Figure 3 in that process S601 is added, and process S308 is replaced by process S602. In S601, CPU 101 calculates an S value that increases as the diameter increases, using the following formula.

[0075]

number

[0076] In S602, the CPU 101 calculates and stores the average value of the pixel values ​​of the fisheye image, centered on the fisheye XY coordinate (x',y'), within a range of ±S horizontally and vertically, as the pixel value of the output image specified by the XY coordinate (x,y). The average of the pixel values ​​is the average of all pixels within the image region defined by the range of ±S.

[0077] In the first embodiment, the image is copied radially as shown in Figures 8(c) and 8(d), so the larger the radial movement, the more unnatural the radial image may appear. According to this embodiment, the larger the radial movement, the wider the range of pixel values ​​can be averaged to reduce the aforementioned unnaturalness. In this embodiment, an example is shown where a smoother value is calculated by averaging a wider range of pixel values ​​as the radial movement increases. However, the S value can be fixed, the smoothed pixel value can be calculated, and then blended so that the smoothing component increases as the radial movement increases, as shown in the third embodiment. In this embodiment, shading as described in the second embodiment may also be used in combination. Alternatively, the average of sampled points may be used for speed improvement. Sampling tends to cause aliasing distortion, which can degrade the image quality of the smoothed image, but by using shading in combination, the visibility of aliasing distortion can be reduced.

[0078] (Example output image) (Use case vi: Edge copy shading LPF range variable) In other words, as specified by equation (8), an image is generated by applying an LPF to the image of the image region specified by an S value that increases as the diameter φ increases, between the moving diameter φ1 and the moving diameter φ2, within the unrestricted image region. That is, the CPU 101 generates an output image by copying the image of the image region specified by an S value that increases as the diameter φ increases, between the moving diameter φ1 and the moving diameter φ2, within the unrestricted image region, which has been subjected to an LPF. The CPU 101 outputs and displays the generated output image. The output image at this time includes a partial image composed of the restricted image region and a partial image composed of the unrestricted image region surrounding the restricted image region. In the restricted image region, the image region used for projection transformation from the wide-angle image is restricted. Such a restricted image region is an image region specified by an image height smaller than the effective image height. The partial image composed of the unrestricted image region is a partial image generated by copying the image of the image region specified by an S value that increases as the diameter φ increases, between the moving diameter φ1 and the moving diameter φ2, which has been subjected to an LPF. The output image, which includes a partial image composed of such an unrestricted image region, yields the following effects: Specifically, in the copied image region within the unrestricted image region, a larger radial diameter φ allows for the generation of pixel values ​​that average a wider range of pixel values. Therefore, the coarseness of the unrestricted image region becomes less noticeable. Consequently, it becomes possible to provide the user with a particularly noticeable sense of immersion.

[0079] (Fifth embodiment) The equirectangular transformation, in which saturation decreases as the radial movement increases, will be explained using Figure 7. Figure 7 is a flowchart illustrating the projection transformation from a fisheye image to an equirectangular image with a 180° field of view according to the fifth embodiment. The details of the processing performed by the image processing device will be explained below using Figure 7. The processing shown in Figure 7 is realized by the CPU 101 executing a program loaded into memory 103. The processing shown in Figure 7 is executed at the timing when the projection transformation processing is started. Note that some or all of the functions of the steps in Figure 7 may be realized by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in the flowchart. Furthermore, the processing shown in Figure 7 can also be realized as a cloud computing configuration in which one function is shared and processed jointly by multiple resources via the internet. Unless otherwise specified or changed, the description will be the same as the description in Figure 3 of the first embodiment.

[0080] The process shown in Figure 7 differs from the process shown in Figure 3 in that processes S701 and S702 are added, and process S308 is replaced by process S703. In S701, CPU 101 calculates the chromatic difference reduction rate β from the radial distance using the following formula. When the reduction rate is 1, the saturation becomes 0, that is, it becomes monochrome.

[0081]

number

[0082] φ'2 represents the radius at which the saturation becomes 0. φ'1 represents the radius at which the saturation begins to decay. In this embodiment, φ'2 is set to a value smaller than the image height at which false color occurs. In S702, the CPU 101 calculates the pixel value v by referring to the pixel value of the fisheye image specified by the fisheye XY coordinates (x',y'). The pixel value v is a vector representing RGB. In S703, the CPU 101 calculates a value obtained by reducing the saturation component of the pixel value v using β as a weight, and stores it as the pixel value of the output image specified by the XY coordinates (x,y). The reduction of the saturation component can be achieved by converting RGB to HSV or HSB, reducing the saturation component by multiplying it by β, and then converting it back to RGB.

[0083] According to this embodiment, by gradually decreasing the saturation as the radial movement increases from the inside of the region where false color occurs, it is possible to suppress the copying of false-color pixels as they are, even if the effective image height is greater than the image height at which false color occurs. This embodiment can also be used in combination with the shading process described in the second embodiment and the LPF process described in the third and fourth embodiments. Furthermore, by combining it with the process of limiting the radial movement to the effective image height threshold described in the first embodiment, it is possible to obtain an image in which the user does not perceive the occurrence of false color.

[0084] (Example output image) (Use case vii: Edge copy shading with variable saturation) In other words, as specified by equation (9), within the unrestricted image region, an image is generated between the radial φ'1 at which saturation begins to decay and the radial φ'2 at which saturation becomes 0, in which case the saturation decreases as the radial φ increases. That is, the CPU 101 generates the output image by copying the image within the unrestricted image region between the radial φ'1 at which saturation begins to decay and the radial φ'2 at which saturation becomes 0, in which case the saturation decreases as the radial φ increases. The CPU 101 outputs and displays the generated output image. The output image at this time includes a partial image composed of the restricted image region and a partial image composed of the unrestricted image region surrounding the restricted image region. In the restricted image region, the image region used for projection transformation from the wide-angle image is restricted. Such a restricted image region is an image region specified by an image height smaller than the effective image height. The partial image formed in the out-of-limitation image region is a partial image generated by copying an image between the radial diameter φ'1, where saturation begins to decrease, and the radial diameter φ'2, where saturation becomes 0, with the saturation decreasing as the radial diameter φ increases. The output image, which includes such a partial image formed in the out-of-limitation image region, yields the following effects. Specifically, in the image region created by copying within the out-of-limitation image region, it is possible to generate pixel values ​​in which saturation gradually decreases from the center of the image circle outward along the radial direction of the radial diameter. Therefore, it is possible to obtain an image in which the user does not perceive the occurrence of false colors in the out-of-limitation image region. Consequently, it is possible to give the user a particularly remarkable sense of immersion.

[0085] Furthermore, although φ'1 and φ'2 were described as fixed values ​​in this embodiment, they may also be configured to be specified by the user via the UI, as will be described later using Figure 9(b). Figure 9(b) shows sliders corresponding to the effective image height threshold, shading start diameter, and shading end diameter, as well as sliders corresponding to the saturation reduction start diameter and saturation reduction end diameter. The values ​​adjusted by the sliders corresponding to the saturation reduction start diameter and saturation reduction end diameter correspond to φ'1 and φ'2, respectively.

[0086] Furthermore, although the reduction of the saturation component was described in this embodiment as a use case calculated in HSB space or HSV space, it is not limited to these. For example, saturation may be reduced by converting RGB to YUV (luminance (Y), chrominance (UV)) and reducing the UV (U: hue, V: chroma) component. Alternatively, saturation may be reduced according to β by calculating in RGB space using the following formula.

[0087]

number

[0088] In equation (10), as β increases, the weight of the RGB components decreases. That is, the RGB values ​​become the same, and the region where β is 1 becomes monochrome.

[0089] (Other embodiments) Although various examples and embodiments of this disclosure have been described above, the spirit and scope of this disclosure are not limited to the specific descriptions herein. This disclosure is not limited to the embodiments described above, and various modifications may be made. Furthermore, this disclosure may combine some of the embodiments described above as appropriate.

[0090] (Variation 1) For example, we have described an example in which a wide-angle image is divided into two image regions by the effective image height for processing, but we are not limited to this. For example, the wide-angle image may be embedded in another image. Specifically, if the wide-angle image is edited to form a single image surrounded by other images, the wide-angle image will appear to be embedded in the other images. Even in such a state, by using the effective image height, it is possible to exclude from the projection transformation processing the image region containing the wide-angle image from the image region where the image with an image height exceeding the effective image height is located. The other images may be, for example, images of real life or computer graphics. Specifically, this could be a system in which an image of real life or a computer graphics image is used as a background image, and an image incorporating the wide-angle image of this disclosure into a part of that background image is displayed on a display means such as an HMD.

[0091] (Modification 2) Furthermore, while we have described an example of projective transformation from an input image represented in a two-dimensional coordinate system to an output image represented in a two-dimensional coordinate system via an image represented in another two-dimensional coordinate system, we are not limited to this example. For instance, an input image represented in a two-dimensional coordinate system may be projected to an output image represented in a two-dimensional coordinate system via an image represented in an arbitrary three-dimensional coordinate system. Alternatively, an input image represented in a two-dimensional coordinate system may be projected to an output image represented in a two-dimensional coordinate system via an image represented in the same coordinate system. In this case, by using an image represented in the same coordinate system, geometric transformations of the image such as scaling, rotation, translation, inversion, and skew become possible as projective transformations. Such geometric transformations allow for correction to remove image distortion. For example, when radial distortion occurs, it is possible to remove the radial distortion by transforming a fisheye image into another fisheye image using geometric transformation within the same coordinate system and comparing the two fisheye images.

[0092] (Variation 3) Furthermore, while examples of UIs for specifying equirectangular transformation parameters have been explained using Figures 9(a) and 9(b), the system may be configured so that the user can select either Figure 9(a) or Figure 9(b). For example, radio buttons may be added to each of the UIs in Figure 9(a) and Figure 9(b). By adding an input form to the UI that allows the user to select one specific option from multiple choices, such as radio buttons, the user can choose any UI.

[0093] (Modification 4) Furthermore, although use cases i to vii have been described as examples of application images, a UI may be configured that allows the user to appropriately select at least one output image from use cases i to vii.

[0094] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. Furthermore, the program may be recorded on a recording medium readable by a computer and provided.

[0095] The disclosure of this embodiment includes configurations represented by the following image processing apparatus, image processing method, and program.

[0096] <Configuration 1> Image acquisition means for acquiring a wide-angle image that exceeds a predetermined angle of view, A restricting means for restricting the image region used for projection transformation from the wide-angle image to a restricted image region, A transformation means for projecting the coordinate system of an image within the restricted image region of the wide-angle image to the coordinate system of the output image, An image processing apparatus characterized by comprising:

[0097] <Configuration 2> The image processing apparatus according to configuration 1, characterized in that the conversion means does not project the coordinate system of an image in an unrestricted image region outside the restricted image region to the coordinate system of the output target image.

[0098] <Structure 3> The image processing apparatus according to configuration 2, further comprising an upper limit acquisition means for acquiring an effective image height, which is an upper limit value of the image height indicating the imaging height from the optical axis center of the imaging optical system used to capture the wide-angle image, as a threshold capable of distinguishing between the restricted image region and the unrestricted image region.

[0099] <Structure 4> The image processing apparatus according to configuration 3, characterized in that the limiting means defines an image region within an image region containing the wide-angle image as the limited image region, where the image region is specified by an image height smaller than the effective image height, and an image region specified by an image height equal to or greater than the effective image height as the unlimited image region.

[0100] <Composition 5> The image processing apparatus according to configuration 3, characterized in that the unrestricted image region may include an image region in which false color occurs.

[0101] <Composition 6> The image processing apparatus according to configuration 5, characterized in that the effective image height is set to a range in which, among the multiple colors constituting the wide-angle image, pixels of a specific color do not exceed a predetermined number of pixels and do not extend into an image area containing pixels of a different color from the specific color.

[0102] <Composition 7> The image processing apparatus according to configuration 6, characterized in that the image acquisition means acquires a fisheye image as the wide-angle image.

[0103] <Structure 8> The image processing apparatus according to configuration 7, characterized in that the conversion means converts the coordinate system of the fisheye image to the orthogonal coordinate system of the equirectangular image as the coordinate system of the image to be output.

[0104] <Composition 9> The image height is determined by the radial distance of the radial element representing the polar coordinate when each pixel position on the image plane is expressed in polar coordinates, starting from the imaging center, which is the intersection point of the optical axis center and the image plane of the imaging optical system. The image processing apparatus according to configuration 8, characterized in that the limiting means limits the radius movement to be equal to or less than the effective image height when the wide-angle image is represented in the polar coordinate system.

[0105] <Composition 10> The image processing apparatus according to configuration 9, further comprising a preprocessing transformation means for transforming the coordinate system of the wide-angle image into the polar coordinate system before limiting the radius movement to be less than the effective image height.

[0106] <Composition 11> The image processing apparatus according to configuration 10, characterized in that the conversion means is set to an orthogonal coordinate system as the coordinate system of the output target.

[0107] <Composition 12> The image processing apparatus according to configuration 11, characterized in that the conversion means converts the coordinate system of the wide-angle image from the polar coordinate system to the Cartesian coordinate system.

[0108] <Composition 13> The image processing apparatus according to configuration 12, characterized in that the output image represented in the coordinate system of the output target is the equirectangular image.

[0109] <Composition 14> The limiting means, when the moving diameter is equal to or greater than the effective image height, is defined as follows: the moving diameter is φ, the effective image height is φ', and the moving diameter that is limited to being equal to or smaller than the effective image height is φNEW. [Mathematics 1] The image processing apparatus according to configuration 9, characterized in that, in accordance with the conditions of TIFF2026106876000012.tif9150, the image area to be used from among the image areas in which the wide-angle image is located is limited to the restricted image area.

[0110] <Composition 15> The limiting means, when the diameter of the movement is equal to or greater than the effective image height, sets the diameter of the movement to φ, the effective image height to φ', and the diameter of the movement that is equal to or less than the effective image height to φ NEW So, [Math 2] The image processing apparatus according to configuration 9, characterized in that, in accordance with the conditions of TIFF2026106876000013.tif11150, the image area to be used from among the image areas containing the wide-angle image is limited to the restricted image area.

[0111] <Composition 16> The effective image height is set to include a lower effective image height and an upper effective image height greater than the lower effective image height. The limiting means, when the moving diameter is φ, the lower effective image height is φ1, the upper effective image height is φ2, and the shading value is α, [Math 3] The image processing apparatus according to configuration 9, characterized in that the pixel values ​​of the image to be output are obtained by multiplying the pixel values ​​of the fisheye image by the shading value in accordance with the conditions of TIFF2026106876000014.tif22150.

[0112] <Composition 17> The image processing apparatus according to configuration 16, characterized in that the pixel values ​​of the output image obtained by multiplying the pixel values ​​of the fisheye image by the shading value show that, between the lower effective image height and the upper effective image height, the pixel values ​​of the output image decrease as the radial movement increases.

[0113] <Composition 18> The image processing apparatus according to configuration 9, further comprising a smoothing means that increases the degree of smoothing of the image in the out-of-limit image region as the diameter of the moving radius increases.

[0114] <Composition 19> The image processing apparatus according to configuration 18, characterized in that the smoothing means increases the degree of smoothing of the image in the unrestricted image region as the diameter of the movement increases, and also increases the image region on which the smoothing is performed.

[0115] <Composition 20> The image processing apparatus according to configuration 16, further comprising a reduction means for reducing the saturation of an image in the out-of-limit image region as the diameter increases.

[0116] <Composition 21> An image acquisition process to acquire a wide-angle image that exceeds a predetermined angle of view, A restriction step in which the image region used for projection transformation from the wide-angle image is restricted to a restricted image region, A transformation step of projecting the coordinate system of the image in the restricted image region of the wide-angle image to the coordinate system of the output image, An image processing method characterized by including

[0117] <Composition 22> A program to cause a computer to function as an image processing method as described in configuration 21. [Explanation of symbols]

[0118] 101 CPU

Claims

1. Image acquisition means for acquiring a wide-angle image that exceeds a predetermined angle of view, A restricting means for restricting the image region used for projection transformation from the wide-angle image to a restricted image region, A transformation means for projecting the coordinate system of an image within the restricted image region of the wide-angle image to the coordinate system of the output image, An image processing apparatus characterized by comprising:

2. The image processing apparatus according to claim 1, characterized in that the conversion means does not project the coordinate system of an image in an unrestricted image region outside the restricted image region to the coordinate system of the output target image.

3. The image processing apparatus according to claim 2, further comprising an upper limit acquisition means for acquiring an effective image height, which is an upper limit value of the image height indicating the imaging height from the optical axis center of the imaging optical system used to capture the wide-angle image, as a threshold capable of distinguishing between the restricted image region and the unrestricted image region.

4. The image processing apparatus according to claim 3, characterized in that the limiting means defines an image region within an image region containing the wide-angle image as the limited image region, where the image region is defined as having an image height smaller than the effective image height, and an image region defined as having an image height equal to or greater than the effective image height as the unlimited image region.

5. The image processing apparatus according to claim 3, characterized in that the unrestricted image region may include an image region in which false color occurs.

6. The image processing apparatus according to claim 5, characterized in that the effective image height is set to a range such that, among the multiple colors constituting the wide-angle image, pixels of a specific color do not extend beyond a predetermined number of pixels into an image area containing pixels of a different color from the specific color.

7. The image processing apparatus according to claim 6, characterized in that the image acquisition means acquires a fisheye image as the wide-angle image.

8. The image processing apparatus according to claim 7, characterized in that the conversion means converts the coordinate system of the fisheye image to the orthogonal coordinate system of the equirectangular image as the coordinate system of the image to be output.

9. The image height is determined by the radial distance of the radial element representing the polar coordinate when each pixel position on the image plane is expressed in polar coordinates, starting from the imaging center, which is the intersection point of the optical axis center and the image plane of the imaging optical system. The image processing apparatus according to claim 8, characterized in that the limiting means limits the radius movement to be equal to or less than the effective image height when the wide-angle image is represented in the polar coordinate system.

10. The image processing apparatus according to claim 9, further comprising a preprocessing transformation means for transforming the coordinate system of the wide-angle image into the polar coordinate system before limiting the radius movement to be less than the effective image height.

11. The image processing apparatus according to claim 10, characterized in that the conversion means is set to an orthogonal coordinate system as the coordinate system of the output target.

12. The image processing apparatus according to claim 11, characterized in that the conversion means converts the coordinate system of the wide-angle image from the polar coordinate system to the Cartesian coordinate system.

13. The image processing apparatus according to claim 12, characterized in that the output image represented in the coordinate system of the output target is the equirectangular image.

14. The limiting means, when the diameter of the movement is equal to or greater than the effective image height, sets the diameter of the movement to φ, the effective image height to φ', and the diameter of the movement that is equal to or smaller than the effective image height to φ NEW So, [Math 1] The image processing apparatus according to claim 9, characterized in that, in accordance with the conditions, the image area to be used from among the image areas in which the wide-angle image is located is limited to the limited image area.

15. The limiting means, when the diameter of the movement is equal to or greater than the effective image height, sets the diameter of the movement to φ, the effective image height to φ', and the diameter of the movement that is equal to or less than the effective image height to φ NEW So, [Math 2] The image processing apparatus according to claim 9, characterized in that, in accordance with the conditions, the image area to be used from among the image areas in which the wide-angle image is located is limited to the limited image area.

16. The effective image height is set to include a lower effective image height and an upper effective image height greater than the lower effective image height. The limiting means sets the diameter of the movement to φ and the lower limit effective image height to φ 1 , the upper limit effective image height is φ 2 If the shading value is α, [Math 3] The image processing apparatus according to claim 9, characterized in that the pixel values ​​of the output image are obtained by multiplying the pixel values ​​of the fisheye image by the shading value in accordance with the conditions.

17. The image processing apparatus according to claim 16, wherein the pixel values ​​of the output image obtained by multiplying the pixel values ​​of the fisheye image by the shading value show that, between the lower effective image height and the upper effective image height, the pixel values ​​of the output image decrease as the radial movement increases.

18. The image processing apparatus according to claim 9, further comprising a smoothing means for increasing the degree of smoothing of the image in the out-of-limit image region as the diameter of the moving part increases.

19. The image processing apparatus according to claim 18, characterized in that the smoothing means increases the degree of smoothing of the image in the unrestricted image region and increases the image region to be smoothed as the diameter of the movement increases.

20. The image processing apparatus according to claim 16, further comprising a reduction means for reducing the saturation of an image in the unrestricted image region as the diameter increases.

21. An image acquisition process to acquire a wide-angle image that exceeds a predetermined angle of view, A restriction step in which the image region used for projection transformation from the wide-angle image is restricted to a restricted image region, A transformation step of projecting the coordinate system of the image in the restricted image region of the wide-angle image to the coordinate system of the output image, An image processing method characterized by including

22. A program for causing a computer to function as the image processing method described in claim 21.

Images