Image processing device and image processing method

Abstract

To provide an image processing device, method, and program, which allow image processing for perspective projection to be applied to a non-perspective projection image without depending on an area in the image when applying image processing to non-perspective projection image, such as a fisheye image, on the premise that the image is viewed on an HMD (Head Mounted Display).SOLUTION: Processing performed by an image processing device comprises: acquiring and dividing a non-perspective projection image 1 S2010; converting an image in each of divided image areas into a perspective projection image S2030; applying image processing to the perspective projection image S2040; and converting the resultant image into a non-perspective projection image 2 S2050.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing technique for non-perspective projection images. [Background technology]

[0002] In recent years, HMD (Head Mounted Display) type XR information processing terminals have started to become popular. XR is a general term for VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality).

[0003] HMD-type XR information processing terminals can display not only CG but also live-action images. A part of a wide-angle image captured using a fisheye lens is cut out by tracking the movement of the head, and is converted into a perspective projection image and displayed. Perspective projection is a projection method in which all points on a three-dimensional object are considered to be radially focused on a single viewpoint. In the strict sense, perspective projection is a projection onto a plane, but a lens is placed in front of the display panel of the HMD, and the image is not displayed on a plane. However, since there is no difference in the fact that it deals with images that are radially focused on a single viewpoint, from now on, these projections will be collectively referred to as perspective projection. Live-action wide-angle image content for XR includes 360-degree images (also called panoramic images) and stereo 180-degree images. 360-degree images are created by stitching together images taken with two or more fisheye cameras. Stereo 180-degree images are obtained by arranging fisheye lenses side by side to take stereo images. When a photograph is taken, it is often recorded in the form of a fisheye image, but for content to be distributed to HMDs, fisheye images with mesh information attached to them that makes it easier to convert them into perspective projection images, or fisheye images converted into equirectangular images are used. Mesh information is the coordinates of the intersections when dividing a fisheye image into triangles and the coordinates of the projection onto a hemisphere. The coordinates projected using mesh information are those that have been freed of lens distortion. When displayed on an HMD, the pixels of the fisheye image are projected onto a hemisphere by leaving the intersections as they are and interpolating the non-intersections, and then converted into a perspective projection image. Equirectangular images are fisheye images converted into equirectangular images, and are converted in a form that removes lens distortion in the same way as mesh images, and are converted into perspective projection images when displayed on an HMD. Fisheye images and equirectangular images can be recorded as 2D image data even in wide-angle areas, but the parts with high image heights (wide-angle areas) are recorded with significant distortion relative to the center. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2020 / 0382755 Summary of the Invention [Problem to be solved by the invention]

[0005] When applying image processing to non-perspective projection images such as fisheye images on the assumption that they will be viewed with an HMD, the technology in Patent Document 1 requires that the image processing be changed according to the projection method and field of view area in order to obtain uniform image processing results in any field of view. This invention provides a technology that makes it possible to apply image processing for perspective projection to non-perspective projection images, regardless of the area in the image. [Means for solving the problem]

[0006] One aspect of the present invention is characterized in that the image processing device includes a conversion unit that converts an image in each image region in a non-perspective projection image into a perspective projection image, performs image processing on the perspective projection image, and then converts it into a non-perspective projection image. Effect of the Invention

[0007] According to the present invention, it is possible to provide a technique that enables image processing for perspective projection to be applied to non-perspectively projected images, regardless of the region in the image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus. [Diagram 2] 4 is a flowchart of a process performed by the image processing device. [Diagram 3] 11A and 11B are diagrams showing the relationship between divided regions and perspective projection images corresponding to the divided regions. [Figure 4] 4 is a flowchart of a process performed by the image processing device. [Diagram 5] FIG. 13 is a diagram showing an example of the configuration of an alpha channel map. [Figure 6] 4 is a flowchart of a process performed by the image processing device. [Figure 7] 4 is a flowchart of a process performed by the image processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] [First embodiment] First, an example of the hardware configuration of an image processing device according to this embodiment will be described with reference to the block diagram of Fig. 1. A CPU 101 executes various processes using computer programs and data stored in a memory 103. As a result, the CPU 101 controls the operation of the entire image processing device, and also executes or controls various processes that will be described as processes performed by the image processing device.

[0011] The non-volatile memory 102 stores an OS (operating system), computer programs and data for causing the CPU 101 and the GPU 105 to execute or control various processes described as processes performed by the image processing device, etc. The computer programs and data stored in the non-volatile memory 102 are loaded into the memory 103 as appropriate under the control of the CPU 101, and become targets for processing by the CPU 101 and the GPU 105.

[0012] The memory 103 has an area for storing computer programs and data loaded from the non-volatile memory 102. The memory 103 also has an area for storing data received from the outside via a general-purpose IF (interface) 106 and a NW / IF (network interface) 107. The memory 103 also has a work area used when the CPU 101 and the GPU 105 execute various processes. In this way, the memory 103 can provide various areas as appropriate.

[0013] The UI device connection unit 104 is an interface for connecting a user interface such as a keyboard, a mouse, a touch panel, etc. Various instructions input by the user by operating the user interface are notified to the CPU 101 via the UI device connection unit 104.

[0014] The GPU 105 performs various types of image processing under the control of the CPU 101. The general-purpose IF 106 is an interface for performing data communication between devices such as a device that is an output destination of an image generated by the image processing device (e.g., a device accessible by an HMD) and a device that can supply an image to the image processing device (e.g., an imaging device or a memory device).

[0015] The NW / IF 107 is an interface for performing data communication with an external device via a wired and / or wireless network such as a LAN, the Internet, etc. Note that the image processing device may perform data communication via the NW / IF 107 with devices such as a device that is an output destination of an image generated by the image processing device, a device that can supply an image to the image processing device, and the like.

[0016] The CPU 101 , the non-volatile memory 102 , the memory 103 , the UI device connection unit 104 , the GPU 105 , the general-purpose IF 106 , and the NW / IF 107 are all connected to the bus 100 .

[0017] Such an image processing device can be, for example, a computer device such as a PC (personal computer), a smartphone, a tablet terminal device, etc. In addition, the hardware configuration applicable to the image processing device according to the present embodiment is not limited to the configuration shown in FIG. 1, and can be modified / changed as appropriate.

[0018] Next, the process performed by the image processing device will be described with reference to the flowchart in Fig. 2. In step S2010, the CPU 101 acquires "non-perspective projection image 1, which is an image in equirectangular projection" in the memory 103. The method of acquiring the non-perspective projection image 1 is not limited to a specific method.

[0019] For example, the CPU 101 may load the non-perspective projection image 1 stored in the non-volatile memory 102 into the memory 103. The CPU 101 may also receive an image (non-perspective projection image 1) captured by an image capturing device via the general-purpose IF 106 or the NW / IF 107 and store it in the memory 103. The image captured by the image capturing device may be an image of each frame in a moving image captured by the image capturing device, or may be a still image captured by the image capturing device periodically or irregularly. The CPU 101 may also receive the non-perspective projection image 1 stored in an external device such as a cloud server via the NW / IF 107 and store it in the memory 103.

[0020] For the purpose of a more specific explanation, we will use as an example a case in which non-perspective projection image 1 is an equirectangular projection image (equirectangular image) with a vertical size of 1,750 pixels, a vertical angle of 175 degrees, a horizontal size of 1,800 pixels, a horizontal angle of 180 degrees, and the pixel values ​​of each pixel being floating-point RGB values.

[0021] Then, CPU 101 divides the acquired non-perspective projection image 1 into a plurality of divided regions. For the purpose of a specific description, the following describes, as an example, a case in which the divided regions are rectangular regions having a size of 25 pixels x 25 pixels.

[0022] In step S2020, CPU 101 selects one of the divided regions obtained in step S2010 that has not been selected as a selected divided region. The selection order of the divided regions is not limited to a specific selection order, and for example, the divided regions in non-perspective projection image 1 are selected in order from the top left to the right and from the top to the bottom.

[0023] In step S2030, the CPU 101 sets an extended selected divided region to an image region of 33 pixels by 33 pixels, which is the selected divided region plus a frame of 4 pixels wide on all four sides, and converts the image in the extended selected divided region (image region) in the non-perspective projection image 1 into a perspective projection image. The pixel position of the center in the non-perspective projection image 1 is set as the origin, and the pixel position of the pixel of interest in the extended selected divided region is set as (x equi , y equi ), the pixel position of the pixel at the center of the extended selected segmentation region is (x equi_c , y equi_c ) where λ=x equi / 10, θ=y equi / 10, λc=x equi_c / 10, θc=y equi_c / 10, s=12.5 / tan(12.5). In this case, the pixel position (x, y) of the corresponding pixel on the perspective projection image that corresponds to the pixel of interest can be obtained according to the following formula (1).

[0024]

number

[0025] Equation (1) is an equation expressing the transformation from the coordinate system of the equirectangular image to the coordinate system of the perspective projection image. Trigonometric functions are calculated in degrees. The CPU 101 obtains the pixel position of a corresponding pixel corresponding to each pixel of interest in the extended selected divided area based on equation (1), and sets the pixel value of the pixel of interest to the pixel value of the corresponding pixel. The pixel values ​​of pixels other than the corresponding pixel in the perspective projection image are obtained, for example, by interpolation using the pixel values ​​of nearby corresponding pixels.

[0026] In this embodiment, s is set so that the magnification ratio of the center of the extended selected divided area is approximately 100%. The reason for adding a 4-pixel wide frame to the selected divided area is to refer to the outside of the area when applying a 9 × 9 Gaussian filter in the next step.

[0027] Here, the relationship between the divided regions and the perspective projection images corresponding to the divided regions will be described with reference to FIG. 3. In FIG. 3, reference numeral 300 indicates a group of divided regions in the non-perspective projection image 1. The extended selected divided region 301 is an image region of 33 pixels×33 pixels obtained by adding a frame of 4 pixels wide to the top, bottom, left, and right of the divided region of 25 pixels×25 pixels, and is converted into a perspective projection image 303 by conversion according to the position of the divided region in the non-perspective projection image 1. The extended selected divided region 302 is an image region of 33 pixels×33 pixels obtained by adding a frame of 4 pixels wide to the top, bottom, left, and right of the divided region of 25 pixels×25 pixels, and is converted into a perspective projection image 304 by conversion according to the position of the divided region in the non-perspective projection image 1. In this way, by performing conversion based on parameters according to the pixel positions of the pixels in the divided region, the divided region is converted (projected) into a perspective projection image of a different shape according to the position of the divided region.

[0028] In step S2040, the CPU 101 performs filtering using a bilateral filter, as an example of image processing, on the perspective projection image generated in step S2030.

[0029] In step S2050, CPU 101 performs an inverse transformation of the transformation performed in step S2030 (the transformation according to equation (1)) on the perspective projection image that has been subjected to image processing in step S2040, to obtain a 33 pixel x 33 pixel non-perspective projection image, and generates a 25 pixel x 25 pixel image as non-perspective projection image 2 by omitting a "4 pixel wide frame on the top, bottom, left, and right sides" from the non-perspective projection image of 33 pixels x 33 pixels.

[0030] In step S2060, CPU 101 specifies a corresponding area in non-perspective projection image 3 that has the same size (the same number of pixels in the vertical direction and the horizontal direction) as non-perspective projection image 1, which corresponds to the selected divided area selected in step S2020, and sets the pixel value of each pixel in the corresponding area to the pixel value of the corresponding pixel in non-perspective projection image 2. This results in a non-perspective projection image 3 in which non-perspective projection image 2 is arranged in the same order as the selected divided areas.

[0031] In step S2070, CPU 101 determines whether or not all divided regions have been selected as selected divided regions. If the result of this determination is that all divided regions have been selected as selected divided regions, the process proceeds to step S2080. On the other hand, if any divided regions remain that have not yet been selected as selected divided regions, the process proceeds to step S2020.

[0032] In step S2080, CPU 101 outputs non-perspective projection image 3 (a combined image of non-perspective projection images 2 arranged in step S2060 above). The non-perspective projection image 2 corresponding to the P-th (P is an integer equal to or greater than 1) divided area from the left end and the Q-th (Q is an integer equal to or greater than 1) divided area from the top end in the group of divided areas in non-perspective projection image 1 is arranged in the P-th position from the left end and the Q-th position from the top end in the group of non-perspective projection images 2 in non-perspective projection image 3. When such a combined image is output as an image for display on an HMD, the combined image becomes an image close to an image seen from the HMD.

[0033] The output destination of the non-perspective projection image 3 is not limited to a specific output destination, and the CPU 101 may transmit the non-perspective projection image 3 to an external device such as a mobile terminal device or a server device held by a user via the general-purpose IF 106 or the NW / IF 107. In addition, the CPU 101 may store the non-perspective projection image 3 in the non-volatile memory 102.

[0034] Even in real life, a part of a wide-angle image using a fisheye lens is cut out by following the movement of the head, and is converted into a perspective projection image and displayed. When converting an equirectangular image into a perspective projection image, the conversion is a horizontal reduction in areas where the image height is higher than the central area. In other words, when a filter process is performed on an equirectangular converted image, the result of image processing applied to the equirectangular image will change depending on the position of the image height at the top and bottom when it is subsequently converted into a perspective projection image. According to this embodiment, by converting each divided area into a perspective projection image, performing image processing, and then projecting and combining again, it is possible to obtain an image processing effect that is not related to the image height even when displayed on an HMD, for example. In particular, the bilateral filter is a process that changes the amount of effect by analyzing the sharpness of the pixels (edges) of the input image. Therefore, rather than directly applying the process to an equirectangular image, which is stretched as the image height at the top and bottom is higher, it is more accurate to obtain the image processing result that is desired when viewing by using this technology to process the perspective projection image.

[0035] This process is equivalent to processing an equirectangular image projected onto a sphere by treating a portion of the sphere, which has been divided finely, as a perspective projection image. It is also possible to obtain the same effect as this technology by changing the size and shape of the filter kernel for each pixel. However, there are disadvantages in that the amount of calculation required to calculate the filter kernel for each pixel increases, and in places where the image height is high both above and below, the filter size also increases, further increasing the amount of calculation. Therefore, this technology is highly useful.

[0036] In this embodiment, a case where a bilateral filter is used as image processing for a perspective projection image has been described, but other types of image processing may be used. For example, image processing such as a band-pass filter such as a low-pass filter, or sharpening processing (super-resolution processing) using deep learning may be used. Since the sharpening processing using deep learning also performs image processing based on the analysis results (characteristics) of the input image, the sharpening effect in this processing is high.

[0037] In addition, in this embodiment, the case where the non-perspective projection image 1 is an equirectangular image has been described, but the present invention is not limited to this, and any projection-type image capable of recording a wide-angle area that cannot be recorded with a perspective projection image may be used. When an image of a wide-angle area is recorded as a two-dimensional image, the image of the wide-angle area is distorted and recorded, so that the full effect of the present technology can be obtained.

[0038] In addition, in this embodiment, the case where the non-perspective projection image 2 is an equirectangular image has been described, but it is not limited to this, and any image of a projection method that records a wide-angle area of ​​an input image may be used, and it does not depend on the input image. Therefore, conversion from a fisheye image to a fisheye image, conversion from a fisheye image to an equirectangular image, conversion from an equirectangular image to an equirectangular image, and conversion from an equirectangular image to a fisheye image are possible. In addition, when the input image is a fisheye image, it is desirable to convert the image without removing lens distortion. This means that even when converting from a fisheye image to a fisheye image, the coordinates of corresponding pixels do not match.

[0039] In addition, in this embodiment, a case where the non-perspective projection image 1 is divided into 70 vertical divisions and 72 horizontal divisions has been described, but the present invention is not limited to this. By reducing the number of divisions, it is possible to realize high-speed processing with reduced processing overhead. Therefore, the number of divisions and the accuracy (internally converted into the number of divisions) may be specified by the user via the UI. Also, the user may select and instruct from a plurality of image processing methods prepared in advance, and the CPU 101 may perform the instructed image processing.

[0040] In addition, in the present embodiment, the conversion to the perspective projection image has been described as being projected pixel by pixel. However, for example, the results of converting the pixel positions of the four corners of the divided region as described above are set as the pixel positions of the four corners of the perspective projection image, and the pixel values ​​of the pixels of the four corners of the divided region are set as the pixel values ​​of the four corners of the perspective projection image. The other pixels in the perspective projection image may be calculated by linearly interpolating the pixel values ​​of the four corners of the perspective projection image. In this embodiment, the image is divided into 70 and 72 parts vertically and horizontally, respectively, which is sufficiently fine, so that linear interpolation causes little error.

[0041] In addition, the non-perspectively projected image 1 may be a monocular fisheye image, a stereo fisheye image, or a 360-degree equirectangular image converted from two or more fisheye images. The term perspective projection in this embodiment refers to a projection method in which all points on a three-dimensional object are radially converged to one viewpoint, and is not limited to strict perspective projection. As long as the points are converged to one viewpoint, the conversion may be to an image with a different aspect ratio or an approximation thereof. In addition, the conversion may be such that the ordinate converges to a horizontal line, or such that the abscissa converges to the vertical direction. For example, by applying a one-dimensional filter in the vertical direction and a one-dimensional filter in the horizontal direction to each converted image, an effect equivalent to that of applying a one-dimensional filter to the perspectively projected image can be obtained.

[0042] [Second embodiment] In the following embodiments, differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. The process performed by the image processing device according to this embodiment will be described with reference to the flowchart in Fig. 4. In Fig. 4, the same process steps as those shown in Fig. 2 are given the same step numbers, and the description of the process steps will be omitted or will be briefly described.

[0043] In step S4015, CPU 101 generates non-perspective projection image 3 of the same size (same number of vertical and horizontal pixels) as non-perspective projection image 1, and initializes the pixel values ​​(RGB values ​​and alpha channel values) of each pixel of non-perspective projection image 3 to 0.

[0044] In step S4020, CPU 101 selects one of the divided regions obtained in step S2010 that has not been selected as a selected divided region, and sets a frame three pixels wide around the selected divided region as a blending region. CPU 101 then sets an image region consisting of the selected divided region and the blending region as an extended selected divided region, and generates an alpha channel map in which the blending ratios of the pixels belonging to the extended selected divided region are registered. The alpha channel map will be described with reference to FIG. 5.

[0045] 5, in this embodiment, the CPU 101 sets a frame three pixels wide as a blending region around the selected divided region 500 having a size of 25 pixels x 25 pixels. The CPU 101 then generates an alpha channel map in which the following blending ratios are registered: "1" for each of the pixels belonging to the selected divided region 500, "0.75" for the blending ratio of the pixel one pixel outside the selected divided region 500, "0.5" for the blending ratio of the pixel two pixels outside the selected divided region 500, and "0.25" for the blending ratio of the pixel three pixels outside the selected divided region 500.

[0046] In step S4025, CPU 101 acquires an image in the extended selected divided region as a selected divided image. That is, each selected partial image acquired in step S4025 has a property that "adjacent selected partial images have overlapping portions." In this embodiment, the following processing may be performed in steps S4020 and S4025.

[0047] That is, in step S4020 for the first time, CPU 101 places a rectangular window of 31 pixels by 31 pixels in the upper left corner of non-perspective projection image 1, and generates an alpha channel map of the image of the image area within the placed rectangular window. Then, in step S4025 for the first time, CPU 101 obtains the image of the image area within the placed rectangular window as a selected divided image.

[0048] In step S4020 for the second time, CPU 101 shifts the previously placed rectangular window 25 pixels to the right and generates an alpha channel map of the image of the image area within the moved rectangular window. Then, in step S4025 for the second time, CPU 101 acquires the image of the image area within the placed rectangular window as a selected divided image.

[0049] In step S4020 for the {72xN (N is an integer equal to or greater than 1)+1} time, CPU 101 moves the rectangular window placed in step S4020 for the (72xN) time by 25 pixels downward and to the left edge of non-perspective projection image 1. Then, in step S4020 for the {72xN (N is an integer equal to or greater than 1)+1} time, CPU 101 generates an alpha channel map of the image of the image area within the moved rectangular window. Then, in step S4025 for the {72xN (N is an integer equal to or greater than 1)+1} time, CPU 101 obtains the image of the image area within the placed rectangular window as a selected divided image.

[0050] Then, CPU 101 assumes that the parameter s used in the above equation (1) is calculated according to the amount of deformation when projecting a non-perspectively projected image onto a perspectively projected image, and obtains s according to the following equations (2) and (3).

[0051]

number

[0052] In step S4030, CPU 101 converts the selected divided image into a perspective projection image according to equation (1) in the same manner as in the first embodiment. In step S4040, CPU 101 performs "convolution integral using a 9×9 Gaussian filter kernel," which is an example of image processing, on the perspective projection image generated in step S4030.

[0053] In step S4050, CPU 101 converts the perspective projection image that has been subjected to image processing in step S4040 into non-perspective projection image 2 by performing a conversion inverse to the conversion (conversion in accordance with equation (1)) performed in step S4030.

[0054] In step S4060, CPU 101 specifies a corresponding area in non-perspective projection image 3 that corresponds to the extended selected divided area, and adds the pixel value of each pixel in the corresponding area to the pixel value of the corresponding pixel in non-perspective projection image 2 generated in step S4050. When adding pixel values, addition of RGB values ​​and addition of alpha channel values ​​are performed.

[0055] In step S2070, CPU 101 determines whether or not all divided regions have been selected as selected divided regions. If the result of this determination is that all divided regions have been selected as selected divided regions, the process proceeds to step S4080. On the other hand, if any divided regions remain that have not yet been selected as selected divided regions, the process proceeds to step S4020.

[0056] In step S4080, CPU 101 obtains a divided value by dividing the RGB value of each pixel in non-perspective projection image 3 by the alpha channel value of that pixel, and updates (normalizes) the pixel value of that pixel to the divided value.

[0057] Depending on the image processing method, the boundary may become clear when a perspective projection image is converted into a non-perspective projection image 2 and recombined. According to this embodiment, the boundary can be made less noticeable by providing an overlapping area when combining and blending. Also, by changing the blending strength according to the amount of deformation when projecting a non-perspective projection image onto a perspective projection image, degradation of image quality due to blending processing can be suppressed.

[0058] [Third embodiment] The process performed by the image processing device according to this embodiment will be described with reference to the flowchart in Fig. 6. In Fig. 6, the same process steps as those shown in Fig. 2 are given the same step numbers, and the description of these process steps will be omitted or will be briefly described. In step S6021, CPU 101 obtains a filter strength σ corresponding to the selected divided region according to the following equations (4) to (6).

[0059]

number

[0060] In step S6022, CPU 101 determines whether filter strength σ is equal to or less than threshold value ε, and if filter strength σ is equal to or less than threshold value, the process proceeds to step S6060, and if filter strength σ is greater than threshold value, the process proceeds to step S2030. In this embodiment, CPU 101 determines whether filter strength σ is 0. If filter strength σ is 0 as a result of this determination, the process proceeds to step S6060, and if filter strength σ is not 0, the process proceeds to step S2030.

[0061] In step S6040, CPU 101 performs, as an example of image processing, a filtering process on the perspective projection image generated in step S2030, in which the perspective projection image is filtered using a 9×9 Gaussian filter with the filter strength σ determined in step S6021.

[0062] In step S6060, CPU 101 identifies a corresponding area in non-perspective projection image 3, which has the same size (the same number of vertical and horizontal pixels) as non-perspective projection image 1, that corresponds to the selected divided area selected in step S2020, and sets the pixel value of each pixel in the corresponding area to the pixel value of the corresponding pixel in the selected divided area that corresponds to the pixel in question.

[0063] In this way, according to this embodiment, the strength of the filter can be changed according to the position from the center of the image while reducing the calculation load such as calculating the filter kernel for each pixel. In this embodiment, it is only necessary to change the filter strength σ as the strength and apply the filter processing to the perspective projection image, and there is no need for complex processing such as changing the aspect ratio of the kernel shape for each position of the selected divided area. In a VR image, the center is more likely to be gazed at, and the degree of gaze decreases as the image height increases. In addition, the visual field range is narrower above and below than to the left and right. By blurring the area that is less likely to be gazed at, the code amount when encoding the image can be effectively reduced according to the visual sensitivity at the time of viewing. Note that in this embodiment, since the amount of deformation is small in the areas with low image heights above and below, a configuration is adopted in which image processing is performed in the original projection space without perspective projection transformation and the images are combined.

[0064] [Fourth embodiment] The process performed by the image processing device according to this embodiment will be described with reference to the flowchart in Fig. 7. In Fig. 7, the same process steps as those shown in Figs. 2 and 6 are given the same step numbers, and the description of these process steps will be omitted or will be briefly described.

[0065] In step S7021, the CPU 101 obtains the filter strength σ corresponding to the selected divided region according to the following equations (7) to (9) (calculating the filter strength σ based on the amount of deformation due to projection of the position of the selected divided region).

[0066]

number

[0067] In this way, according to this embodiment, it is possible to change the filter strength according to the amount of deformation during projection while reducing the computational load of calculating a filter kernel for each pixel. In this embodiment, it is only necessary to change the filter strength σ and apply filtering to the perspective projection image, and there is no need for complex processing such as changing the aspect ratio of the kernel shape for each selected divided region.

[0068] When converting an equirectangular image into a perspective projection image during viewing with an HMD, the image at a high image height is deformed to be horizontally reduced, which tends to cause aliasing. In this embodiment, the larger the amount of deformation, the stronger the low-pass filter applied to the area, which suppresses aliasing.

[0069] [Fifth embodiment] The non-perspective projection image 3 may be incorporated as part of a UI function as necessary and output to a device or file. In this way, various modifications are possible regarding what data the non-perspective projection image 3 is transmitted together with and in what transmission format, and are not limited to a specific modification.

[0070] The steps in the process according to the above flowchart do not necessarily have to be executed in the order shown in the figure. In other words, the order in which some of the process steps are executed may be changed, or some of the process steps may be executed in parallel with other process steps.

[0071] In the above description, the CPU 101 performs all of the processes according to the flowcharts, but some of the processes may be performed by the GPU 105. For example, the GPU 105 may perform some or all of the image-related processes, such as image division, conversion, and image processing, under the control of the CPU 101.

[0072] The process according to the above flow chart may be performed in real time for each frame image that is input sequentially, or may be performed in non-real time for each frame image stored in a memory device.The non-perspectively projected image 3 obtained as a result of the process may be output / transmitted in real time, or may be output / transmitted in non-real time.

[0073] In the above embodiment, the case where the non-perspective projection image 1 is an image of equirectangular projection (equirectangular image) has been described, but the present invention is not limited to this. For example, the non-perspective projection image 1 may be a fisheye image. If the non-perspective projection image 1 is not an equirectangular image, the shape of the divided regions may not be rectangular. Furthermore, the "division of the region" may be achieved by separating the image into a plurality of divided regions, or by dividing the image into a plurality of divided regions such that adjacent divided regions have overlapping portions, and the division method is not limited to a specific division method.

[0074] In addition, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0075] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0076] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0077] The invention of this specification includes the following image processing device, image processing method, and computer program. (Item 1) A conversion means for converting an image in each image region of a non-perspective projection image into a perspective projection image, and converting the perspective projection image into a non-perspective projection image after performing image processing on the perspective projection image. An image processing device comprising: (Item 2) 2. The image processing device according to item 1, characterized in that the conversion means converts the image in the image region into a perspective projection image based on parameters corresponding to pixel positions of pixels in the image region and vertical and horizontal angles of the image region. (Item 3) 3. The image processing device according to item 1 or 2, wherein the conversion means applies a bilateral filter to the perspective projection image. (Item 4) 3. The image processing device according to item 1 or 2, wherein the conversion means applies a band-pass filter to the perspective projection image. (Item 5) 3. The image processing device according to item 1 or 2, wherein the conversion means performs a sharpening process using deep learning on the perspective projection image. (Item 6) 3. The image processing device according to item 1 or 2, wherein the conversion means performs a convolution integral with a Gaussian filter kernel on the perspective projection image. (Item 7) a generating means for generating a combined image based on the non-perspective projection image obtained by the converting means for each of the image regions, the conversion means converts an image in each of adjacent image areas having an overlapping portion into a perspective projection image, performs image processing on the perspective projection image, and converts the image into a non-perspective projection image; The generating means normalizes a pixel value of each pixel of a combined image based on a non-perspective projection image obtained by the converting means for each of the image regions, based on a blending ratio corresponding to the pixel. 7. The image processing device according to any one of items 1 to 6, (Item 8) The image processing device according to any one of items 1 to 7, characterized in that the conversion means performs a filter process with a filter strength corresponding to an image region in the non-perspective projection image on the perspective projection image corresponding to the image region, and then converts the image into the non-perspective projection image. (Item 9) 9. The image processing device according to item 8, wherein the conversion means determines a filter strength based on a position of an image region in the non-perspective projection image. (Item 10) 9. The image processing device according to item 8, wherein the conversion means determines a filter strength based on an amount of deformation caused by projection of a position of an image region in the non-perspective projection image. (Item 11) a generating means for generating a combined image based on the non-perspective projection image obtained by the converting means for each of the image regions, 11. The image processing device according to any one of items 1 to 10, further comprising an output unit that outputs the combined image generated by the generation unit. (Item 12) Item 12. The image processing device according to item 11, wherein the output means outputs the combined image generated by the generation means to an HMD as an image to be displayed. (Item 13) 2. The image processing device according to item 1, wherein the image processing is a plurality of predetermined image processings selected by an instruction from a user. (Item 14) An image processing method performed by an image processing device, comprising: A conversion step in which the conversion means of the image processing device converts the images in each image region in the non-perspective projection image into a perspective projection image, performs image processing on the perspective projection image, and then converts it into a non-perspective projection image. 13. An image processing method comprising: (Item 15) A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 13.

[0078] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0079] 100: Bus 101: CPU 102: Non-volatile memory 103: Memory 104: UI device connection part 105: GPU 106: General-purpose IF 107: NW / IF

Claims

1. an input means for inputting a non-perspective projection image; a conversion means for converting each image in each of a plurality of image regions in the non-perspective projection image into a respective perspective projection image, performing image processing on each of the perspective projection images, and then converting the results of the image processing into a respective non-perspective projection image; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the conversion means converts the respective images in each of the plurality of image areas into the respective perspective projection images based on parameters corresponding to pixel positions in the image areas and vertical and horizontal angles of the image areas.

3. 2. The image processing apparatus according to claim 1, wherein the conversion means applies a bilateral filter to each of the perspective projection images.

4. 2. The image processing apparatus according to claim 1, wherein said conversion means applies a band-pass filter to each of said perspective projection images.

5. The image processing device according to claim 1 , wherein the conversion means performs a sharpening process using deep learning on each of the perspective projection images.

6. 2. The image processing apparatus according to claim 1, wherein the conversion means performs convolution integrals of the respective perspective projection images using a Gaussian filter kernel.

7. generating means for generating a combined image based on the respective non-perspective projection images; the conversion means converts the respective images in each of the plurality of image regions, adjacent image regions having overlapping portions, into the respective perspective projection images, performs image processing on the respective perspective projection images, and then converts the results into the respective non-perspective projection images; The generating means normalizes the pixel value of each pixel of the combined image based on the blending ratio corresponding to the pixel.

2. The image processing device according to claim 1, wherein:

8. 2. The image processing device according to claim 1, wherein the conversion means performs filtering on each of the perspective projection images corresponding to the image regions in the non-perspective projection image at a filter strength corresponding to the image region, and then converts the results into each of the non-perspective projection images.

9. 9. The image processing apparatus according to claim 8, wherein the conversion means determines a filter strength based on the position of each of the image regions in the non-perspective projection image.

10. 9. The image processing apparatus according to claim 8, wherein the conversion means calculates a filter strength based on a deformation amount due to projection of at least one position among the plurality of image regions in the non-perspective projection image.

11. a generating means for generating a combined image based on the non-perspectively projected images obtained by the converting means for the plurality of image regions; an output means for outputting the combined image generated by the generation means; The image processing device according to claim 1 , further comprising:

12. 12. The image processing apparatus according to claim 11, wherein the output means outputs the combined image generated by the generation means to an HMD as an image to be displayed.

13. 2. The image processing apparatus according to claim 1, wherein the image processing is selected from a plurality of predetermined image processing methods in response to an instruction from a user.

14. An image processing method performed by an image processing device, an input step in which an input means of the image processing device inputs a non-perspective projection image; a conversion step in which a conversion means of the image processing device converts each image in each of a plurality of image areas in the non-perspective projection image into a respective perspective projection image, performs image processing on each of the perspective projection images, and then converts the results of the image processing into a respective non-perspective projection image; An image processing method comprising:

15. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 13.

Images