Image display device, image display method, and program

Abstract

To solve the problem in which a viewer feels uncomfortable when the viewer views an image that is synthesized without correcting an image of a two-dimensional model at a distance similar or closer to the height of the viewpoint of a photographing device with respect to an equirectangular projection image.SOLUTION: A PC 50 receives input of a perspective projection image 600 showing an image by perspective projection method combined with an equirectangular projection image 400 obtained by equirectangular projection from a plurality of images captured by a photographing apparatus 10 (step S2001), performs a process of controlling the degree of distortion of the perspective projection image 600 according to distance from the equator of the equirectangular projection image 400 (step S1409), and displays an obtained deformed perspective projection image in the state of combined with the equirectangular projection image 400 on a display 508 (step S1410).SELECTED DRAWING: Figure 29

Description

[Technical field]

[0001] The present disclosure relates to an image display device, an image display method, and a program. [Background technology]

[0002] Special imaging devices that can capture a 360° spherical (panoramic) image (hereinafter simply referred to as a spherical image) in one shot are becoming widespread. In addition, applications that can draw, process, edit, and the like on the spherical image captured by such imaging devices are also known (for example, Patent Document 1).

[0003] The following methods are known as methods for performing the above-mentioned drawing. One is a method of performing drawing while displaying the entire celestial sphere image on a two-dimensional plane using an image (equirectangular projection image) obtained by equirectangular projection. The other is a perspective projection method in which the celestial sphere image is pasted inside a sphere and a rectangular image is created using a virtual camera placed inside the sphere. There is also a method of placing a three-dimensional model on the celestial sphere image. In this way, when a predetermined two-dimensional image (such as a human character) is placed on an image obtained by equirectangular projection, it is assumed that the three-dimensional model is used, or, when a two-dimensional image is used, it is assumed that the two-dimensional image is placed at a location sufficiently far away from the height of the imaging device. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional method, when rendering a celestial sphere image, if the standard latitude is set at 0° (equator) in an equirectangular projection image obtained by equirectangular projection, the image framed by the dotted line expands horizontally near the poles away from the standard latitude, as in the dotted line frame image shown in Fig. 31, resulting in large distortion of the celestial sphere image. This is because the cylindrical projection method directly converts the latitude and longitude into the vertical and horizontal directions of the image, respectively, and the area and angle are incorrect because the image is equidistant with respect to the standard latitude in the vertical direction and expands horizontally away from the standard latitude. In order to project an equirectangular projection image onto a flat image by perspective projection, the equirectangular projection image is pasted inside the sphere, a virtual camera is placed at the center of the sphere, and the image is projected onto a rectangular image as if it were taken from the virtual camera. Here, if you want to further overlay and synthesize a flat image of a two-dimensional model on an equirectangular projection image, you stand the flat image of the two-dimensional model vertically on a virtual ground that is lowered from the center by the height of the shooting device for the equirectangular projection image in a sphere in 3D space, and project it onto a rectangular image so that the equirectangular projection image and the flat image of the two-dimensional model are both captured by a virtual camera. However, this image synthesis method requires processing of the 3D space, which is costly. To reduce the cost of the work, a two-dimensional flat image software application generally overlays and synthesizes the flat image of the two-dimensional model while leaving the equirectangular projection image as it is. However, in order to synthesize the flat image of the two-dimensional model without distortion, the above 3D space processing must be replaced with flat image processing. Therefore, the distance between the flat image of the two-dimensional model that is erected vertically to the horizon in the sphere on which the equirectangular projection image is pasted and the virtual camera must be assumed, and the software application must make corrections according to that distance to reduce the height near the pole and widen the width of the flat image of the two-dimensional model. Due to these characteristics, if an image of a two-dimensional model is synthesized with respect to an equirectangular projection image by using a general two-dimensional flat image software application without correction to a distance that is approximately the same as the height of the viewpoint of the image capturing device or closer than that, a viewer feels an awkwardness in the image when viewing the image of the two-dimensional model superimposed on the spherical image as a projected image onto a flat surface by perspective projection.An image that gives an unnatural impression is one in which, when an image of a two-dimensional model is combined with an equirectangular projection image without correction, the two-dimensional model appears elongated because the necessary conversion of width and height is not performed in the perspective projection image (see the human character images at each distance shown in "No correction" in Figure 29(B) and "No correction" in Figure 29(C)). [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention provides an image display device that displays an equirectangular projection image obtained by equirectangular projection from a plurality of images captured by an image capture device, the image display device including: a perspective projection image receiving means for receiving an input of a perspective projection image indicating an image of a perspective projection method to be synthesized with the equirectangular projection image; and a display control means for displaying on a display means a deformed perspective projection image in which a degree of distortion of the perspective projection image is controlled according to a distance from the equator in the equirectangular projection image, synthesized with the equirectangular projection image. and the display control means displays the deformed perspective projection image, obtained by performing a process to increase the protrusion degree of a protruding portion of an object represented in the perspective projection image, on the display means in a state in which the deformed perspective projection image is synthesized with the equirectangular projection image. An image display device. Effect of the Invention

[0006] According to the present invention, it is possible to effectively display a perspective projection image combined with an equirectangular projection image, as an image that looks as natural as possible. [Brief description of the drawings]

[0007] [Figure 1] 1A is a left side view of the imaging device, (B) is a front view of the imaging device, and (C) is a plan view of the imaging device. [Diagram 2] FIG. 13 is an image of the imaging device in use. [Diagram 3] (A) is a hemispherical image (front) captured by the camera, (B) is a hemispherical image (back) captured by the camera, and (C) is an image represented by equirectangular projection. [Figure 4] (A) A conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (B) a diagram showing a spherical image. [Diagram 5]FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is treated as a stereoscopic sphere in a three-dimensional virtual space. [Figure 6] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 7] FIG. 2 is a diagram illustrating a data structure of image data in a spherical image format. [Figure 8] FIG. 1 is a conceptual diagram illustrating an example of a grid drawing in a three-dimensional Euclidean space. [Figure 9] FIG. 13 is a diagram showing an example of grid distortion in an equirectangular projection image. [Figure 10] 1 is a conceptual diagram showing an example of the relationship between distance and coordinates of a virtual camera, a 2D image, and a mesh. [Figure 11] FIG. 1 is a conceptual diagram showing an example of transformation of a 2D image into an equirectangular projection and unevenness correction (YZ plane). [Figure 12] FIG. 1 is a conceptual diagram showing an example of transformation of a 2D image into an equirectangular projection and unevenness correction (XZ plane). [Figure 13] FIG. 13 is a conceptual diagram showing an example of lateral (φ-axis) movement of a 2D image after unevenness correction in an equirectangular projection. [Figure 14] 1 is a schematic diagram illustrating an example of an image display system according to an embodiment. [Figure 15] FIG. 2 is a diagram illustrating an example of a hardware configuration of an imaging apparatus according to an embodiment. [Figure 16] FIG. 2 is a diagram illustrating an example of a hardware configuration of a PC according to an embodiment. [Figure 17] FIG. 1 is a diagram illustrating an example of a functional configuration of an image display system according to an embodiment. [Figure 18] 4A to 4C are conceptual diagrams illustrating examples of various management tables according to the embodiment. [Figure 19] 11 is a sequence diagram showing an example of a grid setting process and a display image rotation process in a PC according to an embodiment. FIG. [Figure 20] FIG. 4 is a diagram showing an example of a drawing input screen displayed on a PC according to the embodiment. [Figure 21]13 is a flowchart showing an example of a grid display process in a PC according to the embodiment. [Figure 22] 13 is a flowchart showing an example of a generation process of a grid image in a PC according to the embodiment. [Figure 23] 11 is a flowchart showing an example of a display image rotation process in a PC according to an embodiment. [Figure 24] 13 is a flowchart illustrating an example of a drawing process for a spherical image in a PC according to the embodiment. [Diagram 25] FIG. 11 is a sequence diagram showing an example of a 2D image transformation process in a PC according to an embodiment. [Figure 26] FIG. 11 is a diagram showing an example of a 2D image editing screen displayed on a PC according to the embodiment. [Figure 27] 13 is a flowchart showing an example of 2D image transformation processing in a PC according to the embodiment. [Figure 28] 11A and 11B are diagrams illustrating an example of a difference in equirectangular projection between with and without correction according to the embodiment. [Figure 29] 1A and 1B are diagrams illustrating an example of a difference in perspective projection between with and without correction according to an embodiment. [Diagram 30] 11A and 11B are diagrams illustrating an example of a difference in perspective projection caused by the presence or absence of correction of a protrusion according to the embodiment. [Diagram 31] FIG. 1 is a diagram showing an example of a grid display in a conventional equirectangular projection. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] [Overview of the embodiment] <How to generate spherical images> First, the appearance of the photographing device 10 will be described with reference to Fig. 1. The photographing device 10 is a spherical camera for obtaining photographed images that are the basis of a spherical (360°) panoramic image. Fig. 1(A) is a left side view of the photographing device, Fig. 1(B) is a front view of the photographing device, and Fig. 1(C) is a plan view of the photographing device. The photographing device 10 may be any type of camera, including a digital camera, as long as it is capable of photographing a spherical image.

[0010] As shown in Fig. 1(A), the photographing device 10 is sized so that a person can hold it in one hand. Also, as shown in Figs. 1(A), 1(B), and 1(C), an image sensor 103a is provided on the front side (front side) and an image sensor 103b is provided on the back side (rear side) of the upper part of the photographing device 10. These image sensors (image sensors) 103a and 103b are used in conjunction with optical members (for example, lenses 102a and 102b described later) capable of photographing a hemispherical image (angle of view of 180° or more). Also, as shown in Fig. 1(B), an operation unit 115 such as a shutter button is provided on the surface opposite to the front side of the photographing device 10.

[0011] Next, the usage of the photographing device 10 will be described with reference to Fig. 2. Fig. 2 is an image diagram of the photographing device in use. As shown in Fig. 2, the photographing device 10 is used, for example, by a user holding it in his / her hand to photograph subjects around the user. In this case, two hemispherical images can be obtained by photographing subjects around the user using the image sensor 103a and the image sensor 103b shown in Fig. 1, respectively.

[0012] Next, an outline of the process of creating a celestial sphere image from an image captured by the image capture device 10 will be described with reference to Fig. 3 and Fig. 4. Fig. 3(A) is a diagram showing a hemispherical image (front side) captured by the image capture device, Fig. 3(B) is a diagram showing a hemispherical image (rear side) captured by the image capture device, and Fig. 3(C) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 4(A) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 4(B) is a diagram showing a celestial sphere image.

[0013] As shown in Fig. 3(A), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by the lens 102a described below. Also, as shown in Fig. 3(B), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by the lens 102b described below. Then, the photographing device 10 synthesizes the hemispherical image (front side) and the hemispherical image (rear side) flipped by 180° to create an equirectangular projection image EC as shown in Fig. 3(C).

[0014] Then, the photographing device 10 uses OpenGL ES (Open Graphics Library for Embedded Systems) to paste an equirectangular projection image EC so as to cover the sphere as shown in FIG. 4A, and creates a celestial sphere image CE as shown in FIG. 4B. In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize two-dimensional (hereinafter also referred to as "2D" (2-Dimensions)) and three-dimensional (hereinafter also referred to as "3D" (3-Dimensions)). Also, the celestial sphere image CE may be a still image or a video.

[0015] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people when they see it. Therefore, the imaging device 10 can display an image of a predetermined region T (hereinafter, referred to as a "predetermined region image") that is a part of the spherical image CE as a planar image with little curvature, thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to Figs. 5 and 6.

[0016] Fig. 5 is a diagram showing the positions of a virtual camera and a predetermined area when the omnidirectional image is a three-dimensional sphere in a three-dimensional virtual space. The virtual camera IC corresponds to the position of the viewpoint of the final viewer (hereinafter simply referred to as the viewer) who views the omnidirectional image CE displayed as a three-dimensional sphere. In this embodiment, in Fig. 5, the virtual camera IC is described in a state in which it is placed inside the three-dimensional sphere in order to make the diagram closer to actual processing. Note that the virtual camera IC is an example of an imaging device arranged inside the three-dimensional sphere when the omnidirectional image is a three-dimensional sphere in a three-dimensional virtual space.

[0017] In FIG. 5, the omnidirectional image CE shown in FIG. 4 is represented by a three-dimensional solid sphere CS. A predetermined area T in the omnidirectional image CE is a shooting area of ​​a virtual camera IC, and is specified by predetermined area information indicating a shooting direction and a viewing angle of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE. A predetermined area image Q, which is an image of the predetermined area T, is displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC. The predetermined area image Q is represented by predetermined area information (display parameters) that is initially set (default setting). In the following, the shooting direction (ea, aa) and the viewing angle (α) of the virtual camera IC will be used for explanation. Note that the predetermined area T may be represented by the shooting area (x, y, z) of the virtual camera IC, which is the predetermined area T, instead of the viewing angle α and the distance f from the virtual camera IC to the center point CP.

[0018] FIG. 6 is a diagram showing points in a three-dimensional Euclidean space in spherical coordinates. Here, the position coordinates of the center point CP when expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius, polar angle, and azimuth angle, respectively. The radius r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to the center point CP, and is therefore equal to the distance f. FIG. 6 is a diagram showing these relationships. Hereafter, the explanation will be given using the position coordinates (r, θ, φ) of the virtual camera IC.

[0019] Fig. 7 is a diagram showing a data structure of image data in a celestial sphere image format. As shown in Fig. 7(A), image data in the celestial sphere image format is expressed as an array of pixel values ​​whose coordinates are a vertical angle θ made with respect to a predetermined axis and a horizontal angle φ corresponding to the rotation angle around the above-mentioned predetermined axis. Here, the celestial sphere image is expressed in a coordinate system consisting of a vertical angle (latitude in latitude-longitude coordinates) with the zenith direction of the captured scene as the axis, and a horizontal angle (longitude in latitude-longitude coordinates) around the axis of the zenith direction, as shown in Fig. 6. The vertical angle θ is in the range of -90° to 90°, and the horizontal angle φ is in the range of -180° to 180°.

[0020] Each coordinate value (θ,φ) of the omnidirectional format is associated with each point on a sphere representing all directions centered on the shooting point, as shown in Fig. 7(B), and the all directions are associated with the omnidirectional image. The planar coordinates of the image captured by the fisheye lens and the coordinates on the sphere of the omnidirectional image format are associated with each other by applying the above-mentioned projection function and appropriate coordinate conversion.

[0021] In the following description, the two-dimensional coordinates of the omnidirectional image format shown in FIG. 7(A) are a coordinate system with the lower left origin, as shown by the gray axis in FIG. 7(A). In the following description, the two-dimensional coordinates are converted into a horizontal angle value x ranging from 0 to the number of pixels in the horizontal direction, and a vertical angle value y ranging from 0 to the number of pixels in the vertical direction, and are handled. For example, when pixels are configured in increments of 0.1°, the horizontal angle value x ranges from 0 to 3799, and the vertical angle value y ranges from 0 to 1800. The relationship between the horizontal angle value x and the horizontal angle φ, and the relationship between the vertical angle value y and the vertical angle θ are expressed by the following (Equation 1) to (Equation 7). In the following (Equation 1) to (Equation 7), w and h correspond to the image width (e.g., 3600 pixels) and image length (e.g., 1801 pixels) of the omnidirectional image format, respectively.

[0022] The relationship between the two-dimensional coordinates (x, y) of the celestial sphere image shown in Fig. 7(A) and the three-dimensional coordinates (xs, ys, zs) of the surface of the celestial sphere shown in Fig. 7(B) is also expressed by the following (Equation 1) to (Equation 7). The three-dimensional coordinates shown in Fig. 7(B) are a right-handed system, with the center of the sphere as the origin and r representing the radius. Note that the right-handed system (positive system) refers to a coordinate system in which the thumb is the x-axis, the index finger is the y-axis, and the middle finger is the z-axis when the thumb, index finger, and middle finger of the right hand are bent so as to be perpendicular to each other in three-dimensional space.

[0023]

number

[0024]

number

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029]

number

[0030] In the embodiment described below, the relationship between the two images (captured image A and captured image B) captured by the two imaging optical systems (two imaging elements described later) and the image regions on the omnidirectional image is the corresponding relationship shown in "omnidirectional image" in FIG. 7. The composite omnidirectional image is stored in a built-in memory or an external recording medium in a predetermined format. The file format in which the omnidirectional image is stored may be an uncompressed still image such as a bitmap, or a compressed still image such as JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), or PNG (Portable Network Graphics). In still another embodiment, the omnidirectional image may be included as a frame image in a moving image such as MPEG (Moving Picture Experts Group) or AVI (Audio Video Interleave). Note that in the embodiment described below, the omnidirectional image is described as a still image.

[0031] Next, the grid image shown in FIG. 8 will be explained briefly. FIG. 8 is a conceptual diagram showing an example of grid drawing in a three-dimensional Euclidean space. FIG. 8 shows a grid coordinate arrangement in a three-dimensional Euclidean space of a grid image generated based on a grid setting set by an editor (hereinafter, referred to as an editor) who sets and edits various images, among users of a PC 50 described later. The unit of coordinates is, for example, cm (centimeter), m (meter), etc., and can be appropriately changed. For example, when the photographing device 10 is installed at a position 10 cm above the floor surface, the grid setting unit 58 sets the value of the height H to 10.

[0032] Also, each point for drawing the grid is assumed to have grid coordinates (x, y, z). FIG. 8 shows an example of a grid with height 10 (meaning that the grid is placed at position 10 on the Y axis from the origin in the three-dimensional Euclidean space) and grid width 150, divided into six parts. The values ​​of each setting item included in the setting information can be set using a grid setting area on the drawing input screen, which will be described later. An example of processing by the grid ruler processing unit, which will be described later, is shown below. First, the grid ruler processing unit calculates the coordinates of each point obtained by dividing grid coordinate 1 (-75, 30, -75) to grid coordinate 2 (75, 30, -75) into six parts in the X-axis direction. Next, the grid ruler processing unit calculates the coordinates of each point obtained by dividing grid coordinate 2 (75, 30, -75) to grid coordinate 3 (75, 30, 75) into six parts in the Z-axis direction. For example, grid coordinate 5 is (25, 30, -25).

[0033] The grid ruler processing unit converts these grid coordinates into a spherical polar coordinate system to obtain the equirectangular projection coordinates (θ,φ), and generates a grid image to be displayed at the obtained coordinates (θ,φ). Here, the grid coordinates (x,y,z) are converted to the coordinates (θ,φ) using the following (Equation 8) and (Equation 9). The generated grid image is then displayed superimposed on the equirectangular projection image. Note that tan -1 The notation x is the inverse function of tan x, and for convenience it is also written as arctan x.

[0034]

number

[0035]

number

[0036] In addition, in the process described later, when the display image is rotated around the Z axis or the Y axis, the rotation processing unit converts the grid coordinates (x, y, z) into grid coordinates (x'' ,y'',z'') and then converted to CP(r,θ,φ) to obtain the equirectangular projection coordinates (θ,φ). The display control unit then displays a grid image of the obtained coordinates (θ,φ) superimposed on the equirectangular projection image. Note that while FIG. 8 shows an example in which a grid image is displayed on a surface parallel to the X-axis and Z-axis, grid setting may be performed such that a grid image is displayed on a surface parallel to the Y-axis and Z-axis or a surface parallel to the X-axis and Y-axis.

[0037] Fig. 9 is a diagram showing an example of grid distortion in an equirectangular projection image. Fig. 9 shows grid lines and their spacing expressed by equirectangular projection, in which the spacing and distortion of the grid lines increases toward the poles (90.00 or -90.00) with the equator at 0.00 as the center.

[0038] Fig. 10 is a conceptual diagram showing an example of the relationship between the distance and coordinates of the virtual camera, 2D image, and mesh. Fig. 10(A) is a conceptual diagram showing the relationship between the distance of the virtual camera IC, the virtual floor, the virtual platform, and the 2D image in Euclidean space XYZ coordinates.

[0039] In the 2D image deformation process in this embodiment, a process is performed to deform a 2D image into an image that does not look unnatural when synthesized on a perspective projection image based on the height of the virtual camera IC150 shown in FIG. 10(A), the distance Zi from the virtual camera IC150 of the 2D image, the length Yp of the human character (object) 153 (hereinafter simply referred to as the object) including the height ys of the virtual platform 152, and the size (protrusion degree) Yk of the protruding part of the object 153. In the 2D image deformation process, after the height Yic of the virtual camera IC150 is set by an editor, the 2D image is automatically deformed by moving it on a shadow image in equirectangular projection based on the distance Zi between the virtual camera IC150 and the 2D image, the height Ys of the virtual platform 152, and the angle of the φ axis described later. Note that the relationship between the length Yp of the object 153 and the distance Zi may be either an absolute value or a relative value.

[0040] Various settings made by an editor in the embodiment will be described below. First, a virtual floor surface 151 is assumed for the virtual camera IC 150 in Fig. 5, and a distance (height) Yic between the virtual camera IC 150 and the virtual floor surface 151 is set. This virtual floor surface 151 is a floor surface virtually assumed for arranging a 2D image.

[0041] 10A, the position of virtual camera IC 150 on the Z axis is set to 0, and the height of virtual floor surface 151 is indicated by a negative value. For example, if the height of virtual camera IC 150 is 150 cm, virtual camera IC 150 has Y=0, Yic=-150.

[0042] Next, the editor sets the length Yp of the object 153 to be composited into the spherical image (if the object is a human character, the height of the object).

[0043] Furthermore, the editor sets the height ys of the virtual platform, which indicates the distance between the object and the virtual floor surface 151. For example, if the object is on the ground on the virtual floor surface 151, the editor sets 0, and if the object is on a 30 cm platform, the editor inputs 30. Here, in the drawing input screen of FIG. 20 described later, when inputting the distance and the virtual platform value, the editor can also change the inputted value interactively by pressing the "2D image" button in the captured image (equirectangular projection) drawing area displayed in equirectangular projection and operating the drag, wheel, etc.

[0044] 10B is a conceptual diagram showing the relationship between the variables (xi, yi), Xp, Yp, yk, and xj of an image in a 2D image and an equation for transforming a 2D image, which will be described later, in the XY coordinates of the 2D image. Here, the length of an object 153, an example of which is a human character 154, is Yp, the width of the object is Xp, the number of pixels of the width of the 2D image is Xx, the number of pixels of the height is Yx, and the XY coordinates of the human character corresponding to the mesh are (xj, yk). In this case, the variables xi and yi are expressed by the following (Equation 10) to (Equation 13).

[0045]

number

[0046]

number

[0047]

number

[0048]

number

[0049] FIG. 10C is a conceptual diagram showing mesh data for providing information on the unevenness of an object in the XY coordinates of a mesh showing the unevenness of the object. Mesh data 155 is data for providing unevenness information to a 2D image. As an example of mesh data 155, the unevenness of the skirt part of the human character 154 shown in FIG. 10B is expressed as three-dimensional vertex coordinates. In FIG. 10C, the distance in the d-axis direction (distance from the origin to the d-axis direction) di of the skirt showing the mesh shape is set as a protrusion 156. When an editor sets an arbitrary value for the mesh shape using a "mesh" setting button described later, the magnitude of the unevenness (also called the degree of protrusion) of the XY coordinates (xj, yk) of the human character 154 shown in FIG. 10B is expressed as di. In (Equation 10) to (Equation 13), (xj, yk) is converted to (xi, yi), and the magnitude of the unevenness (degree of protrusion) of the (xi, yi) coordinates is expressed as di.

[0050] FIG. 11 is a conceptual diagram showing an example of transformation into equirectangular projection of a 2D image and unevenness correction (YZ plane). FIG. 11(A) is a conceptual diagram showing transformation into equirectangular projection and unevenness correction in YZ coordinates showing the positional relationship between the virtual camera IC and an object included in a 2D image in Euclidean space. In FIG. 11(A), the horizontal distance from the virtual camera IC 150 to the object is zi, and the height from the position of the virtual camera IC 150 to the mesh (protrusion 156) is yi. Furthermore, as described in FIG. 10(C), the size of the unevenness of the mesh (hereinafter referred to as the protrusion degree) is di. Furthermore, the latitude θ before the unevenness correction is θi, and the latitude θ after the unevenness correction is θi'.

[0051] FIG. 11(B) is a conceptual diagram showing images 161 before and 162 after unevenness correction of an object included in a 2D image expressed on an equirectangular projection. As an example, mesh 1 and mesh 2 are set as two meshes in the 2D image. In this case, if the unevenness protrusion degree of yi of mesh 1 is di, yi in Euclidean space (YZ coordinates) moves to yi' (corrected yi) after correction. Therefore, latitude θi as seen from the virtual camera IC becomes θi', and latitude θi in the figure before unevenness correction becomes θi'. In FIG. 11(B), since the protrusion degree; di protrudes toward the direction of the virtual camera IC, the latitude θ increases in angle from θ to θ', and the object is corrected so as to extend downward when viewed vertically from the line of sight of the virtual camera IC. In addition, multiple meshes are set in one 2D image. As an example, if mesh 2 exists, it is processed in the same manner. From the above, when the parameters of xi on the 2D image are also taken into consideration, the relationship between θi before unevenness correction 161 and θi′ after unevenness correction 162 is expressed by the following (Equation 14) and (Equation 15).

[0052]

number

[0053]

number

[0054] Fig. 12 is a conceptual diagram showing an example of transformation of a 2D image into equirectangular projection and unevenness correction (XZ plane). Fig. 12(A) is a conceptual diagram showing transformation of an object into equirectangular projection and unevenness correction in the virtual camera IC in Euclidean space and the XZ coordinates of a 2D image. In Fig. 12(A), the horizontal distance from the virtual camera IC 150 to the object is distance zi, the center of the protrusion of the object is x=0, and the position of the object 153 on the X axis is xi. In addition, the corrected value of xi is xi'.

[0055] φc is an angle of rotation in the longitude φ direction around (x, z)=(0, 0), that is, the Y axis. Details of the movement process of the object 153 in the φ axis direction will be described with reference to FIG.

[0056] FIG. 12(B) is a conceptual diagram showing the state before and after the unevenness correction in a 2D image on an equirectangular projection. In FIG. 12(B), as an example, two meshes, mesh 1 and mesh 2, are set for an object included in the 2D image. In this case, if the distance to the protruding portion 156 of xi of mesh 1 is di, xi in Euclidean space (XZ coordinates) moves to xi' (xi after correction) after correction. Therefore, the longitude φi seen from the virtual camera IC becomes φi' in the 2D image after unevenness correction 172. In FIG. 12(B), since the protruding portion di in the object protrudes toward the virtual camera IC, the longitude φ is corrected so that the angle becomes large and it spreads left and right when viewed horizontally. Note that multiple meshes may be set for one 2D image. Therefore, if mesh 2 exists, it is processed in the same way as mesh 1. The projection process relating to φi before unevenness correction 171 and φi′ after unevenness correction 172 of the object shown in FIG. 12B is expressed by the following (Equation 16) and (Equation 17).

[0057]

number

[0058]

number

[0059] FIG. 13 is a conceptual diagram showing an example of horizontal (φ-axis) movement of a 2D image after unevenness correction on an equirectangular projection. In the horizontal (φ-axis) movement in the equirectangular projection (θφ coordinate axis) shown in FIG. 13, an object included in the 2D image is merely slid horizontally, and is not deformed. In this case, the object is moved by a longitude φc, and projected to a position indicated as "2D image after movement." FIG. 13 shows a state in which the 2D image (object) after unevenness correction and movement is moved by a longitude φc when viewed on the XY plane, and moved to a position indicated as "2D image after movement." Here, a post-deformation 2D image 181 showing the 2D image after deformation, and a post-movement 2D image 182 showing the 2D image after movement are shown in FIG. 13.

[0060] As shown in FIG. 13, the projection process for correcting the unevenness of the object 153 and moving it by the longitude φc to the longitude φi″ is expressed by the following (Equation 18).

[0061]

number

[0062] <Outline of image display system> Next, an outline of the configuration of the image display system according to the present embodiment will be described with reference to Fig. 14. Fig. 14 is a schematic diagram showing an example of the image display system according to the embodiment. The image display system 1 shown in Fig. 14 is a system capable of inputting a predetermined drawn image to a spherical image, which is a captured image acquired by a photographing device 10.

[0063] As shown in Fig. 14, the image display system 1 of this embodiment is composed of a photographing device 10 and a PC (Personal Computer) 50. The photographing device 10 and the PC 50 are communicatively connected via a communication network 9. The communication network 9 is a communication network using the Internet or a LAN (Local Area Network), etc. Note that the communication network 9 may be a communication network in which the Internet and an intranet are connected via a firewall. The communication network 9 may be connected in either a wireless or wired manner.

[0064] As described above, the image capturing device 10 is a special camera for capturing images of subjects, scenery, and the like, and obtaining two hemispherical images that are the basis of a celestial sphere image.

[0065] The PC 50 is an example of an image display device that displays an image (still image or video) acquired from the photographing device 10. The PC 50 acquires a spherical image, which is an image captured by the photographing device 10, via the communication network 9. A specific application is installed in the PC 50, and a user of the PC 50 can perform drawing on the spherical image acquired from the photographing device 10. The PC 50 may be, for example, a mobile phone or a tablet terminal.

[0066] 14 is merely an example, and other examples may be used. For example, the photographing device 10 and the PC 50 may be directly connected to each other without going through the communication network 9, using short-distance wireless communication or a USB (Universal Serial Bus) cable or the like. The photographing device 10 may be configured to communicate with the PC 50 by connecting to the communication network 9 via a communication terminal having a router function, such as a smartphone. Furthermore, the image display system 1 may be configured to include a server device capable of communicating with the photographing device 10 and the PC 50 via the communication network 9. In this case, the server device cooperates with the PC 50 to process a predetermined application.

[0067] <Hardware configuration of the embodiment> Next, the hardware configuration of each device or terminal constituting the image display system 1 will be described with reference to Fig. 15 and Fig. 16. Note that components may be added or deleted from the hardware configurations shown in Fig. 15 and Fig. 16 as necessary.

[0068] <Hardware configuration of the imaging device> First, the hardware configuration of the imaging device 10 will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of the hardware configuration of the imaging device according to the embodiment. In the following, the imaging device 10 is an omnidirectional (all-directional) imaging device using two imaging elements (1) and (2), but any number of imaging elements may be used as long as it is two or more. In addition, the imaging device does not necessarily have to be a device dedicated to omnidirectional imaging, and an omnidirectional imaging unit may be attached to a normal digital camera, smartphone, or the like so as to have substantially the same functions as the imaging device 10.

[0069] As shown in FIG. 15, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a network I / F 119, a gyro sensor 120, an acceleration sensor 121 and a terminal 122. It is composed of:

[0070] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter, when there is no need to distinguish between them, they will be referred to as lenses 102) each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to the lenses. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters required for the operation of the imaging elements are set.

[0071] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an electronic compass 118, a network I / F 119, a gyro sensor 120, an acceleration sensor 121, a terminal 122, and the like.

[0072] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs a predetermined processing on each image data, and then synthesizes the image data to generate data for an equirectangular projection image EC as shown in Figure 3(C).

[0073] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0074] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function corresponding to video display on a display (for example, the display 508 of the PC 50). In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / minute).

[0075] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. Note that, in this embodiment, the imaging device 10 is not provided with a display unit, but may be provided with a display unit.

[0076] The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via the I / F bus, and performs a predetermined process on the sound data.

[0077] The CPU 111 controls the overall operation of the photographing device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104, and data of a processed equirectangular projection image.

[0078] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, etc. An editor operates the operation unit 115 to input various shooting modes, shooting conditions, etc.

[0079] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as necessary.

[0080] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi (registered trademark) via an antenna 117a provided in the photographing device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0081] The electronic compass 118 calculates the direction and tilt (Roll rotation angle) of the image capturing device 10 from the Earth's magnetism, and outputs the direction and tilt information. This direction and tilt information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data such as the capture date and time of the image and the data size of the image data.

[0082] The network I / F 119 is an interface for performing data communication using a communication network 9 such as the Internet via a router or the like.

[0083] The gyro sensor 120 is a sensor that detects changes in angles (Roll angle, Pitch angle, Yaw angle) accompanying the movement of the imaging device 10. The change in angle is an example of related information (metadata) conforming to Exif and is used for image processing such as image correction of the captured image.

[0084] The acceleration sensor 121 is a sensor that detects accelerations in three axial directions. The imaging device 10 calculates the attitude (angle with respect to the gravitational direction) of its own device (imaging device 10) based on the accelerations detected by the acceleration sensor 121. Furthermore, by providing both the gyro sensor 120 and the acceleration sensor 121 in the imaging device 10, the accuracy of image correction is improved.

[0085] As the terminal 122, for example, a concave terminal for Micro-USB is used.

[0086] <Hardware Configuration of the PC> Next, the hardware configuration of the PC 50 will be described with reference to FIG. 16. FIG. 16 is a diagram showing an example of the hardware configuration of the PC according to the embodiment. Note that the PC 50 is constructed by a general computer. The PC 50 includes a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD (Hard Disk Drive) 505, a media drive 507, and a display 508.

[0087] The CPU 501 controls the overall operation of the PC 50. The CPU 501 is a calculation device that realizes each function of the PC 50 by reading programs or data stored in the ROM 502 or the like onto the RAM 503 and executing processing. The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HDD 505 controls the reading or writing of various data from the HD 504 under the control of the CPU 501. The HD 504 stores various data such as programs. The media drive 507 controls the reading or writing (storage) of data from a recording medium 506 such as a flash memory. The display 508 displays various information such as a cursor, a menu, a window, characters, or an image.

[0088] The PC 50 also includes a network I / F 509, a keyboard 511, a mouse 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, an audio input / output I / F 515, a microphone 516, a speaker 517, an external device connection I / F 518, and a short-range communication circuit 519.

[0089] The network I / F 509 is an interface for data communication using the communication network 9. The keyboard 511 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The mouse 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls the reading of various data from a DVD-RW 513 as an example of a removable recording medium. It is to be noted that the DVD-RW drive 514 is not limited to a DVD-RW, and may be a DVD-R or the like. The DVD-RW drive 514 may be a Blu-ray drive that controls the reading or writing of various data from a Blu-ray disc, or a CD-RW drive that controls the reading or writing of various data from a CD-RW (Compact Disc-ReWritable) or the like.

[0090] The microphone 516 is a type of built-in sound collecting means for inputting voice. The sound input / output I / F 515 is a circuit for processing input / output of sound signals between the microphone 516 and the speaker 517 under the control of the CPU 501. The external device connection I / F 518 is an interface for connecting various external devices. The external device connection I / F 518 has a function as a contact sensor for detecting, for example, the touch pen 520 or the user's hand touching the display 508. In addition, the external device connection I / F 518 inputs and detects coordinates corresponding to the position on the display 508 where the touch pen 520 touches, by functioning as a contact sensor. The short-range communication circuit 519 is a communication circuit for enabling communication with an external device by a short-range wireless communication technology such as NFC, Bluetooth, or Wi-Fi. The touch pen 520 is an example of an external input means for performing a predetermined drawing on the display 508.

[0091] 16. The PC 50 also includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 501 shown in FIG.

[0092] Each of the above-mentioned programs may be recorded in a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD, a Blu-ray disc, and an SD card. The recording medium may be provided domestically or internationally as a program product. For example, the PC 50 realizes the image display method according to the present invention by executing the program according to the present invention.

[0093] <Functional configuration of the embodiment> Next, the functional configuration of the photographing device 10 and the PC 50 which constitute a part of the image display system 1 according to the present embodiment will be described with reference to Fig. 17. Fig. 17 is a diagram showing an example of the functional configuration of the image display system according to the embodiment.

[0094] <Functional configuration of the imaging device> First, the functional configuration of the photographing device 10 will be described with reference to Fig. 17. The photographing device 10 has a transmitting / receiving unit 11, a receiving unit 12, a communication unit 13, a sound collecting unit 14, an imaging unit 15, and a memory / reading unit 19. Each of these units is a function or means realized by operating any of the components shown in Fig. 17 in response to an instruction from the CPU 111 in accordance with a program for the photographing device loaded from the SRAM 113 onto the DRAM 114. The photographing device 10 also has a storage unit 1000 constructed by the ROM 112, SRAM 113, and DRAM 114 shown in Fig. 15.

[0095] 15 and the function of transmitting and receiving various data or information to and from other devices such as the PC 50 via the communication network 9. The transmitting and receiving unit 11 transmits, for example, photographed image data acquired by the imaging unit 15 to the PC 50 via the communication network 9.

[0096] The reception unit 12 is mainly realized by the processing of the operation unit 115 and the CPU 111 shown in FIG.

[0097] The communication unit 13 is mainly realized by the input / output I / F 116 or the short-range communication circuit 117 shown in Fig. 15, and processing by the CPU 111, and has a function of communicating with the communication unit 55 of the PC 50 using a short-range wireless communication technology or various cables, etc. The communication unit 13 transmits, for example, photographed image data acquired by the imaging unit 15 to the communication unit 55 of the PC 50. The imaging unit 15 is an example of an imaging means.

[0098] The sound collection unit 14 is mainly realized by the microphone 108, the sound processing unit 109 shown in FIG. 15, and the processing of the CPU 111, and has the function of collecting the sounds around the imaging device 10. The imaging unit 15 is mainly realized by the imaging unit 101, the image processing unit 104, the imaging control unit 105 shown in FIG. 15, and the processing of the CPU 111, and has the function of imaging a subject such as a landscape and obtaining captured image data.

[0099] The storage / reading unit 19 is mainly realized by the processing of the CPU 111 shown in FIG. 15, and has the function of storing various data (or information) in the storage unit 1000 and reading out various data (or information) from the storage unit 1000.

[0100] <Functional Configuration of PC> Similarly, with reference to FIG. 17, the functional configuration of the PC 50 will be described. The PC 50 includes a transmission / reception unit 51, a reception unit 52, a display control unit 53, a determination unit 54, a communication unit 55, a captured image processing unit 56, a captured image memory unit 57, a grid setting unit 58, a rotation processing unit 61, a grid / ruler processing unit 62, a grid / ruler memory unit 63, a perspective projection image processing unit 64, a camera height memory unit 65, a 2D image memory unit 66, a mesh memory unit 67, a 2D image deformation processing unit 68, and a storage / reading unit 59. Each of these units is a function or means realized by any of the components shown in FIG. 16 operating according to an instruction from the CPU 501 according to the program for the PC 50 expanded from the HD 504 onto the RAM 503. Also, the PC 50 has a storage unit 5000 constructed by the ROM 502, the RAM 503, and the HD 504 shown in FIG. 16.

[0101] The transmission / reception unit 51 is mainly realized by the network I / F 509 and the processing of the CPU 501 shown in FIG. 16, and has the function of transmitting and receiving various data or information to and from other devices via the communication network 9. The transmission / reception unit 51 receives (acquires), for example, the captured image data transmitted from the imaging device 10 via the communication network 9.

[0102] The reception unit 52 is mainly realized by the processing of the keyboard 511, the mouse 512, the touch pen 520, and the CPU 501 shown in FIG. 16, and has a function of receiving various selections or inputs from a user (editor). The reception unit 52 receives, for example, a setting of a grid for identifying a relative position of a subject shown in the spherical image by using a drawing input screen 200 described later. The grid is also referred to as, for example, a ruled line, a square, an auxiliary line, a guide, or a grid line. The reception unit 52 also receives, for example, a predetermined drawing input for the spherical image displayed on the display 508. The reception unit 52 is an example of a receiving means.

[0103] The display control unit 53 is mainly realized by the processing of the CPU 501 shown in Fig. 16, and has a function of displaying various screens on the display 508 of the PC 50. The display control unit 53 displays, for example, a drawing input screen 200 described later on the display 508. The display control unit 53 also displays, on the display 508, a celestial sphere image which is a captured image related to captured image data acquired from the photographing device 10. The display control unit 53 is an example of a display control means. The determination unit 54 is mainly realized by the processing of the CPU 501 shown in Fig. 16, and has a function of making various determinations.

[0104] 16 and the processing of the CPU 501, and has a function of communicating with the communication unit 13 of the photographing device 10 using a short-distance wireless communication technology or various cables, etc. The communication unit 55 receives (acquires) photographed image data transmitted from the communication unit 13 of the photographing device 10, for example.

[0105] The captured image processing unit 56 is mainly realized by the processing of the CPU 501 shown in Fig. 16, and has a function of performing image processing on the spherical image displayed on the display 508. The captured image processing unit 56 receives, for example, a predetermined drawing input for the spherical image displayed on the display 508. The captured image processing unit 56 is an example of a drawing unit.

[0106] The captured image memory unit 57 is mainly realized by processing of the ROM 502, the RAM 503, and the CPU 501 shown in FIG. 16, and has a function of temporarily storing the spherical image displayed on the display 508 in order for the captured image processing unit 56 to execute image processing.

[0107] 16, and has a function of setting a grid for identifying a relative position of a subject shown in a spherical image. Furthermore, the grid setting unit 58 stores grid setting values, which will be described later, in a grid setting management DB 5001 of the storage unit 5000 constructed by the ROM 502, the RAM 503, and the HD 504.

[0108] The rotation processing unit 61 is mainly realized by the processing of the CPU 501 shown in Fig. 16, and has a function of rotating the display of the display image displayed on the display 508 in response to a rotation request received from a user. For example, by receiving an input to a Y-axis scroll bar 360 shown in Fig. 20 described later, the rotation processing unit 61 rotates the display image displayed in the captured image (equirectangular projection) drawing area 300, thereby changing the display position or display direction of the display image.

[0109] 16, and has a function of generating a grid image indicating a grid for identifying a relative position of a subject shown in a celestial sphere image and a ruler image indicating a display position of the celestial sphere image. The grid and ruler processing unit 62 generates the grid image based on setting information set in the grid setting management DB 5001, for example.

[0110] The grid / ruler memory unit 63 is mainly realized by the processing of the ROM 502, RAM 503 and CPU 501 shown in FIG. 16, and has the function of temporarily storing the grid image and ruler image generated by the grid / ruler processing unit 62.

[0111] The perspective projection image processing unit 64 is mainly realized by the processing of the CPU 501 shown in Fig. 16, and has a function of executing processing on a perspective projection image showing an image of a predetermined area that is a part of the omnidirectional image. The perspective projection image corresponds to an image of the predetermined area T (predetermined area image Q) seen from the virtual camera IC shown in Fig. 5.

[0112] The camera height memory unit 65 is mainly a part of the storage unit 5000 constructed by the ROM 502, RAM 503 and HD 504 shown in FIG. 16, and stores the values ​​of the distance from the camera to the floor and from the camera to the ceiling for the floor setting described below, and data of the camera height management DB 5002.

[0113] The 2D image memory unit 66 is a part of a storage unit 5000 that is mainly constructed by the ROM 502, RAM 503, and HD 504 shown in FIG.

[0114] The mesh memory unit 67 is a part of the storage unit 5000 which is mainly constructed by the ROM 502, RAM 503 and HD 504 shown in FIG.

[0115] 16, and is a function of performing 2D image transformation processing, that is, transforming a 2D image and synthesizing it with an image displayed in the captured image (equirectangular projection) drawing area 300. After storing the 2D image transformed based on various settings for the 2D image setting area and mesh setting area (described later) in the 2D image management DB 5003, the transformed 2D image is synthesized with the captured image.

[0116] The memory / read unit 59 is mainly realized by the processing of the CPU 501 shown in FIG. 16, and has the function of storing various data (or information) in the memory unit 5000 and reading out various data (or information) from the memory unit 5000.

[0117] ●Various management tables● 18 is a conceptual diagram showing an example of various management tables according to the embodiment. The grid setting management table, the camera height management table, the 2D image management table, and the mesh management table will be described below.

[0118] ○Grid setting management table○ FIG. 18A is a conceptual diagram illustrating an example of a grid setting management table according to an embodiment. A grid setting management DB 5001 configured by a grid setting management table as illustrated in FIG. 18A is constructed in the storage unit 5000. The grid setting management table includes information on a grid number for identifying a grid setting, a grid width indicating the width of the grid, a division number indicating the number of divisions by the grid, and a relative height (distance from the camera) with respect to the viewpoint position of the omnidirectional image. Among these, the relative height with respect to the viewpoint position of the omnidirectional image indicates a position in the omnidirectional image based on the position of the virtual camera IC illustrated in FIG. 5, for example. When an editor inputs various setting information for performing grid setting on a drawing input screen described later, the input setting information is stored in the grid setting management table. Note that the number of grid settings is not limited to this, and may be one or three or more. In this case, the number of records is added to the grid setting management table according to the number of grid settings.

[0119] ○Camera height management table○ FIG. 18(B) is a conceptual diagram showing an example of a camera height management table according to an embodiment. A camera height management DB 5002 configured by a camera height management table as shown in FIG. 18(B) is constructed in the storage unit 5000. The height from the floor to the camera (unit: cm) is included in this camera height management table. When an editor inputs various setting information for setting a 2D image into a drawing input screen described later, the input setting information is stored in the camera height management table. Note that the number of settings related to the camera height is not limited to this, and may be two or more. In this case, the number of records is added to the camera height management table according to the number of various settings.

[0120] ○2D image management table○ FIG. 18(C) is a conceptual diagram showing an example of a 2D image management table according to an embodiment. In the storage unit 5000, a 2D image management DB 5003 configured by a 2D image management table as shown in FIG. 18(C) is constructed. In this 2D image management table, the number of a 2D image, which is an example of a perspective projection image to be synthesized with an equirectangular projection image, the file name of the 2D image, the number of pixels (unit: dot) of the width of the 2D image, the number of pixels (unit: dot) corresponding to the length of the 2D image, the length of the 2D image (unit: cm), the width of the 2D image (unit: cm), the height (unit: cm) of a stand placed under (at the feet) of an object in the 2D image, the horizontal distance (unit: cm) from the virtual camera IC, and the longitude φ (unit: °). The 2D image number indicates a unique number assigned to each setting related to the 2D image. The file name when the 2D image is read and the image data of the image file are stored in the 2D image management DB 5003 by the 2D image memory unit 66. The number of pixels of the width indicates the number of pixels of the width of the 2D image, and is obtained from the image file when the PC 50 reads an arbitrary image file. The number of pixels of the height indicates the number of pixels of the height of the 2D image, and is obtained from the image file when the PC 50 reads an arbitrary file. The 2D image width indicates the width value of the 2D image setting. The height of the platform indicates the height from the floor when the 2D image is set. The distance indicates the distance when the 2D image is set. The longitude (φ) indicates the longitude when the 2D image is set. When the editor inputs various setting information for setting the 2D image into the drawing input screen described later, the input setting information is stored in the 2D image management table. Note that the number of settings related to the 2D image is not limited to this, and may be three or more. In this case, the number of records is added to the 2D image management table according to the number of various settings.

[0121] ○Mesh management table○ FIG. 18(D) is a conceptual diagram showing an example of a mesh management table according to the embodiment. In the storage unit 5000, a mesh management DB 5004 configured by a mesh management table as shown in FIG. 18(D) is constructed. This mesh management table includes a 2D image number which is an example of a perspective projection image to be synthesized with an equirectangular projection image, a mesh number, a mesh shape of a portion (shape) forming a mesh (unevenness including a protrusion 156), an upper left mesh range indicating the upper left coordinate when setting a mesh range, a lower right mesh range indicating the lower right coordinate when setting a mesh range, and a mesh magnification indicating the unevenness magnification of the mesh setting. The 2D image number is a unique number assigned to each setting related to a 2D image. The mesh number is a unique number assigned to each mesh setting related to the setting of a 2D image, and the 2D image number and the mesh number (1, 2, . . .) are primary keys. The mesh shape indicates the shape of the mesh in the mesh setting. In addition, when a file is read by reading a mesh file related to mesh setting, the mesh memory unit 67 stores the file name and mesh data in the mesh management DB 5004. The upper left mesh range is a two-dimensional array of data (x, y) that stores the upper left coordinates of the mesh range of the mesh setting in FIG. 26 described later. The lower right mesh range is a two-dimensional array of data (x, y) that stores the lower right coordinates of the mesh range of the mesh setting in FIG. 26 described later. The mesh magnification indicates the concave / convex magnification in the mesh setting. When the editor inputs various setting information for setting the mesh in the drawing input screen described later, the input setting information is stored in the mesh management table. In addition, the number of settings related to the mesh is not limited to this, and may be four or more. In this case, the number of records is added to the mesh management table according to the number of various settings.

[0122] [Processing or Operation of the Embodiment] FIG. 19 is a sequence diagram showing an example of the grid setting process and the rotation process in the PC according to the embodiment.

[0123] First, the display control unit 53 of the PC 50 starts a predetermined application installed in the PC 50, thereby displaying on the display 508 a drawing input screen 200 for performing drawing input on a captured image (step S11). Fig. 20 is a diagram showing an example of the drawing input screen displayed on the PC according to the embodiment. The drawing input screen 200 shown in Fig. 20(A) is a display screen for an editor to perform a predetermined drawing input on a celestial sphere image displayed in a captured image (equirectangular projection) drawing area 300 (hereinafter simply referred to as drawing area 300).

[0124] The "File" button 210 is pressed when reading out from the storage unit 5000 a file of a captured image for drawing in the drawing area 300. The "Import" button 220 is pressed when importing a captured image from the imaging device 10 connected to the PC 50. An editor can import an image for drawing input into the PC 50 by pressing either the "File" button 210 or the "Import" button 220.

[0125] The grid setting area 230 is an area for inputting setting information of a grid for identifying a relative position of a subject shown in the omnidirectional image. The setting information includes, for example, a grid width indicating the width of the grid, a division number indicating the number of divisions by the grid, and height information of the camera indicating information on the height relative to the viewpoint position of the omnidirectional image. Among them, the grid setting area 230a is an area for inputting setting information of a grid 1 corresponding to the grid image 310a, and the grid setting area 230b is an area for inputting setting information of a grid 2 corresponding to the grid image 310b. For example, the grid image 310a is a grid drawn on the ceiling surface of the subject of the omnidirectional image, and the grid image 310b is a grid drawn on the floor surface of the subject of the omnidirectional image. In this way, the editor can set a plurality of grids by inputting to the grid setting area 230. The number of grids that can be set may be one or may be three or more. In the following description, when it is not necessary to distinguish between the plurality of grid images 310a and 310b, they are referred to as grid images 310.

[0126] The drawing area 300 is an input area for an editor to perform drawing input for the omnidirectional image. In the drawing area 300, for example, an image (equirectangular projection image 400) generated by equirectangular projection that represents the captured omnidirectional image is displayed. In this case, in the equirectangular projection image 400, the coordinates (θ, φ) shown in FIG. 6 or FIG. 7 are represented as the vertical axis θ and the horizontal axis φ. The editor can input a predetermined drawing image such as a character, a line, or a figure to the equirectangular projection image 400 displayed in the drawing area 300 using an input means such as a keyboard 511, a mouse 512, or a touch pen 520. The editor can also paste an image different from the captured omnidirectional image into the drawing area 300.

[0127] The ruler image 330 is an image indicating the display position of the omnidirectional image. The ruler image 330 indicates the display position of the equirectangular projection image 400 representing the omnidirectional image, for example, using latitude (θ) and longitude (φ). In the example shown in FIG. 19(A), the ruler image 330 is displayed with the coordinates (θ,φ) of the center of the drawing area 300 as the origin (0°, 0°). Note that the position of the origin is not limited to this. Furthermore, the drawing area 300 includes a label image indicating the display direction of the omnidirectional image. For example, the label image is labeled every 90° as F for the front (0°, 0°), R for the right (0°, 90°), L for the left (0°, -90°), B for the rear (0°, 180°) and (0°, -180°), T for the ceiling (90°, 0°), and U for the floor (-90°, 0°). Note that the label image is not limited to this, and may be expressed in a state where the editor can visually recognize the display position of the displayed spherical image.

[0128] The Y-axis scroll bar 360 is an operating means for setting the angle of rotation around the Y-axis (horizontal axis direction) as shown in Fig. 6. Here, the range of rotation around the Y-axis is set to a value between -180° and 180°.

[0129] The "perspective projection switching" button 250 is a button (an example of a selection means) that is pressed when a perspective projection image 600 showing a partial area of ​​the omnidirectional image is displayed in the drawing area 300. The "perspective projection switching" button 250 has a function of displaying a screen of the captured image (equirectangular projection) drawing area. In addition, by pressing the "perspective projection switching" button 250, predetermined image data is read from the captured image memory unit 57, and an image (predetermined area image) of the predetermined area T in FIG. 5 is displayed. When the editor presses the "perspective projection switching" button 250, the display control unit 53 displays the perspective projection image 600 shown in FIG. 20(B) in the drawing area 300, instead of the equirectangular projection image 400. Even in this case, the display control unit 53 displays the grid image 310, the ruler image 330 indicating the display position of the perspective projection image 600, and the label image indicating the display direction of the perspective projection image 600 in the drawing area 300. Note that the grid image 310, the ruler image 330, and the label image are displayed superimposed on the celestial sphere image, and therefore do not affect the celestial sphere image stored in the captured image memory unit 57. Furthermore, the display control unit 53 may simultaneously display the equirectangular projection image 400 and the perspective projection image 600 representing the celestial sphere image in the drawing area 300, or may display them separately.

[0130] The "Switch to 2D image" button 255 is a selection button for switching between an editing screen for a captured image and a 2D image shown in Fig. 26, which will be described later. By pressing the "Switch to 2D image" button 255 in the screen state of Fig. 20(A), the editor can switch the screen displayed on the display 508 from the screen display shown in Fig. 20(A) to the 2D image editing screen shown in Fig. 26. The function of the "Switch to 2D image" button 255 is to take over the captured image (equirectangular projection) drawing area stored in the captured image memory unit 57.

[0131] Returning to FIG. 15, when the editor selects the "import" button 220 shown on the drawing input screen 200, the reception unit 52 receives a request to import the photographed image photographed by the photographing device 10 (step S12). The transmission / reception unit 51 or the communication unit 55 transmits a data acquisition request indicating a request for image data of the photographed image to the photographing device 10 (step S13). As a result, the transmission / reception unit 11 or the communication unit 13 of the photographing device 10 receives the data acquisition request transmitted from the PC 50. Next, the storage / readout unit 19 of the photographing device 10 reads out the photographed image data stored in the storage unit 1000 (step S14). The storage unit 1000 stores photographed image data relating to the photographed image photographed by the imaging unit 15 in advance. Then, the transmission / reception unit 11 or the communication unit 13 of the photographing device 10 transmits the photographed image data read out in step S14 to the PC 50 (step S15). As a result, the transmitting / receiving unit 51 or the communication unit 55 of the PC 50 receives the photographed image data transmitted from the photographing device 10.

[0132] On the other hand, when the editor selects the "File" button 210 shown on the drawing input screen 200, the reception unit 52 receives a request to read out the captured image (step S16). The storage / readout unit 59 reads out the captured image data that has been stored in advance in the storage unit 5000 (step S17).

[0133] Then, the display control unit 53 of the PC 50 causes the display 508 to display the omnidirectional image, which is the captured image related to the captured image data received in step S15 or read in step S17, in the drawing area 300 of the drawing input screen 200 (step S18). Specifically, the display control unit 53 causes the drawing area 300 to display an equirectangular projection image 400 (see FIG. 3(C) for example) representing the omnidirectional image. Then, the captured image processing unit 56 causes the captured image memory unit 57 to store the omnidirectional image displayed in step S18 (step S19).

[0134] In the embodiment, the functions of the "Import" button 220 and the "File" button 210 may be interchanged. That is, steps S12 to S15 may be executed by pressing the "File" button 210 by an editor, and steps S16 to S19 may be executed by pressing the "Import" button 220 by an editor. In addition, the various buttons and setting areas in the GUI shown in Fig. 20 are not limited to the configuration shown as an example, and the layout, number and arrangement of buttons may be changed as appropriate according to the usability of the editor.

[0135] <Grid display processing> Next, the PC 50 executes a grid display process for the spherical image displayed in the drawing area 300 (step S20). Here, a process in which the PC 50 displays a grid for the spherical image will be described with reference to Fig. 21. Fig. 21 is a flowchart showing an example of the grid display process in the PC according to the embodiment.

[0136] First, the reception unit 52 receives input of setting information for setting a grid in the grid setting area 230 of the drawing input screen 200 displayed on the display 508 (step S31). The reception unit 52 receives the input of the setting information by storing the grid width, the number of divisions, and the distance from the camera (representing the height from the camera to the ceiling or the height from the camera to the floor surface, and adding a - sign to either of them to distinguish the direction) in at least one of the grid setting areas 230a and 230b illustrated in FIG. 20 by the editor. Next, the grid setting unit 58 sets the grid using the setting information received in step S31 (step S32). Specifically, the grid setting unit 58 stores the values ​​of the setting items included in the received setting information as one record in the grid setting management table.

[0137] Then, the grid / ruler processing unit 62 generates a grid image to be displayed in the drawing area 300 (step S33). Here, Fig. 23 is a flowchart showing an example of a generation process of the grid image in the image display device according to the embodiment.

[0138] First, the grid ruler processing unit 62 of the PC 50 obtains the setting information stored in the grid setting management table (step S51). Here, the number of records of the grid setting obtained by the grid ruler processing unit 62 is set to N, and the index is set to k (k=0). The grid ruler processing unit 62 obtains all records stored in the grid setting management table. Next, the grid ruler processing unit 62 adds 1 to the index k (k+1) (step S52).

[0139] The grid ruler processor 62 calculates the grid coordinate array [Xi, Zj] using the grid width and division number values ​​included in the setting information acquired in step S51 (step S53). Specifically, the grid ruler processor 62 first calculates the width of one grid cell by dividing the grid width value by the division number (width of one grid cell = grid width / division number). Next, the grid coordinates (grid coordinates 1 to 4) for specifying the range of the grid to be set are calculated as follows: Grid coordinate 1 (x,z) = (-1 × grid width, -1 × grid width) Grid coordinate 2(x,z) = (grid width, -1 × grid width) Grid coordinate 3 (x,z) = (grid width, grid width) Grid coordinate 4 (x,z) = (-1 × grid width, grid width) Next, the grid ruler processor 62 creates an array by adding the width of one cell between the calculated grid coordinates 1 to 4 (number of divisions - 1) times. In this case, the grid ruler processor 62 sets the X-axis direction to i and the Z-axis direction to j, and calculates the coordinates of the X-axis and Z-axis by running a double loop (number of divisions - 1) times. A pseudo calculation code is shown below. Loop through i (number of divisions - 1) times Loop through j (number of divisions - 1) times Grid coordinate array [Xi, Zj] = (-1 × grid width + 1 cell width × i, -1 × grid width + 1 cell width × j) End of loop for j End of loop of i" Then, the grid ruler processor 62 uses the value of height H included in the setting information acquired in step S51 to calculate a grid coordinate array [Xi,H,Zj] including the Y-axis coordinate (step S54). In this case, as in step S53, the grid ruler processor 62 sets the X-axis direction to i and the Z-axis direction to j, and calculates the Y-axis coordinate by running a double loop (number of divisions - 1) times. A pseudo calculation code is shown below. Loop through i (number of divisions - 1) times Loop through j (number of divisions - 1) times (x,z) = Grid coordinate array [Xi,Zj] [Xi,H,Zj]=(x,H,z) End of loop for j End of loop of i" Next, the grid / ruler processing unit 62 generates a ruler image 330 indicating the display position of the omnidirectional image and a label image indicating the display direction of the omnidirectional image based on the position of the coordinates (θ,φ)=(0,0) (step S55). Specifically, the grid / ruler processing unit 62 creates a ruler on the horizontal axis with a value of θ=0, and creates a ruler on the vertical axis with a value of φ=0. The ruler image 330 may include scales at equal intervals as shown in FIG. 20, or may include scale values ​​every 30°. In addition, the grid / ruler processing unit 62 generates a label at the corresponding coordinates (θ,φ) so that a user who performs drawing input can visually recognize the display direction of the omnidirectional image. As shown in FIG. 19, the label image includes labels such as symbols or characters that allow a user to visually recognize the display direction of the omnidirectional image, such as front / back, left / right, and up / down, as the display direction.

[0140] The grid ruler processor 62 calculates the equirectangular coordinates (θ,φ) using the grid coordinate array [Xi,H,Zj] and the above-mentioned (Equation 8) and (Equation 9) (step S56). The grid ruler processor 62 extracts, for example, one grid coordinate (x,y,z) from the grid coordinate array [Xi,Zj], and converts the extracted grid coordinate (x,y,z) to the equirectangular coordinate (θ,φ) using (Equation 8) and (Equation 9).

[0141] The grid / ruler processing unit 62 generates a grid image 310 based on the calculated coordinates (θ, φ) (step S57). The grid / ruler processing unit 62 stores the ruler image 330 and label image generated in step S55, and the grid image 310 generated in step S56 in the grid / ruler memory unit 63 (step S58). At this time, the grid ruler processing unit 62 creates rulers for each axis with values ​​of θ=0 and φ=0. In this case, the grid ruler processing unit 62 may add scales at equal intervals. Alternatively, the grid ruler processing unit 62 may write scale values ​​every 30°. Alternatively, the grid ruler processing unit 62 may display labels at the coordinates (θ, φ) as follows so that the direction can be determined: F: (0, 0) L: (-90, 0) R: (90, 0) B: (180, 0) B: (-180, 0) U: (0, -90) T: (0, 90) Furthermore, F: front, L: left, R: right, B: back, U: bottom, and T: ceiling. If the index k is equal to or greater than the number of records N in the grid setting (Yes in step S59), the grid / ruler processing unit 62 ends the process. On the other hand, if the index k is smaller than the number of records N (No in step S59), the grid / ruler processing unit 62 repeats the process from step S52.

[0142] 8 into a spherical polar coordinate system and acquires the equirectangular projection coordinates (θ,φ) by the process of step S56, and generates a grid image 310 to be displayed at the acquired coordinates (θ,φ). Here, the grid coordinates (x, y, z) are converted into the coordinates (θ,φ) using the above-mentioned (Equation 8) and (Equation 9).

[0143] Then, the display control unit 53 causes the generated grid image 310 to be superimposed on the equirectangular projection image 400 and displayed.

[0144] Furthermore, in a process described later, when the display image is rotated around the Y axis, the rotation processing unit 61 converts the grid coordinates (x, y, z) into grid coordinates (x'', y'', z'') according to the coordinate conversion as shown in FIG. 8, and then converts them into CP(r, θ, φ) to acquire the equirectangular projection coordinates (θ, φ). Then, the display control unit 53 displays the grid image 310 of the acquired coordinates (θ, φ) superimposed on the equirectangular projection image 400. Note that, although an example in which the grid image 310 on a surface parallel to the X axis and the Z axis is displayed has been shown in FIG. 8, grid setting may be performed such that the grid image is displayed on a surface parallel to the Y axis and the Z axis or a surface parallel to the X axis and the Y axis. This allows the PC 50 to set and display a grid according to the display state of the omnidirectional image by using the setting information input to the grid setting area 230, thereby improving the convenience of the editor who performs drawing input.

[0145] Returning to FIG. 16, the display control unit 53 of the PC 50 causes the display 508 to display a display image in which the generated grid image 310 is superimposed on the spherical image being displayed (step S34).

[0146] <Display image rotation processing> Returning to FIG. 19, the PC 50 executes a rotation process of the display image displayed on the display 508 (step S21). Details of the rotation process of the display image following this step S21 will be described with reference to FIG. 23. FIG. 23 is a flowchart showing an example of the rotation process of the display image in the PC according to the embodiment. First, the determination unit 54 of the PC 50 determines whether or not a rotation control request of the display image has been accepted by the acceptance unit 52 (step S71). For example, when an input to the Y-axis scroll bar 360 of the drawing input screen 200 is accepted by the acceptance unit 52, the determination unit 54 determines that the rotation control request has been accepted. When the determination unit 54 determines that the rotation control request has been accepted (Yes in step S71), the determination unit 54 shifts the process to step S72. On the other hand, when the determination unit 54 determines that the rotation control request has not been accepted (No in step S71), the process ends.

[0147] Next, the captured image processing unit 56 reads the spherical image stored in the captured image memory unit 57 (step S72). Furthermore, the grid and ruler processing unit 62 reads the grid image 310, the ruler image 330, and the label image stored in the grid and ruler memory unit 63 (step S73). In the following description, the images read in steps S72 and S73 are referred to as "read images."

[0148] Next, when an input to the Y-axis scroll bar 360 is accepted (Yes in step S74), the accepting unit 52 shifts the process to step S75. On the other hand, when an input to the Y-axis scroll bar 360 is not accepted (No in step S74), the accepting unit 52 shifts the process to step S76. Next, the rotation processing unit 61 rotates the read image read in steps S72 and S73 around the Y axis (horizontal axis direction) using the value of the Y-axis scroll bar 360 whose input is accepted in step S74 and the above-mentioned (Equation 8) and (Equation 9) (step S75). Specifically, the rotation processing unit 61 extracts a rotation angle around the Y axis from the value of the Y-axis scroll bar 360, and rotates the display of the read image around the Y axis according to the extracted rotation angle.

[0149] Then, the display control unit 53 superimposes and displays the rotated grid image 310, the ruler image 330, and the label image on the rotated spherical image rotated in step S75 (step S76).

[0150] <Drawing process> 19, the PC 50 executes a drawing process for the spherical image (step S22). FIG 24 is a flowchart showing an example of a drawing process for the spherical image in the PC according to the embodiment.

[0151] First, the reception unit 52 of the PC 50 receives input of a drawn image for the drawing area 300 (step S91). Next, the captured image processing unit 56 of the PC 50 acquires the coordinates (x'', y'', z'') of the drawn image input in step S91 and information on the color of the drawn image (step S92).

[0152] The captured image processing unit 56 uses the value of the Y-axis scroll bar 360 and the above-mentioned (Equation 8) and (Equation 9) to calculate the coordinates (x', y', z') obtained by rotating the coordinates (x'', y'', z'') obtained in step S92 around the Y axis (step S93).

[0153] The captured image processing unit 56 calculates the equirectangular coordinates (θ,φ) converted from the coordinates (x,y,z) calculated in the next step S94 using the above-mentioned (Equation 8) and (Equation 9) (step S94). The captured image processing unit 56 stores color information of the acquired drawn image in the captured image memory unit 57 in association with the calculated coordinates (θ,φ) (step S95).

[0154] <2D image transformation> Fig. 25 is a sequence diagram showing an example of a 2D image transformation process in the image display device according to the embodiment, and Fig. 26 is a diagram showing an example of a 2D image editing screen displayed on the PC according to the embodiment.

[0155] When the editor presses the "Switch to 2D image" button 255, the display control unit 53 of the PC 50 switches from the drawing input screen shown in Fig. 20 to the 2D image editing screen shown in Fig. 26. In this state, the reception unit 52 of the PC 50 receives a press (operation) by the editor of the "Set 2D image" button (tab) 260 displayed on the display 508 (step S2000).

[0156] Moving on to Fig. 26, various buttons, setting areas, and display areas will be described. The "2D image setting" button 260 is a button for switching between 2D image settings, which will be described later, and includes a "2D image setting 1" button 260a and a "2D image setting 2" button 260b. By providing multiple setting buttons, it is possible to deform multiple 2D images and draw them on the captured image.

[0157] The 2D image setting area 265 is an area for inputting setting information for one record of the 2D image management table. The setting information includes, for example, the length (height) of an object included in the 2D image. The length (height) setting area is also a GUI (Graphical User Interface) for setting a value in the 2D image height column of the 2D image management table. In addition, when an editor drags the mouse 512 vertically from a position (θ1) on an object whose height is known in the subject shown in the drawing area 300 to a position (θ2) below the position (θ1) to place a marker, a dialog box for inputting a value is opened by the display control unit 53 of the PC 50. The editor may input a predetermined value into the dialog box, and the calculation may be performed as length (height) = tan θ1 / (tan θ1 - tan θ2). The setting information further includes each of the setting items: width for setting a value in the 2D image width column of the 2D image management table, height from the floor for setting a value in the stand height column of the 2D image management table, distance for setting a distance column of the 2D image management table, and longitude (φ) for setting a longitude (φ) column of the 2D image management table.

[0158] The distance, longitude (φ), and height from the floor are setting items set on the drawing area 300. When the mouse 512 or the touch pen 520 is placed on the post-deformation 2D image showing the post-deformation 2D image and then dragged to move horizontally, the value of the longitude (φ) is updated according to the angle of the φ axis. On the other hand, when the mouse 512 or the touch pen 520 is dragged vertically, the value calculated from the vertical angle (θ) and the distance z, z*tanθ, is updated as the height from the floor. Furthermore, when the wheel of the mouse 512 or the touch pen 520 is rotated toward the user, the distance may be updated to a smaller value, and when the wheel is rotated in the opposite direction, the distance may be updated to a larger value. When each setting item is input or updated with the mouse 512 or the touch pen 520, a post-deformation 2D image is generated according to the sequence diagram of the 2D image deformation and displayed on the drawing area 300.

[0159] The "Switch Mesh Settings" button 270 is a button for switching mesh settings, which will be described later, and sets information for one record in the mesh management table. The "Switch Mesh Settings" button 270 includes a "Mesh 1" button 270a and a "Mesh 2" button 270b. By preparing multiple mesh settings, it is possible to deform a single 2D image to accommodate multiple meshes and render them on the captured image.

[0160] The mesh setting area 275 is an area for inputting setting information for one record of the 2D image management table. The setting information includes, for example, mesh range upper left, which is a setting item for setting the value of the upper left column of the mesh range of the mesh management table, and mesh range lower right, which is a setting item for setting the value of the lower right column of the mesh range of the mesh management table. The mesh range upper left and mesh range lower right may be set by dragging and dropping with the mouse 512 in the 2D image display area 285. The editor sets each setting item of the mesh range upper left and mesh range lower right, so that the mesh data is mapped to the 2D image. Furthermore, mesh shape, which is a setting item for setting the value of the mesh shape column of the mesh management table, may be included, and multiple shapes may be set as presets, in which case the mesh shape is selected from a pull-down list. Furthermore, mesh file read, which is a setting item for setting the mesh shape column of the mesh management table, stores a file name when the storage destination is the mesh management table, and stores the file name and the mesh data itself in FIG. 20 when the storage destination is the mesh memory unit. Furthermore, the setting information includes mesh magnification, which is a setting item for setting the value of the mesh magnification column of the mesh management table (the default setting value is 1).

[0161] When the editor inputs each of the mesh-related settings described above or updates them using the mouse 512 or the stylus 520, a deformed 2D image is generated according to the sequence diagram of 2D image deformation described below and displayed on the drawing area 300.

[0162] The "Load 2D image" button 280 is a button for reading out the file of the 2D image to be drawn on the drawing area 300 from the 2D image memory unit 66. When the editor presses the "Load 2D image" button 280, a file selection dialog box is opened by the display control unit 53. When the editor selects a file in this dialog box, the file of the 2D image to be drawn on the drawing area 300 is read out from the 2D image memory unit 66.

[0163] The 2D image display area 285 is an area for displaying a 2D image loaded with the "Load 2D image" button 280. In addition, the upper left corner and the lower right corner of the mesh described above are set.

[0164] The "switch captured image" button 290 is an operation button for returning to the above-mentioned screen of Fig. 20. When the editor presses this "switch captured image" button 290, the display control unit 53 performs control to return to the drawing input screen 200 of Fig. 20, and provides a GUI that enables the captured image and grid to be set.

[0165] The transformed 2D image 340 (transformed 2D image 181 ) is a 2D image that is transformed and generated according to a sequence of 2D image transformations described below, and is displayed on the drawing area 300 .

[0166] The φ-axis rotation scroll bar is the same as the Y-axis scroll bar 360 described in FIG. 20, and is an operation means for scrolling the captured image in which the transformed 2D image is drawn in the φ-axis direction. Here, when using 3D development software such as Unity or OpenGL, a method may be used in which a UV map is created from mesh data, a 2D image is pasted on the UV map, the captured image is pasted inside the solid sphere in FIG. 5, and the equirectangular projection and perspective projection image 600 are captured from the virtual camera IC. In that case, the image processing in FIG. 10 to FIG. 12 may be processed by the library of the 3D development software. However, since the library of the 3D development software is not used in this embodiment, there is no overhead for 3D conversion and conversion to equirectangular projection, and therefore high-speed processing is possible.

[0167] The "Save" button 370 is an operation button for saving to a file the image in the captured image memory section 57 displayed on the drawing area 300 combined with the transformed 2D image. When the editor presses the "Save" button 370, a file selection dialog box is displayed, and the editor can input and select the name of the file to be saved.

[0168] As in the case of Figure 20, the various buttons and setting areas in the GUI (2D image editing screen) shown in Figure 26 are not limited to the configuration shown in the example, and the layout, number, type and arrangement of buttons may be changed as appropriate depending on the convenience of the editor, etc.

[0169] <Processing sequence> In step S2000, by the operation by the editor described above, the captured image processing unit 56 reads out a predetermined 2D image stored in the 2D image management DB 5003. Next, the reception unit 52 receives a press (operation) of the "Load 2D image" button 280 by the editor (step S2001).

[0170] Following the process of step S2001, the display control unit 53 displays a "Load 2D image" button 280 for loading a 2D image on the display 508. This process allows the editor to select a desired 2D image using the keyboard 511 or mouse 512. The storage / reading unit 59 also reads out a file relating to the desired 2D image selected by the editor from the HD 504, and stores it in the 2D image memory unit 66. Thereafter, the display control unit 53 displays the 2D image stored in the 2D image memory unit 66 in the 2D image display area 285 (step S2002).

[0171] Next, the accepting unit 52 accepts the setting value of the setting item related to length (height) input by the editor (step S2003).

[0172] Next, the display control unit 53 calculates an initial value in cooperation with the perspective projection image processing unit 64 in order to display the transformed 2D image on the drawing area 300. Specifically, the perspective projection image processing unit 64 reads data relating to "2D image height" in the "2D image management table" from the camera height memory unit 65, and sets "length (height)" in "2D image setting 1" to the value of "2D image height."

[0173] Next, the perspective projection image processing unit 64 multiplies the initial value calculated above by the aspect ratio of the image to calculate the value of "width", and sets the result as the initial value of "width" (step S2004). Furthermore, the perspective projection image processing unit 64 sets 0 as the initial value for the setting item "height from floor" in "2D image setting 1", and sets twice the value of "camera height" as the initial value for the setting item "distance" in "2D image setting 1". Furthermore, the perspective projection image processing unit 64 sets 0 as the initial value for the setting item "longitude (φ)" in "2D image setting 1".

[0174] Next, the accepting unit 52 accepts re-input for the setting item "width" in the "2D image settings 1" tab if the editor checks the "width" value and inputs a different value (step S2005).

[0175] The storage / readout unit 59 stores the values ​​of "length (height)" and "width" in the 2D image memory unit 66 (step S2006).

[0176] Next, the reception unit 52 receives inputs by the editor to each of the setting items of "distance" and "longitude (φ)" of "2D image setting 1" (step S2007). The above-mentioned values ​​of "height from floor", "distance" and "longitude (φ)" may be updated by the editor clicking the transformed 2D image displayed on the drawing area 300 with the mouse 512 or the like, and performing drag & drop and wheel operation. For example, when the editor drags the transformed 2D image in the horizontal direction (φ axis), the "longitude (φ)" is updated according to the longitude φ of the drawing area 300. When the editor drags in the vertical direction (θ axis), the value of "height from floor" may be updated using the calculated value of z1 * tanθ from the "distance" (z1) and the latitude θ of the drawing area 300. Furthermore, when the editor turns the wheel of the mouse 512 back and forth, the value of "distance" may be increased or decreased.

[0177] Next, the perspective projection image processing unit 64 passes the "camera height" data in the camera height memory unit 65 and the data of the 2D image number set in the "2D image setting" tab of the 2D image memory unit 66 to the 2D image transformation processing unit 68, and transforms the 2D image (step S2008). Note that the functions of the 2D image transformation processing unit 68 may be collectively performed by the above-mentioned perspective projection image processing unit 64. Furthermore, the functions of the perspective projection image processing unit 64 may be collectively performed by the 2D image transformation processing unit 68.

[0178] Next, the storage / readout unit 59 stores the data of the 2D image number set in the "2D image setting" tab of the 2D image memory unit 66 in the 2D image management DB 5003 (step S2009).

[0179] <Mesh settings> Subsequently, when the editor presses the "switch mesh setting" button 270 in the mesh setting area 275, the reception unit 52 displays various setting items for performing mesh setting (step S2010).

[0180] The reception unit 52 receives input of the coordinate values ​​of the "upper left mesh range" and the "lower right mesh range" of "mesh 1" by the editor. Alternatively, the reception unit 52 may detect that the editor has specified the range of the 2D image display area 285 by dragging and dropping, and update the values ​​of the "upper left mesh range" and the "lower right mesh range" (step S2011).

[0181] The storage / reading unit 59 stores the values ​​set in the respective setting items of "upper left mesh range" and "lower right mesh range" in the mesh management DB 5004 (step S2012).

[0182] The reception unit 52 receives the mesh shape selected from the pull-down menu displayed when the editor presses the "Mesh Shape" button of "Mesh 1". For example, when "Skirt" and "Legs" are displayed in the pull-down menu of "Mesh Shape", the reception unit 52 receives the selection when the editor selects "Skirt". The reception unit 52 may further receive mesh data in response to the editor's selection of a file of mesh data when the display control unit 53 displays a dialog box for file selection in response to the editor pressing the "Load Mesh File" button of "Mesh 1". When the editor selects mesh data, the display control unit 53 displays the file name in "Mesh Shape" of "Mesh 1". After that, the storage / reading unit 59 stores the file of the mesh data received by the reception unit 52 in the mesh memory unit 67 (step S2013).

[0183] The storage / read unit 59 stores the value set in the "mesh shape" in the mesh management DB 5004 (step S2014).

[0184] The reception unit 52 receives an input of the "mesh magnification" of "mesh 1" by the editor (step S2015).

[0185] The storage / reading unit 59 stores the setting value of the "mesh magnification" received by the reception unit 52 in the mesh management DB 5004 (step S2016).

[0186] Furthermore, the memory / read unit 59 stores the data of the 2D image number of the 2D image memory unit 66 set in the "2D image setting area" 265 and the data of the mesh memory unit 67 set in the "mesh setting area" 275 in the 2D image management DB 5003 (step S2017).

[0187] When the 2D image deformation processing unit 68 adjusts the position of the deformed 2D image on the drawing area 300 by operations including clicking, dragging & dropping, and using the mouse wheel by the editor, it accepts and updates the setting items including "height from the floor surface", "distance", and "longitude (φ)" in the same way as in step S2007. After that, when the editor moves the φ-axis rotation scroll bar, the 2D image deformation processing unit 68 updates the value of the scroll bar (step S2018).

[0188] After that, the storage / reading unit 59 stores "height from the floor surface", "distance", and "longitude (φ)" in the 2D image management DB5003 (step S2019).

[0189] Subsequently, the storage / reading unit 59 stores the data of the 2D image number set in the "2D image setting area" 265 of the 2D image memory unit 66 and the data of the mesh memory unit 67 set in the "mesh setting area" 275 in the 2D image management unit 7003 (step S2020).

[0190] The 2D image deformation processing unit 68 gives the value of the φ-axis rotation scroll bar to the rotation processing unit 61, composes the captured image and the deformed 2D image on the drawing area 300 with the grid ruler in the drawing area 300, and the display control unit 53 displays the composite image (step S2021).

[0191] The display control unit 53 displays the composite image synthesized in step S2021 on the display 508 and exits this sequence (step S2022).

[0192] <Processing of PC50> FIG. 27 is a flowchart showing an example of 2D image deformation processing in the PC according to the embodiment. The main functional unit that executes the flowchart of FIG. 27 is, for example, the 2D image deformation processing unit 68.

[0193] First, the reception unit 52 accepts each setting input of the user (editor) for the 2D image editing screen shown in FIG. 26 (step S1401).

[0194] Next, the judgment unit 54 checks whether there has been a change in the 2D image management table in the 2D image memory unit 66 and the mesh management table in the 2D image mesh memory unit 67. If there has been a change, the process proceeds to step S1403, and if there has been no change, the process returns to step S1401 (step S1402).

[0195] If there is a change in the 2D image management table in the 2D image memory unit 66 and the mesh management table in the 2D image mesh memory unit 67 (Yes in step S1402), the 2D image transformation processing unit 68 obtains the 2D image number that has been changed in the 2D image memory unit 66, obtains the data of the 2D image number, and loops (step S1403).

[0196] The 2D image transformation processing unit 68 obtains the values ​​of the width Xp, height Yp, number of pixels in the width Xx, and number of pixels in the height Yx of the 2D image shown in FIG. 20 from the 2D image memory unit 66 and mesh memory unit 67 (step S1404).

[0197] The 2D image transformation processing unit 68 executes a loop process in the range of index k<number of pixels in height Yx, with the initial value of index k set to 0 (step S1405).

[0198] The 2D image transformation processing unit 68 executes a loop process in the range of j<Xx (step S1406).

[0199] The 2D image transformation processing unit 68 obtains the mesh data di from the mesh memory unit 67 having the obtained mesh magnification (m) and the (k, j) coordinates, and overwrites the mesh data di with di×m (step S1407).

[0200] The 2D image transformation processing unit 68 calculates the variables Xi and Yi from the variables k and j using (Equation 10) to (Equation 13) (step S1408).

[0201] The 2D image transformation processing unit 68 calculates the variable θ′ using (Equation 15) and calculates the variable φ″ using (Equation 18) (step S1409). By executing this step S1409, the editor can obtain the “edited” images exemplified at distances of 60 cm and 40 cm in FIG. 28. These “edited” images have larger intervals and distortions of grid lines from the equator 0.00 toward the pole (−90.00), so that the closer to the feet, the more distorted (crushed) the image is displayed. That is, the 2D image transformation processing unit 68 controls the distortion degree of the perspective projection image (2D image) according to the distance from the equator in the equirectangular projection image. In other words, the 2D image transformation processing unit 68 performs processing to increase the distortion degree of the perspective projection image (2D image) the farther it is from the equator in the equirectangular projection image, and to decrease the distortion degree of the perspective projection image (2D image) the closer it is to the equator. Then, the display control unit 53 displays the deformed perspective projection image, deformed by controlling the distortion degree by the 2D image transformation processing unit 68, in a state where the deformed perspective projection image is synthesized with the equirectangular projection image on the display 508. Note that Fig. 28 is a diagram showing an example of the difference in equirectangular projection between with and without correction according to the embodiment.

[0202] Based on each setting input input by the editor, the 2D image transformation processing unit 68 projects a pixel at (j, k) of the 2D image in the 2D image memory unit 66 onto the memory area (θ', φ'') of the captured image memory unit 57. When the 2D image transformation processing unit 68 converts the (θ', φ'') coordinates into (x, y) of the captured image, it calculates (x, y) using (Equation 1) and (Equation 2) and projects (j, k) onto (x, y) (step S1410). Here, the storage / readout unit 59 also stores the original image before projection into the captured image memory unit 57, copies the original image for use in 2D image synthesis, and projects the copied image onto the memory area. By executing the processes of steps S1409 and S1410 described above, the final viewer can view, for example, a human character like the "corrected" image in FIG. 29 and FIG. 30 on the display 508. This allows the final viewer to see the image as it would appear in the real world.

[0203] Fig. 29 is a diagram showing an example of the difference in perspective projection with and without correction according to the embodiment. Fig. 29(A) shows an image viewed on the display 508 without equirectangular correction when the distance from the image capture device 10 to the human character is 200 cm. In this case, the human character is located near the equator, so it is displayed without distortion.

[0204] FIG. 29(B) shows images viewed on the display 508 with and without equirectangular projection correction when the distance from the camera 10 to the human character is 60 cm. Without correction, the human character is displayed as having long, thin legs, and a realistic image cannot be obtained. On the other hand, with correction, because the height of the camera and the line of sight of the human character are almost the same height, the image shows that the legs get shorter (vertically squashed) as they move towards the feet. This is displayed as the same characteristic as an image captured in the real world, and a realistic image is obtained.

[0205] Fig. 29(C) shows images viewed on the display 508 with and without equirectangular projection correction when the distance from the image capture device 10 to the human character is 40 cm. In this case as well, without correction, the human character is displayed as an image with long legs and thinness, and a realistic image cannot be obtained. On the other hand, with correction, an image that looks more like one would see in the real world is obtained, as with the correction in Fig. 29(B).

[0206] Fig. 29(D) shows images viewed on display 508 with and without equirectangular projection correction when the distance from the image capture device 10 to the human character is 40 cm and the FOV is fixed at 60°. In this case, as in Fig. 29(C), an example is shown in which the human character appears far away and cannot be viewed as a realistic image without correction.

[0207] Fig. 30 is a diagram showing an example of the difference in perspective projection with and without correction of protruding parts according to the embodiment. Fig. 30(A) shows an image viewed on a display 508 without correction of protruding parts 156 such as a skirt or shoes of a human character. Since equirectangular projection correction is performed on the human character, the proportions of the equirectangular projection are correct. However, the skirt or shoes are like a picture of a person on a signboard, and it is expressed that only a flat image is obtained, which is difficult to say is a realistic image.

[0208] 30(B) shows an image viewed on display 508 with correction applied to protruding parts 156 of the human character, such as the skirt and shoes. In this case, the human character's protruding parts are corrected by the processes in steps S1408 to S1410 described above, and the image can be viewed as more natural. Note that the process described above is an example of correction in which protruding parts 156 are represented by a mesh and pasted on with a UV map.

[0209] The above-mentioned processes executed by the 2D image transformation processing unit 68 allow a viewer viewing the display 508 to view an image that is as natural as possible, in which the forward protrusions 156 of the skirt, shoes, etc. of the human character that become protrusions 156 when facing forward, as in the "corrected" image shown in Fig. 30. Note that in this embodiment, the skirt and shoes of the human character are exemplified as the protrusions 156 to be corrected, but the present invention is not limited to these. Protrusions 156 may be extracted according to the characteristics of the two-dimensional image, and may be added appropriately to and the setting values ​​changed for each setting item such as "mesh shape" in the GUI (2D image editing screen) exemplified in Fig. 26.

[0210] After step S1410, the 2D image transformation processing unit 68 executes a process of adding 1 to j (j=j+1) as a coordinate movement process (step S1411).

[0211] Next, the 2D image transformation processing unit 68 executes a process of adding 1 to k (k=k+1) as a coordinate movement process (step S1412).

[0212] The 2D image transformation processing unit 68 advances the 2D image number of the changed 2D image to the next one, and returns to the loop of step S1403. If there is no next 2D image number, the 2D image transformation processing unit 68 ends the loop processing (step S1413).

[0213] [Effects of the embodiment] As described above, the PC 50 according to this embodiment accepts input of a perspective projection image 600 indicating an image in a perspective projection method to be synthesized with an equirectangular projection image 400 obtained by equirectangular projection from a plurality of images captured by the imaging device 10 (step S2001), and performs a process of increasing the degree of distortion of the perspective projection image 600 according to the distance of the equirectangular projection image 400 from the equator (step S1409), thereby obtaining a deformed perspective projection image, which is synthesized with the equirectangular projection image 400 and displayed on the display 508 (step S1410).

[0214] According to this embodiment, by performing processing to increase the degree of distortion of the perspective projection image 600 depending on the distance from the equator of the equirectangular projection image 400, it becomes possible to display a two-dimensional image synthesized with the equirectangular projection image 400 as an image that looks as natural as possible.

[0215] Moreover, the PC 50 according to this embodiment displays on the display 508 a transformed image in which the protruding parts 156 of the perspective projection image are emphasized (step S1410), so that the protruding parts 156 of the two-dimensional image (perspective projection image), i.e., the protruding parts of a skirt, shoes, etc., are reproduced more accurately, and the viewer can see an image similar to that seen in the real world. Note that the editor and the viewer in the embodiment may be different persons or may be the same person.

[0216] <Supplementary Information> Each function of the above-mentioned embodiment can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each of the above-mentioned functions.

[0217] In addition, the various tables in the above-mentioned embodiments may be generated by the learning effect of machine learning, and the data of each associated item may be classified by machine learning, making it unnecessary to use tables. Here, machine learning is a technology for making a computer acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates an algorithm required for judgment such as data identification from learning data that is previously captured, and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods, and any learning method for machine learning may be used.

[0218] Although an image display method, a program, an image display device, and an image display system according to one embodiment of the present invention have been described above, the present invention is not limited to the above-described embodiment, and may be modified within the scope of what a person skilled in the art can conceive, such as adding, changing, or deleting other embodiments, and any aspect is within the scope of the present invention as long as it achieves the functions and effects of the present invention. [Explanation of symbols]

[0219] 1 Image display system 9. Communication Networks 10. Photographing equipment (examples of photographing equipment) 50 PC (an example of an image display device) 52 Reception unit (an example of a perspective projection image reception means) 53 Display control unit (an example of a display control means) 56 Photographed image processing unit (an example of a drawing means) 58 Grid setting section 61 Rotation processing section 62 Grid / ruler processing section 64 Perspective projection image processing unit (an example of an image processing means) 68 2D image transformation processing unit 68 (an example of an image processing means) 200 Drawing input screen 300 drawing area 310 Grid Images 330 Ruler Images 400 Equirectangular projection image (example of the first image) 508 Display (an example of a display means) 600 Perspective projection image (an example of the second image) [Prior art documents] [Patent documents]

[0220] [Patent Document 1] JP 2012-80433 A

Claims

1. An image display device that displays an equirectangular projection image from a plurality of images captured by an image capture device, a perspective projection image receiving means for receiving an input of a perspective projection image to be synthesized with the equirectangular projection image; a display control means for controlling a degree of distortion of the perspective projection image in accordance with a distance from the equator in the equirectangular projection image, and displaying the modified perspective projection image on a display means in a state in which the modified perspective projection image is synthesized with the equirectangular projection image; having The display control means An image display device characterized in that the deformed perspective projection image obtained by performing processing to increase the protrusion degree of a protruding portion of an object represented in the perspective projection image is synthesized with the equirectangular projection image and displayed on the display means.

2. The display control means 2. The image display device according to claim 1, characterized in that the deformed perspective projection image obtained by performing a process that increases the degree of distortion of the perspective projection image the farther away from the equator in the equirectangular projection image, is displayed on the display means in a state in which the deformed perspective projection image is synthesized with the equirectangular projection image.

3. 3. The image display device according to claim 2, wherein the protruding portion of the object protrudes toward the center of a solid sphere onto which the equirectangular projection image is attached in a three-dimensional virtual space.

4. a height information receiving means for receiving an input of height information indicating a height from a floor surface within the structure represented by the equirectangular projection image; a length information receiving means for receiving an input of length information indicating a length of the object; a distance information receiving means for receiving an input of distance information indicating a horizontal distance from the center of the three-dimensional sphere to the floor surface with respect to the perspective projection image; an image processing means for performing processing to increase the degree of distortion of the perspective projection image based on the height information, the length information, and the distance information; 4. The image display device according to claim 3, further comprising:

5. a grid information receiving means for receiving an input of grid information indicating a grid for specifying a relative position of the perspective projection image; a grid generating unit configured to generate the grid based on the input of the grid information received by the grid information receiving unit, 5. The image display device according to claim 4, wherein the image processing means combines the grid generated by the grid generating means, the equirectangular projection image, and the modified perspective projection image.

6. The unevenness information receiving means further receives input of unevenness information indicating information on unevenness of the protrusion of the object, 5. The image display device according to claim 4, wherein the image processing means performs processing for increasing the degree of protrusion based on the height information, the length information, the distance information, and the unevenness information.

7. The unevenness information receiving means further receives an input of a magnification of the protrusion, 7. The image display device according to claim 6, wherein the image processing means performs processing to increase the degree of protrusion based on the magnification.

8. 8. The image display device according to claim 4, wherein the image processing means performs the process of increasing the degree of distortion and the process of increasing the degree of prominence by calculations including trigonometric functions.

9. The display control means 9. An image display device according to claim 4, characterized in that at least one of a first screen including the equirectangular projection image and the height information receiving means, and a second screen including the perspective projection image and the length information receiving means is displayed on the display means.

10. An image display method executed by an image display device that displays an equirectangular projection image obtained by equirectangular projection from a celestial sphere image captured by an image capture device, comprising: a perspective projection image receiving step of receiving an input of a perspective projection image to be synthesized with the equirectangular projection image; a display control step of displaying, on a display means, a deformed perspective projection image obtained by controlling a degree of distortion of the perspective projection image according to a distance from the equator in the equirectangular projection image, in a state where the deformed perspective projection image is synthesized with the equirectangular projection image; Perform a process including The display control step includes: An image display method, characterized in that the deformed perspective projection image obtained by performing processing to increase the protrusion degree of a protruding portion of an object represented in the perspective projection image is synthesized with the equirectangular projection image and displayed on the display means.

11. A program for causing a computer to execute the image display method according to claim 10.

Images