Image processing device, image processing system, image processing program, and image processing method

Abstract

To provide an image display device having a viewable area in which a viewing area in which a visibility of a user is lowered or a viewing area changing operation becomes difficult is restricted when upper and lower ends of a captured image of a VR content become the viewing area.SOLUTION: An image processing device of the present invention includes: receiving means of receiving an operation of changing a display range of an image displayed on display means; and display control means of converting a predetermined range of a hemispherical image into a planar image and display a planar image on the display means based on an operation. The display control means performs control such that a region of the planar image corresponding to the predetermined region including at least an upper end position or a lower end position of the hemispherical image is not displayed at a center position of the display range even if the region is displayed in the display range.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an image processing device having a VR playback function. [Background technology]

[0002] Virtual reality (VR) has become popular in recent years. VR content includes photos and videos, which are captured using an imaging device capable of simultaneously capturing a wide-angle image for the right eye and a half-spherical image for the left eye, or a wide-angle image such as a full-spherical image, for the purpose of utilizing the parallax between the eyes to achieve stereoscopic vision.

[0003] The captured VR content is primarily viewed through a non-transparent head-mounted display (HMD) or a display such as a smartphone or computer, resulting in an image obtained by perspective projection transformation that transforms a portion of the viewing area of ​​the captured image. When viewing VR content captured through a display such as a smartphone or computer, the viewing area can be changed by touch operation or by operating a mouse or button, and techniques for limiting the viewable area are known. For example, in Patent Document 1, the viewable area is defined as the area within the captured image, and if the viewing area extends outside the captured image, it can be restored to the viewable area that was previously viewed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-84529 Summary of the Invention [Problem to be solved by the invention]

[0005] When the VR content is a hemispherical image or an image in which the captured image is not present in some areas, it is conceivable that the user will face the edge of the captured image to check whether there is any extraneous reflection, particularly at the edge of the captured image. However, with the technology disclosed in the above-mentioned patent document, the viewable area is limited to within the captured image, so the edge of the captured image can only be displayed at the edge of the viewing area and cannot be checked from the front.

[0006] Furthermore, if there is no limit to the viewable area, it is possible to view the captured image by facing the edge of the captured image, but the area outside the captured image will be displayed in the viewing area, as shown in Figure 5(c). Such areas outside the captured image are generally represented by being filled in with black. In particular, when the viewing area is at the top or bottom of the captured image and the captured image and the blacked-out areas outside the captured image are simultaneously displayed, if an operation to change the viewing area to the left or right is performed, the captured image and the blacked-out areas outside the captured image will be displayed as if they are rotating and transitioning. When displayed in this manner, the image displayed on the screen changes drastically, reducing user visibility. Furthermore, the more the operation to change the viewing area to the left or right is performed at the center of the captured image, the greater the amount of rotation of the displayed image in the viewing area, making it difficult for the user to determine the desired position of the viewing area.

[0007] Therefore, the object of the present invention is to provide an image display device having a viewable area that limits the viewing area, which can reduce user visibility and make it difficult to change the viewing area when the upper and lower edges of a captured image of VR content become the viewing area. [Means for solving the problem]

[0008] One aspect of the present invention is an image processing device including: a receiving means for receiving an operation to change a display range of an image displayed on a display means; and a display control means for converting a predetermined range of a semi-spherical image into a planar image based on the operation and controlling the image to be displayed on the display means, wherein the display control means controls the planar image so that an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the semi-spherical image is not displayed at a center position of the display range even if it is displayed in the display range. [Effects of the Invention]

[0009] According to the present invention, even when the upper and lower edges of a captured image of VR content are included in the viewing area, a decrease in visibility for the user can be prevented, and the user can easily check the captured image and change the viewing area. [Brief explanation of the drawings]

[0010] [Figure 1] 1A is an external view of a display device 100. FIG. 1B is a configuration block diagram of the display device 100. [Figure 2] 1A is a circular fisheye image in VR content, FIG. 1B is a diagram showing an area to be subjected to perspective projection transformation in the circular fisheye image, and FIG. 1C is a diagram for explaining an image subjected to perspective projection transformation. [Figure 3] 10A and 10B are diagrams for explaining the correspondence between a circular fisheye image and a hemisphere in a three-dimensional virtual space. [Figure 4] FIG. 10 is a diagram showing the positions of a virtual camera in a three-dimensional virtual space and an area where perspective projection transformation is performed in a hemispherical image. [Figure 5] 10A and 10B are diagrams illustrating an image generated by performing perspective projection transformation on a hemispherical image in a three-dimensional virtual space. [Figure 6] 1 is a flowchart showing a processing flow according to the first embodiment of the present invention. [Figure 7] 1 is a flowchart showing a processing flow according to the first embodiment of the present invention. [Figure 8] FIG. 2 is a diagram showing a restricted range of an area where perspective projection transformation is performed according to the first embodiment of the present invention. [Figure 9] FIG. 2 is a diagram showing a restricted range of an area where perspective projection transformation is performed according to the first embodiment of the present invention. [Figure 10] 10 is a flowchart showing a processing flow according to a second embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing a movement amount reduction range of an area where perspective projection transformation is performed according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] (Embodiment 1) FIG. 1(a) shows an example of an external view of a display control device 100, which is a type of electronic device.

[0013] The display 105 is a display unit that displays images and various information. As will be described later, the display 105 is configured integrally with a touch panel 106a, and is capable of detecting touch operations on the display surface of the display 105. The display control device 100 is capable of displaying a perspective projection converted image, which is a planar image obtained by performing perspective projection conversion on a photograph or video of VR content on the display 105. The display of the perspective projection converted image is a display format equivalent to the display format when viewing with an HMD, and will hereinafter be referred to as VR display.

[0014] As shown in the figure, the operation unit 106 includes a touch panel 106a and operation units 106b, 106c, 106d, and 106e. The operation unit 106b is a power button that accepts an operation to switch the power of the display control device 100 on and off. The operation units 106c and 106d are buttons to which various operations can be assigned. For example, when VR is displayed on the display 105, an operation to change the viewing area or the zoom level, or an operation to increase or decrease the volume of the audio output from the audio output unit 112 can be assigned. Here, the operation to change the zoom level includes an operation to enlarge and an operation to reduce. The operation unit 106e is a home button for displaying a home screen on the display 105. Note that the operation unit may be a device separate from the display device, and may be an image processing system including at least a display device having a display unit and a controller having an operation unit for operating the display device.

[0015] The audio output terminal 112a is an earphone jack for outputting audio to earphones, an external speaker, etc. The speaker 112b is a built-in speaker for producing audio.

[0016] 1(b) shows an example of the configuration of a display control device 100 as an example of a device to which the present invention can be applied. The display control device 200 can be configured using a display device such as a smartphone.

[0017] 1, a CPU 101, a volatile memory 102, a nonvolatile memory 103, an image processing unit 104, a display 105, an operation unit 106, a recording medium I / F 107, an external I / F 109, a communication I / F 110, and an audio output unit 112 are connected to an internal bus 150. The units connected to the internal bus 150 are capable of exchanging data with each other via the internal bus 150.

[0018] The volatile memory 102 is, for example, a RAM (a volatile memory using a semiconductor element, etc.). The CPU 101 is a processor that controls each unit of the display control device 100 using the volatile memory 102 as a work memory in accordance with a program stored in, for example, the nonvolatile memory 103. The nonvolatile memory 103 stores image data, audio data, other data, various programs for the operation of the CPU 101, etc. The nonvolatile memory 103 is, for example, formed from a hard disk (HD) or a ROM.

[0019] The image processing unit 104 is a processor that performs various image processing operations on image data stored in the nonvolatile memory 103 or the recording medium 108, video signals acquired via the external I / F 109, and image data acquired via the communication I / F 110 under the control of the CPU 101. The image processing operations performed by the image processing unit 104 include A / D conversion, D / A conversion, image data encoding, compression, decoding, enlargement / reduction (resizing), perspective projection conversion, noise reduction, and color conversion. The image processing unit 104 may be configured with a dedicated circuit block for performing specific image processing operations. Depending on the type of image processing, the CPU 101 may perform image processing according to a program without using the image processing unit 104. The image processing unit 104 may also be incorporated into the CPU 101. The image processing unit 104 may also be implemented by, for example, a graphics processing unit (GPU). The GPU may be independent of the CPU 101 or may be located on the same chip as the CPU 101.

[0020] Display 105 displays images, GUI screens constituting a GUI (Graphical User Interface), and the like under the control of CPU 101. CPU 101 generates a display control signal in accordance with a program, and controls each unit of display control device 100 to generate a video signal to be displayed on display 105 and output it to display 105. Display 105 displays an image based on the output video signal. Note that the display control device 100 itself is only provided with an interface for outputting a video signal to be displayed on display 105, and display 105 may be configured as an external monitor (such as a television).

[0021] The operation unit 106 is an input device for accepting user operations, and includes a character information input device such as a keyboard, a pointing device such as a mouse or touch panel, a button, a dial, a joystick, a touch sensor, a touchpad, etc. The operation unit 106 is a user interface that allows the user to select and input a display method for image data stored in the recording medium 108. The display method can be selected from at least a circular fisheye display of the circular fisheye image itself, and a perspective projection transformation display (VR display) in which perspective projection transformation is applied to the circular fisheye image.

[0022] The touch panel is an input device that is configured as a plane overlaid on the display 105 and outputs coordinate information according to the touched position.

[0023] The storage medium I / F 107 allows a storage medium 108 such as a memory card, CD, or DVD to be attached, and reads data from the attached storage medium 108 and writes data to the storage medium 108 based on the control of the CPU 101.

[0024] The external I / F 109 is an interface for connecting to an external device via a wired cable or wirelessly, and for inputting and outputting video signals and audio signals.

[0025] The communication I / F 110 is an interface for communicating with external devices directly or via the Internet to send and receive various data such as files and commands.

[0026] The audio output unit 112 outputs audio from video and music data, operation sounds, ringtones, various notification sounds, etc. The audio output unit 112 includes an audio output terminal for connecting earphones or the like, and a speaker, but audio output may also be performed via wireless communication or the like.

[0027] The display device may be a head-mounted display or AR glasses. In the case of a head-mounted display, the image displayed on the display may be image-processed so that the user perceives it as a planar image. For example, a circular fisheye image may be converted into a planar image by perspective projection transformation, and then the image may be processed so that the user perceives it as a planar image, and the image may be displayed on a display for the right eye and a display for the left eye.

[0028] The inertial measurement unit 113 is a sensor for detecting the position and orientation of the display control device 100. The inertial measurement unit 113 has an inertial measurement unit (IMU) configured with inertial sensors such as an acceleration sensor and an angular acceleration sensor. The inertial measurement unit 113 is used to acquire position information and orientation information of a user, and the CPU 101 acquires the user's position information and orientation information from the inertial measurement unit 113. Note that the inertial measurement unit 113 may be capable of detecting only orientation information, only position information, or both orientation information and position information. In other words, it may be capable of detecting at least one of orientation information and position information. Note that the inertial measurement unit 113 may include a geomagnetic sensor that is a sensor for detecting the orientation of the display control device 100. The CPU 101 acquires information on the orientation of the display control device 100 from the geomagnetic sensor.

[0029] The image processing unit 104 includes a perspective projection transformation processing unit 104a. The perspective projection transformation processing unit 104a performs perspective projection transformation processing when the user selects VR display of image data stored in the recording medium 108 via the operation unit 106. Since perspective projection transformation is performed by setting a viewing angle, only a partial area of ​​the captured image is transformed and generated. Here, the perspective projection transformed image generation method according to the first embodiment will be described in detail with reference to FIGS. 2 and 3, taking the case of capturing a hemispherical image as an example.

[0030] Fig. 2(a) is a diagram showing an image captured when a fisheye lens is used with the imaging device according to the first embodiment. As shown in Fig. 2(a), the image data stored on the recording medium 108 is a circularly cropped and distorted image. For example, by performing perspective projection transformation on a portion of the image, such as the area surrounded by the dotted line in Fig. 2(b), it is possible to produce a display similar to what the user would see on an HMD, as shown in Fig. 2(c).

[0031] The perspective projection transformation processing unit 104a first uses a 3D computer graphics library or the like to draw a hemisphere as shown in Fig. 3(a) and pastes a circular fisheye image inside it. An example of a 3D computer graphics library is Open GL ES (Open Graphics Library For Embedded Systems).

[0032] When a circular fisheye image is pasted inside a hemisphere, for example, as shown in FIG. 3(b), the circular fisheye image is associated with a coordinate system consisting of a vertical angle θ with the zenith direction of the captured image as the axis and a horizontal angle φ around the zenith axis. In this case, if the field of view range of the circular fisheye image is 180°, the vertical angle θ and horizontal angle φ will be in the range of -90° to 90°. The coordinate values ​​(θ, φ) of the circular fisheye image can be associated with each point on the spherical surface representing the hemispherical image shown in FIG. 3(a). If the center of the hemisphere shown in FIG. 3(a) is set to 0 and the three-dimensional coordinates on the spherical surface are (X, Y, Z), the relationship between the circular fisheye image and the two-dimensional coordinates can be expressed by (Equation 1), (Equation 2), and (Equation 3). Here, r is the radius of the hemisphere. By pasting the circular fisheye image inside the hemisphere based on the coordinate correspondences shown by these equations, a hemispherical image can be generated in a three-dimensional virtual space.

[0033]

number

[0034]

number

[0035]

number

[0036] When generating a spherical image, circular fisheye images 180° in front of and 180° behind the photographer are acquired, and each half-spherical image is generated by the above-mentioned means, and then these are stitched together to generate a 360° spherical image.

[0037] As described above, the spherical image and the semi-spherical image are images pasted so as to cover the spherical surface, and therefore are not images that the user can view on the display control device 100 as they are.

[0038] FIG. 4 is a diagram showing the positional relationship between a virtual camera 401 in a three-dimensional virtual space in a hemispherical image and a region 410 for determining the region where perspective projection transformation is performed. The virtual camera 401 corresponds to the viewpoint of a user viewing the hemispherical image displayed as a three-dimensional hemisphere. The region 410 is determined by the range of the virtual camera's viewing angle, with the virtual camera's direction determined based on three-dimensional coordinates (X, Y, Z) 411 on the sphere as the center. Perspective projection transformation is performed on an image of the region on the sphere included in the region 410 as viewed from the virtual camera 401, and the image is displayed on the display 105. In this embodiment, the region where perspective projection transformation is performed is defined as the viewing range, and the viewing range is changed by changing the orientation of the virtual camera 401 in response to a user's operation on the operation unit 106. Here, the viewing range corresponds to the display range displayed on the display 105. Furthermore, the zoom is changed by changing the vertical viewing angle α412 of the virtual camera in response to a user's operation on the operation unit 106. Here, the change in scaling is correlated such that the smaller the viewing angle α 412, the larger the VR display becomes. Furthermore, it is assumed that w 413 in FIG.

[0039] In this embodiment, the VR content will be described as a hemispherical circular fisheye image, but the VR content is not limited to this example as long as it is a photograph or video. For example, an equirectangular image converted from a circular fisheye image may also be used.

[0040] If the operation unit 106 includes a touch panel 106a, the CPU 101 can detect the following operations or states on the touch panel 106a. A finger or pen that has not been touching the touch panel 106a touches the touch panel 106a again, that is, the start of touching (hereinafter referred to as touch-down). The touch panel 106a is in a state of being touched with a finger or a pen (hereinafter referred to as Touch-On). Touching the touch panel 106a with a finger or a pen and moving it (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 106a is released, that is, the touch ends (hereinafter referred to as "touch-up"). A state in which nothing is touching the touch panel 106a (hereinafter referred to as Touch-Off).

[0041] When touch down is detected, touch on is also detected at the same time. After touch down, touch on will usually continue to be detected unless touch up is detected. Touch move is also detected when touch on is detected. Even if touch on is detected, touch move will not be detected unless the touch position moves. Once it is detected that all fingers or pens that were touching have touched up, touch off occurs.

[0042] These operation states and the position coordinates of the finger or pen touching the touch panel 106a are notified to the CPU 101 via the internal bus, and the CPU 101 determines what kind of operation (touch operation) has been performed on the touch panel 106a based on the notified information. Regarding touch-move, the movement direction of the finger or pen moving on the touch panel 106a can also be determined for each vertical and horizontal component on the touch panel 106a based on changes in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 106a, moved quickly for a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced across the touch panel 106a as if flicking it. If a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is then detected, it can be determined that a flick has been performed (it can be determined that a flick occurred following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched simultaneously and the touch positions are brought closer together is called a pinch in, and a touch operation in which the touch positions are moved farther apart is called a pinch out. Pinch out and pinch in are collectively called a pinch operation (or simply pinch). The touch panel 106a may be of any of various types, such as a resistive film type, a capacitance type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, or an optical sensor type. Depending on the type, there are types that detect a touch by contact with the touch panel, and types that detect a touch by the approach of a finger or a pen to the touch panel, but either type is acceptable.

[0043] In the above-described embodiments, the present invention has been described with reference to a smartphone as an example, but is not limited to this example. That is, the present invention may also be applied to an imaging device such as a digital camera. That is, the present invention may also be applied when a captured image recorded on a recording medium readable by the digital camera, such as a memory card, is played back and displayed on a display such as a rear LCD device of the digital camera. Furthermore, the present invention may also be applied to a personal computer, a tablet terminal, a portable image viewer, a printer device with a display, a digital photo frame, a music player, a game console, or an e-book reader. It may also be a head-mounted display device worn on the user's head, such as an HMD or AR glasses.

[0044] As an example of an embodiment, a situation will be described in which, when perspective projection conversion is performed by the perspective projection conversion processing unit 104a of the display control device 100, if the VR content is a hemispherical circular fisheye image, the perspective projection conversion is performed with the region including the upper end of the captured image as the viewing region. Figure 5 is a diagram showing this situation.

[0045] FIG. 5(a) shows a hemisphere 502 drawn in a three-dimensional solid-line area, with a circular fisheye image pasted inside it. In other words, in FIG. 5(a), a circular fisheye image is drawn inside the surface of the hemisphere 502 in the positive z-axis direction from the xy plane. A virtual camera 501 is placed at the three-dimensional center (0,0,0) and is looking up toward the top of the three-dimensional y-axis. As a result, a viewing area 510 is positioned perpendicular to the y-axis. The radius of the hemisphere 502 is set to 1. FIG. 5(b) shows a top view of FIG. 5(a). FIG. 5(c) shows a view generated by perspective projection transformation when the virtual camera 501 in FIG. 5(a) is rotated around the y-axis (commonly called pan rotation) while looking toward the top of the y-axis. Figures 5(c), 5(d), 5(e), 5(f), and 5(g) correspond to the arrows (1), (2), (3), (4), and (5) in Figure 5(a), respectively, and the arrows indicate the downward direction of each image after perspective projection transformation. Note that Figure 5(a) corresponds to (1), (2), (3), (4), and (5) in Figure 5(b), respectively.

[0046] As shown in Figure 5(e), when virtual camera 501 is facing upward on the y-axis, the viewing area of ​​the perspective projection transformation includes captured image portion 511 and portion outside captured image 512. Note that there are several ways to represent portion outside captured image 512, but in this example, it is displayed in black. When virtual camera 501 is panned in this situation, captured image portion 511 and portion outside captured image 512 are displayed on display 105 so as to rotate and transition on the VR display, as shown in Figures 5(c), 5(d), 5(e), 5(f), and 5(g).

[0047] The rotational transitions of the captured image portion 511 and the portion outside the captured image 512 in Figures 5(c), 5(d), 5(e), 5(f), and 5(g) are significantly reflected in the change in the image displayed on the display 105, which is thought to lead to a decrease in the visibility of the image for the user.

[0048] Furthermore, if the user's pan rotation input is not a three-dimensional input but a two-dimensional input, such as touching and moving the touch panel 106a, it must be converted into a three-dimensional input relative to the hemisphere before perspective projection conversion. One possible method for achieving this is to move the two-dimensional coordinates on the circular fisheye image along the horizontal axis and then convert that position to three-dimensional coordinates. Looking at the coordinate system in Figure 3(b), which maps the circular fisheye image to the vertical and horizontal angles of the three-dimensional coordinates on the spherical surface, the closer the horizontal angle φ corresponding to the pan rotation is to the zenith axis of the circular fisheye image, the larger the scale on the circular fisheye image, especially the closer it is to the edge of the zenith axis. Therefore, when the virtual camera 501 is panned from the situation in Figure 5(c) by inputting two-dimensional coordinates, the amount of rotation becomes large, especially in the ranges of Figures 5(d), 5(e), and 5(f), making it difficult for the user to move to the desired position.

[0049] As a process for solving the above problem, an example of a process for restricting the viewing area for the upper and lower edges of a captured image of VR content, which is a feature of the present invention, will be described with reference to Figures 6 to 9. Note that the upper edge of the captured image of VR content is an area that includes at least the upper edge position of the captured image of VR content, and the lower edge of the captured image of VR content is an area that includes at least the lower edge position of the captured image of VR content.

[0050] 6 shows a flow for performing perspective projection conversion on VR content, displaying it in VR, and accepting a change in the viewpoint area. The display control device 100 executes the process when it is displaying an image that can be displayed in VR by perspective projection conversion, such as a circular fisheye image, on the display 105, or when it has accepted a user operation to display it.

[0051] In step S601, if CPU 101 receives a user operation to start VR display from operation unit 106, it proceeds to step S602, and if not, it returns to step S601. The selection to start VR display may be received for an image that can be displayed in VR as a circular fisheye image on display 105, or when selecting an image to display on display 105, a selection of the display method, whether to display as a circular fisheye image or to display in VR, may be received.

[0052] In step S602, CPU 101 initializes and stores viewing area parameters that determine the viewing area of ​​the VR display, and then proceeds to step S603. In this embodiment, the viewing area parameters are the coordinates that form the center of the viewing area during perspective projection transformation on the three-dimensional coordinate system, or the corresponding two-dimensional coordinates on the circular fisheye image and the vertical viewing angle during perspective projection transformation (viewing angle α412 in FIG. 4). Among the three-dimensional coordinates on the circular fisheye image, the coordinates that form the center of the viewing area during perspective projection transformation are coordinates 411 in FIG. 4, and will hereinafter be referred to as the viewing area center coordinates. In this embodiment, the two-dimensional viewing area center coordinates on the circular fisheye image and the vertical viewing angle during perspective projection transformation are stored. The initialization may be performed using any value, or the parameter values ​​used during the immediately preceding perspective projection transformation may be used.

[0053] In step S603, the CPU 101 notifies the perspective projection transformation processing unit 104a of the circular fisheye image and the viewing area parameters, commands perspective projection transformation processing, and proceeds to step S604. The perspective projection transformation processing unit 104a performs perspective projection transformation processing on the circular fisheye image in response to the command, generating an image for VR display. Note that the process of adjusting the horizontal and vertical resolutions of the generated image to those of the display 105 may be performed by the perspective projection transformation processing unit 104a, or a resizing process or the like included in the image processing unit 104 may be used.

[0054] In step S604, the CPU 101 receives the generated image, displays it on the display 105, and the process proceeds to step S605.

[0055] In step S605, CPU 101 determines whether a change in the viewing area has been received through a user operation on operation unit 106. If CPU 101 determines that a change in the viewing area has been received, the process proceeds to step S606. If CPU 101 does not determine that a change in the viewing area has been received, the process proceeds to step S607. For example, when a touch-and-move is performed on the VR display on touch panel 106a, or when operation units 106c and 106d are pressed while directional key operations are assigned to them, CPU 101 can determine that a change in the viewing area involves a movement of position. Also, when a pinch-in or pinch-out is performed on the VR display on touch panel 106a, CPU 101 can determine that a change in the viewing area involves enlargement or reduction. Note that an operation for changing the enlargement or reduction level may be assigned to operation units 106c and 106d, and when operation unit 106c or operation unit 106d is pressed, CPU 101 can determine that a change in the viewing area involves enlargement or reduction.

[0056] In step S606, CPU 101 determines the viewing area based on the user operation in step S605 using the viewing area control process described below. After determining the viewing area, the process proceeds to step S603, where perspective projection conversion processing and updating of the VR display are performed.

[0057] In step S607, if the CPU 101 determines that a user operation to end the VR display has been received from the operation unit 106, the processing ends. If the CPU 101 does not determine that a user operation to end the VR display has been received, the processing proceeds to S605, where the VR display state continues. When the processing ends, a circular fisheye image of the image that was being VR displayed may be displayed, or the state may return to one in which the image display can be selected. For example, the processing may end in conjunction with a power-off process in response to a user operation received from the operation unit 106b, or in conjunction with a transition to the home screen in response to a user operation received from the operation unit 106e.

[0058] Next, a specific example of the line-of-sight area control process in step S606 in FIG. 6 will be described with reference to FIGS.

[0059] FIG. 7 shows a flow of processing for restricting the viewing area at the top and bottom of a captured image of VR content.

[0060] 8 is a diagram showing the range (hereinafter referred to as viewing restriction range) in which a viewing area restriction is applied to a hemispherical circular fisheye image of VR content. The viewing restriction range is restricted so that the viewing area center coordinates corresponding to the center coordinates of the display area displayed on display 105 do not fall within the area. The method for determining the viewing restriction range will be described later, but rectangular ranges 811 and 812, and ranges 813 and 814 are paired because restrictions are applied to the top and bottom, respectively. Furthermore, ranges 811 and 813, and ranges 812 and 814 may overlap each other.

[0061] The viewing restriction range may overlap with the viewing area of ​​the perspective projection transformation, and in this embodiment, a restriction is imposed on the two-dimensional coordinates on the circular fisheye image before it is converted into coordinates that become the center of the viewing area during the perspective projection transformation in three dimensions. For example, if ranges 811 and 813 are set as viewing restriction ranges, processing is performed so that the viewing area center coordinates 801 cannot be set within the viewing restriction range. Note that a viewpoint area restriction may also be imposed on a three-dimensional coordinate system converted from a two-dimensional coordinate system into a state in which the circular fisheye image is pasted onto a hemisphere.

[0062] 8, the movable area of ​​the viewing area center coordinates is limited at the top and bottom of the circular fisheye image, but not at the right and left. If necessary, the movable area of ​​the viewing area center coordinates may also be limited at the right and left ends. For example, the movable area of ​​the viewing area center coordinates may be limited at the right and left ends, narrower than the limited areas at the top and bottom.

[0063] FIG. 9 is a diagram showing the positions of viewing area center coordinates 901a and 901b when the viewing restricted area is changed from range 911 to range 913 by enlargement / reduction processing.

[0064] The processing in FIG. 7 starts when the line-of-sight area control processing in step S606 in FIG. 6 is entered.

[0065] In step S701, CPU 101 determines the changed viewpoint area based on the viewpoint area parameters changed in the user operation received in step S605 of Fig. 6. For example, if the received user operation is an operation to move the position of the viewing area, the center coordinates of the viewing area are changed. Also, if the received user operation is a change in scaling, the vertical field of view angle during perspective projection transformation is changed.

[0066] In step S702, CPU 101 determines whether the current display state is VR display. If so, the process proceeds to step S703; if not, the process proceeds to step S707. This is useful when the display state is a mixture of circular fisheye image display and VR display, and the same flow is used for operations in an enlarged state while displaying a circular fisheye image, for example, and the viewpoint control area is not required. Therefore, if the flow always passes through in VR display state, it can be omitted.

[0067] In step S703, the CPU 101 determines whether the input from the operation unit 106 in the user operation accepted in step S605 of Fig. 6 was a two-dimensional operation that functions as a directional key, such as the touch panel 106a or the operation units 106c and 106d. If it is determined that the input was a two-dimensional operation, the process proceeds to step S704; if it is determined that the input was not a two-dimensional operation, the process proceeds to step S707. Step S703 is useful when three-dimensional operation is possible using a gyro sensor that detects the orientation of the display control device 100 as input, that is, when the center coordinate information of the three-dimensional viewing area can be changed by three-dimensional operation and when a viewpoint control area is not required.

[0068] In step S704, CPU 101 obtains the viewable range from the vertical viewing angle of the viewpoint area parameter. The viewable range can be set using a calculation formula with the vertical viewing angle as a parameter relative to the vertical axis of the circular fisheye image. The range not included in the viewable range becomes viewing restriction ranges 811-814 in FIG. 8 and viewing restriction ranges 911 and 913 in FIG. 9. Specifically, the center of the vertical axis of the circular fisheye image is set to 0, and boundary values ​​for the positive and negative vertical axes are calculated using a calculation formula including a constant value with the vertical viewing angle as a parameter. The viewable range is defined between the positive and negative boundary values. The larger the vertical viewing angle, the closer the boundary value is to the center of the vertical axis of the circular fisheye image, and the smaller the vertical viewing angle, the closer it is to the edge of the vertical axis of the circular fisheye image. Note that if the available vertical viewing angles are predetermined, the corresponding viewable range may be stored in advance. Up to this point, we have described the range of the viewable range relative to the vertical axis of the circular fisheye image, but the range relative to the horizontal axis does not necessarily have to be specified. When determining the range, for example, the width of the circular fisheye image may be set as the range, or the imaging range of the circular fisheye image may be set as the range.

[0069] In step S705, if the changed viewing area calculated in step S701 exceeds the viewable range, CPU 101 proceeds to step S706, otherwise proceeds to step S707. As described above in the explanation of the viewing restriction range, the viewing area exceeding the viewable range means that the center coordinates of the viewing area on the circular fisheye image of the viewing area parameters exceed the boundary value of the viewable range.

[0070] In step S706, CPU 101 changes the viewing area center coordinates on the circular fisheye image of the viewing area parameters to the boundary value of the viewable range. If the positive boundary value of the viewable range is exceeded, the value of the vertical axis of the viewing area center coordinates is changed to the positive boundary value. Also, if the negative boundary value is exceeded, the value of the vertical axis of the viewing area center coordinates is changed to the negative boundary value. In other words, the viewing area center coordinates are restricted so that they do not move to the upper or lower end positions of the circular fisheye image.

[0071] Furthermore, when the vertical viewing angle is changed in step S701, the viewable range is also changed in step S704 from the viewable range of the previous viewing range. After the viewable range is changed, the boundary value of the new viewable range may be closer to the center of the image. For example, in FIG. 9(a), the viewing range center coordinate 901a is located at the boundary value of the previous viewing restriction range 913 (the same as the boundary value of the viewable range). If the operation unit 106 accepts a user operation for a reduction process, i.e., an operation to widen the viewing range, the new viewing restriction range changes to range 911. In this case, as shown in FIG. 9(b), the value of the vertical axis of the viewing range center coordinate is changed to coordinate 901b so that it is located on the boundary value of the viewing restriction range 911. Conversely, if the operation unit 106 accepts a user operation for an enlargement process in the state shown in FIG. 9(b), since the previous viewing range center coordinate was the boundary value of the previous viewpoint restriction range, the new viewing range center coordinate may be changed to the boundary value of the new viewpoint restriction range. That is, when an operation to narrow the viewing area is received, the state of FIG. 9(b) may be changed to the state of FIG. 9(a).

[0072] In step S707, CPU 101 stores the changed viewpoint area and ends the viewing area control process.

[0073] It should be noted that when the viewing area is changed by a three-dimensional operation, such as an operation by changing the posture of the display control device 100, it is not necessary to set a viewing restriction range.

[0074] (Embodiment 2) Next, a display device and processing according to a second embodiment of the present invention will be described. In this embodiment, the configuration of the display device 100 is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0075] An example of a process for reducing the amount of horizontal movement (amount of change) for a partial range at the top and bottom of a captured image of VR content, which is a feature of the present invention, will be described with reference to Figures 6, 10, and 11. Note that the flowchart in Figure 6 is the same as that in the first embodiment, and therefore its description will be omitted. Also, the flowchart in Figure 10 will omit a description of the processing common to the flowchart in Figure 7 described in the first embodiment, and will only describe the differences.

[0076] FIG. 10 is a flowchart of a process for reducing the amount of horizontal movement for a partial range of the upper and lower ends of a captured image of VR content.

[0077] 11 is a diagram showing 1111 to 1114 (hereinafter referred to as movement amount reduction ranges) in which the amount of horizontal movement of VR content is reduced for a hemispherical circular fisheye image. The movement amount reduction ranges are paired, with rectangular ranges 1111 and 1112, and ranges 1113 and 1114, to impose restrictions on the top and bottom, respectively. Furthermore, ranges 1111 and 1113, and ranges 1112 and 1114 may overlap each other.

[0078] In this embodiment, the movement amount reduction range is set on the two-dimensional coordinates of the circular fisheye image before it is converted into coordinates that become the center of the viewing area during three-dimensional perspective projection transformation. Note that the range may also be set on a three-dimensional coordinate system converted from the two-dimensional coordinate system into a state in which the circular fisheye image is pasted onto a hemisphere.

[0079] The flow in FIG. 10 starts when the gaze area control process in S606 in FIG. 6 is entered.

[0080] Steps S1001 to S1003 are common processes to steps S701 to S703 in FIG. 7 of the first embodiment.

[0081] In step S1004, CPU 101 acquires a movement amount reduction range from the vertical viewing angle of the viewpoint area parameter. The movement amount reduction range for the vertical axis on the circular fisheye image is the same as the viewing restriction range described in embodiment 1. The range for the horizontal axis is set to 0 at the image center of the horizontal axis of the circular fisheye image, and positive and negative boundary values ​​are used. Note that the boundary value for the horizontal axis may be a predetermined value, or may be determined, for example, from the vertical viewing angle.

[0082] In step S1005, if the changed viewing area calculated in step S1001 is within the movement amount reduction range, CPU 101 proceeds to step S706, and if not, proceeds to step S707.

[0083] In step S1006, CPU 101 reduces the amount of movement of the viewpoint area parameter relative to the horizontal axis of the viewing area center coordinates, and recalculates the changed viewing area. The amount of movement to be reduced is calculated according to a calculation formula for reducing the amount of movement, such as multiplying by a predetermined numerical value equal to or less than 1.

[0084] Step S1007 is the same process as step S707 in FIG. 7 of the first embodiment.

[0085] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0086] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0087] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0088] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0089] In addition, in each of the examples described above, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0090] [Configuration 1] a receiving means for receiving an operation to change a display range of an image displayed on a display means; a display control means for converting a predetermined range of the hemispherical image into a planar image based on the operation and controlling the display means to display the planar image; The display control means controls the display so that an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image is not displayed at a center position of the display range even if it is displayed in the display range. 1. An image processing device comprising:

[0091] [Configuration 2] The receiving means receives an operation via a touch panel or a button. 2. The image processing device according to configuration 1,

[0092] [Configuration 3] The receiving means receives an operation to change the position of the display range or an operation to change the size of the display range. 3. The image processing device according to configuration 1 or 2.

[0093] [Configuration 4] The display control means converts the hemispherical image into a planar image by perspective projection transformation. 4. The image processing device according to any one of configurations 1 to 3.

[0094] [Configuration 5] When the receiving means receives an operation to change the size of the display range, the receiving means changes the position and size of the display range so that an area of ​​the planar image corresponding to a predetermined area including at least the upper end position or the lower end position of the hemispherical image is not displayed at the center of the display range. 5. The image processing device according to any one of configurations 1 to 4.

[0095] [Configuration 6] The display control means changes the amount of change in the position of the display range based on an operation to change the position of the display range, depending on whether an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image is displayed on the display means or not. 6. The image processing device according to any one of configurations 1 to 5,

[0096] [Configuration 7] When an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the semi-spherical image is displayed on the display means, the display control means controls the amount of change in the position of the display range based on an operation to change the position of the display range to be smaller than when an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the semi-spherical image is not displayed on the display means. 7. The image processing device according to configuration 6,

[0097] [Configuration 8] The display means is further included. 8. The image processing device according to any one of configurations 1 to 7, wherein:

[0098] [Configuration 9] further comprising an imaging means for capturing a circular fisheye image; The display control means converts the circular fisheye image captured by the imaging means into a planar image by perspective projection transformation and displays the planar image on the display means. 9. The image processing device according to any one of configurations 1 to 8, wherein:

[0099] [Configuration 10] Of the image displayed on the display means, the area that does not correspond to the planar image is displayed in a predetermined color. 10. The image processing device according to any one of configurations 1 to 9, wherein:

[0100] [Configuration 11] the display control means controls the display means to display the planar image or the circular fisheye image; When the circular fisheye image is displayed on the display means, the area of ​​the circular fisheye image corresponding to a predetermined area including at least the upper end position or the lower end position of the hemispherical image is not limited to a displayable range. 11. The image processing device according to any one of configurations 1 to 10,

[0101] [Configuration 12] When the display range is changed based on a three-dimensional operation, the display control means does not impose any limitation on the displayable range of an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image. 12. The image processing device according to claim 1, wherein the image processing device is a processor.

[0102] [Configuration 13] the receiving means receives an operation to change the display range by a change in attitude of a head-mounted display including the display means, When the display control means changes the display range based on a change in the attitude of the head-mounted display, the display control means does not impose any limitation on the displayable range of an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image. 13. The image processing device according to configuration 12.

[0103] [Configuration 14] a receiving means for receiving an operation to change a display range of an image displayed on a display means; a display control means for converting a predetermined range of the hemispherical image into a planar image based on the operation and controlling the display means to display the planar image; The display control means controls the flat image so that the position of the flat image corresponding to the upper end position or the lower end position of the semi-spherical image is not displayed at the center of the display range even if the flat image is displayed in the display range. 2. The image processing device according to configuration 1,

[0104] [Control method] a receiving step of receiving an operation to change a display range of an image displayed on a display means; a display control step of converting a predetermined range of the hemispherical image into a planar image based on the operation and controlling the display means to display the planar image; The display control step controls the display so that an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image is not displayed at a center position of the display range even if it is displayed in the display range. 2. A method for controlling an image processing apparatus comprising:

[0105] [program] A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 14.

[0106] [system] a receiving device that receives an operation to change a display range of an image displayed on a display device; a display control device that controls the display device to convert a predetermined range of the hemispherical image into a planar image based on the operation, and to display the planar image on the display device; The display control device controls the display so that an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image is not displayed at a center position of the display range even if it is displayed in the display range. An image processing system comprising:

Claims

1. a receiving means for receiving an operation to change a display range of an image displayed on a display means; a display control means for converting a predetermined range of the hemispherical image into a planar image based on the operation and controlling the display means to display the planar image; The display control means controls the display so that an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image is not displayed at a center position of the display range even if it is displayed in the display range.

1. An image processing device comprising:

2. The receiving means receives an operation via a touch panel or a button.

2. The image processing device according to claim 1, wherein:

3. The receiving means receives an operation to change the position of the display range or an operation to change the size of the display range.

2. The image processing device according to claim 1, wherein:

4. The display control means converts the hemispherical image into a planar image by perspective projection transformation.

2. The image processing device according to claim 1, wherein:

5. When the receiving means receives an operation to change the size of the display range, the receiving means changes the position and size of the display range so that an area of ​​the planar image corresponding to a predetermined area including at least the upper end position or the lower end position of the hemispherical image is not displayed at the center position of the display range.

2. The image processing device according to claim 1, wherein:

6. The display control means changes the amount of change in the position of the display range based on an operation to change the position of the display range, depending on whether an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image is displayed on the display means or not.

2. The image processing device according to claim 1, wherein:

7. When an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the semi-spherical image is displayed on the display means, the display control means controls the amount of change in the position of the display range based on an operation to change the position of the display range to be smaller than when an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the semi-spherical image is not displayed on the display means.

7. The image processing device according to claim 6,

8. The display means is further included.

2. The image processing device according to claim 1, wherein:

9. further comprising an imaging means for capturing a circular fisheye image; The display control means converts the circular fisheye image captured by the imaging means into a planar image by perspective projection transformation and displays the planar image on the display means.

2. The image processing device according to claim 1, wherein:

10. Of the image displayed on the display means, the area that does not correspond to the planar image is displayed in a predetermined color.

2. The image processing device according to claim 1, wherein:

11. the display control means controls the display means to display the planar image or the circular fisheye image; When the circular fisheye image is displayed on the display means, the area of ​​the circular fisheye image corresponding to a predetermined area including at least the upper end position or the lower end position of the hemispherical image is not limited to a displayable range.

2. The image processing device according to claim 1, wherein:

12. When the display range is changed based on a three-dimensional operation, the display control means does not impose a limit on a displayable range of an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image.

2. The image processing device according to claim 1, wherein:

13. the receiving means receives an operation to change the display range by a change in attitude of a head-mounted display including the display means, When the display control means changes the display range based on a change in the attitude of the head-mounted display, the display control means does not impose any limitation on the displayable range of an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image.

13. The image processing device according to claim 12.

14. a receiving step of receiving an operation to change a display range of an image displayed on a display means; a display control step of converting a predetermined range of the hemispherical image into a planar image based on the operation and controlling the display means to display the planar image; The display control step controls the display so that an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image is not displayed at a center position of the display range even if the area is displayed in the display range.

2. A method for controlling an image processing apparatus comprising:

15. A program for causing a computer to function as each of the means of the image processing apparatus according to claim 1.

16. a receiving device that receives an operation to change a display range of an image displayed on a display device; a display control device that controls the display device to convert a predetermined range of the hemispherical image into a planar image based on the operation, and to display the planar image on the display device; The display control device controls the display so that an area of ​​the planar image corresponding to a predetermined area including at least an upper end position or a lower end position of the hemispherical image is not displayed at a center position of the display range even if it is displayed in the display range. An image processing system comprising:

Images