Image processing system, image processing method, and program
Patent Information
- Application Number
- JP2021214448
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-12-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-12-28
AI Technical Summary
Existing image processing technologies for HMDs struggle to convert wide-angle images into perspective projection images suitable for human vision, making it difficult to confirm the perspective projection image generated from a partial image area corresponding to a predetermined angle of view.
An image processing device that includes parameter acquisition, generation, and output means to create a perspective projection image from a wide-angle image, characterized by including area information indicating the image area corresponding to the viewing angle, using methods like central projection to generate a planar developed image.
Facilitates easier confirmation of the perspective projection image generated from a partial image region corresponding to a predetermined angle of view, allowing for intuitive image editing and composition adjustment.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical Field]
[0001] This disclosure relates to an image processing technique for converting wide-angle images into perspective projection images. [Background technology]
[0002] In recent years, HMD (Head Mounted Display) type XR information processing terminals have begun to spread. XR is a general term encompassing VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality). HMDs come in various field-of-view specifications depending on cost and application. Image content for HMDs generally uses ultra-wide-angle images, such as fisheye images (also called panoramic images) with a field of view of 180 degrees or more, obtained by shooting with a wide-angle fisheye lens. The image actually displayed on the HMD is a limited image area from the wide-angle image that matches the field of view of the HMD. Therefore, it is necessary to cut out the image area that matches the field of view determined for each HMD model from the wide-angle image and convert it into a perspective projection image unfolded on a plane (see Non-Patent Literature 1). Here, a perspective projection image unfolded on a plane is an image obtained using a normal lens suitable for human vision and is suitable for display on a flat display. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] Makoto Shoji, et al., "Special Feature A: New Video and Audio Experiences, Chapter 2: Omnidirectional Photography," Journal of the Institute of Image Information and Television Engineers, Vol. 69, No. 7, pp. 652-657 (2015). [Overview of the project] [Problems that the invention aims to solve]
[0004] When editing image content for HMDs, it is necessary to know whether the required objects are included in the image displayed on the HMD. However, the image area cropped for HMDs has a narrow field of view, making it difficult to capture objects, and wide-angle images are not suitable for human viewing on a normal flat display. Therefore, there is a problem in that it is difficult to check the perspective projection image generated from a portion of the image area corresponding to a given field of view of a wide-angle image. [Means for solving the problem]
[0005] The technology of the present disclosure is an image processing apparatus for generating a perspective projection image from a wide-angle image, comprising: parameter acquisition means for acquiring a first parameter relating to a predetermined field of view angle; a first generation means for generating the perspective projection image from the wide-angle image based on the first parameter; and output means for outputting the generated perspective projection image to a display device, wherein the perspective projection image includes region information indicating an image region corresponding to the predetermined field of view angle in the wide-angle image. [Effects of the Invention]
[0006] The technology disclosed herein makes it easier to verify a perspective projection image generated from a portion of the image region corresponding to a predetermined angle of view of a wide-angle image. [Brief explanation of the drawing]
[0007] [Figure 1] A diagram illustrating an example of the hardware configuration of an image processing apparatus according to an embodiment. [Figure 2] Flowchart for generating a perspective projection image in Embodiment 1 [Figure 3] Diagram illustrating an example of the UI screen in Embodiment 1. [Figure 4] Flowchart for generating the UI screen in Embodiment 2 [Figure 5] Diagram illustrating an example of a UI screen in Embodiment 2. [Figure 6] Flowchart for generating a perspective projection image in Embodiment 3 [Figure 7]Figure for explaining the UI screen example in Embodiment 3 [Figure 8] Flowchart for generating a perspective projection image in Embodiment 4 [Figure 9] Figure for explaining the UI screen example in Embodiment 4 [Figure 10] Flowchart for generating a perspective projection image in Embodiment 5 [Figure 11] Flowchart for generating a perspective projection image in Embodiment 6 [Figure 12] Flowchart for generating a perspective projection image in Embodiment 7 [Figure 13] Figure for explaining the screen example representing the center of the viewpoint [Figure 14] Figure for explaining the software configuration example of the image processing apparatus according to the embodiment
Modes for Carrying Out the Invention
[0008] [Embodiment 1] An image processing apparatus that can highlight an image area corresponding to a predetermined angle of view in a super-wide-angle image while converting the super-wide-angle image into a planar展开画像 will be described. For the sake of convenience in explanation, among wide-angle images, those with a large degree of wide angle may also be referred to as super-wide-angle images. Projection methods used for generating a fisheye image, which is a super-wide-angle image, include stereographic projection, equidistant projection, equal solid angle projection, etc., but the projection method used for generating a planar展开画像 is the central projection method. An image generated by the central projection method is called a perspective projection image. Therefore, a projective transformation is required to obtain a perspective projection image that becomes a planar展开画像 from a fisheye image.
[0009] In FIG. 1, a diagram for explaining the hardware configuration example of the image processing apparatus according to the present embodiment is shown.
[0010] Bus 100 enables data communication between the various components of the information processing device. CPU 101 executes software, including programs for image processing, and controls the entire information processing device. ROM 102 stores programs executed by CPU 101 and various data. RAM 103 loads programs and data stored in ROM 102 when the information processing device starts up, functions as a work area for CPU 101, and stores acquired and generated images. UI (User Interface) device connection unit 104 can accept input of posture information from a keyboard, mouse, and HMD. GPU 105 performs image processing and can output image data to a display device such as an HMD or a standard flat-panel display. General-purpose IF (Interface) 106 allows a capture device to be connected to input camera images. NW (Network) / IF 107 connects to a network via wired or wireless connection and can input and output data to and from external devices via the network. In this embodiment, images are described as being acquired from ROM 102 or UI device connection unit 104, but images may also be input in real time from a capture device via general-purpose IF 106, or acquired via a network via NW / IF 107. Furthermore, images generated by the image processing device of this embodiment can be incorporated as part of the UI function as needed, displayed on a display, or output to a file.
[0011] Figure 14 shows a diagram illustrating an example of the software configuration of the image processing device according to this embodiment. The image acquisition unit 1401 acquires an ultra-wide-angle image including field of view information from the ROM 102 and NW / IF 107. The region information acquisition unit 1402 acquires information regarding the size of the perspective projection image, parameters related to the optical axis direction and field of view of the virtual camera, and the field of view area to be drawn superimposed on the perspective projection image from the ROM 102, NW / IF 107, and the UI unit 1403, which will be described later. The UI unit 1403 generates a UI screen and outputs the UI screen, which includes the image output by the region information drawing unit 1404, which will be described later, to the image device, and acquires user input input via the UI screen. The image conversion unit 1405 generates a perspective projection image by projecting the ultra-wide-angle image based on the size of the perspective projection image, the optical axis direction and field of view of the virtual camera acquired by the region information acquisition unit 1402. The region information drawing unit 1404 superimposes the information indicating the field of view acquired by the region information acquisition unit 1402 onto the perspective projection image and outputs it to the UI unit 1403. The HMD image generation unit 1406 extracts the image region corresponding to the field of view acquired by the region information acquisition unit 1402 from the perspective projection image, generates an image for the HMD, and outputs it to the HMD.
[0012] The coordinate system for the image used in this embodiment is such that the origin is the center of the image, and the width and height of the image are w, respectively. h h h Assuming the X coordinate of the left edge of the image is -w h / 2, the X coordinate of the right edge of the image is w h / 2, the y coordinate of the top edge of the image is -h h / 2, the y coordinate of the bottom edge of the image is h h The result will be / 2. Furthermore, each pixel constituting the image will have RGB color channels, and the pixel value of each pixel will consist of values for each RGB color channel.
[0013] Figure 2 shows a flowchart illustrating the process of generating a perspective projection image for each frame in Embodiment 1. Unless otherwise specified, these steps are to be executed in the order indicated by the arrows in the figure. However, the order can be changed as long as there is no dependence on the data input / output relationships of each step. In addition, each step is to be executed by the CPU 101, but some of the processing, such as drawing, may be performed by the GPU 105. The images generated by this flow are displayed on a display connected via the GPU 105. If the display frame rate of the input images is 60fps, then 60 images will be read, generated, and displayed per second.
[0014] In S2010, the image acquisition unit 1401 acquires an ultra-wide-angle image. In this embodiment, a fisheye image obtained using an equidistant projection fisheye lens with a maximum field of view of 180 degrees is assumed to be input as the ultra-wide-angle image. It is assumed that the optical distortion of the fisheye image acquired here has been removed by digital image processing. In this embodiment, the radius of the lens circle of the ultra-wide-angle image is r', and the width and height of the fisheye image are assumed to be 2r'+1.
[0015] In S2020, the region information acquisition unit 1402 acquires the optical axis direction and field of view of a virtual camera for defining a confirmation image set for the ultra-wide-angle image. In this embodiment, an example of obtaining the optical axis direction and field of view of the virtual camera from the application's UI unit 1403 is described. Initially, the optical axis direction of the virtual camera is the front direction of the acquired ultra-wide-angle image, and the field of view of the virtual camera is 120 degrees (60 degrees to the left and right). As a method for acquiring these virtual camera parameters, the amount of movement in the up, down, left, and right directions obtained by dragging the mouse is taken as the amount of change in the up, down, left, and right directions of the virtual camera's optical axis direction, and this change amount is added to the above initial value and obtained. In addition, by clicking the up, down, left, and right buttons, the optical axis direction of the virtual camera is changed by 10 degrees and obtained. Furthermore, the field of view of the virtual camera can be increased or decreased by rotating the mouse scroll wheel. Note that a virtual camera is a virtual camera different from an actually installed imaging device that is capable of obtaining a virtual viewpoint image as an image representing the view from a specified virtual viewpoint, and is a concept for conveniently explaining the virtual viewpoint related to the generation of the virtual viewpoint image. The optical axis direction and field of view of the viewpoint in the virtual image generated based on the fisheye image and the specified virtual viewpoint can be expressed as the optical axis direction and field of view of the virtual camera. The position of the virtual camera is the position of the imaging device that acquires the ultra-wide-angle image.
[0016] In S2030, the area information acquisition unit 1402 acquires predetermined viewing angle information. In this embodiment, as the viewing angle information of the HMD, a fixation stable viewing angle of 60 degrees (30 degrees each for up, down, left, and right) is acquired from the data stored in the ROM 102 or the like in advance. Note that there are various classifications of the human viewing angle, and the effective viewing angle where one can fixate only with eye movement and instantaneously capture specific information is within about 15 degrees each for left and right, about 8 degrees upward, and about 12 degrees downward. The viewing angle within which fixation is possible without difficulty with both eye movement and head movement is called the fixation stable viewing angle, and is within 30 to 45 degrees each for left and right, 20 to 30 degrees upward, and 25 to 40 degrees downward. With respect to this fixation stable viewing angle, there is only the discrimination ability to determine the presence of the presented information, but the viewing angle that affects the human spatial coordinate sense is called the induction viewing angle, which is 30 to 100 degrees each for left and right, and 20 to 85 degrees each for up and down.
[0017] In S2040, the image conversion unit 1405 converts the fisheye image acquired as the ultra-wide-angle image in S2010 into a perspective projection image serving as a confirmation image based on the optical axis direction and the angle of view of the virtual camera acquired in S2020. Let the width of the perspective projection image be w, the height be h, the optical axis direction of the virtual camera be (φ x , φ y ), the angle of view of the virtual camera be 2a degrees (a degrees each for up, down, left, and right), the coordinate values be (x, y), and the coordinate values of the fisheye shadow image be (x’, y’). Also, let the pixel image height indicated by an angle be (θ x , θ y ). (θ x , θ y ) is made the same value for the fisheye image and the perspective projection image based on the following formula. In this embodiment, h = 1080 and w = 1920.
[0018]
Equation
[0019]
Equation
[0020]
Equation
[0021]
number
[0022]
number
[0023]
number
[0024]
number
[0025] Here, r is the distance from the center to the edge of the perspective projection image, and r' is the radius of the lens circle in the fisheye image. Also, max(w,h) is a function that outputs the larger of w and h. The pixel value at each coordinate on the transformed perspective projection image is calculated based on the pixel value at the corresponding coordinate on the original fisheye image, which is associated by equations (1) to (7) above. Note that since the coordinates where pixel values exist on the fisheye image are real numbers and not integers, if a coordinate for which no pixel value exists is calculated, the pixel value is interpolated using the bicubic method and used as the pixel value in the perspective projection image. Although the focal length of the perspective projection image is often set, in this embodiment the focal length is set automatically according to the size of the generated image.
[0026] In S2050, the area information drawing unit 1404 calculates the field of view area based on the field of view angle information of the virtual camera's field of view, and draws a rectangle representing the calculated field of view area superimposed on the perspective projection image. In this embodiment, in order to highlight the field of view area, a rectangle with sides positioned at -30 degrees to the upper left and 30 degrees to the lower right from the optical axis of the virtual camera is drawn. The Y coordinates of the top and bottom sides of the rectangle, the X coordinates of the left and right sides are calculated by equations (4) and (5), respectively. If a portion of the rectangle representing the field of view area does not fit within the perspective projection image, the rectangle is deformed and drawn so that it fits within the perspective projection image.
[0027] Figure 3 shows an example of a rectangle indicating the field of view in the UI screen of this embodiment. In Figure 3, a perspective projection image 301, which is a confirmation image, is displayed on the UI screen, and a rectangle 302 indicating the field of view is drawn as a dashed line in the center of the perspective projection image 301. Note that the line used to draw the rectangle 302 indicating the field of view is not limited to a dashed line; any line that is easy to see, such as a continuous green line, may be used. The size of the rectangle 302 indicating the field of view will change relative to the perspective projection image 301 according to the field of view of the second virtual camera, but in this embodiment, it is always drawn as a constant size in the center of the perspective projection image 301. Figure 3(a) shows the state where the field of view of the virtual camera is narrow, and Figure 3(b) shows the state where the field of view of the virtual camera is wide. As described above, the field of view of the virtual camera can be changed according to the field of view of the virtual camera input by the user via the UI unit 1403.
[0028] In S2060, the region information drawing unit 1404 draws character information 303 based on the field of view information on the perspective projection image 301, which is a confirmation image. In this embodiment, the string "gaze-stable field of view" is drawn in the upper left corner of the rectangle indicating the field of view area, in the same color as the rectangle. If the field of view set in the acquired field of view information is not the gaze-stable field of view, then a string of characters appropriate to the field of view information, such as "guided field of view," will be drawn.
[0029] Since viewers do not perform editing tasks such as cropping parts of the perspective projection image, when they view a perspective projection image cropped within a stable gaze field of view using an HMD, the field of view is not wider than a person's field of view, allowing them to view the image without being conscious of the effective field of view.
[0030] On the other hand, while the photographer must consider the field of view and composition when taking images, and the editor must refine the image by cropping parts of it, it is difficult to grasp how much of the field of view the subject occupies when viewing the cropped perspective projection image on a flat display. In particular, if there is a so-called zoom function that changes the field of view of the virtual camera, it is necessary to be aware of the current field of view (zoom magnification) and the relationship between the size of the object and the field of view, making it difficult to concentrate on shooting.
[0031] Therefore, in this embodiment, by displaying a guide rectangle based on a predetermined field of view on the perspective projection image for editing and shooting, it becomes easier to frame the subject within the intended field of view, enabling intuitive shooting. For example, when checking the image immediately after shooting, it is possible to determine whether or not the subject is within the field of view based on the rectangle indicating the field of view area based on the field of view, so if it is determined that the subject is not within the field of view, the image can be reshot immediately. Conversely, it can also be used to photograph the subject larger than the guide rectangle in order to create a sense of impact.
[0032] In this embodiment, the gaze-stable field of view is obtained as field of view information, but this is not limited to this. For example, any field of view, such as the field of view or effective field of view of a typical HMD, may be obtained as field of view information, and the field of view may be displayed numerically along with the rectangle corresponding to that field of view. Also, although the string to be drawn is "gaze-stable field of view," this is not limited to this, and any text information for the highlighted area is acceptable, such as "gaze-stable field of view (60 degrees)" or simply "60 degrees." Furthermore, the value converted to the full-frame focal length (mm) may be displayed. Also, the text information may be omitted. In other words, S2060 can be skipped.
[0033] In this embodiment, a rectangle is drawn to indicate the field of view, assuming that the field of view is the same in all directions (up, down, left, and right) relative to the HMD wearer's line of sight. However, the method of drawing this rectangle is not limited to this. For example, as shown in Figure 3(c), a rectangle may be drawn at positions of -30 degrees to the left, 30 degrees to the right, 20 degrees up, and -25 degrees down from the center to represent a stable gaze field of view.
[0034] In this embodiment, a fisheye image with an equidistant projection and a 180-degree field of view is used as the input ultra-wide-angle image, but the input ultra-wide-angle image is not limited to this. The fisheye image used as the ultra-wide-angle image may have a field of view of 180 degrees or less, or greater than 180 degrees, and may be an image recorded using a projection method other than equidistant projection. Furthermore, an equirectangular image obtained by transforming the ultra-wide-angle image into an equirectangular image, or a 360-degree image may be input. In this embodiment, the ultra-wide-angle image refers to an ultra-wide-angle image that can secure a sufficient field of view for an HMD capable of head-tracking display. For example, the field of view of a normal lens is expressed by its focal length, and even a wide-angle lens has a focal length of about 8mm (the sum of the left and right fields of view is 130 degrees for a full-frame sensor). This is too narrow a field of view to be displayed on an HMD that tracks head movement. While this technology can be applied if a narrow field of view is acceptable, it is preferable to input fisheye images or omnidirectional images that can ensure sufficient image quality (pixel density) even in the wide-angle region, since the perspective projection image is generated by cropping from an arbitrary angular direction. Furthermore, even with images with a narrow field of view, sufficient field of view and image quality can be obtained by creating a pseudo-ultra-wide-angle image by combining and interpolating multiple images with different orientations.
[0035] Although this embodiment has been described as being executed on an image processing device having a configuration equivalent to that of a PC, it is not limited to that, and may also be executed on an image processing unit on an imaging device, with the perspective projection image displayed on the viewfinder or rear display of the imaging device.
[0036] In this embodiment, a video was used as an example of the input ultra-wide-angle image, but the input ultra-wide-angle image may also be a still image. Furthermore, although the input ultra-wide-angle image is described as an image obtained by capturing with an imaging device, it may also be a computer-generated image (CG) as long as it contains field-of-view information.
[0037] [Embodiment 2] In Embodiment 1, a perspective projection image was generated by drawing a rectangle indicating the field of view area based on the field of view angle from an ultra-wide-angle image. In this embodiment, however, an image with UI widgets drawn is generated. Figure 4 is a flowchart for generating the UI screen in Embodiment 2. Unless otherwise specified, the explanation shall be the same as the explanation in Figure 2 of Embodiment 1. The UI screen generated in this embodiment consists of a display area 501 for displaying the perspective projection image, and widgets such as a checkbox 502, a slider 503 for controlling the playback position of the perspective projection image, and a play button 504. Since the operation of the widgets is general, the explanation of widgets unrelated to this technology will be omitted. The state of the checkboxes 502 described later is assumed to be unchecked by default.
[0038] In S4030, the area information acquisition unit 1402 acquires multiple field of view information. In this embodiment, it obtains information such as 60 degrees (stable field of view when looking at), 90 degrees (product A), 100 degrees (product B), and 110 degrees (product B), as well as information on whether the checkbox corresponding to each item is checked or not.
[0039] In S4045, the UI unit 1403 obtains user selection information indicating whether the checkbox selected by the user is selected or not.
[0040] In S4050, the area information drawing unit 1404 calculates the field of view area based on the virtual camera's field of view, field of view information, and user selection information, and draws one or more rectangles representing the calculated field of view area on the perspective projection image. In this embodiment, rectangles corresponding to the field of view for which the corresponding checkbox is checked are drawn from the acquired field of view information. Figure 5 shows the UI screen of a video player as an example of this drawing.
[0041] In S4060, the UI unit 1403 draws character information 501 based on viewing angle information on the UI screen. In this embodiment, this character information based on viewing angle information is not superimposed on the perspective projection image as shown in Figure 5, but is drawn as a widget outside the display area where the perspective projection image is displayed. At this time, a checkbox indicating whether or not a rectangle is drawn is drawn with a checkmark or a checkmark. The state of whether or not the checkbox is selected is retained even after this flow ends and can be retrieved when this flow is executed again.
[0042] In S4070, the UI unit 1403 draws other widgets. In this embodiment, it draws a play button 504, a slider bar 503 that shows the video playback status, a field of view control button 507 that specifies the field of view of the virtual camera, and a Fit button 506 that sets the field of view of the virtual camera to fit the window.
[0043] In this embodiment, the user can set the target angle or HMD field of view by drawing a rectangle and selecting the field of view they want to highlight using checkboxes, and then highlight that field of view. Furthermore, by drawing rectangles corresponding to multiple field of view information simultaneously, the user can, for example, compare and confirm the gaze area or HMD field of view on a perspective projection image that previews the field of view areas for multiple field of view.
[0044] In this embodiment, four types of viewing angle information were acquired in S4030, but the number and content are not limited to these. Alternatively, a configuration may be used in which viewing angle information and user input are acquired via the network and the displayed content is updated.
[0045] [Embodiment 3] In Embodiments 1 and 2, the rectangle indicating the field of view was drawn fixedly relative to the perspective projection image. In this embodiment, the rectangle indicating the field of view is drawn while moving relative to the perspective projection image so as to track a specific object in the input ultra-wide-angle fisheye image. Figure 6 is a flowchart for generating the perspective projection image in Embodiment 3. Unless otherwise specified, the explanation shall be the same as the explanation in Figure 2 of Embodiment 1. The object to be tracked may be selected by the user each time, or a predetermined object may be detected.
[0046] In S6045, the region information acquisition unit 1402 acquires object tracking mode information, indicating whether the user has selected an object tracking mode. In this embodiment, as shown in Figure 7, it obtains information on whether a checkbox on the UI screen is checked or not. Figure 7 is a diagram illustrating an example of the UI screen in Embodiment 3.
[0047] In S6050, the area information drawing unit 1404 determines the coordinate values of the field of view area based on the optical axis direction and field of view, field of view information of the virtual camera, and object tracking information, and draws a rectangle indicating the calculated field of view area on the perspective projection image. Figure 7 shows an example of the UI screen at this time. Figure 7 shows an example of the screen when the object tracking mode is selected for object tracking information and the optical axis direction (front direction) of the virtual camera is shifted to the right, and accordingly, the rectangle indicating the field of view area is also drawn shifted to the right. When object tracking mode is selected, in order to obtain the coordinate values of the field of view area, the coordinate values of the top left, top right, bottom left, and bottom right points of the rectangle indicating the field of view area are calculated based on the coordinate values of the object position included in the object tracking information and equations (2) and (3). Then, the top, bottom, left, and right sides of the rectangle are drawn based on the calculated coordinate values. The calculation method when object tracking mode information is not selected is because the optical axis direction of the virtual camera is ignored, so (φ x , φ ySet )=(0,0) and calculate similarly. An example of a perspective projection image when the object tracking mode information is not selected and the above coordinate values are not obtained, and the rectangle is drawn in the center of the screen, will be the same as shown in Figure 3.
[0048] In this embodiment, by correcting the four points of the rectangle indicating the field of view, the rectangle is displayed not in the center of the perspective projection image, but along the optical axis of the virtual viewpoint camera, as shown in Figure 7. This allows for the drawing of a rectangle indicating the field of view along the optical axis of the virtual camera, providing an image that makes it easy to recognize the field of view in front while simultaneously viewing subjects outside the field of view while manipulating the optical axis of the virtual camera with the mouse.
[0049] [Embodiment 4] While Embodiments 1 to 3 showed examples of highlighting using rectangles, this embodiment will explain, using Figure 8, how to draw circular guidelines using the highlighting method. Figure 8 is a flowchart for generating a perspective image in Embodiment 4. Unless otherwise specified, the explanation will be the same as the explanation in Figure 2 of Embodiment 1.
[0050] In S8030, the area information acquisition unit 1402 acquires field of view information. In this embodiment, information of 30 degrees is obtained as the radius of the gaze-stable field of view.
[0051] In S8050, the region information drawing unit 1404 calculates the field of view region based on the optical axis direction and field of view of the virtual camera, as well as the field of view angle information, and draws circular guidelines on the perspective projection image. In this embodiment, based on the field of view angle information of 30 degrees obtained in S8030, the radius of the circle is obtained by setting θ=30 using the following formula, and a circle is drawn from the center of the perspective projection image with radius radius.
[0052]
number
[0053] An example of the UI screen in this case is shown in Figure 9. Figure 9 is a diagram illustrating an example of the UI screen in Embodiment 4. Figure 9(a) is an example of the UI screen in which a circle, as explained in S8050, is drawn. The field of view of the HMD and the field of view of a human are closer to a circle than to a rectangle. Therefore, it is possible to highlight an area that is closer to the actual view. In this embodiment, a perfect circle is drawn, but the circle does not have to be a perfect circle; it may be an ellipse, a rectangle with rounded corners, a rhombus with rounded corners, or a combination of a circle and a rectangle. For example, as shown in Figures 9(b) and 9(c). Note that if one tries to reproduce the field of view based on human visual characteristics more faithfully, the shape will be distorted compared to the shape of a rectangle or a circle. On the other hand, a distorted shape reduces the aesthetic appeal. Considering that the highlighted field of view area is merely a guide, neither is necessarily correct, but rather it is a matter of balancing fidelity and aesthetic appeal, and should be decided according to the target user.
[0054] [Embodiment 5] In this embodiment, an example of highlighting the field of view by drawing a circle on a fisheye image will be explained using Figure 10. Figure 10 is a flowchart for generating a perspective projection image in Embodiment 5. Unless otherwise specified, the explanation will be the same as the explanation in Figure 2 of Embodiment 1.
[0055] In S10010, the image acquisition unit 1401 acquires a fisheye image. The operation is equivalent to that in S2010, except that the image to be acquired has changed.
[0056] In S10035, the region information drawing unit 1404 calculates the field of view region based on the field of view and field of view information of the virtual camera, and draws circles indicating the field of view region on the fisheye image. In this embodiment, the ultra-wide-angle image is treated as a fisheye image, and circles corresponding to the field of view based on the field of view information are drawn on concentric circles centered on the optical axis. The circles drawn on the concentric circles of the fisheye maintain their shape even when converted to a perspective projection image.
[0057]
number
[0058] In S10050, the image conversion unit 1405 converts the fisheye image into a perspective projection image based on the optical axis direction and field of view of the virtual camera. This is the same operation as in S2050. However, the circles drawn on the fisheye image will also be converted.
[0059] In this embodiment, circles were drawn on a fisheye image instead of a perspective projection image. Even when guidelines are drawn on a fisheye image instead of a perspective projection image, the same effects as in other embodiments can be achieved.
[0060] In this embodiment, the ultra-wide-angle image (fisheye image) on which the guidelines are drawn is converted into a perspective projection image based on the optical axis direction and field of view of the virtual camera. As a result, the guidelines are drawn in the direction of the optical axis direction of the virtual camera. In other words, the guidelines are drawn in the mode that tracks the object as described in Embodiment 3.
[0061] [Embodiment 6] In this embodiment, an example of relatively emphasizing the area within the field of view by suppressing the image area outside the field of view will be explained using Figure 11. Figure 11 is a flowchart for generating a perspective projection image in Embodiment 6. In this embodiment, suppressing the image means adjusting the gain to darken the image, reduce the contrast, and make specific image areas less prominent. Unless otherwise specified or changed, the explanation will be the same as the explanation in Figure 2 of Embodiment 1.
[0062] In S11050, the region information rendering unit 1404 calculates the field of view region based on the field of view and field of view information of the virtual camera, and reduces the brightness outside the field of view region of the perspective projection image.
[0063] According to this embodiment, in addition to drawing rectangles as in Embodiment 1, the field of view can be enhanced by processing the image.
[0064] In this embodiment, an example of suppressing the image area is shown as gain adjustment, but this is not the only method. For example, it is also possible to completely black out the area outside the field of view. It is also possible to suppress the image by applying a low-pass filter outside the field of view. In this case, instead of turning the low-pass filter ON and OFF inside and outside the field of view, a stronger low-pass filter may be applied further outside the field of view.
[0065] In this embodiment, the boundary between the inside and outside of the field of view was described as being rectangular, but it is not limited to this, and may be circular as described in Embodiment 4.
[0066] [Embodiment 7] In this embodiment, the field of view is displayed in conjunction with the head movement of the HMD. Figure 12 is a flowchart for generating the perspective projection image in Embodiment 7. Unless otherwise specified or modified, the explanation shall be the same as that given in Figure 2 of Embodiment 1.
[0067] In this embodiment, the generated perspective projection image is output to both a flat panel display and an HMD, and the display area on the HMD is highlighted on the flat panel display. The HMD displays an image corresponding to the field of view in the optical axis direction that follows head movement, and the flat panel display displays an image with a fixed field of view direction unless there is input from the user viewing the flat panel display.
[0068] In S12030, the region information acquisition unit 1402 acquires the field of view information and HMD orientation information of the connected HMD. At this time, the field of view of the connected HMD is obtained from the model name of that model. Alternatively, the field of view may be obtained directly instead of by model name. Regarding the HMD orientation information, as described later, it is sufficient to obtain the front direction of the HMD, and there are no restrictions on its format.
[0069] In S12050, the dual-device information rendering unit 1404 calculates the field of view area based on the HMD attitude information and superimposes a rectangle indicating the calculated field of view area onto the perspective projection image. x , φ'y ) is calculated. At this time, (φ x , φ y )=(φ' x , φ' y Using equations (2) and (3), the coordinates of the top-left, top-right, bottom-left, and bottom-right points of the rectangle of the field of view are calculated, and the top, bottom, left, and right sides of the rectangle are drawn as lines. As a result, the position of the drawn rectangle changes in accordance with the head movement of the HMD.
[0070] Conventionally, when outputting to both an HMD and a flat-screen display simultaneously, the flat-screen display often showed images from the HMD that followed head movements. While these head-following images are natural for the HMD wearer, they cause significant blurring and motion sickness for users viewing on a regular flat-screen display. In this embodiment, by displaying a perspective projection image on the display from the UI unit 1403, which is a superimposed rectangle representing the HMD's field of view, it is possible to display a fixed image that is less likely to cause motion sickness while still confirming the HMD wearer's gaze direction.
[0071] Furthermore, as shown in Figure 13, a cross, an "x," or a dot may be drawn on the rectangle to be displayed, with the center of the rectangle as the intersection point. Figure 13 is an example of a screen that represents the center of the viewpoint, which can represent the direction of the viewpoint. Note that if it is not necessary to represent the field of view, it is not necessary to draw a rectangle, and it is sufficient to simply draw a cross, an "x," or a dot to indicate the direction of the viewpoint.
[0072] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]
[0073] 1401 Image acquisition unit 1402 Area information acquisition unit 1403 UI section 1404 Area information drawing section 1405 Image Conversion Unit
Claims
1. 1. An image processing device for generating a perspective projection image from a wide-angle image, comprising: a first generating means for generating the perspective projection image from the wide-angle image based on a first parameter related to a predetermined viewing angle; an output means for outputting the generated perspective projection image; Equipped with The perspective projection image includes area information indicating an image area corresponding to the predetermined viewing angle in the wide-angle image.
13. An image processing device comprising:
2. The region information includes a frame indicating an image region corresponding to the predetermined viewing angle.
2. The image processing device according to claim 1,
3. The region information includes a character string indicating the viewing angle.
3. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing device.
4. The viewing angle is a viewing angle of a predetermined HMD, The region information includes a character string indicating a model of the predetermined HMD.
4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.
5. The apparatus further includes a parameter acquisition unit for acquiring a second parameter related to the wide-angle image and a third parameter related to the perspective projection image, the second parameter includes an angle of view of the wide-angle image, and the third parameter includes an angle of view of the perspective projection image and coordinates in the wide-angle image corresponding to reference coordinates in the perspective projection image; 5. The image processing device according to claim 1, wherein the first generating means generates the perspective projection image from the wide-angle image based on the second parameter, the third parameter, and the first parameter.
6. The wide-angle image is one of an image obtained based on a fisheye lens projection method, an equirectangular image, and a panoramic image.
6. The image processing device according to claim 1,
7. the region information includes region information for a plurality of image regions for a plurality of viewing angles; 7. The image processing device according to claim 1,
8. coordinate value acquisition means for acquiring coordinate values that designate coordinates in the wide-angle image; The first generating means determines a position at which the region information is to be superimposed on the perspective projection image based on the coordinate values acquired by the coordinate value acquiring means.
8. The image processing device according to claim 1,
9. the coordinate value acquisition means acquires coordinate values at which a specific object is located in the wide-angle image; 9. The image processing device according to claim 8,
10. The first generating means generates the perspective projection image so as to change at least one of brightness and contrast inside and outside the image region.
10. The image processing device according to claim 1,
11. a second generating means for generating an HMD image by cutting out the image region based on the region information; Equipped with the output means outputs the perspective projection image and the HMD image to separate display devices.
11. The image processing device according to claim 1,
12. 1. An image processing method for generating a perspective projection image from a wide-angle image, comprising: a first generation step of generating the perspective projection image from the wide-angle image based on a first parameter related to a predetermined viewing angle; an output step of outputting the generated perspective projection image; Equipped with The perspective projection image includes area information indicating an image area corresponding to the predetermined viewing angle in the wide-angle image.
13. An image processing method comprising:
13. A program for causing a computer to function as the image processing device according to any one of claims 1 to 11.
