Image processing device, image processing method and program

Abstract

To provide an image processing system that facilitates confirmation of a perspective projection image generated from a partial image area corresponding to a predetermined angle of view of a wide-angle image.SOLUTION: An image acquisition unit acquires an ultra-wide-angle image. An area information acquisition unit acquires an optical axis direction and an angle of view of a virtual camera for defining a confirmation image set for the ultra-wide-angle image and acquires predetermined visual angle information. An image conversion unit converts, on the basis of the acquired optical axis direction and the angle of view of the virtual camera, the acquired ultra-wide-angle image into a perspective projection image as the confirmation image. An area information drawing unit calculates a visual field area on the basis of the visual angle information on the angle of view of the virtual camera, and draws a rectangle indicating the calculated visual field area by superimposing the same on the perspective projection image.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present disclosure relates to image processing techniques for converting wide-angle images into perspective projection images. [Background technology]

[0002] In recent years, head-mounted display (HMD)-type XR information processing devices have become widespread. XR is a collective term that encompasses virtual reality (VR), augmented reality (AR), and mixed reality (MR). HMDs are available with a variety of viewing angles depending on their cost and intended use. Image content for HMDs typically uses ultra-wide-angle images, such as fisheye images (also known as panoramic images) with a viewing angle of 180 degrees or more, captured using a fisheye lens with a wide viewing angle. The image actually displayed on an HMD is a limited image region of the wide-angle image that is tailored to the viewing angle of the HMD. Therefore, it is necessary to crop an image region tailored to the viewing angle specified for each HMD model from the wide-angle image and convert it into a perspective projection image projected onto a plane (see Non-Patent Document 1). A perspective projection image projected onto a plane is an image obtained using a normal lens that is suited to human vision and is suitable for display on a flat display. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Makoto Shoji and two others, "Special Feature A: New Visual and Audio Experiences, Chapter 2: Full-Dome Shooting," Journal of the Institute of Image Information and Television Engineers, Vol. 69, No. 7, pp. 652-657 (2015) Summary of the Invention [Problem to be solved by the invention]

[0004] When editing image content for an HMD, it is necessary to know whether the necessary objects are included in the image displayed on the HMD. However, the image area cropped for the HMD has a narrow field of view, making it difficult to capture objects, and wide-angle images are not suitable for human viewing on a regular flat display. This poses a challenge: it is difficult to confirm the perspective projection image generated from a portion of the image area corresponding to a specific field of view of the wide-angle image. [Means for solving the problem]

[0005] The technology disclosed herein is an image processing device for generating a perspective projection image from a wide-angle image, the image processing device including: a first generating means for generating the perspective projection image from the wide-angle image based on a first parameter related to a predetermined viewing angle; and an output means for outputting the generated perspective projection image, the perspective projection image including area information indicating an image area in the wide-angle image that corresponds to the predetermined viewing angle. the region information includes a character string according to the viewing angle; It is characterized by: [Effects of the Invention]

[0006] According to the technology of the present disclosure, it becomes easier to check a perspective projection image generated from a partial image region corresponding to a predetermined angle of view of a wide-angle image. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of the hardware configuration of an image processing apparatus according to an embodiment. [Figure 2] Flow diagram for generating a perspective projection image in the first embodiment [Figure 3] FIG. 1 is a diagram illustrating an example of a UI screen according to the first embodiment. [Figure 4] Flow diagram for generating a UI screen in the second embodiment [Figure 5] FIG. 10 is a diagram illustrating an example of a UI screen in the second embodiment. [Figure 6] Flow diagram for generating a perspective projection image in the third embodiment [Figure 7] FIG. 10 is a diagram illustrating an example of a UI screen in the third embodiment. [Figure 8] Flow diagram for generating a perspective projection image in the fourth embodiment [Figure 9] FIG. 10 is a diagram illustrating an example of a UI screen in the fourth embodiment. [Figure 10] Flow diagram for generating a perspective projection image in the fifth embodiment [Figure 11] Flow diagram for generating a perspective projection image in the sixth embodiment [Figure 12] Flow diagram for generating a perspective projection image in embodiment 7 [Figure 13] A diagram illustrating an example of a screen that shows the center of view [Figure 14] FIG. 1 is a diagram illustrating an example of the software configuration of an image processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] [Embodiment 1] This paper describes an image processing device that can convert an ultra-wide-angle image into a planar unfolded image while emphasizing an image region in the ultra-wide-angle image corresponding to a predetermined angle of view on a perspective projection image. For convenience of explanation, a wide-angle image with a large degree of wide-angle may be referred to as an ultra-wide-angle image. Projection methods used to generate a fisheye image, which is an ultra-wide-angle image, include stereoscopic projection, equidistant projection, and equirectangular solid angle projection, but the projection method used to generate a planar unfolded image is the central projection method. An image generated using the central projection method is called a perspective projection image. Therefore, a projective transformation is required to obtain a perspective projection image, which is a planar unfolded image, from a fisheye image.

[0009] FIG. 1 is a diagram illustrating an example of the hardware configuration of an image processing apparatus according to this embodiment.

[0010] The bus 100 enables data communication between the various components of the information processing device. The CPU 101 executes software, including programs for image processing, and controls the entire information processing device. The ROM 102 stores programs executed by the CPU 101 and various data. The RAM 103 loads programs and data stored in the ROM 102 when the information processing device starts up, functioning as a work area for the CPU 101 and storing acquired and generated images. The UI (user interface) device connection unit 104 can accept input from a keyboard, mouse, and HMD orientation information. The GPU 105 can process images and output image data to a display device such as an HMD or a standard flat panel display. The general-purpose IF (interface) 106 can connect a capture device to input camera images. The NW (network) / IF 107 connects to a network via wired or wireless connections and can input and output data to and from external devices via the network. In this embodiment, the image is described as being acquired from the ROM 102 or the UI device connection unit 104, but the image may be input in real time from a capture device via the general-purpose IF 106, or may be acquired over a network via the NW / IF 107. Furthermore, the image generated by the image processing device of this embodiment can be incorporated as part of the UI function as needed, and displayed on a display or output to a file.

[0011] FIG. 14 illustrates an example of the software configuration of an image processing device according to this embodiment. The image acquisition unit 1401 acquires an ultra-wide-angle image, including angle-of-view information, from the ROM 102 or the NW / IF 107. The region information acquisition unit 1402 acquires information regarding the size of the perspective projection image, parameters related to the optical axis direction and angle of view of the virtual camera, and information regarding the field of view region to be superimposed on the perspective projection image, from the ROM 102, the NW / IF 107, or a UI unit 1403 (described later). The UI unit 1403 generates a UI screen, outputs the UI screen, including an image output by a region information rendering unit 1404 (described later), to the image device, and acquires user input via the UI screen. The image conversion unit 1405 performs projective transformation on the ultra-wide-angle image to generate a perspective projection image based on the size of the perspective projection image, the optical axis direction and angle of view of the virtual camera acquired by the region information acquisition unit 1402. An area information drawing unit 1404 draws information indicating the field of view acquired by the area information acquisition unit 1402 superimposed on the perspective projection image and outputs the information to a UI unit 1403. An HMD image generation unit 1406 cuts out an image area corresponding to the field of view acquired by the area information acquisition unit 1402 from the perspective projection image, generates an HMD image, and outputs the image to the HMD.

[0012] The coordinate system of the image handled in this embodiment has the center of the image as the origin, and the width and height of the image are w h , h h The X coordinate of the left edge of the image is -w h / 2, the X coordinate of the right edge of the image is w h / 2, the y coordinate of the top edge of the image is -h h / 2, the y coordinate of the bottom edge of the image is h h It is also assumed that each pixel constituting an image has an RGB color channel, and the pixel value of each pixel consists of a value for each RGB color channel.

[0013] FIG. 2 shows a flowchart for explaining the process of generating a perspective projection image for each frame in the first embodiment. Unless otherwise specified, these steps are executed in the order indicated by the arrows in the figure. However, the order of execution can be changed as long as there is no dependency on the data input / output relationship between each step. Each step is executed by the CPU 101, but some of the processing, such as rendering processing, may be executed by the GPU 105. Images generated by this flow are displayed on a display connected via the GPU 105. If the display frame rate of input images is 60 fps, then 60 images are read, generated, and displayed per second.

[0014] In S2010, the image acquisition unit 1401 acquires an ultra-wide-angle image. In this embodiment, it is assumed that a fisheye image acquired using an equidistant projection fisheye lens with a maximum angle of view of 180 degrees is input as the ultra-wide-angle image. It is assumed that optical distortion has been removed from the fisheye image acquired as the ultra-wide-angle image by digital image processing. In this embodiment, it is assumed that the radius of the lens circle of the ultra-wide-angle image is r', and the width and height of the fisheye image are 2r'+1.

[0015] In S2020, the region information acquisition unit 1402 acquires the optical axis direction and the angle of view of a virtual camera for defining a confirmation image set for the ultra-wide-angle image. In this embodiment, an example will be described in which the optical axis direction and the angle of view of the virtual camera are acquired from the UI unit 1403 of an application. In the initial state, the optical axis direction of the virtual camera is set to the front direction of the acquired ultra-wide-angle image, and the angle of view of the virtual camera is 120 degrees (60 degrees left and right). To acquire these virtual camera parameters, the amount of up / down / left / right movement obtained by dragging the mouse is used as the amount of change in the optical axis direction of the virtual camera, and this change is added to the initial value. Clicking the up / down / left / right buttons acquires a change in the optical axis direction of the virtual camera by 10 degrees. The angle of view of the virtual camera can be increased or decreased by rotating the scroll wheel of the mouse. The virtual camera is a virtual camera, different from an actually installed imaging device, that can acquire a virtual viewpoint image representing the view from a specified virtual viewpoint as a captured image. It is a concept used for conveniently explaining the virtual viewpoint related to the generation of a virtual viewpoint image. The optical axis direction and the angle of view of the virtual camera, which are generated based on the fisheye image and the specified virtual viewpoint, can be expressed as the optical axis direction and the angle of view of the virtual camera, where the position of the virtual camera is the position of the imaging device that captured the ultra-wide-angle image.

[0016] In S2030, the area information acquisition unit 1402 acquires predetermined field of view information. In this embodiment, the field of view information of the HMD is a stable gaze field of view of 60 degrees (30 degrees each up, down, left, and right) acquired from data stored in advance in the ROM 102 or the like. Human field of view angles are classified into various categories. The effective field of view, which allows gaze solely through eye movement and instantaneous capture of specific information, is approximately 15 degrees left and right, approximately 8 degrees up, and approximately 12 degrees down. The field of view that allows gaze comfortably with eye and head movement is called the stable gaze field, which is approximately 30 to 45 degrees left and right, 20 to 30 degrees up, and 25 to 40 degrees down. Compared to this stable gaze field, the field of view that only allows the ability to distinguish the presence of presented information but that influences the human sense of spatial coordinates is called the induced field of view, which is approximately 30 to 100 degrees left and right, and 20 to 85 degrees up and down.

[0017] In S2040, the image conversion unit 1405 converts the fisheye image acquired as the super wide-angle image in S2010 into a perspective projection image to be used as a confirmation image, based on the optical axis direction and angle of view of the virtual camera acquired in S2020. The width of the perspective projection image is w, the height is h, and the optical axis direction of the virtual camera is (φ x ,φ y ), the angle of view of the virtual camera is 2a degrees (a degrees up, down, left, and right), the coordinate value is (x, y), and the coordinate value of the fisheye image is (x', y'). Also, the image height of the pixel expressed as an angle is (θ x ,θ y ) (θ x ,θ y ) is set to the same value for the fisheye image and the perspective projection image based on the following formula: In this embodiment, h=1080, w=1920.

[0018]

number

[0019]

number

[0020]

number

[0021]

number

[0022]

number

[0023]

number

[0024]

number

[0025] Here, r is the distance from the center of the perspective projection image to the edge of the image, and r' is the radius of the lens circle of the fisheye image. Furthermore, max(w,h) is a function that outputs the larger of w and h. The pixel value at each coordinate on the perspective projection image after transformation is calculated based on the pixel value at the corresponding coordinate on the fisheye image before transformation, which is associated using the above equations (1) to (7). Note that since coordinates where pixel values ​​exist on the fisheye image are real numbers, not integers, when a coordinate where no pixel value exists is calculated, the pixel value is interpolated using the bicubic algorithm and used as the pixel value of the perspective projection image. Note that a focal length is often set for perspective projection images, but in this embodiment, the focal length is automatically set according to the size of the image to be generated.

[0026] In S2050, the area information drawing unit 1404 calculates the field of view based on field of view angle information about the angle of view of the virtual camera, and draws a rectangle representing the calculated field of view by superimposing it on the perspective projection image. In this embodiment, to highlight the field of view, a rectangle is drawn with sides positioned at -30 degrees to the top left and 30 degrees to the bottom right from the optical axis of the virtual camera. The Y coordinate of the top side of the rectangle, the Y coordinate of the bottom side, the X coordinate of the left side, and the X coordinate of the right side are calculated using equations (4) and (5), respectively. Note that if part of the rectangle representing the field of view does not fit within the perspective projection image, the rectangle is deformed and drawn so that it fits within the perspective projection image.

[0027] FIG. 3 shows an example of a rectangle indicating a field of view on a UI screen in this embodiment. In FIG. 3, a perspective projection image 301, which is a confirmation image, is displayed on the UI screen, and a rectangle 302 indicating the field of view is drawn in a dashed line at the center of the perspective projection image 301. Note that the line used to draw the rectangle 302 indicating the field of view is not limited to a dashed line and may be any line that is easily visible, such as a continuous green line. The rectangle 302 indicating the field of view changes size relative to the perspective projection image 301 depending on the angle of view of the second virtual camera. In this embodiment, however, it is always drawn at a constant size at the center of the perspective projection image 301. FIG. 3(a) shows a state in which the angle of view of the virtual camera is narrow, and FIG. 3(b) shows a state in which the angle of view of the virtual camera is wide. As described above, the angle of view of the virtual camera can be changed depending on the angle of view of the virtual camera input by the user via the UI unit 1403.

[0028] In S2060, the region information drawing unit 1404 draws text information 303 based on the viewing angle information on the perspective projection image 301, which is the confirmation image. In this embodiment, the text "stable gaze viewing angle" is drawn in the upper left of the rectangle indicating the viewing region in the same color as the rectangle. Note that if the viewing angle set in the acquired viewing angle information is not a stable gaze viewing angle, a text string in accordance with the viewing angle information, such as "guided viewing field," is drawn.

[0029] Viewers do not need to perform editing tasks such as cutting out parts of perspective projection images. Therefore, when viewers view perspective projection images cut out at a stable gaze viewing angle on an HMD, the angle of view is not wider than that of a human, so they can view the image without being conscious of their effective field of view.

[0030] On the other hand, photographers need to take images while considering the angle of view and composition, and editors need to create the image by cropping parts, but when viewing the cropped perspective projection image on a flat panel display, it is difficult to grasp the angle of view in which the subject fits. In particular, when there is a so-called zoom function that changes the angle of view of the virtual camera, it is necessary to understand the relationship between the size of the object and the viewing angle while also understanding the current angle of view (zoom magnification), making it difficult to concentrate on shooting.

[0031] Therefore, in this embodiment, by displaying a guide rectangle based on a predetermined field of view on the perspective projection image for editing and shooting, it becomes easier to fit the subject within the expected field of view, enabling intuitive shooting. For example, when checking the image immediately after shooting, it is possible to determine whether the subject fits within the field of view based on the rectangle indicating the field of view area based on the field of view, so if it is determined that the subject does not fit within the field of view, it is possible to immediately reshoot. Conversely, it can also be used for purposes such as making the subject appear larger than the guide rectangle to create a more impactful image.

[0032] In this embodiment, the viewing angle information is a stable gaze viewing angle, but this is not intended to be limiting. For example, any viewing angle, such as a typical HMD viewing angle or effective field of view, may be acquired as viewing angle information, and the viewing angle may be displayed numerically along with a rectangle corresponding to that viewing angle. Furthermore, the character string to be drawn is "stable gaze viewing field," but this is not intended to be limiting. Any text information for the highlighted area may be used, such as "stable gaze viewing angle (60 degrees)" or simply "60 degrees." The value (mm) converted to the focal length of a full-frame image may also be displayed. The text information may also be omitted. In other words, S2060 can be skipped.

[0033] In this embodiment, a rectangle representing the visual field is drawn assuming that the angle of view is the same in all directions relative to the line of sight of the HMD wearer, but the method of drawing this rectangle is not limited to this. For example, as shown in Figure 3(c), a rectangle may be drawn at a position -30 degrees left, 30 degrees right, 20 degrees up, and -25 degrees down from the center as the stable gaze field.

[0034] In this embodiment, an equidistantly projected fisheye image with a 180-degree angle of view is input as the ultra-wide-angle image. However, the input ultra-wide-angle image is not limited to this. The fisheye image used as the ultra-wide-angle image may be less than or greater than 180 degrees, and may be an image recorded using a projection method other than equidistant projection. Furthermore, an equirectangular image obtained by converting an ultra-wide-angle image into an equirectangular image, or a panoramic image may be input. In this embodiment, the ultra-wide-angle image refers to an ultra-wide-angle image that can ensure a sufficient field of view in an HMD capable of head-tracking display. For example, the field of view of a typical lens is expressed by focal length, and even a wide-angle image has a focal length of approximately 8 mm (the total horizontal field of view for a full-frame sensor is 130 degrees). This field of view is too narrow to be displayed in an HMD that tracks head movement. This technology can be applied if a narrow angle of view is acceptable, but since perspective projection images are generated by cropping in any angular direction, it is preferable to input a fisheye image or a panoramic image, which can ensure sufficient image quality (pixel density) even in wide-angle areas. Even with images with a narrow angle of view, sufficient angle of view and image quality can be obtained when creating a pseudo-ultra-wide-angle image by synthesizing and interpolating multiple images taken in different directions.

[0035] This embodiment has been described as being executed on an image processing device having a configuration equivalent to that of a PC, but this is not limited to this, and the configuration may also be such that the image processing device is executed on an image capturing device and a perspective projection image is displayed on the viewfinder or rear display of the image capturing device.

[0036] In this embodiment, a video image is used as an example of the input ultra-wide-angle image, but the input ultra-wide-angle image may be a still image. Also, although the input ultra-wide-angle image is described as an image captured by an imaging device, it may be a CG image as long as it has information about the angle of view.

[0037] [Embodiment 2] In the first embodiment, a perspective projection image was generated by drawing a rectangle indicating a field of view based on the field of view angle from an ultra-wide-angle image. In this embodiment, however, an image is generated by drawing widgets for the UI. FIG. 4 is a flow diagram for generating a UI screen in the second embodiment. Unless otherwise specified or changed, the description will be the same as that of FIG. 2 in the first embodiment. The UI screen generated in this embodiment is composed of widgets such as a display area 501 that displays a perspective projection image, a check box 502, a slider 503 that controls the playback position of the perspective projection image, and a play button 504. Note that the widget operations are common, and therefore descriptions of widgets unrelated to this technology will be omitted. Note that the default state of the check boxes 502, which will be described later, is that all check boxes 502 are unchecked.

[0038] In S4030, the area information acquisition unit 1402 acquires information on a plurality of viewing angles. In this embodiment, the information acquired is 60 degrees (stable gaze field), 90 degrees (product A), 100 degrees (product B), and 110 degrees (product B), as well as information on whether the checkboxes corresponding to each item are checked.

[0039] In S4045, the UI unit 1403 acquires user selection information indicating whether or not the check box selected by the user is selected.

[0040] In S4050, the area information drawing unit 1404 calculates the field of view based on the angle of view of the virtual camera, the field of view information, and the user selection information, and draws one or more rectangles indicating the calculated field of view on the perspective projection image. In this embodiment, the rectangles corresponding to the field of view angles for which the corresponding checkboxes are checked are drawn from the obtained field of view information. As an example of this drawing, FIG. 5 shows the UI screen of a video player.

[0041] In S4060, the UI unit 1403 draws character information 501 based on the viewing angle information on the UI screen. In this embodiment, this character information based on the viewing angle information is not superimposed on the perspective projection image as shown in FIG. 5, but is drawn as a widget outside the display area displaying the perspective projection image. At this time, a check box indicating whether or not a rectangle is drawn is drawn as checked. Note that the state of whether or not the check box is selected is retained even after this flow ends, and can be acquired when this flow is re-executed.

[0042] In S4070, the UI unit 1403 draws other widgets. In this embodiment, the UI unit 1403 draws a play button 504, a slider bar 503 that indicates the playback status of the video, a view angle control button 507 that specifies the view angle of the virtual camera, and a fit button 506 that sets the view angle of the virtual camera so that it fits the window.

[0043] In this embodiment, the user can set the target angle or HMD field of view by drawing a rectangle and selecting the field of view they want to emphasize using a check box, and then highlight that field of view. Also, by simultaneously drawing rectangles corresponding to multiple field of view information, the user can, for example, compare and check the gaze area or HMD field of view on a perspective projection image that previews the field of view for multiple field of view.

[0044] In this embodiment, four types of viewing angle information are acquired in S4030, but the number and content are not limited to this. Also, a configuration may be adopted in which viewing angle information and user input are acquired via a network and the display content is updated.

[0045] [Embodiment 3] In the first and second embodiments, the rectangle indicating the field of view is drawn fixedly relative to the perspective projection image. In this embodiment, the rectangle indicating the field of view is drawn while moving relative to the perspective projection image so as to track a specific object in the input fisheye image, which is an ultra-wide-angle image. Figure 6 is a flow diagram for generating a perspective projection image in the third embodiment. Unless there are any changes or special descriptions, the description shall conform to the description of Figure 2 in the first embodiment. Note that the object to be tracked may be selected by the user each time, or a predetermined object may be detected.

[0046] In S6045, the area information acquisition unit 1402 acquires object tracking mode information indicating whether the user has selected the object tracking mode. In this embodiment, information is acquired as to whether a check box in the UI screen is checked, as shown in Fig. 7. Fig. 7 is a diagram illustrating an example of a UI screen in the third embodiment.

[0047] In S6050, the area information drawing unit 1404 determines the coordinate values ​​of the field of view area based on the optical axis direction and angle of view of the virtual camera, field of view angle information, and object tracking information, and draws a rectangle indicating the calculated field of view area on the perspective projection image. FIG. 7 shows an example of a UI screen at this time. FIG. 7 shows an example of a screen when the optical axis direction (front direction) of the virtual camera is shifted to the right when the object tracking mode is selected as the object tracking information, and accordingly, the rectangle indicating the field of view area is also drawn shifted to the right. When the object tracking mode is selected, first, to obtain the coordinate values ​​of the field of view area, the coordinate values ​​of the four points of the upper left, upper right, lower left, and lower right of the rectangle indicating the field of view are calculated based on the coordinate values ​​of the object position included in the object tracking information and equations (2) and (3). Then, the top, bottom, left, and right sides of the rectangle are drawn based on the calculated coordinate values. When the object tracking mode information is not selected, the calculation method ignores the optical axis direction of the virtual camera, so (φ x , φ y) = (0, 0), and perform the calculation in the same way. An example of a perspective projection image when the object tracking mode information is not selected, the above coordinate values ​​are not acquired, and the rectangle is drawn in the center of the screen will be the same as that shown in Figure 3.

[0048] In this embodiment, by correcting four points of the rectangle indicating the field of view, the rectangle is not fixed at the center of the perspective projection image but is displayed in the direction of the optical axis of the virtual viewpoint camera as shown in Fig. 7. This allows the rectangle indicating the field of view to be drawn in the direction of the optical axis of the virtual camera, and an image can be provided that makes it easy to recognize the front field of view while also checking subjects outside the field of view by manipulating the optical axis direction of the virtual camera with the mouse.

[0049] [Embodiment 4] In the first to third embodiments, examples of highlighting using a rectangle were shown, but in this embodiment, a method of drawing a circular guideline using a highlighting method will be described with reference to Fig. 8. Fig. 8 is a flow diagram of generating a perspective image in the fourth embodiment. Unless there are any changes or special notes, the description will be the same as that in Fig. 2 of the first embodiment.

[0050] In S8030, the region information acquisition unit 1402 acquires viewing angle information. In this embodiment, information that the radius of the gaze stable viewing angle is 30 degrees is acquired.

[0051] In S8050, the area information drawing unit 1404 calculates the field of view area based on the optical axis direction and angle of view of the virtual camera and the field of view angle information, and draws a circular guide line on the perspective projection image. In this embodiment, based on the field of view angle information of 30 degrees obtained in S8030, θ=30 is set, and the radius of the circle is obtained by the following formula, and a circle with radius radius is drawn from the center of the perspective projection image.

[0052]

number

[0053] An example of the UI screen at this time is shown in FIG. 9. FIG. 9 is a diagram illustrating an example of a UI screen in the fourth embodiment. FIG. 9(a) is an example of a UI screen in which the circle described in S8050 is drawn. The viewing angle of an HMD and a human viewing angle are closer to a circle than to a rectangle. Therefore, it is possible to highlight an area that more closely resembles the actual appearance. In this embodiment, a perfect circle is drawn, but the circle does not have to be a perfect circle. It may be an ellipse or a rectangle with circular corners, a diamond with circular corners, or a combination of a circle and a rectangle. For example, it may look like FIG. 9(b) or 9(c). Note that an attempt to more faithfully reproduce the field of view based on human visual characteristics results in a distorted shape compared to a rectangle or circle. On the other hand, a distorted shape reduces the design quality. Considering that the field of view area to be highlighted is merely a guide, neither is correct; it is a matter of balancing fidelity and design, and should be determined based on the target user.

[0054] [Embodiment 5] In this embodiment, an example of highlighting a field of view by drawing a circle on a fisheye image will be described with reference to Fig. 10. Fig. 10 is a flow diagram for generating a perspective projection image in embodiment 5. Unless otherwise changed or otherwise specified, the description will be the same as that of Fig. 2 in embodiment 1.

[0055] In S10010, the image acquisition unit 1401 acquires a fisheye image. The operation is the same as in S2010, except that the image to be acquired has changed.

[0056] In S10035, the area information drawing unit 1404 calculates the field of view based on the angle of view and viewing angle information of the virtual camera, and draws a circle indicating the field of view on the fisheye image. Assuming that the ultra-wide-angle image in this embodiment is a fisheye image, circles corresponding to the viewing angle based on the viewing angle information are drawn on concentric circles centered on the optical axis. The circles drawn on the concentric circles of the fisheye image maintain their circular shape even when converted to a perspective projection image.

[0057]

number

[0058] In S10050, the image conversion unit 1405 converts the fisheye image into a perspective projection image based on the optical axis direction and angle of view of the virtual camera. This is the same as the operation in S2050. However, the circle drawn on the fisheye image is also converted.

[0059] In this embodiment, a circle is drawn on a fisheye image instead of a perspective projection image. In this way, even if a guide line is drawn on a fisheye image instead of a perspective projection image, it is possible to achieve the same effect as in the other embodiments.

[0060] In this embodiment, the ultra-wide-angle image (fisheye image) on which the guide lines are drawn is converted into a perspective projection image based on the optical axis direction and angle of view of the virtual camera, so the guide lines are drawn in the direction of the optical axis of the virtual camera. In other words, the guide lines are drawn in the object tracking mode described in the third embodiment.

[0061] [Embodiment 6] In this embodiment, an example of relatively emphasizing the inside of the field of view by suppressing the image area outside the field of view will be described with reference to FIG. 11. FIG. 11 is a flow diagram showing the generation of a perspective projection image in the sixth embodiment. In this embodiment, suppressing the image means adjusting the gain to darken the image and reduce the contrast, thereby making specific image areas less noticeable. Note that unless there are any changes or special descriptions, the description will be the same as that of FIG. 2 in the first embodiment.

[0062] In S11050, the area information drawing unit 1404 calculates the field of view area based on the angle of view of the virtual camera and the field of view angle information, and reduces the brightness outside the field of view area of ​​the perspective projection image.

[0063] According to this embodiment, not only is a rectangle drawn as in the first embodiment, but the visual field area can also be emphasized by processing the image.

[0064] In this embodiment, an example of adjusting the gain has been shown as a way to suppress the image area, but this is not limiting. For example, a method of completely blacking out the area outside the viewing area is also possible. It is also possible to suppress the image by applying a low-pass filter to the outside of the viewing angle. In this case, rather than turning the low-pass filter on and off inside and outside the viewing area, a stronger low-pass filter may be applied to the area further outside the viewing area.

[0065] In this embodiment, the boundary between the inside and outside of the visual field area is described as being rectangular, but this is not limiting and may be circular as described in the fourth embodiment.

[0066] [Embodiment 7] In this embodiment, the visual field is displayed in conjunction with head movement of the HMD. Fig. 12 is a flow diagram for generating a perspective projection image in embodiment 7. Unless otherwise specified or changed, the explanation follows the explanation in Fig. 2 of embodiment 1.

[0067] In this embodiment, the generated perspective projection image is output to both the flat display and the HMD, and the display area of ​​the HMD is highlighted on the flat display. The HMD displays an image corresponding to the field of view in the optical axis direction that follows head movement, and the flat display displays an image with a fixed field of view unless there is input from the user looking at the flat display.

[0068] In S12030, the area information acquisition unit 1402 acquires the field of view angle information and HMD orientation information of the connected HMD. At this time, the field of view of the connected HMD is acquired from the model name of that model. Note that the field of view may be acquired directly instead of the model name. The HMD orientation information may be in any format as long as it can acquire the front direction of the HMD, as described below.

[0069] In S12050, the both-device information drawing unit 1404 calculates the field of view based on the HMD posture information, and draws a rectangle indicating the calculated field of view on the perspective projection image in a superimposed manner. x , φ'y ) is calculated. At this time, (φ x , φ y )=(φ' x , φ' y ) using equations (2) and (3), the coordinates of the four points (top left, top right, bottom left, and bottom right) of the rectangle in the field of view are calculated, and the top, bottom, left, and right sides of the rectangle are drawn as lines. This allows the position of the drawn rectangle to change according to the head movement of the HMD.

[0070] Conventionally, when simultaneously outputting to an HMD and a flat display, an HMD image that follows head movement is often displayed on the flat display. Images that follow head movement appear natural to the HMD wearer, but users viewing them on a regular flat display have a problem of significant blurring, making them prone to motion sickness. In this embodiment, by displaying a perspective projection image on the display from the UI unit 1403, in which a rectangle indicating the HMD's field of view is superimposed, it is possible to display a fixed image that is less likely to cause motion sickness while confirming the line of sight of the HMD wearer.

[0071] Note that the displayed rectangle may be drawn with a cross, an X, or a dot intersecting the center of the rectangle, as shown in Figure 13. Figure 13 is a diagram illustrating an example of a screen that represents the center of the viewpoint, which allows the viewpoint direction to be expressed. Note that if it is not necessary to represent the viewing angle, it is not necessary to draw a rectangle, and it is sufficient to simply draw a cross, an X, or a dot that indicates the viewpoint direction.

[0072] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0073] 1401 Image acquisition unit 1402 Area information acquisition unit 1403 UI section 1404 Area information drawing section 1405 Image conversion unit

Claims

1. 1. An image processing device for generating a perspective projection image from a wide-angle image, comprising: a first generating means for generating the perspective projection image from the wide-angle image based on a first parameter related to a predetermined viewing angle; an output means for outputting the generated perspective projection image; Equipped with the perspective projection image includes area information indicating an image area in the wide-angle image that corresponds to the predetermined viewing angle, the region information includes a character string according to the viewing angle; 1. An image processing device comprising:

2. the region information includes a frame indicating an image region corresponding to the predetermined viewing angle; 2. The image processing device according to claim 1, wherein:

3. The character string is a character string indicating the viewing angle.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. The field of view is a field of view of a predetermined HMD, The area information includes a character string indicating a model of the predetermined HMD.

4. The image processing device according to claim 1, wherein the image processing device is a computer.

5. further comprising a parameter acquisition means for acquiring a second parameter related to the wide-angle image and a third parameter related to the perspective projection image; the second parameter includes an angle of view of the wide-angle image, and the third parameter includes an angle of view of the perspective projection image and coordinates in the wide-angle image corresponding to reference coordinates in the perspective projection image; 5. The image processing device according to claim 1, wherein the first generating means generates the perspective projection image from the wide-angle image based on the second parameter, the third parameter, and the first parameter.

6. The wide-angle image is one of an image obtained based on a fisheye lens projection method, an equirectangular image, and a panoramic image.

6. The image processing device according to claim 1, wherein the image processing device is a computer.

7. the region information includes region information for a plurality of image regions for a plurality of viewing angles; 7. The image processing device according to claim 1, wherein the image processing device is a computer.

8. coordinate value acquisition means for acquiring coordinate values ​​specifying coordinates in the wide-angle image; the first generating means determines a position on the perspective projection image at which the region information is to be superimposed, based on the coordinate values ​​acquired by the coordinate value acquiring means; 8. The image processing device according to claim 1, wherein the image processing device is a computer.

9. the coordinate value acquisition means acquires coordinate values ​​at which a specific object is located in the wide-angle image; 9. The image processing device according to claim 8,

10. the first generating means generates the perspective projection image so as to change at least one of brightness and contrast inside and outside the image region.

10. The image processing device according to claim 1, wherein the image processing device is a computer.

11. a second generating means for generating an HMD image by cutting out the image region based on the region information; Equipped with the output means outputs the perspective projection image and the HMD image to separate display devices.

11. The image processing device according to claim 1,

12. 1. An image processing method for generating a perspective projection image from a wide-angle image, comprising: a first generation step of generating the perspective projection image from the wide-angle image based on first parameters related to a predetermined viewing angle; an output step of outputting the generated perspective projection image; Equipped with the perspective projection image includes area information indicating an image area in the wide-angle image that corresponds to the predetermined viewing angle, the region information includes a character string according to the viewing angle; An image processing method comprising:

13. A program for causing a computer to function as the image processing device according to any one of claims 1 to 11.

Images