Image processing apparatus, image processing method, and program

Abstract

To reduce processing loads for generating an image representing the view from a viewpoint different from that of a captured image.SOLUTION: When the drawing range of a second image is projected onto a first image, if the drawing range of the second image does not fit into the first image, a second image generation mode is set to mode 2. In the mode 2, a viewpoint conversion unit converts a captured image to a second viewpoint based on first and second viewpoint information and outputs an actual image cut from the captured image after viewpoint conversion to a second image generation unit to fit a display view angle of an HMD. A second CG image generation unit generates a two-dimensional CG image by performing rendering processing based on the second viewpoint information on CG data for generating a CG image to be superimposed on a generated viewpoint-converted live-action image. The second image generation unit generates a second image by performing synthesis processing on the generated viewpoint-converted live-action image and the generated CG image for the left eye and the right eye, respectively.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to an image processing technique for generating images to be displayed on a head mounted display (HMD). [Background technology]

[0002] Conventionally, a video see-through type HMD presents a display image to the wearer of the HMD by displaying on a display device such as an LCD a display image in which a CG (computer graphics) image is superimposed on a real-life image captured by an imaging device mounted on the HMD. Some such HMDs have a screenshot function to save the display image observed by the HMD wearer as an image file, but in general, a screenshot saves the display image as is as an image file. The composition of the display image needs to be determined by the wearer of the HMD by moving their head, but it may not be easy to obtain the desired composition within the limited movable range of a person's head.

[0003] Patent Document 1 discloses a technique for simultaneously displaying a first virtual viewpoint image corresponding to a virtual viewpoint determined based on the position and orientation of a head, and a second virtual viewpoint image corresponding to a virtual viewpoint determined using a controller capable of manipulating the position and orientation independently of the head. The first virtual image and the second virtual image are generated based on a plurality of captured images obtained in advance by synchronously capturing images using a plurality of imaging devices installed around a user wearing an HMD. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2020-135222 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology in Patent Document 1 requires high-load processing, such as foreground / background separation of the captured image and generation of a 3D shape model, and there is a problem in that it is difficult to generate and display a virtual viewpoint image in real time in response to user operations. [Means for solving the problem]

[0006] The present disclosure is an image processing device comprising: an image acquisition means for acquiring an captured image; a viewpoint information acquisition means for acquiring viewpoint information; a first image generation means for generating a first image by cutting out a portion of an image area of ​​the captured image; and a second image generation means for converting the viewpoint of the captured image or the first image based on second viewpoint information different from first viewpoint information corresponding to the captured image and the first image, and generating a second image by cutting out a portion of an image area from the image after viewpoint conversion, wherein the step of generating the second image by the second image generation means is characterized in that, when a drawing range of the second image is projected onto the first image, if the drawing range of the second image does not fit within the first image, the captured image is converted from the viewpoint to generate the second image. Effect of the Invention

[0007] The present disclosure makes it possible to reduce the processing load for generating an image that represents an appearance from a viewpoint different from that of a captured image. [Brief description of the drawings]

[0008] [Figure 1] A diagram showing an example of the configuration of a video see-through type HMD system as an image display system. [Diagram 2] A diagram showing an example of the internal configuration of an HMD [Diagram 3] FIG. 1 is a diagram showing an example of a hardware configuration of an image processing apparatus; [Figure 4] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to a first embodiment. [Diagram 5] 1 is a flowchart of a process executed by the image processing apparatus according to the first embodiment. [Figure 6] A diagram illustrating a real image, a CG image, and a first image. [Figure 7] A diagram illustrating an example of an HMD user and an image displayed on the HMD. [Figure 8] FIG. 2 is a diagram illustrating a second image generating method. [Figure 9] FIG. 13 is a diagram showing a drawing range for each generation mode of a second image; [Figure 10] FIG. 13 is a diagram illustrating an example of mode switching based on the distance between the viewpoint and a CG object in another embodiment. [Figure 11] FIG. 13 is a diagram for explaining an example of mode switching based on the relative positional relationship between the viewpoint and a CG object in another embodiment. [Figure 12] FIG. 13 is a diagram illustrating an example of mode switching based on the occupancy rate of a CG object in a second image in another embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a warning display when an unwanted object is captured in the second image in another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present disclosure. Note that the same components will be described with the same reference numerals.

[0010] [Embodiment 1] In the first embodiment, a screenshot image of a video see-through head mounted display (HMD) can be generated as an image that shows a view from a viewpoint different from that of the HMD wearer. In this embodiment, a screenshot image can be generated by converting a captured image obtained by an imaging device on the head or a CG composite image obtained by combining the captured image with CG to a different viewpoint.

[0011] 1 shows a configuration example of a video see-through type HMD system as an image display system in the present disclosure, which includes a video see-through type HMD 101, an image processing device 102, and a controller 103. These can communicate image data, control signals, and the like by wired connection using a video signal line such as an HDMI (registered trademark) cable or a data signal line such as a USB cable, or wireless connection such as WiFi. In addition, the HMD 101 has various sensors (not shown) for acquiring the positions and orientations of the HMD 101 and the controller 103.

[0012] 2 is a diagram showing an example of the internal configuration of the HMD 101. The video see-through HMD 101 has a left-eye background imaging device 201 and a right-eye background imaging device 202 for acquiring an actual image of the real world. The imaging angles of view of these background imaging devices 201 and 202 are a left-eye imaging angle of view 209 and a right-eye imaging angle of view 210, respectively. Note that in this embodiment, for the sake of simplicity, it is assumed that the left and right imaging angles of view are equal.

[0013] The HMD 101 also has a left-eye display 203 and a right-eye display 206, which are configured with display panels such as a liquid crystal panel or an organic EL panel, in order to present an image to the wearer of the HMD 101. Furthermore, a left-eye eyepiece 204 and a right-eye eyepiece 207 are disposed in front of these displays, and the wearer of the HMD 101 observes an enlarged virtual image of the image displayed on the displays 203 and 206 through these lenses. The display angles of view of the display units configured with these displays and lenses are a left-eye display angle of view 205 and a right-eye display angle of view 208, respectively. Note that in this embodiment, for the sake of simplicity of explanation, the left and right display angles of view are assumed to be equal, and are referred to simply as display angles of view. The display angles of view are assumed to be designed to be narrower than the imaging angle of view.

[0014] The image processing device 102 acquires captured images from the left-eye background imaging device 201 and the right-eye background imaging device 202 of the HMD 101, and generates a first image by superimposing a CG image on a real-life image obtained by cutting out a part of an image area of ​​the captured image so as to match the display angle of the HMD 101. In addition, the image processing device 102 performs viewpoint conversion on the first image, or superimposes a CG image generated based on the converted viewpoint on a real-life image cut out from the captured image after viewpoint conversion, thereby generating a CG composite image at a position and orientation specified by the controller as a second image. Furthermore, the image processing device 102 superimposes the second image on the first image to generate a left-eye display image and a right-eye display image. When the image processing device 102 generates these display images, it outputs them to the HMD 101, and causes them to be displayed on the left-eye display 203 and the right-eye display 206 of the HMD 101, respectively.

[0015] In this embodiment, the image processing device 102 will be described as being an independent system configuration from the HMD 101, but the image processing device 102 may be built into the HMD 101 as an integrated HMD system.

[0016] 3 is a diagram showing an example of the hardware configuration of the image processing device 102 of the present disclosure. The image processing device 102 has a CPU 301, a RAM 302, a ROM 303, a HDD 304, a general-purpose I / F 305, a video output I / F 306, and a video input I / F 307, which are connected to each other via a main bus 300 so as to be able to communicate with each other.

[0017] The CPU 301 is a processor that comprehensively controls each unit in the image processing device 102. The RAM 302 functions as a main memory, a work area, etc. of the CPU 301. The ROM 303 stores a group of programs executed by the CPU 301. The HDD 304 stores applications executed by the CPU 301, data used in image processing, etc. The HDD 304 may be another storage device such as an SSD or a flash memory. The general-purpose I / F 305 is a serial bus interface such as USB or IEEE1394, and is connected to a user input device 308 such as a keyboard, a mouse, or a controller. A user can input commands such as a screenshot and position and orientation information of a controller to the image processing device 102 using the user input device 308. The general-purpose I / F 305 is also connected to the HMD 101, and is used to acquire captured images of the left and right background imaging devices and to acquire position and orientation information of the HMD 101. The video output I / F 306 is an I / F such as an HDMI or a display port, and is used to output a display image to the HMD 101.

[0018] 4 is a block diagram showing an example of the functional configuration of the image processing device 102 in this embodiment. The image processing device 102 includes a first viewpoint information acquisition unit 401, a first CG image generation unit 402, a captured image acquisition unit 403, a first image generation unit 404, a second viewpoint information acquisition unit 405, an image generation method determination unit 406, a viewpoint conversion unit 407, and a second CG image generation unit 408. The image processing device 102 also includes a second image generation unit 409, a display image generation unit 410, a display control unit 411, and an image storage unit 412.

[0019] A first viewpoint information acquisition unit 401 acquires position and orientation information of the HMD 101 from the HMD 101 , and outputs the acquired position and orientation information of the HMD 101 to a first CG image generation unit 402 , an image generation method determination unit 406 , and a viewpoint conversion unit 407 .

[0020] The first CG image generating unit 402 performs rendering processing on the CG data based on the first viewpoint information, generates a two-dimensional first CG image to be superimposed on the captured image, and outputs the first CG image to a first image generating unit 404 .

[0021] The captured image acquisition unit 403 acquires captured images from the left eye background imaging device 201 and the right eye background imaging device 202, outputs an actual image cut out from the captured image to match the display angle of the HMD 101 to the first image generation unit 404, and outputs the captured image to the viewpoint conversion unit 407.

[0022] The first image generation unit 404 generates a first image based on the captured image and the first CG image, and outputs it to the display image generation unit 410 and the viewpoint conversion unit 407. If there is no first CG image, the first image generation unit 404 outputs the real-life image as it is as the first image. If there is a first CG image, the first image generation unit 404 outputs a composite image obtained by combining the real-life image and the first CG image as the first image.

[0023] A second viewpoint information acquisition unit 405 acquires position and orientation information of the controller 103 from the controller 103 , and outputs the information to an image generation method determination unit 406 , a second CG image generation unit 408 , and a viewpoint conversion unit 407 .

[0024] The image generation method determination unit 406 determines an image generation mode from the first viewpoint information and the second viewpoint information, and outputs the determined image generation mode to the viewpoint conversion unit 407, the second CG image generation unit 408, and the second image generation unit 409.

[0025] The viewpoint conversion unit 407 performs viewpoint conversion of the acquired image based on the image generation mode determined by the image generation method determination unit 406. When the image generation mode is mode 1, the first composite image is subjected to viewpoint conversion and output to the display image generation unit 410 and the image storage unit 412. When the image generation mode is mode 2, the captured image is subjected to viewpoint conversion and the captured image after viewpoint conversion is output to the second image generation unit.

[0026] When the second image generation mode is mode 2, the second CG image generation unit 408 performs rendering processing on the CG data based on the second viewpoint information to generate a two-dimensional second CG image to be superimposed on the live-action image after viewpoint conversion, and outputs the image to the second image generation unit 409.

[0027] The second image generation unit 409 generates a second image based on the captured image after viewpoint conversion and the second CG image, and outputs the second image to the display image generation unit 410 and the image storage unit 412. When there is no second CG image, the second image generation unit 409 outputs the real-life image after viewpoint conversion cut out from the captured image after viewpoint conversion as the second image as is. When there is a second CG image, the second image generation unit 409 outputs a composite image obtained by combining the real-life image cut out from the captured image after viewpoint conversion and the second CG image as the second image.

[0028] The display image generating unit 410 synthesizes the acquired first image and second image to generate a display image, and outputs the display image to the display control unit 411.

[0029] The display control unit 411 causes the display images input from the display image generation unit 410 to be displayed on the left eye display 203 and the right eye display 206 of the HMD 101, respectively.

[0030] The image storage unit 412 stores the second image input from the second image generation unit 409 as an image file.

[0031] 7 shows an example in which a person wearing an HMD 101 is previewing a screenshot image. A person 701 wearing an HMD 101 is viewing an image 702 displayed on the HMD 101. A preview image 703 of the screenshot is displayed in the image 702. Parameters of the preview image 703 such as the position and orientation of the viewpoint are adjusted by a controller 103 operated by the wearer 701. Since parameters such as the position and orientation of the viewpoint of the screenshot can be set by operating the controller 103 held in the hand, it is easier to set the viewpoint of the screenshot than to set the viewpoint of the screenshot from the viewpoint of the HMD wearer.

[0032] FIG. 8 shows an overview of the processing in each mode for generating a screenshot image. In this embodiment, a second image, which is a composite image representing the view from the second viewpoint, is saved as a screenshot image. The second image is generated by switching between two image generation modes. As shown in FIG. 8(a), in mode 1, the first image displayed on the HMD 101 is converted to the second viewpoint to generate the second image. As shown in FIG. 8(b), in mode 2, the second image is generated by synthesizing an image obtained by converting a captured image of the first viewpoint to the second viewpoint and a CG image generated based on the second viewpoint information. Compared to mode 1, generation in mode 2 requires generation and synthesis of a CG image in addition to viewpoint conversion processing, so the processing load is high. In this embodiment, the second image can be generated while adjusting the processing load by switching between mode 1 and mode 2. In this embodiment, the screenshot image will be described as being generated using a background image for the left eye or a first image for the left eye.

[0033] FIG. 9 shows an example of the drawing range for each generation mode of the second image. If the drawing range of the screenshot image does not fit within the drawing range of the first image, when the second image is generated in mode 1, an area not shown in the first image will occur in the second image as an area 901 that cannot be drawn. In contrast, in mode 2, viewpoint conversion is performed on the captured image before the live-action image is cut out, and a part of the image area is cut out from the captured image after the viewpoint conversion to generate the live-action image, so that the drawing range of the captured image can be set as the drawing range of the second image. The drawing range can be expanded. In addition, since the CG image is generated based on the second viewpoint information, the CG image can also be rendered in the entire drawing range of the second image. As a result, in mode 2, a wider range can be drawn than in mode 1. Therefore, if the entire second image can be drawn in mode 1, the second image is generated in mode 1, and if an area that cannot be drawn in mode 1 occurs, the second image is generated by switching to mode 2. As a result, the drawing range of the second image can be expanded while suppressing the processing load.

[0034] The following describes the flow of processing executed by the image processing device 102 of this embodiment when it receives a command from a user to start the screenshot mode. FIG. 5 is a flowchart of the processing executed by the image processing device 102. The CPU 301 reads out a program for implementing the flowchart shown in FIG. 5 stored in the ROM 303 or HDD 304, and executes it while using the RAM 302 as a work area. In this way, the CPU 301 fulfills the roles of each of the functional components shown in FIG. 4. Note that in the following flowcharts, each process (step) will be represented as "S".

[0035] In S501, the first viewpoint information acquisition unit 401 acquires viewpoint information of the HMD 101 from the HMD 101. The viewpoint information is information including information necessary for rendering such as the position, orientation, and angle of view of the viewpoint. The first viewpoint information includes viewpoint information of the left eye and the right eye.

[0036] In S502, the captured image acquisition unit 403 acquires captured images from the left-eye background imaging device 201 and the right-eye background imaging device 202, and generates an actual image by cutting out a part of the captured image to fit the display angle of view of the HMD 101. An example of the actual image is shown in Fig. 6(a). The left-eye background imaging device 201 and the right-eye background imaging device 202 are disposed apart from each other, and the two actual images captured by the respective imaging devices are stereo images with parallax.

[0037] In S503, the first CG image generating unit 402 reads out into the RAM 302 the CG data to be superimposed on the live-action image acquired in S502, and generates a two-dimensional CG image by performing rendering processing based on the first viewpoint information. An example of the CG image is shown in Fig. 6(b). The CG image is an image in which a CG object 602 rendered based on the first viewpoint information is placed in a non-display area 603. For the synthesis processing in the subsequent steps, the non-display area 603 is filled with a color that can be clearly distinguished from the CG object 602, or an alpha value is set to make the area distinguishable from the CG object 602.

[0038] In S504, the first image generating unit 404 generates a first image by performing a synthesis process on the real-life image generated in S502 and the CG image generated in S503 for the left eye and the right eye, respectively. FIG. 6(c) shows an example of the first image. The first image is an image in which a real-life image is synthesized with a CG object 602 excluding a non-display area 603 in the CG image. The synthesis process is performed by blending the real-life image and the CG image for each pixel. The blending ratio may be determined based on the front-back relationship between the CG object 602 and the real-life image based on the distance information acquired from the distance image, by separately generating a distance image in which distance information to the object is recorded for each pixel for both the real-life image and the CG image. The blending ratio may be a binary value of 0 or 1. The distance image may be generated by performing a stereo matching process on the real-life images for the left eye and the right eye, or by providing a distance sensor separately in the HMD 101.

[0039] In S505, the second viewpoint information acquisition unit 405 acquires second viewpoint information including position and orientation information of the controller 103. The position and orientation information included in the second viewpoint information may be the position and orientation of the controller 103 itself, or may be a position and orientation offset from the position and orientation of the controller 103. Note that the position information in the second viewpoint information specifies the relative positional relationship between the position information of the first viewpoint information and a reference point. The position and orientation of the controller are calculated based on output values ​​from sensors attached to the controller 103 or the HMD 101. Parameters other than the position and orientation, such as the angle of view, are calculated based on the amount of operation of a button attached to the controller 103.

[0040] In S506, the image generation method determination unit 406 determines the second image generation mode based on the first viewpoint information and the second viewpoint information. When the drawing range of the second image is projected onto the first image, if the drawing range of the second image is within the first image, the second image generation mode is set to mode 1, and if the drawing range of the second image is not within the first image, the second image generation mode is set to mode 2. Four points at the top left, top right, bottom left, and bottom right ends of the second image are projected onto the first image, and if all points are projected within the first image, the drawing range of the second image is deemed to be within the drawing range of the first image. When projecting points on the drawing range of the second image onto the drawing range of the first image, the points on the drawing range of the second image are assumed to be at a certain distance from the first viewpoint, and the two-dimensional points on the drawing range of the second image are converted into three-dimensional points and projected onto the drawing range of the first image.

[0041] In S507, the information processing unit 400 determines whether the generation mode of the second image determined in S506 is mode 1, in which case the process proceeds to S508, and whether the generation mode is mode 2, in which case the process proceeds to S509.

[0042] In S508, the viewpoint conversion unit 407 converts the first image to the second viewpoint based on the first viewpoint information and the second viewpoint information to generate a second image, and outputs the generated second image to the display control unit 411 and the image storage unit 412. Assuming that the first image is projected onto a plane at a predetermined distance from the first viewpoint in three-dimensional space, the first image is projected onto a plane perpendicular to the line of sight from the second viewpoint based on the first viewpoint information and the second viewpoint information. In this way, a second image obtained by viewpoint conversion of the first image is generated.

[0043] In S509, the viewpoint conversion unit 407 converts the captured image to the second viewpoint based on the first viewpoint information and the second viewpoint information, and outputs an actual image cut out from the captured image after the viewpoint conversion so as to match the display angle of view of the HMD 101 to the second image generation unit 409. As in S508, assuming that the captured image is projected onto a plane at a predetermined distance from the first viewpoint in three-dimensional space, the captured image is projected and converted onto a plane orthogonal to the line of sight from the second viewpoint based on the first viewpoint information and the second viewpoint information. This generates a viewpoint-converted captured image.

[0044] In S510, the second CG image generation unit 408, similar to S503, generates a two-dimensional CG image by performing rendering processing based on the second viewpoint information on the CG data for generating a CG image to be superimposed on the viewpoint-converted live-action image generated in S509.

[0045] In S511, similarly to S504, the second image generating unit 409 generates a second image by synthesizing the actual image after viewpoint conversion generated in S509 and the CG image generated in S510 for the left eye and the right eye, respectively.

[0046] In S512, the display image generating unit 410 superimposes the second image on the acquired first image to generate a display image for the left eye and a display image for the right eye, and outputs the generated image to the display control unit 411. As shown in FIG. 7, the second image 703, which is a screenshot image, is superimposed on the first image 702, so that the wearer 701 of the HMD 101 can preview the screenshot image. This allows the wearer 701 to check the angle of view and composition of the screenshot image while grasping the surrounding situation from the first image. The superimposition position of the second image 703 may be fixed, or may be determined in conjunction with the relative position of the controller 103 with respect to the HMD 101. In addition, the display image may be corrected so as to suppress distortion when the display image is observed through the eyepiece of the HMD 101.

[0047] In S513, the display control unit 411 displays the left eye display image and the right eye display image generated in S512 on the left eye display 203 and the right eye display 206, respectively.

[0048] In S514, if a command to start taking a screenshot has been received, the image storage unit 412 records the second image in the HDD 304 as an image file.

[0049] Through the above processing, a screenshot image that shows the view from a viewpoint different from that of the wearer of the HMD 101 can be generated with a reduced processing load in a video see-through type HMD 101 equipped with an imaging device. This allows the HMD wearer to easily obtain a screenshot image with a desired composition.

[0050] [Other embodiments] In the first embodiment, the second image generation mode is set to mode 1 if all the four corner points of the drawing range of the second image are projected into the first image, but it may be set to mode 1 if all the four corner points of the first image projected onto the plane on which the second image is drawn are projected outside the drawing range of the second image. Also, instead of a predetermined distance for the first image, a distance may be calculated for each object and a distance may be set for each image area corresponding to each object, and the surface onto which the first image is projected may be a spherical surface or a cylindrical surface other than a plane. Also, points other than the four corner points of the drawing range of the second image may be used to determine the second image generation mode.

[0051] In the first embodiment, an example in which the generation mode of the second image is switched based on the rendering range of the first viewpoint and the second viewpoint has been described, but the criterion for switching the mode is not limited to this. For example, as shown in Fig. 10, switching may be based on the distance d between the first viewpoint and the CG object. As shown in Fig. 10(a), when the distance d is equal to or greater than a threshold dth, switching is made to mode 1, and when the distance d is less than the threshold dth, switching is made to mode 2 as shown in Fig. 10(b).

[0052] Also, as shown in Fig. 11, the mode may be switched based on the relative positional relationship between the first viewpoint, the second viewpoint, and the CG object. As shown in Fig. 11(a), when the angle θ between the first viewpoint, the CG object, and the second viewpoint is less than a threshold value θth, the mode is switched to mode 1, and as shown in Fig. 11(b), when the angle θ is equal to or greater than a threshold value θth, the mode is switched to mode 2. By switching the generation mode of the second image based on the distance and relative positional relationship between the viewpoint and the CG object, the second image can be generated using mode 2 when an occlusion area is likely to occur, and using mode 1 with a small load when an occlusion area is unlikely to occur. The CG object for which the distance is calculated may be selected by any method, such as the nearest CG object or a CG object designated as a CG object of interest.

[0053] Also, as shown in FIG. 12, the generation mode of the second image may be switched based on the occupancy rate of the CG object on the screenshot image. A bounding box that contains the CG object is projected onto the drawing range of the second image, and the occupancy rate p of the CG object in the second image is calculated based on the area of ​​the projected region 1202. When the occlusion rate is less than the threshold pth as shown in FIG. 12(a), the mode is switched to mode 1, and when the occlusion rate is equal to or greater than the threshold pth as shown in FIG. 12(b), the mode is switched to mode 2. This allows the second image to be generated using mode 2 when an occlusion region is likely to occur, and mode 1, which has a small processing load, when an occlusion region is unlikely to occur. Here, an example in which the generation mode of the second image is switched based on the occupancy rate of the CG object in the second image has been described, but the generation mode of the second image may be switched based on the occupancy rate of the CG object in the first image.

[0054] In the first embodiment, a configuration has been described in which the generation mode of the second image is determined for each frame in a moving image, but the generation mode of the second image may be determined for each set of frames. In addition, in order to maintain the continuity between frames of the second image, a filter may be applied so that the change in the generation mode becomes smooth over time.

[0055] In the first embodiment, an example has been described in which the viewpoint conversion of an image is performed by assuming a predetermined distance from the first viewpoint to the plane onto which the first image is projected. However, the predetermined distance may be a distance determined in advance, or may be a distance determined based on the distance to the CG object being drawn. Moreover, the viewpoint conversion may be performed using a distance image that stores the distance to the object on the first image. The distance image of the first image can be calculated by generating a distance image of the first image during the synthesis process of the first image. The distance image of the real image may be generated by performing a stereo matching process on the left and right captured images obtained by capturing images using the left and right background image capturing devices 201 and 202, or by providing a distance sensor separately in the HMD 101.

[0056] 13, when the controller 103 for specifying the position of the second viewpoint or the like is shown in the second image 1301, a warning may be displayed that the controller 103 or the like is shown in the screenshot. Whether the controller 103 is shown in the second image is determined by projecting a bounding box surrounding the controller 103 based on the position and orientation of the controller 103 onto the drawing range of the second image. This makes it possible to prevent unnecessary objects from being unintentionally shown in the screenshot image.

[0057] In the first embodiment, the screenshot image is generated using the captured image for the left eye or the first image for the left eye, but the screenshot image may be generated using an image for the left eye or the right eye that is closer to the second viewpoint. By shortening the distance between the viewpoints before and after the viewpoint conversion, a high-quality image with less occlusion can be generated.

[0058] In the first embodiment, an example of saving a still image as a screenshot in a file has been described, but the same effect can be obtained when saving a video as a file. In this case, the screenshot images generated for each frame can be saved as a video file.

[0059] (Other Examples) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. The present disclosure includes the following configurations and methods. [Configuration 1] An image acquisition means for acquiring a captured image; A viewpoint information acquisition means for acquiring viewpoint information; a first image generating means for generating a first image by cutting out a partial image area of ​​the captured image; a second image generating means for converting a viewpoint of the captured image or the first image based on second viewpoint information different from first viewpoint information corresponding to the captured image and the first image, and generating a second image by cutting out a part of an image area from the image after the viewpoint conversion; Equipped with The step of generating the second image includes performing viewpoint conversion on the captured image to generate the second image when the drawing range of the second image does not fit within the first image when the drawing range of the second image is projected onto the first image. 13. An image processing device comprising: [Configuration 2] A CG image generating means for generating a CG image based on the viewpoint information is further provided, the first image generating means generates the first image by combining the CG image generated by the CG image generating means based on the first viewpoint information, When the second image generating means performs viewpoint conversion on the captured image based on the second viewpoint information, the second image generating means generates the second image by synthesizing the image after viewpoint conversion with the CG image generated by the CG image generating means based on the second viewpoint information. 2. The image processing device according to claim 1, [Configuration 3] the second image generating means performs viewpoint conversion on the first image to generate the second image when the shortest distance between the position specified in the second viewpoint information and the CG object in the CG image is equal to or greater than a predetermined threshold, and performs viewpoint conversion on the captured image to generate the second image when the shortest distance is less than the predetermined threshold. 3. The image processing device according to configuration 2. [Configuration 4] the second image generating means performs viewpoint conversion on the first image to generate the second image when an occupancy rate of the CG object included in the CG image in the second image is less than a predetermined threshold, and performs viewpoint conversion on the captured image to generate the second image when the occupancy rate is equal to or greater than the predetermined threshold. 3. The image processing device according to configuration 2. [Configuration 5] the second image generating means, when a drawing range of the second image is projected onto the first image and the drawing range of the second image falls within the first image, performs viewpoint conversion on the first image to generate the second image; 5. The image processing device according to any one of configurations 1 to 4. [Configuration 6] the second image generating means acquires distance information between a position specified in the first viewpoint information and an object included in the first image, and performs the viewpoint conversion based on the distance information; 6. The image processing device according to any one of configurations 1 to 5. [Configuration 7] the captured image and the first image are stereo images, the second image generating means generates the second image based on an image in which a position specified in the first viewpoint information is close to a position specified in the second viewpoint information, among the captured image or a stereo image of the first image. 7. The image processing device according to any one of configurations 1 to 6. [Configuration 8] a synthesis means for generating a synthesis image by synthesizing the first image with the second image; 8. The image processing device according to any one of configurations 1 to 7, further comprising: [Configuration 9] The viewpoint information includes information on a position and orientation in a three-dimensional space, and an angle of view. 9. The image processing device according to any one of configurations 1 to 8. [Configuration 10] The captured image is an image obtained by capturing an image using an imaging device mounted on a video see-through type head-mounted display. 10. The image processing device according to any one of configurations 1 to 9. [Configuration 11] Obtaining a captured image; Obtaining viewpoint information; generating a first image by cutting out a part of an image area of ​​the captured image; performing viewpoint conversion on the captured image or the first image based on second viewpoint information different from first viewpoint information corresponding to the captured image and the first image, and generating a second image by cutting out a part of an image area from the image after the viewpoint conversion; having and generating the second image by performing viewpoint conversion on the captured image when the drawing range of the second image does not fit within the first image when the drawing range of the second image is projected onto the first image. 13. An image processing method comprising: [Configuration 12] A program for causing a computer to function as the image processing device according to any one of configurations 1 to 10.

Claims

1. An image acquisition means for acquiring a captured image; A viewpoint information acquisition means for acquiring viewpoint information; a first image generating means for generating a first image by cutting out a partial image area of ​​the captured image; a second image generating means for converting a viewpoint of the captured image or the first image based on second viewpoint information different from first viewpoint information corresponding to the captured image and the first image, and generating a second image by cutting out a part of an image area from the image after the viewpoint conversion; Equipped with The step of generating the second image includes performing viewpoint conversion on the captured image to generate the second image when the drawing range of the second image does not fit within the first image when the drawing range of the second image is projected onto the first image.

13. An image processing device comprising:

2. A CG image generating means for generating a CG image based on the viewpoint information is further provided, the first image generating means generates the first image by combining the CG image generated by the CG image generating means based on the first viewpoint information, When the second image generating means performs viewpoint conversion on the captured image based on the second viewpoint information, the second image generating means generates the second image by synthesizing the image after viewpoint conversion with a CG image generated by the CG image generating means based on the second viewpoint information.

2. The image processing device according to claim 1,

3. the second image generating means performs viewpoint conversion on the first image to generate the second image when the shortest distance between the position specified in the second viewpoint information and the CG object in the CG image is equal to or greater than a predetermined threshold, and performs viewpoint conversion on the captured image to generate the second image when the shortest distance is less than the predetermined threshold.

3. The image processing device according to claim 2.

4. the second image generating means performs viewpoint conversion on the first image to generate the second image when an occupancy rate of the CG object included in the CG image in the second image is less than a predetermined threshold, and performs viewpoint conversion on the captured image to generate the second image when the occupancy rate is equal to or greater than the predetermined threshold.

3. The image processing device according to claim 2.

5. the second image generating means, when a drawing range of the second image is projected onto the first image and the drawing range of the second image falls within the first image, performs viewpoint conversion on the first image to generate the second image; 2. The image processing device according to claim 1,

6. the second image generating means acquires distance information between a position specified in the first viewpoint information and an object included in the first image, and performs the viewpoint conversion based on the distance information; 2. The image processing device according to claim 1,

7. the captured image and the first image are stereo images, the second image generating means generates the second image based on an image in which a position specified in the first viewpoint information is close to a position specified in the second viewpoint information, among the captured image or a stereo image of the first image.

2. The image processing device according to claim 1,

8. a synthesis means for generating a synthesis image by synthesizing the first image with the second image; The image processing device according to claim 1 , further comprising:

9. The viewpoint information includes information on a position and orientation in a three-dimensional space, and an angle of view.

2. The image processing device according to claim 1,

10. The captured image is an image obtained by capturing an image using an imaging device mounted on a video see-through type head-mounted display.

10. The image processing device according to claim 1,

11. Obtaining a captured image; Obtaining viewpoint information; generating a first image by cutting out a part of an image area from the captured image; performing viewpoint conversion on the captured image or the first image based on second viewpoint information different from first viewpoint information corresponding to the captured image and the first image, and generating a second image by cutting out a part of an image area from the image after the viewpoint conversion; having and generating the second image includes performing viewpoint conversion on the captured image to generate the second image when the drawing range of the second image does not fit within the first image when the drawing range of the second image is projected onto the first image.

13. An image processing method comprising:

12. A program for causing a computer to function as the image processing device according to claim 9.

Images