Image processing device, image processing method, and program

Abstract

To save a proper screenshot, i.e., images with display angles of view of an HMD, and also enable a person to check a situation even outside the display angles of view at the time of saving the screenshot.SOLUTION: When receiving a notification that a screen capture command has been received from a command receiver, an image file generating part acquires wide angle images from a wide angle image generating part. The image file generating part acquires display angle-of-view information from a display image generating part. The image file generating part sets the wide angle images as image data to be stored in a main image region of an image file, and sets numerical values of a vertical angle of view and a horizontal angle of view of a display image to be displayed on an HMD, which is the acquired display angle-of-view information, as data to be stored in a metadata region. The image file generating part generates an image file of a screenshot according to the settings, and saves the image file in an HDD.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to image processing technology in a head-mounted display (HMD) or the like.

Background Art

[0002] Conventionally, in a video see-through type head-mounted display (HMD), mixed reality (MR) is realized by displaying an image in which a CG (computer graphics) image is superimposed on a captured image obtained by capturing an object with an imaging device. In such a video see-through type HMD, the viewing angle of the displayed image (hereinafter referred to as the display viewing angle) is made narrower than the viewing angle of the captured image (hereinafter referred to as the captured viewing angle) for reasons such as facilitating alignment between the actually displayed image (hereinafter referred to as the display image) and the captured image. Therefore, when saving a screenshot in an HMD, generally, an image file is saved as a display image with a narrower viewing angle rather than a captured image. Therefore, from the saved image file, it is not possible to view the area outside the display viewing angle that was viewable in the captured image.

[0003] Patent Document 1 discloses a method of generating an image with a wider viewing angle than the display image displayed on an HMD and displaying it on an external display device.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] If an image with a wider field of view than the display image shown on the HMD generated by this method is saved as a screenshot, it becomes possible to view areas outside the display field of view that were visible in the captured image from the saved image file. However, an image with a wider field of view than the display image shown on the HMD, generated by the method disclosed in Patent Document 1, is not the image that was displayed on the HMD and actually viewed by the viewer, and therefore, in that sense, it is not a true screenshot. [Means for solving the problem]

[0006] This invention is NAIO Display images based on the image. Image processing device for head-mounted display, which allows screenshot of keep User instructions Receiving means for receiving, and the User instructions were given. Sometimes a wide-angle image corresponding to the display image shown on the head-mounted display. and , The angle of view information, including the angle of view of the displayed image, is obtained, Wide-angle image and the aforementioned field of view information and of In the specified file format Store Set to do so The wide-angle image is provided as a means for generating a file, and the wide-angle image is The aforementioned The field of view is wider than the displayed image. but Wide image ru It is characterized by the following: [Effects of the Invention]

[0007] This invention makes it possible to not only save screenshots that have the same field of view as the HMD, but also to check the situation outside the field of view at the time the screenshot was saved. [Brief explanation of the drawing]

[0008] [Figure 1] Configuration diagram of a video see-through HMD system as an image display system. [Figure 2] Internal configuration diagram of the HMD [Figure 3] Diagram showing the system configuration of the image processing device. [Figure 4] Block diagram showing the functional configuration of the image processing apparatus in Embodiment 1 [Figure 5] Flowchart of the process for outputting a display image to the HMD executed by the image processing apparatus in Embodiment 1 [Figure 6] Flowchart of the process for saving a screenshot executed by the image processing apparatus in Embodiment 1 [Figure 7] Diagram explaining the captured image, rendered image, and wide-angle image [Figure 8] Diagram explaining an example of display viewing angle information [Figure 9] Diagram explaining the display image [Figure 10] Diagram explaining the structure of the screenshot image file [Figure 11] Block diagram showing the functional configuration of the image processing apparatus in Embodiment 2 [Figure 12] Flowchart of the process executed by the image processing apparatus in Embodiment 2 [Figure 13] Block diagram showing the functional configuration of the image processing apparatus in Embodiment 3 [Figure 14] Flowchart of the process executed by the image processing apparatus in Embodiment 3 [Figure 15] Diagram explaining the relationship between the display viewing angle and the CG object

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all combinations of the features described in this embodiment are essential for the solution means of the present invention. For the same configuration, the same reference numerals will be used for description.

[0010] <Embodiment 1> In Embodiment 1, the process of saving the display image of the video see-through type HMD as an image file will be described. In this embodiment, regarding the process of saving an image file configured such that an image of the imaging angle obtained by imaging with the background camera mounted on the HMD is stored in the main image area and information indicating the display angle of the HMD is stored in the metadata area, it will be described.

[0011] FIG. 1 shows the configuration of a video see-through type HMD system as an image display system according to the present invention. The video see-through type HMD 101 and the image processing device 102 are connected by a video signal line such as an HDMI cable or a data signal line such as a USB cable, and communication of image data, control signals, etc. is possible between them.

[0012] Figure 2 shows the internal configuration of the HMD 101. The HMD 101, being a video see-through type, has a left-eye background camera 201 and a right-eye background camera 202 for acquiring images of the surroundings of the HMD 101. The imaging angles of these background cameras are the left-eye imaging angle 209 and the right-eye imaging angle 210, respectively. In this embodiment, for the sake of simplicity, these left and right imaging angles are assumed to be equal, and are simply referred to as imaging angles. Furthermore, to present images to the viewer of the HMD 101, it has a left-eye display 203 and a right-eye display 206, which are composed of display panels such as liquid crystal panels and organic EL panels. In addition, a left-eye eyepiece 204 is placed in front of the left-eye display 203, and a right-eye eyepiece 207 is placed in front of the right-eye display 206, and the viewer views an enlarged virtual image of the display image shown on the displays through these lenses. The display unit, composed of these displays and lenses, has a display field of view of 205 for the left eye and a display field of view of 208 for the right eye, respectively. In this embodiment, for the sake of simplicity, these left and right display fields of view are assumed to be equal, and are simply referred to as display fields of view. The display unit is set so that the entire display surface of the display fits within the viewer's field of view, and the display field of view matches the field of view of the display image shown on the display. In addition, the HMD101 also has various sensors (not shown) for acquiring the viewer's position and posture.

[0013] Here, the imaging angles of the background cameras 201 and 202, namely the left-eye imaging angle 209 and the right-eye imaging angle 210, are designed to be wider than the display angles of the display unit, namely the left-eye display angle 205 and the right-eye display angle 208, respectively. The HMD 101 is worn on the head of a viewer (not shown) and can display a left-eye display image and a right-eye display image (magnified virtual image) to the viewer's left and right eyes.

[0014] The image processing device 102 generates a wide-angle image by superimposing a CG rendering image onto the captured images acquired from the left-eye background camera 201 and the right-eye background camera 202 of the HMD 101. Then, it extracts an image area corresponding to the display field of view of the HMD 101 from the wide-angle image to create the left-eye display image and the right-eye display image. These display images are then displayed on the left-eye display 203 and the right-eye display 206 of the HMD 101.

[0015] In this embodiment, the image processing device 102 is described as a system configuration independent of the HMD 101, but a configuration such as an integrated HMD system in which the image processing device 102 is included inside the HMD 101 may also be adopted.

[0016] Figure 3 shows the system configuration of the image processing apparatus 102 of the present invention. The image processing apparatus 102 has a CPU 301, RAM 302, ROM 303, HDD 304, general-purpose I / F 305, video output I / F 306, and video input I / F 307 internally, which are connected via the main bus 300.

[0017] The CPU 301 is a processor that comprehensively controls each part of the image processing unit 102. The RAM 302 functions as the main memory and work area of ​​the CPU 301. The ROM 303 stores the programs executed by the CPU 301. The HDD 304 stores applications executed by the CPU 301 and data used for image processing. The general-purpose I / F 305 is a serial bus interface such as USB or IEEE1394 and is connected to user input devices 308 such as a keyboard, mouse, and controller. Through these user input devices 308, the user can transmit screen capture commands for taking screenshots, etc., to the image processing unit 102. The general-purpose I / F 305 is also connected to the HMD 101 and is used to acquire images captured by the left and right background cameras and to acquire position and orientation information. The video output I / F 306 is an interface such as HDMI (registered trademark) or DisplayPort and is used to output images for display to the HMD 101.

[0018] Figure 4 is a block diagram showing the functional configuration of the image processing device 102 in this embodiment. In Figure 4, the image processing device 102 includes an image acquisition unit 401, a rendering image generation unit 402, a wide-angle image generation unit 403, a display image generation unit 404, a command receiving unit 405, and an image file generation unit 406. The configuration shown in Figure 4 includes a configuration for generating a display image to be output to the HMD 101 and a configuration for generating a screenshot image file.

[0019] The image acquisition unit 401 acquires two images with parallax from the left eye background camera 201 and the right eye background camera 202, and outputs them to the wide-angle image generation unit 403.

[0020] The rendering image generation unit 402 generates a two-dimensional rendering image for superimposing onto the captured image by performing CG rendering processing on the CG data, and outputs it to the wide-angle image generation unit 403.

[0021] The wide-angle image generation unit 403 generates wide-angle images for the left and right eyes by combining the rendered image with the captured images for the left and right eyes, respectively, and outputs these images to the image file generation unit 406 and the display image generation unit 404.

[0022] The display image generation unit 404 acquires a wide-angle image from the wide-angle image generation unit 403 and obtains display field of view information for the left and right eye displays of the HMD 101 from the ROM 303 or HDD 304. Based on the acquired wide-angle image and display field of view information, the display image generation unit 404 generates a display image and outputs it to the HMD 101. The display image output from the display image generation unit 404 to the HMD 101 is a lens distortion corrected image so that it appears without distortion when viewed through the eyepieces 204 and 207 included in the display unit of the HMD 101.

[0023] When the command receiving unit 405 receives a screen capture command from the user input device 308, it instructs the image file generation unit 406 to acquire a wide-angle image from the wide-angle image generation unit 403.

[0024] When the image file generation unit 406 receives an instruction from the command receiving unit 405 to generate a screenshot image file, it acquires a wide-angle image from the wide-angle image generation unit 403 and display angle information from the display image generation unit 404. Then, based on the acquired wide-angle image and display angle information, the image file generation unit 406 generates an image file and saves it to the HDD 304. In the image file generated by the image file generation unit 406, the wide-angle image is stored in the main image area, and the display angle information is stored in the metadata area. Note that the display angle information may be acquired directly from the ROM 303 or HDD 304 instead of from the display image generation unit 404.

[0025] The following describes the processing flow executed by the image processing device 102 in this embodiment when it receives a screen capture command from a user who is viewing an image on the HMD 101, instructing them to save a screenshot. Figure 5 is a flowchart of the process executed by the image processing device 102 to generate a display image and output it to the HMD 101, and Figure 6 is a flowchart of the process executed when it receives a screen capture command from the user. The CPU 301 reads the program that implements the flowchart shown in Figure 5, which is stored in the ROM 303 or HDD 304, and executes it using the RAM 302 as a work area. In this way, the CPU 301 fulfills the role of each functional configuration shown in Figure 4. In the following flowcharts, each step will be denoted as "S".

[0026] In S501, the display image generation unit 404 acquires the display field of view information of the HMD 101. The display field of view information is acquired by reading the vertical field of view and horizontal field of view that are pre-stored in the ROM 303 or HDD 304 as device information of the HMD 101. In this embodiment, it is assumed that the center of the wide-angle image and the center of the display image coincide, and that the display image can be extracted from the wide-angle image based on the display field of view information. If the centers of the wide-angle image and the display image are misaligned, information regarding the misalignment of the centers should be included in the display field of view information. Here, an example has been described in which the vertical field of view and horizontal field of view that are pre-stored as device information of the HMD 101 are used as the display field of view information, but the display field of view information is not limited to this. For example, as shown in Figure 8, the top-left coordinate (x1, y1), width Wview, and height Hview of the rectangular region 801 corresponding to the display field of view on the wide-angle image, calculated from the relationship between the vertical field of view, horizontal field of view, and imaging field of view, may be acquired as the display field of view information. Alternatively, the top-left coordinate (x1, y1) and bottom-right coordinate (x2, y2), which are the coordinates of opposite vertices of the rectangular region 801 corresponding to the display field of view on the wide-angle image, may be obtained as display field of view information.

[0027] Steps S502 to S507 are steps for generating and outputting each frame of the display image, which are repeated until the image processing device 102 receives a command to stop displaying the display image on the HMD 101.

[0028] In S502, the image acquisition unit 401 acquires images of the imaging field of view from the left eye background camera 201 and the right eye background camera 202. An example of an acquired image is shown in Figure 7(a). The left eye background camera 201 and the right eye background camera 202 are positioned at different locations so that the acquired images for the left and right eyes become stereo images with parallax.

[0029] In step S503, the rendering image generation unit 402 reads the CG data of the rendering image to be superimposed on the captured image into RAM 302 and generates a two-dimensional rendering image by performing CG rendering processing. When performing CG rendering processing, the field of view setting of the rendering image is set to match the field of view of the captured image and the rendering process is performed. Figure 7(b) shows an example of a rendering image. 702 indicates the CG object to be used for display on the rendering image, and 703 indicates the transparent area other than the CG object 702 used for display. For the compositing process in subsequent steps, the transparent area 703 is either filled with a color that can be reliably distinguished from the CG object 702, or an alpha value is set to make it distinguishable from the CG object 702.

[0030] In S504, the wide-angle image generation unit 403 generates left and right wide-angle images by combining the left and right captured images acquired in S501 and the left and right rendered images generated in S502. An example of a wide-angle image is shown in Figure 7(c). The CG object 702, with the transparent region 703 in the rendered image removed, is combined with the captured image. The combining process is performed by blending the captured image and the rendered image pixel by pixel. The blending ratio can be determined by separately generating distance images for both the captured image and the rendered image, recording distance data to the object for each pixel, and then determining the front-to-back relationship between the CG object 702 and the captured image based on these distance images. The blending ratio can be either 0 or 1 (a binary value). The distance image for the captured image can be generated by stereo matching processing of the left and right captured images with parallax captured by the left and right background cameras, or by separately providing a distance sensor in the HMD 101. The distance image for the rendered image can be generated by setting the distance for the pixels corresponding to the CG object 702 according to a predetermined distance determination method. Through the processing described so far, the wide-angle image is generated with an imaging field of view wider than the display field of view of the HMD101. Note that the wide-angle image only needs to be generated based on the captured image; it may be a wide-angle image generated solely from the captured image without any rendering images being superimposed.

[0031] In step S505, the display image generation unit 404 extracts regions from the wide-angle image corresponding to the vertical and horizontal fields of view specified by the display field of view information, performs lens distortion correction on the extracted images to generate left and right display images, and outputs them to the HMD 101. Figure 9 shows an example of a display image. 901 is an auxiliary line indicating the size of the wide-angle image. 902 shows the display image, which is an image extracted from a portion of the wide-angle image 901.

[0032] In S506, the command receiving unit 405 determines whether or not it has received a screen capture command. If it has received a screen capture command, it proceeds to S507; otherwise, it returns to S502.

[0033] In S507, the image file generation unit 406 executes the process of saving the screenshot. Details of S507 will be described later with reference to Figure 6. Although S501 to S507 are processed sequentially here, S507 may also be processed in parallel with S502 to S506.

[0034] Next, in this embodiment, the details of the screenshot saving process (S507) that the image processing device 102 executes when it receives a screen capture command from the user input device 308 will be explained with reference to Figure 6.

[0035] In S601, when the image file generation unit 406 receives a notification from the command receiving unit 405 indicating that a screen capture command has been received, it acquires a wide-angle image from the wide-angle image generation unit 403. However, if a significant delay occurs between the reception of the screen capture command and the output of the wide-angle image, the wide-angle image output from the wide-angle image generation unit 403 to the image file generation unit 406 may be a wide-angle image generated a predetermined time earlier.

[0036] In S602, the image file generation unit 406 obtains display field of view information from the display image generation unit 404. If the image file generation unit 406 has already obtained the display field of view information, this step may be omitted.

[0037] In S603, the image file generation unit 406 sets the wide-angle image acquired in S601 as image data to be stored in the main image area 1004 of the image file. Figure 10 shows an example of a common image file format such as Exif-JPEG. The image file 1001 for saving a screenshot has a metadata area 1002 for storing metadata such as image size and resolution, a thumbnail image area 1003 for storing thumbnail images, and a main image area 1004 for storing the main image. In this embodiment, the wide-angle image is set as image data to be stored in the main image area 1004.

[0038] In S604, the image file generation unit 406 sets the numerical values ​​of the vertical and horizontal field of view of the display image to be displayed on the HMD 101, which are the display field of view information acquired in S602, as data to be stored in the metadata area 1002.

[0039] In S605, the image file generation unit 407 generates a screenshot image file and saves it to the HDD 304 according to the settings in S505 and S506.

[0040] Through the above process, when saving a screenshot of the HMD101, it is possible to generate an image file that stores an image with a wider field of view than the HMD101's display field of view as the main image, and also stores the HMD101's display field of view information as metadata. As a result, when opening the image file and viewing the screenshot, it is possible to display not only the image with the HMD101's display field of view, but also surrounding images that were not visible to the viewer when the screenshot was saved.

[0041] <Embodiment 2> Embodiment 2 describes the process of setting a display image generated at the display field of view as a thumbnail for the screenshot image file, in addition to the process of Embodiment 1. Note that the process of generating a display image and outputting it to the HMD 101, which is performed by the image processing device 102, is the same as in Embodiment 1, so the explanation is omitted.

[0042] Figure 11 is a block diagram showing the functional configuration of the image processing apparatus 102 in this embodiment. The image processing apparatus 102 in this embodiment has a resizing unit 1107 in addition to the configuration of the image processing apparatus 102 in Embodiment 1. Here, we will omit the explanation of configurations 1101 to 1106, which have the same functions as the configuration of Embodiment 1 shown in Figure 4, and will only explain the points related to the resizing unit 1107, which is a difference from Embodiment 1.

[0043] When the resizing unit 1107 receives notification from the command receiving unit 1105 that a screen capture command has been received, it acquires the display image before lens distortion correction, which was generated by the display image generation unit 404. The resizing unit 1107 then reduces the file size of the acquired display image to generate a reduced image of the display image and outputs it to the image file generation unit 1106.

[0044] The details of the screenshot saving process (S507) executed by the image processing device 102 when it receives a screen capture command from the user input device 308 in this embodiment will be explained below with reference to Figure 12. Figure 12 is a flowchart of the screenshot saving process executed by the image processing device 102. The CPU 301 reads the program that implements the flowchart shown in Figure 12, which is stored in the ROM 303 or HDD 304, and executes it using the RAM 302 as the work area. In this way, the CPU 301 fulfills the role of each functional configuration shown in Figure 11. In the following flowchart, each process (step) will be denoted as "S".

[0045] Since steps S1201, S1204-S1206 are the same processes as steps S601-S605 in the flowchart shown in Figure 6 of Embodiment 1, their explanation is omitted here.

[0046] In S1202, the resizing unit 1107 acquires the display image before lens distortion correction from the display image generation unit 1104.

[0047] In S1203, the resizing unit 1107 performs a thumbnail image generation process to reduce the size of the display image acquired in S1202 to a size that conforms to the specifications of the thumbnail image of the image file format, thereby generating a thumbnail image. The resizing unit 1107 then outputs the generated thumbnail image of the display image to the image file generation unit 1106.

[0048] In S1207, the image file generation unit 1106 sets the thumbnail image generated in S1203 as image data to be stored in the thumbnail image area 1003.

[0049] In S1208, the image file generation unit 1106 generates a screenshot image file according to the settings in S1205 to S1207 and saves it to the HDD 304.

[0050] Through the above process, when saving a screenshot of the HMD101, it is possible to generate an image file in which an image with a wider field of view than the HMD101's display field of view is stored as the main image, and an image with the HMD101's display field of view is stored as the thumbnail image. As a result, the thumbnail image is the same as the screenshot image, making it easier to identify the image file. Furthermore, when opening the image file and viewing the screenshot, it becomes possible to display not only the HMD101's display field of view, but also the surrounding image that was not visible to the viewer when the screenshot was saved.

[0051] <Embodiment 3> In video see-through HMDs, when CG is placed within the real world for viewing, the CG often becomes the main subject. Therefore, Embodiment 3 describes a process that determines whether or not the CG object is within the display field of view and switches the image set in the main image area 1004 based on the determination result.

[0052] Figure 13 is a block diagram showing the functional configuration of the image processing apparatus 102 in this embodiment. The image processing apparatus 102 in this embodiment has a region determination unit 1308 in addition to the configuration of the image processing apparatus 102 in Embodiment 2. Here, we will omit the explanation of configurations 1301 to 1307, which have the same functions as the configuration of Embodiment 2 shown in Figure 11, and will only explain the points related to the region determination unit 1308, which is a difference from Embodiment 2.

[0053] The region determination unit 1308 determines whether the CG object on the rendered image is contained within the display field of view based on the display field of view information, and outputs the determination result to the image file generation unit 1306. The image file generation unit 1306 stores the wide-angle image or the displayed image in the main image area 1004 of the image file according to the determination result of the region determination unit 1308.

[0054] The following describes the processing flow executed by the image processing device 102 of this embodiment when it receives a screen capture command from a user. Figure 14 is a flowchart of the processing executed by the image processing device 102. The CPU 301 reads the program that implements the flowchart shown in Figure 14, which is stored in the ROM 303 or HDD 304, and executes it using the RAM 302 as the work area. In this way, the CPU 301 fulfills the role of each functional configuration shown in Figure 13. In the following flowchart, each process (step) will be denoted as "S".

[0055] Since steps S1401 to S1405 are the same as steps S1201 to S1204 in the flowchart shown in Figure 12 of Embodiment 2, their explanation will be omitted.

[0056] In S1405, the region determination unit 1308 determines whether the CG object on the rendering image generated in S503 is contained within the display field of view, based on the vertical and horizontal field of view information, which is the display field of view information acquired in S1404. Figure 15 shows the relationship between the CG object 1501 on the left and right rendering images generated with the imaging field of view, the bounding box 1502 which is a rectangle surrounding the CG object 1501, and the image region 1503 corresponding to the display field of view. In this step, in order to determine whether the CG object 1501 is contained within the display field of view, the positional relationship between the bounding box 1502 of the CG object 1501 and the image region 1503 corresponding to the display field of view on the rendering image is compared. If the bounding box 1502 is completely contained within the image region 1503, the process proceeds to S1406. If the bounding box 1502 extends beyond the image region 1503, as shown in Figure 15, the process proceeds to S1407.

[0057] In S1406, the image file generation unit 1306 sets the display image acquired in S1402 as image data to be stored in the main image area 1004 of the image file.

[0058] In S1407, the image file generation unit 1306 sets the wide-angle image acquired in S1401 as image data to be stored in the main image area 1004 of the image file.

[0059] S1408 to S1410 are the same as S1206 to S1208 in the flowchart shown in Figure 12 of Embodiment 2.

[0060] In S1410, the image file generation unit 1306 generates a screenshot image file and saves it to the HDD 304 according to S1406, or the settings in S1406 and the settings in S1408 and S1409.

[0061] Furthermore, if the display image is set in S1406 as image data to be stored in the main image area 1004 of the image file, S1408, which sets the display field of view information as data to be stored in the metadata area 1002, may be omitted.

[0062] Through the above process, when saving a screenshot of the HMD101, the image set as the main image can be switched between the image of the capture field of view and the image of the display field of view, depending on whether the target CG object fits within the display field of view. This allows the user to later open the image file and view the screenshot, even if the entire CG object did not fit within the HMD's display field of view when the screenshot was saved, allowing the portion of the CG object that extends beyond the display field of view to be displayed later. On the other hand, if the entire target CG object fits within the display field of view, the file size of the saved image file can be reduced by setting the display image, which has a smaller image size than the wide-angle image, as the main image. Furthermore, when the display image is set as the main image, it can be confirmed that the target CG object does not extend beyond the display field of view.

[0063] <Other Embodiments> In embodiments 1 to 3, the wide-angle image and display image stored in the image file are configured to use images generated for output to the HMD101, but the configuration is not limited to this. For example, the wide-angle image and display image stored in the image file may be newly generated separately from the image generated for output to the HMD101 when a screen capture command is received.

[0064] Furthermore, while embodiments 1 to 3 described examples of saving still images as screenshots, the same effect can be obtained when saving videos as screenshots. In this case, display angle information can be stored in the metadata area of ​​the video file format, image data extracted and reduced from any single frame of the displayed image can be stored in the thumbnail image area, and the video data of the wide-angle image or displayed image can be stored in the main image area.

[0065] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0066] This disclosure includes the following configurations and methods:

[0067] (Composition 1) An image processing device for a video see-through type head-mounted display equipped with an imaging device, A receiving means that receives a screen capture command instructing the saving of a screenshot, A file generation means that, upon receiving the aforementioned screen capture command, acquires a wide-angle image corresponding to the display image shown on the head-mounted display and generates an image file containing the wide-angle image, Equipped with, The wide-angle image is an image with a wider field of view than the display image shown on the head-mounted display, which is generated based on the captured image obtained by the imaging device. The file generation means acquires the field of view information of the displayed image and sets the field of view information as metadata for the image file. An image processing apparatus characterized by the following:

[0068] (Configuration 2) The file generation means sets the wide-angle image as the main image in the image file. The image processing apparatus according to configuration 1, characterized in that...

[0069] (Composition 3) The wide-angle image is an image generated based on an image obtained by combining the captured image and the rendered image. An image processing apparatus according to configuration 1 or 2, characterized by the above.

[0070] (Composition 4) An image processing device for a video see-through type head-mounted display equipped with an imaging device, A receiving means that receives a screen capture command instructing the saving of a screenshot, A file generation means that, upon receiving the aforementioned screen capture command, acquires the display image that was displayed on the head-mounted display or a wide-angle image corresponding to said display image, and generates an image file containing the displayed display image or the wide-angle image, Equipped with, The wide-angle image is an image with a wider field of view than the display image shown on the head-mounted display, which is generated based on the captured image obtained by the imaging device. The file generation means acquires the field of view information of the displayed image, and if the entirety of a predetermined object included in the rendered image fits within the image area specified by the field of view information, it generates an image file containing the displayed image, and if the entirety of the predetermined object does not fit within the image area, it sets the field of view information as metadata and generates an image file containing the wide-angle image. An image processing apparatus characterized by the following:

[0071] (Composition 5) When the file generation means generates an image file containing the display image, it sets the display image as the main image in the image file, and when it generates an image file containing the wide-angle image, it sets the wide-angle image as the main image in the image file. The image processing apparatus according to configuration 4, characterized in that...

[0072] (Composition 6) The system further includes a reduced image generation means for acquiring the displayed image and generating a reduced image of the displayed image, The file generation means sets the reduced image as a thumbnail image in the image file. An image processing apparatus according to any one of configurations 1 to 5, characterized by the above.

[0073] (Composition 7) The aforementioned field of view information is a value indicating the horizontal field of view and the vertical field of view. An image processing apparatus according to any one of configurations 1 to 6, characterized in that

[0074] (Composition 8) The field of view information is the coordinates of the vertices of the rectangular region in the first image. An image processing apparatus according to any one of configurations 1 to 6, characterized in that

[0075] (Composition 9) A first image generation means that acquires the captured image from the imaging device and generates the wide-angle image, A second image stabilization means that acquires the wide-angle image from the first image generation means and generates the display image, An image processing apparatus according to any one of configurations 1 to 8, further comprising:

[0076] (Composition 10) An image processing method for a video see-through type head-mounted display equipped with an imaging device, A receiving step that receives a screen capture command instructing the saving of a screenshot, A file generation step which involves obtaining a wide-angle image corresponding to the display image shown on the head-mounted display when the aforementioned screen capture command is received, and generating an image file containing the wide-angle image, It has, The wide-angle image is an image with a wider field of view than the display image shown on the head-mounted display, which is generated based on the captured image obtained by the imaging device. The file generation step involves obtaining the field of view information of the displayed image and setting the field of view information as metadata for the image file. An image processing method characterized by the following:

[0077] (Composition 11) An image processing method for a video see-through type head-mounted display equipped with an imaging device, A receiving step that receives a screen capture command instructing the saving of a screenshot, A file generation step which involves acquiring the display image or a wide-angle image corresponding to the display image that was displayed on the head-mounted display when the screen capture command is received, and generating an image file that stores the displayed image or the wide-angle image, It has, The wide-angle image is an image with a wider field of view than the display image shown on the head-mounted display, which is generated based on the captured image obtained by the imaging device. The file generation step involves obtaining the field of view information of the displayed image, generating an image file containing the displayed image if the entirety of a predetermined object included in the rendered image fits within the image area specified by the field of view information, and generating an image file containing the wide-angle image if the entirety of the predetermined object does not fit within the image area, setting the field of view information as metadata. An image processing method characterized by the following:

[0078] (Composition 12) A program for causing a computer to perform each step of the image processing method described in configuration 10 or 11. [Explanation of Symbols]

[0079] Command receiving unit 405 Image file generation unit 406

Claims

1. An image processing device for a head-mounted display that displays an image based on an captured image, A receiving means for receiving user instructions to save a screenshot, A file generation means that acquires a wide-angle image corresponding to the display image that was displayed on the head-mounted display when the user instruction was given, and field-of-view information including the field of view of the display image, and sets the wide-angle image and the field-of-view information to be stored in an image file configured in a predetermined file format, thereby generating the image file; A reduction image generation means that acquires the display image and generates a reduced image of the display image, Equipped with, The wide-angle image is an image with a wider field of view than the displayed image. The aforementioned predetermined file format includes at least a metadata area, a thumbnail image area, and a main image area. The file generation means sets the wide-angle image as image data to be stored in the main image area, sets the reduced image as a thumbnail image to be stored in the thumbnail image area, sets the field of view information as metadata to be stored in the metadata area, and the field of view information is information that identifies the image area corresponding to the display image in the wide-angle image. An image processing apparatus characterized by the following:

2. The wide-angle image is an image generated based on an image obtained by combining the captured image and the rendered image. The image processing apparatus according to feature 1.

3. The file generation means generates an image file in which the display image is stored in the main image area instead of the wide-angle image when the entirety of a predetermined object included in the wide-angle image fits within the image area specified by the field of view information. The image processing apparatus according to feature 1.

4. The aforementioned field of view information is a value indicating the horizontal field of view and the vertical field of view. The image processing apparatus according to feature 1.

5. The field of view information is the coordinates of the vertices of the rectangular region in the wide-angle image. The image processing apparatus according to feature 1.

6. A first image generation means that acquires the captured image and generates the wide-angle image, A second image stabilization means that acquires the wide-angle image from the first image generation means and generates the display image, The image processing apparatus according to any one of claims 1 to 5, further comprising:

7. The image processing apparatus according to claim 1, characterized in that the head-mounted display is a video see-through type head-mounted display equipped with an imaging device.

8. An image processing method for a head-mounted display that displays an image based on an captured image, A receiving step that receives a user instruction to save a screenshot, A file generation step involves acquiring a wide-angle image corresponding to the display image shown on the head-mounted display when the user instruction was given, and field-of-view information including the field of view of the display image, and generating an image file by setting the wide-angle image and the field-of-view information to be stored in an image file configured in a predetermined file format, A reduction image generation step involves acquiring the display image and generating a reduced image of the display image, It has, The wide-angle image is an image with a wider field of view than the displayed image. The aforementioned predetermined file format includes at least a metadata area, a thumbnail image area, and a main image area. The file generation step involves setting the wide-angle image as image data to be stored in the main image area, setting the reduced image as a thumbnail image to be stored in the thumbnail image area, setting the field of view information as metadata to be stored in the metadata area, and the field of view information being information that identifies the image area corresponding to the display image in the wide-angle image. An image processing method characterized by the following:

9. A program for causing a computer to perform each step of the image processing method described in claim 8.

Images