Image processing apparatus, image processing method, and program

Abstract

To save a proper screenshot, i.e., images with display angles of view of an HMD, and also enable a person to check a situation even outside the display angles of view at the time of saving the screenshot.SOLUTION: When receiving a notification that a screen capture command has been received from a command receiver, an image file generating part acquires wide angle images from a wide angle image generating part. The image file generating part acquires display angle-of-view information from a display image generating part. The image file generating part sets the wide angle images as image data to be stored in a main image region of an image file, and sets numerical values of a vertical angle of view and a horizontal angle of view of a display image to be displayed on an HMD, which is the acquired display angle-of-view information, as data to be stored in a metadata region. The image file generating part generates an image file of a screenshot according to the settings, and saves the image file in an HDD.SELECTED DRAWING: Figure 6

Description

Technical Field

[0004] , , , , , , , ,

[0005] , , , ,

[0001] The present invention relates to image processing technology in a head-mounted display (HMD) or the like.

Background Art

[0002] Conventionally, in a video see-through type head-mounted display (HMD), mixed reality (MR) is realized by displaying an image in which a CG (computer graphics) image is superimposed on a captured image obtained by capturing an object with an imaging device. In such a video see-through type HMD, the viewing angle of the display image (hereinafter referred to as the display viewing angle) is made narrower than the viewing angle of the captured image (hereinafter referred to as the captured viewing angle) for reasons such as facilitating alignment between the actually displayed image (hereinafter referred to as the display image) and the captured image. Therefore, when saving a screenshot in the HMD, generally, instead of the captured image, the display image with a narrower viewing angle is saved as an image file. Therefore, from the saved image file, it is not possible to view the area outside the display viewing angle that was viewable in the captured image.

[0003] Patent Document 1 discloses a method of generating an image with a wider viewing angle than the display image displayed on the HMD and displaying it on an external display device.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] If an image with a wider field of view than the display image shown on the HMD generated by this method is saved as a screenshot, it becomes possible to view areas outside the display field of view that were visible in the captured image from the saved image file. However, an image with a wider field of view than the display image shown on the HMD, generated by the method disclosed in Patent Document 1, is not the image that was displayed on the HMD and actually viewed by the viewer, and therefore, in that sense, it is not a true screenshot. [Means for solving the problem]

[0006] The present invention relates to an image processing device for a video see-through type head-mounted display equipped with an imaging device, comprising: a receiving means for receiving a screen capture command instructing the saving of a screenshot; and a file generating means for acquiring a wide-angle image corresponding to the display image that was displayed on the head-mounted display when the screen capture command was received, and generating an image file storing the wide-angle image, wherein the wide-angle image is an image with a wider field of view than the display image displayed on the head-mounted display, generated based on an image captured by the imaging device, and the file generating means acquires field of view information of the display image and sets the field of view information as metadata for the image file. [Effects of the Invention]

[0007] This invention makes it possible to not only save screenshots that have the same field of view as the HMD, but also to check the situation outside the field of view at the time the screenshot was saved. [Brief explanation of the drawing]

[0008] [Figure 1] Configuration diagram of a video see-through HMD system as an image display system. [Figure 2] Internal configuration diagram of the HMD [Figure 3] Diagram showing the system configuration of the image processing device. [Figure 4]Block diagram showing the functional configuration of the image processing apparatus in Embodiment 1 [Figure 5] Flowchart of the process by which the image processing device in Embodiment 1 outputs a display image to the HMD. [Figure 6] Flowchart of the process for saving a screenshot performed by the image processing device in Embodiment 1 [Figure 7] Diagram illustrating captured image, rendered image, and wide-angle image. [Figure 8] A diagram illustrating an example of display field of view information. [Figure 9] Diagram explaining the displayed image [Figure 10] A diagram illustrating the structure of screenshot image files. [Figure 11] Block diagram showing the functional configuration of the image processing apparatus in Embodiment 2 [Figure 12] Flowchart of the process performed by the image processing device in Embodiment 2 [Figure 13] Block diagram showing the functional configuration of the image processing apparatus in Embodiment 3 [Figure 14] Flowchart of the process performed by the image processing device in Embodiment 3 [Figure 15] A diagram illustrating the relationship between the display field of view and CG objects. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. Note that the following embodiments are not limiting to the present invention, and not all combinations of features described in these embodiments are essential to the solution of the present invention. Identical components will be denoted by the same reference numerals.

[0010] <Embodiment 1> Embodiment 1 describes the process of saving the display image of a video see-through HMD as an image file. In this embodiment, the image file is configured such that the image of the captured field of view obtained by the background camera mounted on the HMD is stored in the main image area, and information indicating the display field of view of the HMD is stored in the metadata area.

[0011] Figure 1 shows the configuration of a video see-through HMD system as an image display system in the present invention. The video see-through HMD 101 and the image processing device 102 are connected by video signal lines such as HDMI cables and data signal lines such as USB cables, enabling communication of image data and control signals between them.

[0012] Figure 2 shows the internal configuration of the HMD 101. The HMD 101, which is of the video see-through type, has a left-eye background camera 201 and a right-eye background camera 202 for acquiring imaging images around the HMD 101. The imaging angular fields of these background cameras are the left-eye imaging angular field 209 and the right-eye imaging angular field 210, respectively. In this embodiment, for the sake of simplifying the explanation, it is assumed that the left and right imaging angular fields are equal, and the expression "imaging angular field" is also used to refer to them simply. Also, in order to present an image to the viewer of the HMD 101, it has a left-eye display 203 and a right-eye display 206 composed of display panels such as a liquid crystal panel or an organic EL panel. Further, a left-eye eyepiece lens 204 is arranged in front of the left-eye display 203, and a right-eye eyepiece lens 207 is arranged in front of the right-eye display 206. The viewer views an enlarged virtual image of the display image displayed on the display through these lenses. The display angular fields of the display unit composed of these displays and lenses are the left-eye display angular field 205 and the right-eye display angular field 208, respectively. In this embodiment, for the sake of simplifying the explanation, it is assumed that the left and right display angular fields are equal, and the expression "display angular field" is also used to refer to them simply. Also, it is assumed that the display unit is set such that the entire display surface of the display is within the viewer's field of view, and the display angular field coincides with the angular field of the display image displayed on the display. In addition, the HMD 101 also has various sensors (not shown) for acquiring the position and posture of the viewer.

[0013] Here, it is assumed that the left-eye imaging angular field 209 and the right-eye imaging angular field 210, which are the imaging angular fields of the background cameras 201 and 202, are designed to be wider than the left-eye display angular field 205 and the right-eye display angular field 208, which are the display angular fields of the display unit, respectively. The HMD 101 is worn on the head of the viewer (not shown) and can show the left-eye display image and the right-eye display image (enlarged virtual image) to the left and right eyes of the viewer.

[0014] The image processing device 102 generates a wide-angle image by superimposing a CG rendering image on the captured images acquired from the left-eye background camera 201 and the right-eye background camera 202 of the HMD 101. Then, an image area corresponding to the display angle of view of the HMD 101 is cut out from the wide-angle image to obtain a left-eye display image and a right-eye display image. These display images are then displayed on the left-eye display 203 and the right-eye display 206 of the HMD 101.

[0015] In the present embodiment, the image processing device 102 is described as an independent system configuration from the HMD 101, but it may also have a configuration such as an integrated HMD system including the image processing device 102 inside the HMD 101.

[0016] FIG. 3 is a diagram showing the system configuration of the image processing device 102 of the present invention. The image processing device 102 has a CPU 301, a RAM 302, a ROM 303, a HDD 304, a general-purpose I / F 305, a video output I / F 306, and a video input I / F 307 inside, and these are connected via a main bus 300.

[0017] The CPU 301 is a processor that comprehensively controls each part inside the image processing device 102. The RAM 302 functions as the main memory, work area, etc. of the CPU 301. The ROM 303 stores a group of programs executed by the CPU 301. The HDD 304 stores applications executed by the CPU 301, data used for image processing, etc. The general-purpose I / F 305 is a serial bus interface such as USB or IEEE 1394, and is connected to a user input device 308 such as a keyboard, mouse, or controller. The user can transmit a screen capture command for taking a screen shot to the image processing device 102 through these user input devices 308. Also, the general-purpose I / F 305 is connected to the HMD 101 and is used for acquiring the captured images of the left and right background cameras and acquiring position and orientation information, etc. The video output I / F 306 is an I / F such as HDMI (registered trademark) or DisplayPort, and is used for outputting a display image to the HMD 101.

[0018] Figure 4 is a block diagram showing the functional configuration of the image processing device 102 in this embodiment. In Figure 4, the image processing device 102 includes an image acquisition unit 401, a rendering image generation unit 402, a wide-angle image generation unit 403, a display image generation unit 404, a command receiving unit 405, and an image file generation unit 406. The configuration shown in Figure 4 includes a configuration for generating a display image to be output to the HMD 101 and a configuration for generating a screenshot image file.

[0019] The image acquisition unit 401 acquires two images with parallax from the left eye background camera 201 and the right eye background camera 202, and outputs them to the wide-angle image generation unit 403.

[0020] The rendering image generation unit 402 generates a two-dimensional rendering image for superimposing onto the captured image by performing CG rendering processing on the CG data, and outputs it to the wide-angle image generation unit 403.

[0021] The wide-angle image generation unit 403 generates wide-angle images for the left and right eyes by combining the rendered image with the captured images for the left and right eyes, respectively, and outputs these images to the image file generation unit 406 and the display image generation unit 404.

[0022] The display image generation unit 404 acquires a wide-angle image from the wide-angle image generation unit 403 and obtains display field of view information for the left and right eye displays of the HMD 101 from the ROM 303 or HDD 304. Based on the acquired wide-angle image and display field of view information, the display image generation unit 404 generates a display image and outputs it to the HMD 101. The display image output from the display image generation unit 404 to the HMD 101 is a lens distortion corrected image so that it appears without distortion when viewed through the eyepieces 204 and 207 included in the display unit of the HMD 101.

[0023] When the command receiving unit 405 receives a screen capture command from the user input device 308, it instructs the image file generation unit 406 to acquire a wide-angle image from the wide-angle image generation unit 403.

[0024] When the image file generation unit 406 receives an instruction from the command receiving unit 405 to generate a screenshot image file, it acquires a wide-angle image from the wide-angle image generation unit 403 and display angle information from the display image generation unit 404. Then, based on the acquired wide-angle image and display angle information, the image file generation unit 406 generates an image file and saves it to the HDD 304. In the image file generated by the image file generation unit 406, the wide-angle image is stored in the main image area, and the display angle information is stored in the metadata area. Note that the display angle information may be acquired directly from the ROM 303 or HDD 304 instead of from the display image generation unit 404.

[0025] The following describes the processing flow executed by the image processing device 102 in this embodiment when it receives a screen capture command from a user who is viewing an image on the HMD 101, instructing them to save a screenshot. Figure 5 is a flowchart of the process executed by the image processing device 102 to generate a display image and output it to the HMD 101, and Figure 6 is a flowchart of the process executed when it receives a screen capture command from the user. The CPU 301 reads the program that implements the flowchart shown in Figure 5, which is stored in the ROM 303 or HDD 304, and executes it using the RAM 302 as a work area. In this way, the CPU 301 fulfills the role of each functional configuration shown in Figure 4. In the following flowcharts, each step will be denoted as "S".

[0026] In S501, the display image generation unit 404 acquires the display field of view information of the HMD 101. The display field of view information is acquired by reading the vertical field of view and horizontal field of view that are pre-stored in the ROM 303 or HDD 304 as device information of the HMD 101. In this embodiment, it is assumed that the center of the wide-angle image and the center of the display image coincide, and that the display image can be extracted from the wide-angle image based on the display field of view information. If the centers of the wide-angle image and the display image are misaligned, information regarding the misalignment of the centers should be included in the display field of view information. Here, an example has been described in which the vertical field of view and horizontal field of view that are pre-stored as device information of the HMD 101 are used as the display field of view information, but the display field of view information is not limited to this. For example, as shown in Figure 8, the top-left coordinate (x1, y1), width Wview, and height Hview of the rectangular region 801 corresponding to the display field of view on the wide-angle image, calculated from the relationship between the vertical field of view, horizontal field of view, and imaging field of view, may be acquired as the display field of view information. Alternatively, the coordinates of the top-left (x1, y1) and bottom-right (x2, y2) of the rectangular region 801 corresponding to the display field of view on the wide-angle image may be obtained as display field of view information.

[0027] Steps S502 to S507 are steps for generating and outputting each frame of the display image, which are repeated until the image processing device 102 receives a command to stop displaying the display image on the HMD 101.

[0028] In S502, the image acquisition unit 401 acquires images of the imaging field of view from the left eye background camera 201 and the right eye background camera 202. An example of an acquired image is shown in Figure 7(a). The left eye background camera 201 and the right eye background camera 202 are positioned at different locations so that the acquired images for the left and right eyes become stereo images with parallax.

[0029] In step S503, the rendering image generation unit 402 reads the CG data of the rendering image to be superimposed on the captured image into RAM 302 and generates a two-dimensional rendering image by performing CG rendering processing. When performing CG rendering processing, the field of view setting of the rendering image is set to match the field of view of the captured image and the rendering process is performed. Figure 7(b) shows an example of a rendering image. 702 indicates the CG object to be used for display on the rendering image, and 703 indicates the transparent area other than the CG object 702 used for display. For the compositing process in subsequent steps, the transparent area 703 is either filled with a color that can be reliably distinguished from the CG object 702, or an alpha value is set to make it distinguishable from the CG object 702.

[0030] In S504, the wide-angle image generation unit 403 generates left and right wide-angle images by combining the left and right captured images acquired in S501 and the left and right rendered images generated in S502. An example of a wide-angle image is shown in Figure 7(c). The CG object 702, with the transparent region 703 in the rendered image removed, is combined with the captured image. The combining process is performed by blending the captured image and the rendered image pixel by pixel. The blending ratio can be determined by separately generating distance images for both the captured image and the rendered image, recording distance data to the object for each pixel, and then determining the front-to-back relationship between the CG object 702 and the captured image based on these distance images. The blending ratio can be either 0 or 1 (a binary value). The distance image for the captured image can be generated by stereo matching processing of the left and right captured images with parallax captured by the left and right background cameras, or by separately providing a distance sensor in the HMD 101. The distance image for the rendered image can be generated by setting the distance for the pixels corresponding to the CG object 702 according to a predetermined distance determination method. Through the processing described so far, the wide-angle image is generated with an imaging field of view wider than the display field of view of the HMD101. Note that the wide-angle image only needs to be generated based on the captured image; it may be a wide-angle image generated solely from the captured image without any rendering images being superimposed.

[0031] In step S505, the display image generation unit 404 extracts regions from the wide-angle image corresponding to the vertical and horizontal fields of view specified by the display field of view information, performs lens distortion correction on the extracted images to generate left and right display images, and outputs them to the HMD 101. Figure 9 shows an example of a display image. 901 is an auxiliary line indicating the size of the wide-angle image. 902 shows the display image, which is an image extracted from a portion of the wide-angle image 901.

[0032] In S506, the command receiving unit 405 determines whether or not it has received a screen capture command. If it has received a screen capture command, it proceeds to S507; otherwise, it returns to S502.

[0033] In S507, the image file generation unit 406 executes the process of saving the screenshot. Details of S507 will be described later with reference to Figure 6. Although S501 to S507 are processed sequentially here, S507 may also be processed in parallel with S502 to S506.

[0034] Next, in this embodiment, the details of the screenshot saving process (S507) that the image processing device 102 executes when it receives a screen capture command from the user input device 308 will be explained with reference to Figure 6.

[0035] In S601, when the image file generation unit 406 receives a notification from the command receiving unit 405 indicating that a screen capture command has been received, it acquires a wide-angle image from the wide-angle image generation unit 403. However, if a significant delay occurs between the reception of the screen capture command and the output of the wide-angle image, the wide-angle image output from the wide-angle image generation unit 403 to the image file generation unit 406 may be a wide-angle image generated a predetermined time earlier.

[0036] In S602, the image file generation unit 406 obtains display field of view information from the display image generation unit 404. If the image file generation unit 406 has already obtained the display field of view information, this step may be omitted.

[0037] In S603, the image file generation unit 406 sets the wide-angle image acquired in S601 as image data to be stored in the main image area 1004 of the image file. Figure 10 shows an example of a common image file format such as Exif-JPEG. The image file 1001 for saving a screenshot has a metadata area 1002 for storing metadata such as image size and resolution, a thumbnail image area 1003 for storing thumbnail images, and a main image area 1004 for storing the main image. In this embodiment, the wide-angle image is set as image data to be stored in the main image area 1004.

[0038] In S604, the image file generation unit 406 sets the numerical values ​​of the vertical and horizontal field of view of the display image to be displayed on the HMD 101, which are the display field of view information acquired in S602, as data to be stored in the metadata area 1002.

[0039] In S605, the image file generation unit 407 generates a screenshot image file and saves it to the HDD 304 according to the settings in S505 and S506.

[0040] Through the above process, when saving a screenshot of the HMD101, it is possible to generate an image file that stores an image with a wider field of view than the HMD101's display field of view as the main image, and also stores the HMD101's display field of view information as metadata. As a result, when opening the image file and viewing the screenshot, it is possible to display not only the image with the HMD101's display field of view, but also surrounding images that were not visible to the viewer when the screenshot was saved.

[0041] <Embodiment 2> Embodiment 2 describes the process of setting a display image generated at the display field of view as a thumbnail for the screenshot image file, in addition to the process of Embodiment 1. Note that the process of generating a display image and outputting it to the HMD 101, which is performed by the image processing device 102, is the same as in Embodiment 1, so the explanation is omitted.

[0042] Figure 11 is a block diagram showing the functional configuration of the image processing apparatus 102 in this embodiment. The image processing apparatus 102 in this embodiment has a resizing unit 1107 in addition to the configuration of the image processing apparatus 102 in Embodiment 1. Here, we will omit the explanation of configurations 1101 to 1106, which have the same functions as the configuration of Embodiment 1 shown in Figure 4, and will only explain the points related to the resizing unit 1107, which is a difference from Embodiment 1.

[0043] When the resizing unit 1107 receives notification from the command receiving unit 1105 that a screen capture command has been received, it acquires the display image before lens distortion correction, which was generated by the display image generation unit 404. The resizing unit 1107 then reduces the file size of the acquired display image to generate a reduced image of the display image and outputs it to the image file generation unit 1106.

[0044] The details of the screenshot saving process (S507) executed by the image processing device 102 when it receives a screen capture command from the user input device 308 in this embodiment will be explained below with reference to Figure 12. Figure 12 is a flowchart of the screenshot saving process executed by the image processing device 102. The CPU 301 reads the program that implements the flowchart shown in Figure 12, which is stored in the ROM 303 or HDD 304, and executes it using the RAM 302 as the work area. In this way, the CPU 301 fulfills the role of each functional configuration shown in Figure 11. In the following flowchart, each process (step) will be denoted as "S".

[0045] Since steps S1201, S1204-S1206 are the same processes as steps S601-S605 in the flowchart shown in Figure 6 of Embodiment 1, their explanation is omitted here.

[0046] In S1202, the resizing unit 1107 acquires the display image before lens distortion correction from the display image generation unit 1104.

[0047] In S1203, the resizing unit 1107 performs a thumbnail image generation process to reduce the size of the display image acquired in S1202 to a size that conforms to the specifications of the thumbnail image of the image file format, thereby generating a thumbnail image. The resizing unit 1107 then outputs the generated thumbnail image of the display image to the image file generation unit 1106.

[0048] In S1207, the image file generation unit 1106 sets the thumbnail image generated in S1203 as image data to be stored in the thumbnail image area 1003.

[0049] In S1208, the image file generation unit 1106 generates a screenshot image file according to the settings in S1205 to S1207 and saves it to the HDD 304.

[0050] Through the above process, when saving a screenshot of the HMD101, it is possible to generate an image file in which an image with a wider field of view than the HMD101's display field of view is stored as the main image, and an image with the HMD101's display field of view is stored as the thumbnail image. As a result, the thumbnail image is the same as the screenshot image, making it easier to identify the image file. Furthermore, when opening the image file and viewing the screenshot, it becomes possible to display not only the HMD101's display field of view, but also the surrounding image that was not visible to the viewer when the screenshot was saved.

[0051] <Embodiment 3> In video see-through HMDs, when CG is placed within the real world for viewing, the CG often becomes the main subject. Therefore, Embodiment 3 describes a process that determines whether or not the CG object is within the display field of view and switches the image set in the main image area 1004 based on the determination result.

[0052] Figure 13 is a block diagram showing the functional configuration of the image processing apparatus 102 in this embodiment. The image processing apparatus 102 in this embodiment has a region determination unit 1308 in addition to the configuration of the image processing apparatus 102 in Embodiment 2. Here, we will omit the explanation of configurations 1301 to 1307, which have the same functions as the configuration of Embodiment 2 shown in Figure 11, and will only explain the points related to the region determination unit 1308, which is a difference from Embodiment 2.

[0053] The region determination unit 1308 determines whether the CG object on the rendered image is contained within the display field of view based on the display field of view information, and outputs the determination result to the image file generation unit 1306. The image file generation unit 1306 stores the wide-angle image or the displayed image in the main image area 1004 of the image file according to the determination result of the region determination unit 1308.

[0054] The following describes the processing flow executed by the image processing device 102 of this embodiment when it receives a screen capture command from a user. Figure 14 is a flowchart of the processing executed by the image processing device 102. The CPU 301 reads the program that implements the flowchart shown in Figure 14, which is stored in the ROM 303 or HDD 304, and executes it using the RAM 302 as the work area. In this way, the CPU 301 fulfills the role of each functional configuration shown in Figure 13. In the following flowchart, each process (step) will be denoted as "S".

[0055] Since steps S1401 to S1405 are the same as steps S1201 to S1204 in the flowchart shown in Figure 12 of Embodiment 2, their explanation will be omitted.

[0056] In S1405, the region determination unit 1308 determines whether the CG object on the rendering image generated in S503 is contained within the display field of view, based on the vertical and horizontal field of view information, which is the display field of view information acquired in S1404. Figure 15 shows the relationship between the CG object 1501 on the left and right rendering images generated with the imaging field of view, the bounding box 1502 which is a rectangle surrounding the CG object 1501, and the image region 1503 corresponding to the display field of view. In this step, in order to determine whether the CG object 1501 is contained within the display field of view, the positional relationship between the bounding box 1502 of the CG object 1501 and the image region 1503 corresponding to the display field of view on the rendering image is compared. If the bounding box 1502 is completely contained within the image region 1503, the process proceeds to S1406. If the bounding box 1502 extends beyond the image region 1503, as shown in Figure 15, the process proceeds to S1407.

[0057] In S1406, the image file generation unit 1306 sets the display image acquired in S1402 as image data to be stored in the main image area 1004 of the image file.

[0058] In S1407, the image file generation unit 1306 sets the wide-angle image acquired in S1401 as image data to be stored in the main image area 1004 of the image file.

[0059] S1408 to S1410 are the same as S1206 to S1208 in the flowchart shown in Figure 12 of Embodiment 2.

[0060] In S1410, the image file generation unit 1306 generates a screenshot image file and saves it to the HDD 304 according to S1406, or the settings in S1406 and the settings in S1408 and S1409.

[0061] Furthermore, if the display image is set in S1406 as image data to be stored in the main image area 1004 of the image file, S1408, which sets the display field of view information as data to be stored in the metadata area 1002, may be omitted.

[0062] Through the above process, when saving a screenshot of the HMD101, the image set as the main image can be switched between the image of the capture field of view and the image of the display field of view, depending on whether the target CG object fits within the display field of view. This allows the user to later open the image file and view the screenshot, even if the entire CG object did not fit within the HMD's display field of view when the screenshot was saved, allowing the portion of the CG object that extends beyond the display field of view to be displayed later. On the other hand, if the entire target CG object fits within the display field of view, the file size of the saved image file can be reduced by setting the display image, which has a smaller image size than the wide-angle image, as the main image. Furthermore, when the display image is set as the main image, it can be confirmed that the target CG object does not extend beyond the display field of view.

[0063] <Other Embodiments> In embodiments 1 to 3, the wide-angle image and display image stored in the image file are configured to use images generated for output to the HMD101, but the configuration is not limited to this. For example, the wide-angle image and display image stored in the image file may be newly generated separately from the image generated for output to the HMD101 when a screen capture command is received.

[0064] Furthermore, while embodiments 1 to 3 described examples of saving still images as screenshots, the same effect can be obtained when saving videos as screenshots. In this case, display angle information can be stored in the metadata area of ​​the video file format, image data extracted and reduced from any single frame of the displayed image can be stored in the thumbnail image area, and the video data of the wide-angle image or displayed image can be stored in the main image area.

[0065] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0066] This disclosure includes the following configurations and methods:

[0067] (Composition 1) An image processing device for a video see-through type head-mounted display equipped with an imaging device, A receiving means that receives a screen capture command instructing the saving of a screenshot, A file generation means that, upon receiving the aforementioned screen capture command, acquires a wide-angle image corresponding to the display image shown on the head-mounted display and generates an image file containing the wide-angle image, Equipped with, The wide-angle image is an image with a wider field of view than the display image shown on the head-mounted display, which is generated based on the captured image obtained by the imaging device. The file generation means acquires the field of view information of the displayed image and sets the field of view information as metadata for the image file. An image processing apparatus characterized by the following:

[0068] (Configuration 2) The file generation means sets the wide-angle image as the main image in the image file. The image processing apparatus according to configuration 1, characterized in that...

[0069] (Composition 3) The wide-angle image is an image generated based on an image obtained by combining the captured image and the rendered image. An image processing apparatus according to configuration 1 or 2, characterized by the above.

[0070] (Composition 4) An image processing device for a video see-through type head-mounted display equipped with an imaging device, A receiving means that receives a screen capture command instructing the saving of a screenshot, A file generation means that, upon receiving the aforementioned screen capture command, acquires the display image that was displayed on the head-mounted display or a wide-angle image corresponding to said display image, and generates an image file containing the displayed display image or the wide-angle image, Equipped with, The wide-angle image is an image with a wider field of view than the display image shown on the head-mounted display, which is generated based on the captured image obtained by the imaging device. The file generation means acquires the field of view information of the displayed image, and if the entirety of a predetermined object included in the rendered image fits within the image area specified by the field of view information, it generates an image file containing the displayed image, and if the entirety of the predetermined object does not fit within the image area, it sets the field of view information as metadata and generates an image file containing the wide-angle image. An image processing apparatus characterized by the following:

[0071] (Composition 5) When the file generation means generates an image file containing the display image, it sets the display image as the main image in the image file, and when it generates an image file containing the wide-angle image, it sets the wide-angle image as the main image in the image file. The image processing apparatus according to configuration 4, characterized in that...

[0072] (Composition 6) The system further includes a means for acquiring the display image and generating a reduced image of the display image, The file generation means sets the reduced image as a thumbnail image in the image file. An image processing apparatus according to any one of configurations 1 to 5, characterized by the above.

[0073] (Composition 7) The aforementioned field of view information is a value indicating the horizontal field of view and the vertical field of view. An image processing apparatus according to any one of configurations 1 to 6, characterized in that

[0074] (Composition 8) The field of view information is the coordinates of the vertices of the rectangular region in the first image. An image processing apparatus according to any one of configurations 1 to 6, characterized in that

[0075] (Composition 9) A first image generation means that acquires the captured image from the imaging device and generates the wide-angle image, A second image stabilization means that acquires the wide-angle image from the first image generation means and generates the display image, An image processing apparatus according to any one of configurations 1 to 8, further comprising:

[0076] (Composition 10) An image processing method for a video see-through type head-mounted display equipped with an imaging device, A receiving step that receives a screen capture command instructing the saving of a screenshot, A file generation step which involves obtaining a wide-angle image corresponding to the display image shown on the head-mounted display when the aforementioned screen capture command is received, and generating an image file containing the wide-angle image, It has, The wide-angle image is an image with a wider field of view than the display image shown on the head-mounted display, which is generated based on the captured image obtained by the imaging device. The file generation step involves obtaining the field of view information of the displayed image and setting the field of view information as metadata for the image file. An image processing method characterized by the following:

[0077] (Composition 11) An image processing method for a video see-through type head-mounted display equipped with an imaging device, A receiving step that receives a screen capture command instructing the saving of a screenshot, A file generation step which acquires the display image or a wide-angle image corresponding to the display image that was displayed on the head-mounted display when the screen capture command is received, and generates an image file that stores the displayed image or the wide-angle image, It has, The wide-angle image is an image with a wider field of view than the display image shown on the head-mounted display, which is generated based on the captured image obtained by the imaging device. The file generation step involves obtaining the field of view information of the displayed image, generating an image file containing the displayed image if the entirety of a predetermined object included in the rendered image fits within the image region specified by the field of view information, and generating an image file containing the wide-angle image if the entirety of the predetermined object does not fit within the image region, setting the field of view information as metadata. An image processing method characterized by the following:

[0078] (Composition 12) A program for causing a computer to perform each step of the image processing method described in configuration 10 or 11. [Explanation of Symbols]

[0079] Command receiving unit 405 Image file generation unit 406

Claims

1. An image processing device for a head mounted display that displays a display image based on a captured image, receiving means for receiving a user instruction to save the screenshot; a file generating means for acquiring a wide-angle image corresponding to a display image displayed on the head mounted display when the user instruction was given and information on an angle of view of the display image, the wide-angle image and the information on the angle of view being stored in a predetermined file format; Equipped with The wide-angle image is an image having a wider angle of view than the display image.

13. An image processing device comprising:

2. the file generating means sets the wide-angle image as a main image in the images stored in the file format; 2. The image processing device according to claim 1,

3. The wide-angle image is an image generated based on an image obtained by combining the captured image and a rendering image.

2. The image processing device according to claim 1,

4. The wide-angle image is the display image or a wide-angle image corresponding to the display image, the file generation means acquires angle of view information of the display image, and generates an image file storing the display image if a predetermined object included in the wide-angle image fits entirely within an image area specified by the angle of view information, and generates an image file storing the wide-angle image by setting the angle of view information as metadata if the predetermined object does not fit entirely within the image area.

2. The image processing device according to claim 1,

5. When generating an image file storing the display image, the file generation means sets the display image as a main image in the image file, and when generating an image file storing the wide-angle image, the file generation means sets the wide-angle image as a main image in the image file.

2. The image processing device according to claim 1,

6. a reduced image generating means for acquiring the display image and generating a reduced image of the display image; the file generating means sets the reduced image as a thumbnail image in the image stored in the file format; 2. The image processing device according to claim 1,

7. The angle of view information is a value indicating a horizontal angle of view and a vertical angle of view.

2. The image processing device according to claim 1,

8. The angle of view information is coordinates of opposite vertices of a rectangular area in the wide-angle image.

2. The image processing device according to claim 1,

9. A first image generating means for acquiring the captured image and generating the wide-angle image; a second image freeze means for acquiring the wide-angle image from the first image generating means and generating the display image; 9. The image processing device according to claim 1, further comprising:

10. The image processing device described in Claim 1, characterized in that the head-mounted display is a video see-through type head-mounted display equipped with an imaging device.

11. An image processing method for a head mounted display that displays a display image based on a captured image, comprising: receiving a user instruction to save the screenshot; a file generating step of acquiring a wide-angle image corresponding to a display image displayed on the head mounted display when the user instruction was given and angle-of-view information including an angle of view of the display image, and setting the wide-angle image and the angle-of-view information to be stored in a predetermined file format; having The wide-angle image is an image having a wider angle of view than the display image.

13. An image processing method comprising:

12. A program for causing a computer to execute each step of the image processing method according to claim 11.

Images