Image processing device, image processing method and program

Abstract

To provide a mechanism which allows an observer to quickly confirm information on the outside of a display field angle observed by the observer without confirming a main image by opening an image file.SOLUTION: An image processing device comprises: an image acquisition unit 403 which acquires a first thumbnail image having a first field angle recorded on an image file; an image generation unit 404 which generates a second thumbnail image having a second field angle different from the first field angle; an image setting unit 405 which sets a thumbnail image to be displayed on a thumbnail display region according to whether or not a position of a cursor operated by a user is located on the thumbnail display region of a screen; and a display unit 406 which displays the thumbnail image set by the image setting unit 405 on the thumbnail display region of the screen. The image setting unit 405 sets the second thumbnail image as the thumbnail image to be displayed on the thumbnail display region when the position of the cursor is located on the thumbnail display region of the screen.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to image processing for processing thumbnail images on a display. [Background technology]

[0002] Conventionally, video see-through head-mounted displays (HMDs) display images on a display element such as an LCD, superimposing a computer graphics (CG) image onto a captured image of a real subject, and then direct light from the display element to the observer's eyes through a display optical system. In order to facilitate alignment between the captured image and the display image, such video see-through HMDs generally have a wider field of view (hereinafter referred to as the "captured field of view") than the field of view displayed on the display element (hereinafter referred to as the "display field of view"). Furthermore, such HMDs use a screenshot function to record the displayed image being viewed by the observer as an image file. However, screenshots generally record the displayed image at the display field of view as an image file. Therefore, when the screenshot image recorded in the image file is later played back and viewed on an external display, it is not possible to confirm the surroundings of the displayed field of view.

[0003] Patent Document 1 discloses a technology for generating an image corresponding to a wider angle of view than the display image displayed on the HMD in such a video see-through HMD. Patent Document 1 describes displaying an image corresponding to a wider angle of view than the display image on an external display. When generating a screenshot image file, instead of setting an image with the display angle of view as the main image, an image with a wider angle of view than the display angle of view is generated using the method disclosed in Patent Document 1 and set as the main image. Meanwhile, an image with the display angle of view is set as a thumbnail image as in the conventional method. Then, this main image and thumbnail image are recorded together in an image file. This allows the viewer to confirm the image they were actually observing by displaying the thumbnail image, and further allows the viewer to confirm the surrounding situation by displaying the main image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-23026 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the conventional technology described above, it was not possible to check information outside the viewing angle of the HMD observer when saving the screenshot from the thumbnail image alone, so it was necessary to open the image file once and check the main image, which posed a problem in that it took a long time to display the main image.

[0006] The present invention has been made in view of such problems, and aims to enable a viewer to quickly check information outside the display angle of view that he or she has been observing, without having to open an image file and check the main image. [Means for solving the problem]

[0007] The image processing device of the present invention performs processing to display thumbnail images on a screen, and includes an input means for inputting an image file including a main image having a wider angle of view than the thumbnail image, and a control means for controlling the main image to be displayed on the screen as a thumbnail image when a cursor is positioned over a thumbnail display area on the screen. [Effects of the Invention]

[0008] According to the present invention, the viewer can quickly check information outside the display angle of view that he or she has been observing, without opening an image file and checking the main image. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram showing an example of a schematic configuration of an image display system (a video see-through HMD system) according to a first embodiment. [Figure 2] 2 is a diagram illustrating an example of the internal configuration of the HMD illustrated in FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of a system configuration of the image processing device illustrated in FIG. [Figure 4] FIG. 4 illustrates a first embodiment and shows an example of the configuration of the image processing device illustrated in FIGS. 1 and 3. [Figure 5] 5 is a flowchart illustrating an example of a processing procedure of an image processing method performed by the image processing device illustrated in FIGS. 3 and 4 according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating a first embodiment and explains a method for displaying thumbnail images that changes depending on whether the cursor position is located on a thumbnail display area for an image file on the screen. [Figure 7] FIG. 10 shows a third embodiment and is a diagram illustrating a method for displaying thumbnail images when the cursor is positioned on a thumbnail display area for an image file on the screen. [Figure 8] FIG. 4 illustrates a second embodiment and is a diagram illustrating an example of a functional configuration of the image processing device illustrated in FIGS. 1 and 3. [Figure 9] 10 is a flowchart illustrating an example of a processing procedure of an image processing method performed by the image processing device illustrated in FIGS. 3 and 8 according to the second embodiment. [Figure 10] FIG. 10 shows a second embodiment and is a diagram illustrating a method of displaying thumbnail images that changes depending on whether the cursor position is located on a thumbnail display area for an image file on the screen. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, with reference to the drawings, a description will be given of a mode (embodiment) for carrying out the present invention. At this time, the embodiment described below does not limit the present invention, and not all of the combinations of features described in the embodiment are necessarily essential to the solution of the present invention. Note that the same components will be described with the same reference numerals.

[0011] (First embodiment) First, the first embodiment will be described.

[0012] In the first embodiment, when a file list using thumbnail images is displayed on a screen for image files that include main images with a wider angle of view than thumbnail images, the following processing is performed: Specifically, in the first embodiment, processing is performed to change the angle of view of the displayed thumbnail images according to the position of a cursor operated by a user who is the viewer.

[0013] 1 is a diagram showing an example of a schematic configuration of an image display system (video see-through type HMD system) 100 according to the first embodiment. As shown in FIG. 1, the image display system 100 is configured to include a video see-through type HMD 101, an image processing device 102, and a user input device 103.

[0014] The HMD 101 is a head-mounted display worn on the head of an observer, and allows the observer to observe captured images for the left and right eyes through the observer's left and right eyes, respectively. This HMD 101 is a video see-through type HMD.

[0015] The image processing device 102 is a device that processes images obtained by the HMD 101. In this embodiment, the image processing device 102 and the HMD 101 are connected by a video signal line such as an HDMI (registered trademark) cable or a data signal line such as a USB cable, and perform communication of image data, control signals, and the like.

[0016] The user input device 103 is a device that allows a user, who is an observer wearing the HMD 101 on his / her head, to input to the image processing device 102. The user input device 103 is composed of, for example, a keyboard, a mouse, a controller, and the like.

[0017] FIG. 2 is a diagram showing an example of the internal configuration of the HMD 101 shown in FIG. As shown in FIG. 2, the HMD 101 is provided corresponding to the left eye El and right eye Er of the observer, and includes a left-eye background camera 201 and a right-eye background camera 202 for capturing live images of the real world. The imaging angles of these background cameras 201 and 202 are a left-eye imaging angle of view 209 and a right-eye imaging angle of view 210, respectively. In this embodiment, for the sake of simplicity, the left-eye imaging angle of view 209 and the right-eye imaging angle of view 210 are assumed to be equal, and are also referred to simply as "imaging angle of view." As shown in FIG. 2, the HMD 101 is provided corresponding to the left eye El and right eye Er of the observer, and includes a left-eye display 203 and a right-eye display 206, each of which is configured with a display panel such as a liquid crystal panel or an organic EL panel, for allowing the observer to view images. Furthermore, as shown in FIG. 2 , the HMD 101 has a left eye eyepiece 204 and a right eye eyepiece 207 disposed in front of the displays 203 and 206. The viewer views enlarged virtual images of the images displayed on the displays 203 and 206 through the eyepieces 204 and 207, respectively. The display angles of view of the display units formed by the displays 203 and 206 and the eyepieces 204 and 207 are a left eye display angle of view 205 and a right eye display angle of view 208, respectively. Note that in this embodiment, for the sake of simplicity, the left eye display angle of view 205 and the right eye display angle of view 208 are assumed to be equal, and are also referred to simply as the "display angle of view." The HMD 101 also has various sensors (not shown) for acquiring the position and orientation of the viewer. Here, the imaging angles of view of the background cameras 201 and 202, that is, the left eye imaging angle of view 209 and the right eye imaging angle of view 210, are respectively designed to be wider than the left eye display angle of view 205 and the right eye display angle of view 208, that is, the display angles of view of the display unit.

[0018] The HMD 101 is worn on the head of an observer (not shown), and allows the observer's left eye El and right eye Er to observe (enlarged virtual images of) a left eye display image and a right eye display image, respectively.

[0019] The image processing device 102 generates a first image by superimposing a CG rendering image on a real-life image acquired from the left-eye background camera 201 and the right-eye background camera 202 of the HMD 101. Then, the image processing device 102 acquires an image area corresponding to the display angle of the HMD 101 from the first image and generates a left-eye display image and a right-eye display image. Then, the image processing device 102 displays these display images on the left-eye display 203 and the right-eye display 206 of the HMD 101.

[0020] In this embodiment, the image processing device 102 will be described as being a system configuration independent of the HMD 101, but the HMD 101 may also be configured as an integrated HMD system including the image processing device 102 inside. Also, the user input device 103 will be described as being a system configuration connected to the image processing device 102 by wire, but the user input device 103 may also be configured as being connected to the image processing device 102 by known wireless technology such as Bluetooth.

[0021] FIG. 3 is a diagram showing an example of the system configuration of the image processing device 102 shown in FIG. As shown in FIG. 3, the image processing apparatus 102 has a system configuration including a CPU 301, a RAM 302, a ROM 303, an HDD 304, a general-purpose I / F 305, a video output I / F 306, and a main bus 307.

[0022] The CPU 301 is a processor that comprehensively controls each unit within the image processing device 102. The RAM 302 functions as the main memory, work area, etc. of the CPU 301. The ROM 303 stores a group of programs executed by the CPU 301. The HDD 304 stores applications executed by the CPU 301, data used for image processing, etc. The general-purpose I / F 305 is a serial bus interface such as USB or IEEE 1394, and is connected to the user input device 103. A user who is an observer can send commands such as screenshots to the image processing device 102 through the user input device 103. The general-purpose I / F 305 is also connected to the HMD 101 and is used to acquire images captured by the left-eye background camera 201 and the right-eye background camera 202, as well as position and orientation information. In this embodiment, the general-purpose I / F 305 is configured as an input unit that inputs an image file including a main image with a wider angle of view than a thumbnail image to the CPU 301 or the like (including the RAM 302 and HDD 304) in the image processing device 102. The video output I / F 306 is an I / F such as HDMI or display port, and is used to display a display image on the HMD 101. The main bus 307 is a bus that communicatively connects the CPU 301, RAM 302, ROM 303, HDD 304, general-purpose I / F 305, and video output I / F 306.

[0023] FIG. 4 shows the first embodiment and is a diagram illustrating an example of the configuration of the image processing device 102 shown in FIGS. 4 includes a cursor position acquisition unit 401, a cursor position determination unit 402, a first thumbnail image acquisition unit 403, a second thumbnail image generation unit 404, a thumbnail image setting unit 405, and a display unit 406. For example, the CPU 301 shown in FIG. 3 executes a program stored in the ROM 303, thereby realizing the components 401 to 406 shown in FIG. 4.

[0024] The image processing device 102 performs processing to display a file list using thumbnail images on the screens of the left eye display 203 and the right eye display 206 for image files that include a main image with a wider angle of view than a thumbnail image with a first angle of view.

[0025] The cursor position acquisition unit 401 is a position acquisition unit that acquires the position of a cursor operated by a user, who is an observer wearing the HMD 101 on his / her head, to select an image file. The cursor position acquisition unit 401 then outputs the acquired cursor position to the cursor position determination unit 402.

[0026] The cursor position determination unit 402 is a position determination unit that determines whether the cursor position acquired by the cursor position acquisition unit 401 is located on the thumbnail display area (on the thumbnail image) of the image file on the screen. Then, the cursor position determination unit 402 outputs the determination result to the thumbnail image setting unit 405.

[0027] The first thumbnail image acquisition unit 403 is an image acquisition unit that acquires a thumbnail image having a first angle of view recorded in an image file as a first thumbnail image. The first thumbnail image acquisition unit 403 then outputs the acquired first thumbnail image to the thumbnail image setting unit 405.

[0028] The second thumbnail image generation unit 404 is an image generation unit that generates a second thumbnail image having a second angle of view that is different from the first angle of view of the first thumbnail image described above. In this embodiment, the second thumbnail image generation unit 404 generates a thumbnail image having a second angle of view (wider than the first angle of view) that is the same as the angle of view of the main image recorded in the image file. In other words, the main image is input as the second thumbnail image. The second thumbnail image generation unit 404 then outputs the generated second thumbnail image to the thumbnail image setting unit 405.

[0029] The thumbnail image setting unit 405 is an image setting unit that sets a thumbnail image to be displayed in the thumbnail display area according to the result of the determination by the cursor position determination unit 402 as to whether the cursor is located in the thumbnail display area of ​​the image file on the screen. Specifically, when the cursor is located in the thumbnail display area of ​​the image file on the screen, the thumbnail image setting unit 405 sets the second thumbnail image as the thumbnail image to be displayed in the thumbnail display area. Furthermore, when the cursor is not located in the thumbnail display area of ​​the image file on the screen, the thumbnail image setting unit 405 sets the first thumbnail image as the thumbnail image to be displayed in the thumbnail display area. The thumbnail image setting unit 405 then outputs the set thumbnail image to the display unit 406. Alternatively, the thumbnail image setting unit 405 may output a control signal for controlling the display unit 406 to display the set thumbnail image on the display unit 406. In this case, the thumbnail image setting unit 405 is configured as a control unit in the image processing device 102. In addition to the thumbnail image setting unit 405, this control means may also include a cursor position acquisition unit 401, a cursor position determination unit 402, a first thumbnail image acquisition unit 403, and a second thumbnail image generation unit 404.

[0030] The display unit 406 is a display means that displays the thumbnail images set by the thumbnail image setting unit 405 in thumbnail display areas for image files on the screens of the left-eye display 203 and right-eye display 206 .

[0031] Next, a processing procedure that the image processing device 102 of this embodiment executes based on a cursor operation by a user who is an observer wearing the HMD 101 on his / her head will be described. Fig. 5 is a flowchart showing an example of the processing procedure of an image processing method by the image processing device 102 shown in Figs. 3 and 4 according to the first embodiment. The CPU 301 shown in Fig. 3 reads out a program for executing the processing of the flowchart shown in Fig. 5 stored in the ROM 303 or the HDD 304, and executes the processing using the RAM 302 as a work area. As a result, the CPU 301 shown in Fig. 3 fulfills the roles of the respective functional components shown in Fig. 4. In the flowchart of Fig. 5, each process (step) is represented by "S".

[0032] In S501, the cursor position acquisition unit 401 acquires the position of the cursor due to a user operation input via the user input device 103 to select an image file. At this time, the cursor position acquisition unit 401 acquires the cursor position as a coordinate value in an orthogonal coordinate system having the upper left corner of the screens of the left-eye display 203 and the right-eye display 206 as the origin, for example.

[0033] In S502, the cursor position determination unit 402 determines whether the cursor position acquired in S501 is located on the thumbnail display area of ​​the image file on the screen. The processing of S502 will be described with reference to FIG.

[0034] 6 shows the first embodiment and is a diagram illustrating a thumbnail image display method that changes depending on whether the cursor position is located on a thumbnail display area for an image file on a screen 600. FIG. 6(a) shows an example in which the cursor 601 is not located on the thumbnail display area on the screen 600 of the left eye display 203 and the right eye display 206. FIG. 6(b) shows an example in which the cursor 601 is located on the thumbnail display area on the screen 600 of the left eye display 203 and the right eye display 206. In S502 of FIG. 5, the position of the cursor 601 can be determined by comparing the coordinate values ​​indicating the position of the cursor 601 with the coordinate values ​​indicating the thumbnail display area in which the thumbnail images are displayed (for example, the top left coordinate and bottom right coordinate of the thumbnail display area).

[0035] Here, we return to the explanation of FIG. As a result of the determination in S502, if the position of the cursor acquired in S501 is located on the thumbnail display area of ​​the image file on the screen (S502 / Yes), the process proceeds to S503. In S503, the second thumbnail image generation unit 404 reduces the main image included in the image file designated by the cursor using a known method such as bilinear interpolation, thereby generating a second thumbnail image having a second angle of view corresponding to the angle of view of the main image. The second thumbnail image generated in S503 is cached in RAM 302 shown in Fig. 3. If a cache of the second thumbnail image has already been generated, the generation process of the second thumbnail image can be replaced by reading out this image data, and the above-mentioned reduction process of the main image does not need to be performed each time.

[0036] Next, in S504, the thumbnail image setting unit 405 sets the thumbnail image to be displayed in the thumbnail display area for the image file on the screen to the second thumbnail image generated in S503.

[0037] Furthermore, if it is determined in S502 that the cursor position acquired in S501 is not located on the thumbnail display area of ​​the image file on the screen (S502 / No), the process proceeds to S505. In S505, the first thumbnail image acquisition unit 403 acquires the thumbnail image having the first angle of view recorded in the image file as the first thumbnail image. The first angle of view of the first thumbnail image is narrower than the second angles of view of the main image and the second thumbnail image.

[0038] Next, in S506, the thumbnail image setting unit 405 sets the thumbnail image to be displayed in the thumbnail display area for the image file on the screen to the first thumbnail image acquired in S505.

[0039] When the process of S504 is completed, or when the process of S506 is completed, the process proceeds to S507. When the process proceeds to S507, the display unit 406 displays the thumbnail images set in S504 or S506 in the thumbnail display areas of the corresponding image files on the screens of the left-eye display 203 and the right-eye display 206. In FIG. 6(a), the cursor 601 is positioned outside the display area of ​​the thumbnail image 602 on the screen 600, so the first thumbnail image 602 recorded in the image file is displayed. On the other hand, in FIG. 6(b), the cursor 601 is positioned inside the display area of ​​the thumbnail image 603, so the second thumbnail image 603, which has a wider angle of view than the first thumbnail image 602, is displayed. Here, in the example shown in FIG. 6(b), the second thumbnail image generation unit 404 generates the second thumbnail image 603, which is larger than the first thumbnail image 602. Furthermore, in this embodiment, as described above, the angle of view (second angle of view) of the second thumbnail image 603 is the same as (the same as) the angle of view of the main image recorded in the image file.

[0040] When the process of S507 ends, the process of the flowchart in FIG. 5 ends.

[0041] The image processing device 102 according to the first embodiment described above performs the following processing when displaying a file list using thumbnail images on the screen for image files including main images with a wider angle of view than the thumbnail images. Specifically, the cursor position acquisition unit 401 acquires the position of a cursor operated by a user who is an observer wearing the HMD 101 on his or her head. The first thumbnail image acquisition unit 403 acquires a first thumbnail image having a first angle of view recorded in the image file. The second thumbnail image generation unit 404 generates a second thumbnail image having a second angle of view different from the first angle of view of the first thumbnail image (in this embodiment, a wider angle of view than the first angle of view). The thumbnail image setting unit 405 then sets a thumbnail image to be displayed in the thumbnail display area depending on whether the position of the cursor acquired by the cursor position acquisition unit 401 is located on the thumbnail display area of ​​the image file on the screen. Specifically, when the cursor is positioned over a thumbnail display area for an image file on the screen, the thumbnail image setting unit 405 sets the second thumbnail image as the thumbnail image to be displayed in the thumbnail display area. Then, the display unit 406 displays the thumbnail image set by the thumbnail image setting unit 405 in the thumbnail display area for the image file on the screens of the left-eye display 203 and the right-eye display 206. With this configuration, when a file list using thumbnail images is displayed on the screen for an image file that includes a main image with a wider angle of view than a thumbnail image, the angle of view of the displayed thumbnail image can be changed according to the position of the cursor operated by the user, allowing the viewer to quickly check information outside the display angle of view that they have been observing, without having to open the image file and check the main image.

[0042] (Second embodiment) Next, a second embodiment will be described. In the following description of the second embodiment, matters common to the first embodiment will be omitted, and only matters different from the first embodiment will be described.

[0043] In the second embodiment, when a file list using thumbnail images is displayed on the screen for an image file that includes a main image with a wider angle of view than the thumbnail image, the following processing is performed: Specifically, in the second embodiment, when the cursor operated by the user who is the viewer is positioned on the thumbnail display area of ​​the image file on the screen, the angle of view of the displayed thumbnail image is changed to the angle of view obtained by cropping out the center of the main image.

[0044] The schematic configuration of the image display system (video see-through type HMD system) according to the second embodiment is similar to the schematic configuration of the image display system (video see-through type HMD system) 100 according to the first embodiment shown in Fig. 1. The internal configuration of the HMD 101 according to the second embodiment is similar to the internal configuration of the HMD 101 according to the first embodiment shown in Fig. 2. The system configuration of the image processing device 102 according to the second embodiment is similar to the system configuration of the image processing device 102 according to the first embodiment shown in Fig. 3.

[0045] FIG. 8 illustrates the second embodiment and is a diagram showing an example of the functional configuration of the image processing apparatus 102 illustrated in FIGS. The image processing device 102 in Figure 8 has a cursor position acquisition unit 801, a cursor position determination unit 802, a first thumbnail image acquisition unit 803, a second thumbnail image generation unit 804, a thumbnail image setting unit 805, a display unit 806, and a display area information acquisition unit 807.

[0046] The cursor position acquisition unit 801 has the same function as the cursor position acquisition unit 401 in Fig. 4. That is, the cursor position acquisition unit 801 is a position acquisition means that acquires the position of a cursor operated by a user who is an observer wearing the HMD 101 on his head to select an image file. Then, the cursor position acquisition unit 801 outputs the acquired cursor position to the cursor position determination unit 802.

[0047] The cursor position determination unit 802 has the same function as the cursor position determination unit 402 in Fig. 4. That is, the cursor position determination unit 802 is a position determination means that determines whether the cursor position acquired by the cursor position acquisition unit 801 is located on the thumbnail display area of ​​the image file on the screen (on the thumbnail image). Then, the cursor position determination unit 802 outputs this determination result to the thumbnail image setting unit 805.

[0048] The first thumbnail image acquisition unit 803 has the same function as the first thumbnail image acquisition unit 403 in Fig. 4. In other words, the first thumbnail image acquisition unit 803 is an image acquisition unit that acquires a thumbnail image having a first angle of view recorded in an image file as a first thumbnail image. The first thumbnail image acquisition unit 803 then outputs the acquired first thumbnail image to the thumbnail image setting unit 805.

[0049] The display area information acquisition unit 807 is an information acquisition unit that acquires display area information indicating the size of the partial area (image area) of the main image corresponding to the first thumbnail image from the metadata of the image file. In this embodiment, the display area information includes information on the width and height of the partial area of ​​the main image corresponding to the first thumbnail image. The display area information acquisition unit 807 then outputs the acquired display area information to the second thumbnail image generation unit 804.

[0050] The second thumbnail image generation unit 804 has the same function as the second thumbnail image generation unit 404 in Fig. 4. That is, the second thumbnail image generation unit 804 is an image generation unit that generates a second thumbnail image having a second angle of view that is different from the first angle of view of the first thumbnail image described above. Specifically, the second thumbnail image generation unit 804 generates, as the second thumbnail image, an image having an angle of view obtained by cropping the central part of the main image to the size of the partial area of ​​the main image that corresponds to the first thumbnail image, based on the display area information from the display area information acquisition unit 807. Then, the second thumbnail image generation unit 804 outputs the generated second thumbnail image to the thumbnail image setting unit 805.

[0051] The thumbnail image setting unit 805 has the same function as the thumbnail image setting unit 405 in Fig. 4. That is, the thumbnail image setting unit 805 is an image setting unit that sets a thumbnail image to be displayed in the thumbnail display area in accordance with the determination result by the cursor position determination unit 802 as to whether or not the cursor is located in the thumbnail display area for the image file on the screen. Specifically, when the cursor is located in the thumbnail display area for the image file on the screen, the thumbnail image setting unit 805 sets the second thumbnail image as the thumbnail image to be displayed in the thumbnail display area. Furthermore, when the cursor is not located in the thumbnail display area for the image file on the screen, the thumbnail image setting unit 805 sets the first thumbnail image as the thumbnail image to be displayed in the thumbnail display area. Then, the thumbnail image setting unit 805 outputs the set thumbnail image to the display unit 806.

[0052] The display unit 806 has the same function as the display unit 406 in Fig. 4. That is, the display unit 806 is a display means that displays the thumbnail images set by the thumbnail image setting unit 805 in thumbnail display areas for image files on the screens of the left-eye display 203 and the right-eye display 206.

[0053] Next, a processing procedure that the image processing device 102 of this embodiment executes based on a cursor operation by a user who is an observer wearing the HMD 101 on his / her head will be described. Fig. 9 is a flowchart showing an example of the processing procedure of an image processing method by the image processing device 102 shown in Figs. 3 and 8 according to the second embodiment. The CPU 301 shown in Fig. 3 reads out a program for executing the processing of the flowchart shown in Fig. 9 stored in the ROM 303 or the HDD 304, and executes the processing using the RAM 302 as a work area. As a result, the CPU 301 shown in Fig. 3 fulfills the roles of the respective functional components shown in Fig. 4. In the flowchart of Fig. 9, each process (step) is represented by "S".

[0054] In S901, the cursor position acquisition unit 801 acquires the position of the cursor due to a user operation input via the user input device 103 to select an image file. At this time, the cursor position acquisition unit 801 acquires the cursor position as, for example, coordinate values ​​in an orthogonal coordinate system with the upper left corner of the screens of the left-eye display 203 and the right-eye display 206 as the origin.

[0055] In S902, the cursor position determination unit 802 determines whether or not the cursor position acquired in S901 is located on the thumbnail display area of ​​the image file on the screen. The processing of this 902 will be described with reference to FIG.

[0056] Fig. 10 shows a second embodiment and is a diagram for explaining a thumbnail image display method that changes depending on whether the cursor position is located on a thumbnail display area of ​​an image file on the screen. Fig. 10(a) shows an example where the cursor 1001 is not located on the thumbnail display area on the screens of the left eye display 203 and the right eye display 206. Fig. 10(b) shows an example where the cursor 1001 is located on the thumbnail display area on the screens of the left eye display 203 and the right eye display 206. In S902 of Fig. 9, the position of the cursor 1001 can be determined by comparing the coordinate values ​​indicating the position of the cursor 1001 with the coordinate values ​​indicating the thumbnail display area in which the thumbnail images are displayed (for example, the upper left coordinate and the lower right coordinate of the thumbnail display area).

[0057] Here, we return to the explanation of FIG. As a result of the determination in S902, if the position of the cursor acquired in S901 is located on the thumbnail display area of ​​the image file on the screen (S902 / Yes), the process proceeds to S903. In S903, the display area information acquisition unit 807 acquires display area information indicating the size of the partial area of ​​the main image corresponding to the first thumbnail image from the metadata of the image file. Here, the first thumbnail image 1002 shown in FIG. 10(a) has a narrower angle of view than the main image 1003, and is therefore an image corresponding to the partial area of ​​the main image 1003. In other words, the display area information acquired in S903 is information for specifying the size of the partial area of ​​the main image 1003 corresponding to this first thumbnail image 1002. The acquired display area information includes information on the width and height of the partial area of ​​the main image 1003 corresponding to the first thumbnail image 1002. Note that the acquired display area information may also include information such as the top left coordinates and bottom right coordinates of the partial area of ​​the main image 1003 corresponding to the first thumbnail image 1002, or the horizontal and vertical angles of view of the partial area relative to the horizontal and vertical angles of view of the main image 1003. That is, any information may be acquired as the display area information as long as it can identify the size (width and height) of the partial area of ​​the main image 1003 corresponding to the first thumbnail image 1002.

[0058] Next, in S904, the second thumbnail image generation unit 804 generates a second thumbnail image having a second angle of view different from the first angle of view of the first thumbnail image, based on the display area information acquired in S903. Specifically, the second thumbnail image generation unit 804 generates the second thumbnail image by reducing an image having an angle of view obtained by cropping the center portion of the main image included in the image file designated by the cursor to the size of the partial area of ​​the main image corresponding to the first thumbnail image. This second thumbnail image generated in S904 is cached in RAM 302 shown in FIG. 3. Note that if a cache of the second thumbnail image has already been generated, the generation process of the second thumbnail image can be replaced by reading out this image data, and the above-mentioned reduction process of the main image does not need to be performed each time.

[0059] Next, in S905, the thumbnail image setting unit 805 sets the thumbnail image to be displayed in the thumbnail display area for the image file on the screen to the second thumbnail image generated in S904.

[0060] Furthermore, if it is determined in S902 that the cursor position acquired in S901 is not located on the thumbnail display area of ​​the image file on the screen (S902 / No), the process proceeds to S906. In S906, the first thumbnail image acquisition unit 803 acquires the thumbnail image having the first angle of view recorded in the image file as the first thumbnail image. The first angle of view of the first thumbnail image is narrower than that of the main image.

[0061] Next, in S907, the thumbnail image setting unit 805 sets the thumbnail image to be displayed in the thumbnail display area for the image file on the screen to the first thumbnail image acquired in S906.

[0062] When the process of S905 is completed, or when the process of S907 is completed, the process proceeds to S908. When the process proceeds to S908, the display unit 806 displays the thumbnail images set in S905 or S907 in the thumbnail display areas of the corresponding image files on the screens of the left-eye display 203 and the right-eye display 206. In FIG. 10(a), the cursor 1001 is located outside the display area of ​​the thumbnail image 1002, so the first thumbnail image 1002 recorded in the image file is displayed. The example shown in FIG. 10(a) indicates that a partial area slightly to the left of the center of the main image 1003 is used as the angle of view of the first thumbnail image 1002. On the other hand, in FIG. 10(b), the cursor 1001 is located inside the display area of ​​the thumbnail image 1004. Therefore, in FIG. 10(b), a second thumbnail image 1004 is displayed, which is generated by cropping the center of the main image 1003 and has a second angle of view different from the first angle of view of the first thumbnail image 1002. Here, the reason why a second thumbnail image 1004 is generated and displayed as shown in FIG. 10(b) is that the main subject is generally often located in the center of the main image 1003.

[0063] The image processing device 102 according to the second embodiment described above performs the following processing when displaying a file list using thumbnail images on a screen for an image file including a main image with a wider angle of view than the thumbnail image. Specifically, the display area information acquisition unit 807 acquires display area information indicating the size of the partial area of ​​the main image corresponding to the first thumbnail image from the metadata of the image file. Then, based on the display area information from the display area information acquisition unit 807, the second thumbnail image generation unit 804 generates, as a second thumbnail image, an image with an angle of view obtained by cropping the central portion of the main image to the size of the partial area of ​​the main image corresponding to the first thumbnail image. With this configuration, when the cursor operated by the user is positioned over the thumbnail display area of ​​the image file on the screen (S902 / Yes), the angle of view of the thumbnail image can be changed to an angle of view obtained by cropping out the central area of ​​the main image, allowing the viewer to quickly check information outside the display angle of view they were observing (in the example shown in FIG. 10, information on the right edge of the second thumbnail image 1004 that is not shown in the first thumbnail image 1002) without having to open the image file and check the main image.

[0064] (Third embodiment) Next, a third embodiment will be described. In the following description of the third embodiment, matters common to the first and second embodiments will be omitted, and only matters different from the first and second embodiments will be described.

[0065] The schematic configuration of the image display system (video see-through type HMD system) according to the third embodiment is similar to the schematic configuration of the image display system (video see-through type HMD system) 100 according to the first embodiment shown in FIG. 1. The internal configuration of the HMD 101 according to the third embodiment is similar to the internal configuration of the HMD 101 according to the first embodiment shown in FIG. 2. Furthermore, the system configuration of the image processing device 102 according to the third embodiment is similar to the system configuration of the image processing device 102 according to the first embodiment shown in FIG. 3. Furthermore, the functional configuration of the image processing device 102 according to the third embodiment is similar to the functional configuration of the image processing device 102 according to the second embodiment shown in FIG. 8, for example.

[0066] Fig. 7 shows the third embodiment and is a diagram for explaining a method for displaying thumbnail images when a cursor 701 is positioned on a thumbnail display area for an image file on a screen 700. Specifically, Fig. 7 shows a modified example of the method for displaying thumbnail images in the first embodiment shown in Fig. 6(b).

[0067] 7(a) and 7(b) respectively show cases where the second thumbnail image generation unit 804 generates second thumbnail images 703 and 705 that are larger than the first thumbnail image 602 shown in Fig. 6(a). In this case, in Fig. 7(a), the second thumbnail image generation unit 804 generates the second thumbnail image 703 to which a rectangle 702 is added as information identifying the area of ​​the first thumbnail image 602 shown in Fig. 6(a) (the area corresponding to the angle of view of the first thumbnail image 602). In Fig. 7(b), the second thumbnail image generation unit 804 generates the second thumbnail image 705 by reducing the brightness of an area 704 that corresponds to the outside of the angle of view of the first thumbnail image 602 shown in Fig. 6(a) as the above-mentioned identifying information. 7(a) and 7(b), in order to calculate the area corresponding to the angle of view of the first thumbnail image 602, the functional configuration of the image processing device 102 is configured to include the display area information acquisition unit 807 shown in the second embodiment. The second thumbnail image generation unit 804 then generates the second thumbnail image 703 using display area information acquired by the display area information acquisition unit 807 that indicates the size of the partial area (image area) of the main image that corresponds to the angle of view of the first thumbnail image 602.

[0068] 7(c) shows a case where the second thumbnail image generation unit 804 generates a second thumbnail image 706 by changing the angle of view without changing the size of the display area of ​​the thumbnail image displayed on the screen. That is, in FIG. 7(c), the second thumbnail image generation unit 804 generates a second thumbnail image 706 of the same size as the first thumbnail image 602 shown in FIG. 6(a). Note that the second thumbnail image 706 shown in FIG. 7(c) may also be subjected to processing such as adding the rectangle 702 shown in FIG. 7(a) as information identifying the area of ​​the first thumbnail image 602 shown in FIG. 6(a) or reducing the brightness of the area 704 shown in FIG. 7(b). While the above description has been given of an example in which the second thumbnail image is displayed in the area where the first thumbnail image is displayed, the second thumbnail image may also be displayed in an area different from the area where the first thumbnail image is displayed. For example, the second thumbnail image may be shifted from the area displaying the first thumbnail image, or may be displayed as a pop-up image shifted from the area displaying the first thumbnail image.

[0069] In any of the examples shown in Figures 7(a) to 7(c), the following effects can be achieved when a file list using thumbnail images is displayed on the screen for image files that include main images with a wider angle of view than the thumbnail images. Specifically, the angle of view of the thumbnail images displayed on the screen can be changed according to the position of the cursor operated by the user who is the viewer. This allows the viewer to quickly check information outside the display angle of view that they have been observing, without having to open the image file and check the main image.

[0070] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. This program and a computer-readable storage medium storing the program are included in the present invention.

[0071] It should be noted that the above-described embodiments of the present invention are merely illustrative examples of the implementation of the present invention, and the technical scope of the present invention should not be construed as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0072] The disclosure of this embodiment includes the following configuration, method, and program. [Configuration 1] An image processing device that performs processing to display thumbnail images on a screen, an input means for inputting an image file including a main image having a wider angle of view than the thumbnail image; a control means for controlling the display of the main image as a thumbnail image on the screen when the cursor is positioned on a thumbnail display area on the screen; 1. An image processing device comprising: [Configuration 2] The control means a position acquisition means for acquiring the position of the cursor; an image acquisition means for acquiring a first thumbnail image having a first angle of view recorded in the image file; an image generating means for generating a second thumbnail image having a second angle of view different from the first angle of view; an image setting means for setting a thumbnail image to be displayed in the thumbnail display area depending on whether the cursor is positioned on the thumbnail display area; and 2. The image processing device according to configuration 1, wherein the control means controls the display of the set thumbnail image on the screen. [Configuration 3] 3. The image processing device according to configuration 2, wherein the second angle of view is wider than the first angle of view. [Configuration 4] 4. The image processing device according to configuration 2 or 3, wherein the second angle of view is the same as the angle of view of the main image recorded in the image file. [Configuration 5] 5. The image processing device according to any one of configurations 2 to 4, wherein the image generating means generates the second thumbnail image that is larger than the first thumbnail image. [Configuration 6] 6. The image processing device according to configuration 5, wherein the image generating means generates the second thumbnail image to which information for identifying an area of ​​the first thumbnail image is added. [Configuration 7] 5. The image processing device according to any one of configurations 2 to 4, wherein the image generating means generates the second thumbnail image having the same size as the first thumbnail image. [Configuration 8] further comprising information acquisition means for acquiring, from metadata of the image file, display area information indicating the size of a partial area of ​​the main image corresponding to the first thumbnail image; The image processing device according to configuration 2, wherein the image generating means generates, as the second thumbnail image, an image having a field of view obtained by cutting out a central part of the main image to a size in the partial area based on the display area information. [Configuration 9] 9. The image processing device according to configuration 8, wherein the display area information includes information about the width and height of the partial area. [Method 1] An image processing method for performing processing to display thumbnail images on a screen, an input step of inputting an image file including a main image having a wider angle of view than the thumbnail image; a control step of performing control to display the main image on the screen as a thumbnail image when the cursor is located on a thumbnail display area on the screen; An image processing method comprising: [Program 1] A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 9. [Explanation of symbols]

[0073] 100: Image display system (video see-through type HMD system), 101: HMD, 102: Image processing device, 103: User input device, 201: Background camera for left eye, 202: Background camera for right eye, 203: Display for left eye, 204: Eyepiece for left eye, 205: Display angle of view for left eye, 206: Display for right eye, 207: Eyepiece for right eye, 208: Display angle of view for right eye, 209: Image capture angle of view for left eye, 210: Image capture for right eye Corner, 301: CPU, 302: RAM, 303: ROM, 304: HDD, 305: General-purpose I / F, 306: Video output I / F, 307: Main bus, 401, 801: Cursor position acquisition unit, 402, 802: Cursor position determination unit, 403, 803: First thumbnail image acquisition unit, 404, 804: Second thumbnail image generation unit, 405, 805: Thumbnail image setting unit, 406, 806: Display unit, 807: Display area information acquisition unit

Claims

1. An image processing apparatus that performs processing for displaying a thumbnail image on a screen, comprising: input means for inputting an image file including a main image having a wider angle of view than the thumbnail image; determination means for determining the position of a cursor on the screen; control means for performing control to display the thumbnail image on the screen when the cursor is not located on the thumbnail display area on the screen, and to display the main image having a size smaller than the size of the screen and larger than the size of the thumbnail image as a thumbnail image on the screen when the cursor is located on the thumbnail display area on the screen; An image processing apparatus characterized by comprising the above.

2. The control means includes: position acquisition means for acquiring the position of the cursor; image acquisition means for acquiring a first thumbnail image having a first angle of view recorded in the image file; image generation means for generating a second thumbnail image having a second angle of view different from the first angle of view; image setting means for setting a thumbnail image to be displayed in the thumbnail display area according to whether the position of the cursor is located on the thumbnail display area; The control means performs control to display the set thumbnail image on the screen. The image processing apparatus according to claim 1, characterized in that The control means performs control to display the set thumbnail image on the screen. The image processing apparatus according to claim 1, characterized in that

3. The image processing apparatus according to claim 2, characterized in that the second angle of view is wider than the first angle of view.

4. The image processing apparatus according to claim 2, characterized in that the second angle of view is the same as the angle of view of the main image recorded in the image file.

5. The image processing apparatus according to claim 2, characterized in that the image generation means generates the second thumbnail image larger than the first thumbnail image.

6. The image processing apparatus according to claim 5, characterized in that the image generation means generates the second thumbnail image with information for identifying the area of the first thumbnail image attached thereto.

7. The image processing apparatus according to claim 2, characterized in that the image generation means generates the second thumbnail image having the same size as the first thumbnail image.

8. The image processing apparatus further includes information acquisition means for acquiring display area information indicating the size in a partial area of the main image corresponding to the first thumbnail image from metadata of the image file. The image processing apparatus according to claim 2, wherein the image generation means generates, as the second thumbnail image, an image of an angle of view obtained by cutting out a central portion of the main image in the size in the partial region based on the display region information.

9. The image processing apparatus according to claim 8, wherein the display region information includes information on a width and a height in the partial region.

10. An image processing method for performing processing for displaying a thumbnail image on a screen, comprising: an input step of inputting an image file including a main image having a wider angle of view than the thumbnail image; a determination step of determining a position of a cursor on the screen; a control step of, when the cursor is not located on the thumbnail display region on the screen, displaying the thumbnail image on the screen, and when the cursor is located on the thumbnail display region on the screen, performing control to display, on the screen, the main image having a size smaller than the size of the screen and larger than the size of the thumbnail image as the thumbnail image; An image processing method characterized by comprising the above.

11. A program for causing a computer to function as each means of the image processing apparatus according to claim 1.

Images