Information processing device, method, and program

Abstract

To make it possible to check an area of interest even when the area of interest is out of the angle of view of a captured image.SOLUTION: An information processing device (110) is for displaying on a display unit (112) a captured image by an imaging apparatus (101). The information processing device (110) includes display control means (116) for controlling to display on the display unit (112) an image of an area of interest included in a captured image. When the area of interest is in the angle of view of a frame of the captured image, the display control means (116) displays an image of the area of interest included in the frame. When the area of interest is out of the angle of view of the frame of the captured image, the display control means (116) displays an image of the area of interest included in a frame previous to the frame.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a method, and a program suitable for use in remotely supporting, for example, work performed by an operator.

Background Art

[0002] There is provided a remote support system used to remotely support work performed by an operator. In the remote support system, when an operator photographs a work target with a camera, the photographed video is transferred in real time via a network to a personal computer or the like of a supporter at a remote location. The supporter can grasp the work situation and support the operator by checking the transferred photographed video on a monitor. As a camera for photographing the work target, for example, a digital camera mounted on a head-mounted display (hereinafter abbreviated as HMD) is used. By using the digital camera mounted on the HMD, it is possible to photograph at a position close to the operator's line of sight, and since both hands are not blocked for photographing, the workability is excellent, and the applicable scenes are expanding.

[0003] Patent Document 1 discloses an operation support device that improves the visibility during an operation on a designated area of an operation target, which has a configuration of cutting out the video of the designated area from the camera video and displaying it on a display device as the designated area video.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In remote support systems, if the work object moves out of the field of view of the camera's captured image, the supporter will be unable to see the work object on the monitor, making it difficult to support the worker. For example, when using a digital camera mounted on an HMD, the movement of the worker's head can cause the captured image to blur, resulting in the work object repeatedly moving out of and into the field of view of the captured image. Patent Document 1 does not consider how to deal with the situation when the work object moves out of the field of view of the camera image (captured image).

[0006] This invention has been made in view of the above-mentioned points, and aims to enable the confirmation of the area of ​​interest even when the area of ​​interest is outside the field of view of the captured video. [Means for solving the problem]

[0007] The present invention is an information processing device that displays captured video images taken by an imaging device on a display unit, and includes display control means for controlling the display unit to display an image of a region of interest included in the captured video. The image processing means performs a geometric transformation on the image of the area of ​​interest so as to maintain the same field of view when the field of view of the imaging device changes. The display control means is characterized in that, when the area of ​​interest is within the field of view of the frame of the captured video, it displays a live view image which is an enlarged image of the area of ​​interest included in that frame, and when the area of ​​interest is outside the field of view of the frame of the captured video, it displays a still image which is an enlarged image of the area of ​​interest included in the frame immediately preceding the frame in which the area of ​​interest was within the field of view of the frame of the captured video. [Effects of the Invention]

[0008] According to the present invention, the area of ​​interest can be confirmed even when it is outside the field of view of the captured video. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example configuration of a network system according to the first embodiment. [Figure 2]This figure shows an example of the display on the display unit of the remote support device according to the first embodiment. [Figure 3] This is a flowchart showing the process performed by the remote support device according to the first embodiment. [Figure 4] This figure shows an example of the display on the display unit of the remote support device according to the second embodiment. [Figure 5] This figure shows an example of the display on the display unit of the remote support device according to the third embodiment. [Figure 6] This is a flowchart showing the processes performed by the remote support device according to the third embodiment. [Figure 7] This diagram shows a modified example of the display on the display unit of a remote support device. [Modes for carrying out the invention]

[0010] Preferred embodiments of the present invention will be described below with reference to the attached drawings. (First embodiment) Figure 1 shows an example configuration of a network system according to the first embodiment. This network system functions as a remote support system used to remotely assist workers performing tasks.

[0011] The HMD camera unit 101 is a digital camera mounted on an HMD worn on the worker's head, and transmits video data of the work situation to the remote support device 110 via a wireless or wired network connection. In this embodiment, the HMD camera unit 101 is used as an example, but the imaging device referred to in this invention may be a network-connectable digital camera, a digital camera mounted on a smartphone or tablet, etc.

[0012] The remote support device 110 communicates with the HMD camera unit 101 and displays the video captured by the HMD camera unit 101 (hereinafter referred to as camera video) on the display unit 112. The camera video is a video. The remote support device 110 comprises a receiving unit 111, a display unit 112, a spatial recognition unit 113, a focus area designation unit 114, a focus area placement unit 115, and a display control unit 116, with each unit 111 to 116 connected via a bus 117. The remote support device 110 is composed of a computer device equipped with, for example, a CPU, memory, and storage, and the functions of each unit 111 to 116 are realized by the CPU executing a predetermined program.

[0013] The receiving unit 111 receives camera video data from the HMD camera unit 101 via the network.

[0014] The display unit 112 is a monitor composed of an LCD (liquid crystal display device) or the like, and under the control of the display control unit 116, it displays camera images received by the receiving unit 111 and images of the area of ​​interest, which will be described later.

[0015] The spatial recognition unit 113 recognizes the space from the camera image received by the receiving unit 111 and calculates three-dimensional environmental map coordinates. One method for calculating environmental map coordinates is to use Visual SLAM (Simultaneous Localization and Mapping). Visual SLAM is a technology that can simultaneously estimate the camera's own position and create environmental map coordinates in an unknown environment.

[0016] The focus area designation unit 114 designates a focus area based on the camera image. The focus area designation unit 114 receives the designation of a focus area on the camera image displayed on the display unit 112 and stores the coordinates of the focus area, which are positioned relative to the 3D environmental map coordinates calculated by the spatial recognition unit 113, as focus area information. The method for designating the focus area can utilize known techniques; for example, as described in Patent Document 2, a 3D area designation method can be used, which involves using a pointing device such as a general mouse to designate a 3D area on a 2D display screen.

[0017] Based on the area of interest specified by the area of interest specifying unit 114, the area of interest placement unit 115 maps the area of interest to the three-dimensional environmental map coordinates calculated by the spatial recognition unit 113.

[0018] The display control unit 116 performs control to display the camera video received by the reception unit 111 on the display unit 112. Further, based on the area of interest mapped by the area of interest placement unit 115, the display control unit 116 cuts out the area of interest from the camera video received by the reception unit 111, enlarges the image of the area of interest, and performs control to display it on the display unit 112. Here, when the area of interest is within the viewing angle of the frame of the camera video, the display control unit 116 cuts out the area of interest from the frame, enlarges the image of the area of interest, and displays it on the display unit 112. The display control unit 116 has a function of holding the image of the area of interest for one frame, and updates the held image of the area of interest to the latest one each time the area of interest is cut out from the frame of the camera video. Further, when the area of interest goes out of the viewing angle of the frame of the camera video, the display control unit 116 enlarges the held image of the area of interest, that is, the image of the area of interest of the frame before the frame, and displays it on the display unit 112.

[0019] Referring to FIG. 2, a display example of the display unit 112 will be described. FIGS. 2(A-1), (A-2), (A-3), and (A-4) show the time-series change of the camera video, and the frames constituting the camera video are arranged in time-series order. In this example, the HMD camera unit 101 is used to shoot so as to pan to the right direction, and the subject is moving to the left direction on the camera video.

[0020] FIGS. 2(B-1), (B-2), (B-3), and (B-4) schematically show the state in which the areas of interest 201 to 204 are mapped by the area of interest placement unit 115 for the frames of (A-1), (A-2), (A-3), and (A-4), respectively.

[0021] Figures 2(C-1), (C-2), and (C-3) show the areas of interest extracted from frames (A-1), (A-2), and (A-3), respectively, with the images of these areas enlarged and displayed. Figure 2(C-4) is the same image of the area of ​​interest as in (C-3).

[0022] Specifically, the area of ​​interest 201 in (B-1) is the area designated by the area of ​​interest designation unit 114 in the frame of (A-1). The area of ​​interest 201 is the area mapped to three-dimensional environmental map coordinates by the area of ​​interest placement unit 115 based on the area of ​​interest information stored in the area of ​​interest designation unit 114. Since the area of ​​interest 201 is within the field of view of the camera image frame, the image of the area of ​​interest 201 is enlarged and displayed on the display unit 112, as shown in (C-1). The area of ​​interest 202 in (B-2) is an area mapped to three-dimensional environmental map coordinates by the area of ​​interest placement unit 115 based on the area of ​​interest information of the area of ​​interest specified in (B-1). The three-dimensional environmental map coordinates are calculated by the spatial recognition unit 113 from the frame in (A-2). Since the area of ​​interest 202 is within the field of view of the camera image frame, the image of the area of ​​interest 202 is enlarged and displayed on the display unit 112, as shown in (C-2). The area of ​​interest 203 in (B-3) is an area mapped to three-dimensional environmental map coordinates by the area of ​​interest placement unit 115 based on the area of ​​interest information of the area of ​​interest specified in (B-1). The three-dimensional environmental map coordinates are calculated by the spatial recognition unit 113 from the frame in (A-3). Since the area of ​​interest 203 is within the field of view of the camera image frame, the image of the area of ​​interest 203 is enlarged and displayed on the display unit 112, as shown in (C-3). As shown in Figures 2(C-1) to (C-3), when the area of ​​interest is within the field of view of the camera image frame, the area of ​​interest is sequentially extracted from each frame and displayed, so the area of ​​interest is displayed as a live view image.

[0023] The area of ​​interest 204 in (B-4) is an area mapped to three-dimensional environmental map coordinates by the area of ​​interest placement unit 115 based on the area of ​​interest information of the area of ​​interest specified in (B-1). The three-dimensional environmental map coordinates are calculated by the spatial recognition unit 113 from the frame in (A-4). As the area of ​​interest 204 moves out of the field of view of the camera image frame, the image of the area of ​​interest 203 from the previous frame (see (C-3)) is enlarged and displayed as a still image on the display unit 112, as shown in (C-4). The image of the area of ​​interest 203 continues to be displayed on the display unit 112 until the area of ​​interest enters the field of view of the camera image frame.

[0024] As described above, when the area of ​​interest is within the field of view of the camera image frame, an image of the area of ​​interest extracted from that frame is displayed, allowing the supporter to view the area of ​​interest as a live view image. When the area of ​​interest moves out of the field of view of the camera image frame, an image of the area of ​​interest extracted from a frame prior to that frame is displayed, allowing the supporter to continue viewing the area of ​​interest as a still image.

[0025] Next, with reference to Figure 3, the processes performed by the remote support device 110 according to the first embodiment will be described. Figure 3 is a flowchart showing the processes performed by the remote support device 110 according to the first embodiment. In step S301, the display control unit 116 displays the camera image received by the receiving unit 111 on the display unit 112. In step S302, the area of ​​focus designation unit 114 receives the designation of the area of ​​focus on the camera image displayed on the display unit 112 in step S301, and stores the coordinates of the area of ​​focus as area of ​​focus information.

[0026] In step S303, the spatial recognition unit 113 calculates 3D environmental map coordinates from the current frame of the camera image. In step S304, the area of ​​interest placement unit 115 maps the area of ​​interest to the three-dimensional environmental map coordinates calculated in step S303, based on the area of ​​interest information stored in step S302.

[0027] In step S305, the display control unit 116 determines whether the area of ​​interest, mapped to the 3D environmental map coordinates in step S304, is within the field of view of the current frame of the camera image. If the area of ​​interest is within the field of view, the process proceeds to step S306; otherwise, the process proceeds to step S307. The criteria for determining whether the area of ​​interest is within the field of view of the current frame can be predetermined. For example, the system may determine that the area of ​​interest is outside the field of view if it is even slightly outside the field of view of the current frame, or it may determine that the area of ​​interest is outside the field of view if a predetermined percentage of the area of ​​interest (for example, half of the area of ​​interest) is outside the field of view of the current frame. In step S306, the display control unit 116 extracts the region of interest mapped in step S304 from the current frame of the camera image and generates an image of the region of interest. The display control unit 116 also stores this image of the region of interest for one frame. In step S307, the display control unit 116 retrieves the image of the area of ​​interest that has been held.

[0028] In step S308, the display unit 112 displays an enlarged image of the area of ​​interest generated in step S306 or the image of the area of ​​interest extracted in step S307 on the display unit 112. In step S309, the display control unit 116 updates the camera image to the next frame. In step S310, the remote support device 110 determines whether or not there is a termination instruction from the supporter. If there is no termination instruction, it proceeds to step S303. If there is a termination instruction, this flowchart is terminated.

[0029] As described above, in the first embodiment, even when the area of ​​interest moves out of the field of view of the camera image frame, the image of the area of ​​interest included in the frame prior to that frame is displayed, allowing the area of ​​interest to be confirmed. This makes it possible to continue providing support to the worker.

[0030] Furthermore, the components described as being provided by the remote support device 110 may also be provided on the HMD camera unit 101 (or the HMD on which it is mounted). For example, the spatial recognition unit 113 and the focus area placement unit 115 may be provided on the HMD camera unit 101 (or the HMD on which it is mounted). Furthermore, although the display control unit 116 of the remote support device 110 is described as performing the process of extracting the area of ​​interest from the camera video frame and generating an image of the area of ​​interest, this process may also be performed by the HMD camera unit 101 (or the HMD on which it is mounted). In the flowchart of Figure 3, for example, the HMD camera unit 101 (or the HMD on which it is mounted) may perform the processes in steps S303 to S307. In this case, the focus area information based on the specification in step S302 should be sent from the remote support device 110 to the HMD. In this embodiment, an example was described in which the remote support device 110 functions as an information processing device as defined in the present invention. However, the imaging device as defined in the present invention may also function as an information processing device as defined in the present invention, controlling the display on the supporter's monitor.

[0031] (Second embodiment) In the second embodiment, a configuration having a function to perform geometric transformations on the image of the region of interest will be described. The configuration of the remote support device 110 is the same as in the first embodiment, and the same reference numerals will be used below, their descriptions will be omitted, and the differences from the first embodiment will be the focus of the description. In this embodiment, a function to perform geometric transformations on the image is added to the display control unit 116, and it functions as an image processing means as defined in the present invention. For example, an affine transformation is used as the geometric transformation of the image.

[0032] Figures 4(A-1) and (A-2) show the time-series changes in the camera image, with the frames constituting the camera image arranged in chronological order. In this example, the worker wearing the HMD moves, and the field of view of the HMD camera unit 101 changes. In this case, simply cutting out the area of ​​interest from the camera image and enlarging the image of the area of ​​interest to display on the display unit 112 may make the area of ​​interest difficult to see. Therefore, in the second embodiment, the display control unit 116 extracts the area of ​​interest from the camera image received by the receiving unit 111 based on the area of ​​interest arranged by the area of ​​interest arrangement unit 115, performs an affine transformation to maintain the same field of view based on the area of ​​interest information stored in the area of ​​interest designation unit 114, and then performs control to enlarge the image of the area of ​​interest and display it on the display unit 112.

[0033] Referring to Figure 4, an example of the display on the display unit 112 will be explained. Figures 4(A-1) and (A-2) show the time-series changes in the camera image, with the frames constituting the camera image arranged in chronological order. In this example, the worker wearing the HMD moves, and the field of view of the HMD camera unit 101 changes.

[0034] Figures 4(B-1) and (B-2) schematically show the state in which the areas of interest 401 and 402 are mapped to the frames (A-1) and (A-2), respectively, by the area of ​​interest placement unit 115.

[0035] Figure 4(C-1) shows the result of extracting the region of interest from the frame (A-1) and displaying an enlarged image of the region of interest. Figure 2(C-2) shows the result of extracting the region of interest from the frame (A-2), performing an affine transform, and then displaying an enlarged image of the region of interest. Specifically, the area of ​​interest 401 in (B-1) is the area designated by the area of ​​interest designation unit 114 in the frame of (A-1). The area of ​​interest 401 is the area mapped to three-dimensional environmental map coordinates by the area of ​​interest placement unit 115 based on the area of ​​interest information stored in the area of ​​interest designation unit 114. Since the area of ​​interest 401 is within the field of view of the camera image frame, the image of the area of ​​interest 401 is enlarged and displayed on the display unit 112 as a live view image, as shown in (C-1). The area of ​​interest 402 in (B-2) is an area mapped to three-dimensional environmental map coordinates by the area of ​​interest placement unit 115 based on the area of ​​interest information of the area of ​​interest specified in (B-1). The three-dimensional environmental map coordinates are calculated by the spatial recognition unit 113 from the camera image frame in (A-2). In this case, as shown in (D-2), the area of ​​interest 402 is cut out, and an affine transformation is performed to maintain the same field of view based on the area of ​​interest information stored in the area of ​​interest designation unit 114 (area of ​​interest 412 after affine transformation). Since the area of ​​interest 402 is within the field of view of the camera image frame, as shown in (C-2), the image of the area of ​​interest 412 after affine transformation is enlarged and displayed on the display unit 112 as a live view image.

[0036] Next, the processing performed by the remote support device 110 according to the second embodiment will be described. The processing performed by the remote support device 110 is basically the same as the flowchart in Figure 3, but in step S306, the affine transformation process is added. In step S306 of the second embodiment, the display control unit 116 extracts the area of ​​interest mapped in step S304 from the current frame of the camera image, performs an affine transformation to maintain the same field of view based on the area of ​​interest information of the area of ​​interest specified by the area of ​​interest designation unit 114, and generates an image of the area of ​​interest. The display control unit 116 also stores this image of the area of ​​interest for one frame.

[0037] As described above, in the second embodiment, similar to the first embodiment, even when the area of ​​interest moves out of the field of view of the camera image frame, the image of the area of ​​interest included in the frame prior to that frame is displayed, allowing the area of ​​interest to be confirmed. This makes it possible to continue providing support to the worker. Furthermore, in the second embodiment, even when the field of view of the HMD camera unit 101 changes due to the worker wearing the HMD moving, the image of the area of ​​interest can be maintained at the same field of view, providing an easy-to-view image.

[0038] (Third embodiment) In the third embodiment, a configuration having the function of specifying multiple areas of interest and selecting and displaying a desired area of ​​interest will be described. The configuration of the remote support device 110 is the same as in the first embodiment, and the same reference numerals will be used below, their descriptions will be omitted, and the differences from the first embodiment will be the main focus of the description. In this embodiment, the area of ​​interest specification unit 114 has been added with the function of specifying multiple areas of interest and storing information for each area of ​​interest. Furthermore, the display control unit 116 has been added with the function of displaying images of multiple areas of interest in a wipe display and displaying the selected image from the wipe display as the main image in a larger size.

[0039] In this embodiment, the focus area designation unit 114 receives designations for multiple focus areas on the camera image displayed on the display unit 112 and stores information for each focus area. The display control unit 116 extracts the area of ​​interest from the camera video received by the receiving unit 111 based on the area of ​​interest mapped by the area of ​​interest placement unit 115, resizes the image of the area of ​​interest, and displays it as a wipe on the display unit 112. In this case, as in the first embodiment, when the area of ​​interest is within the field of view of the camera video frame, the display control unit 116 extracts the area of ​​interest from that frame, resizes the image of the area of ​​interest, and displays it as a wipe on the display unit 112. The display control unit 116 has a function to hold images of one frame's worth of area of ​​interest for multiple areas of interest, and updates the held image of the area of ​​interest to the latest one each time an area of ​​interest is extracted from the camera video frame. Furthermore, when the area of ​​interest is outside the field of view of the camera video frame, the display control unit 116 resizes the held image of the area of ​​interest, i.e., the image of the area of ​​interest from a frame prior to that frame, and displays it as a wipe on the display unit 112. The display control unit 116 then controls the display to enlarge the selected wipe display as the main image.

[0040] Referring to Figure 5, an example of the display on the display unit 112 will be explained. Figure 5(A) shows the camera image frame. Figure 5(B) schematically shows the state in which multiple areas of interest 501 to 503 are mapped to the frame of interest in (A) by the area of ​​interest arrangement unit 115. Figure 5(C) shows the state in which multiple areas of interest have been extracted from the frame of (A), and images 511 to 513 of the areas of interest are displayed in a wipe. It also shows the state in which an image selected from the wipe display, in this case image 511, is displayed as the main video 510.

[0041] Next, with reference to Figure 6, the processes performed by the remote support device 110 according to the third embodiment will be described. Figure 6 is a flowchart of the processes performed by the remote support device 110 according to the third embodiment. Steps S601, S603, S609, and S610 are the same processes as steps S301, S303, S309, and S310 of the first embodiment, and their explanation will be omitted. In step S602, the area of ​​focus designation unit 114 receives designations for multiple areas of focus on the camera image displayed on the display unit 112 in step S601, and stores the coordinates of each area of ​​focus as area of ​​focus information. In step S604, the area of ​​interest placement unit 115 maps multiple areas of interest to the three-dimensional environmental map coordinates calculated in step S603, based on the multiple areas of interest information stored in step S602.

[0042] In step S605, the display control unit 116 determines whether the target area of ​​interest, which was mapped to the three-dimensional environmental map coordinates in step S604, is within the field of view of the current frame of the camera image. If the area of ​​interest is within the field of view, the process proceeds to step S606; if the area of ​​interest is outside the field of view, the process proceeds to step S607. The criteria for determining whether the area of ​​interest is within the field of view of the current frame are as described in the first embodiment. In step S611, which will be described later, the multiple areas of interest are processed sequentially. In step S606, the display control unit 116 extracts the region of interest mapped in step S604 from the current frame of the camera image and generates an image of the region of interest. The display control unit 116 also stores this image of the region of interest for one frame. In step S607, the display control unit 116 retrieves the image of the area of ​​interest that has been held. In step S611, the display control unit 116 determines whether all of the multiple areas of interest information specified in step S602 have been processed. If all have been processed, the system proceeds to step S612; otherwise, it proceeds to step S605.

[0043] In step S612, the display control unit 116 resizes the image of the area of ​​interest generated in step S306 or the image of the area of ​​interest extracted in step S307 and displays it as a wipe on the display unit 112. In step S613, the display control unit 116 displays the selected wipe display from those displayed in step S612 as the main image (main display).

[0044] As described above, in the third embodiment, similar to the first embodiment, even when the area of ​​interest moves out of the field of view of the camera image frame, the image of the area of ​​interest included in the frame prior to that frame is displayed, allowing the area of ​​interest to be confirmed. This makes it possible to continue providing support to the worker. Furthermore, in the third embodiment, multiple areas of focus can be specified, and a desired area of ​​focus can be selected and displayed, thereby improving the efficiency of remote support.

[0045] Although various embodiments have been described above, the following functions may be added. For example, by sending information indicating the area of ​​focus specified and mapped by the support staff to the HMD, and displaying the area of ​​focus with a frame or other indicator on the HMD panel, the worker can also recognize the area of ​​focus specified by the support staff.

[0046] Furthermore, as shown in Figure 7, the display unit 112 clearly indicates whether the image of the area of ​​interest displayed is an image of the area of ​​interest when it is within the field of view or an image of the area of ​​interest when it is outside the field of view. Figures 7(C-1) and (C-2) show the image of the area of ​​interest in enlarged view. In Figure 7(C-1), mark 701 is displayed to indicate that the image is of the area of ​​interest when it is within the field of view (live view image). In Figure 7(C-2), mark 702 is displayed to indicate that the image is of the area of ​​interest when it is outside the field of view (still image).

[0047] Although the present invention has been described above along with its embodiments, these embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features. (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0048] This embodiment includes the following configurations and methods. [Configuration 1] An information processing device that displays images captured by an imaging device on a display unit, The system includes a display control means for controlling the display unit to display an image of a region of interest included in the captured video. The display control means is When the area of ​​interest is within the field of view of the frame of the captured video, the image of the area of ​​interest included in that frame is displayed. An information processing device characterized in that, when the area of ​​interest falls outside the field of view of the frame of the captured video, it displays an image of the area of ​​interest included in a frame prior to that frame. [Configuration 2] A spatial recognition means for recognizing space from the aforementioned captured video footage, The information processing apparatus according to configuration 1, further comprising: a focus area placement means for mapping the focus area to the space recognized by the spatial recognition means. [Configuration 3] The information processing apparatus according to configuration 1 or 2, characterized by comprising a means for specifying a region of focus based on the captured video footage. [Structure 4] An information processing apparatus according to any one of configurations 1 to 3, characterized by comprising image processing means for performing geometric transformations on an image of the region of interest. [Composition 5] There are multiple areas of interest, The information processing apparatus according to any one of configurations 1 to 4, characterized in that the display control means displays a plurality of images of the area of ​​interest in a wipe display, and displays a selected image from among the wipe displays in a larger size. [Composition 6] The information processing apparatus according to any one of configurations 1 to 5, characterized in that the display control means clearly indicates whether the image of the area of ​​interest is an image of the area of ​​interest when it is within the field of view or an image of the area of ​​interest when it is outside the field of view. [Composition 7] The information processing apparatus according to any one of configurations 1 to 6, characterized in that the imaging device is a digital camera mounted on a head-mounted display. [method] An information processing method performed by an information processing device that displays images captured by an imaging device on a display unit, The system includes a display control step that controls the display unit to display an image of a region of interest included in the captured video. In the aforementioned display control step, When the area of ​​interest is within the field of view of the frame of the captured video, the image of the area of ​​interest included in that frame is displayed. An information processing method characterized by displaying an image of the area of ​​interest included in a frame prior to the frame when the area of ​​interest falls outside the field of view of the frame of the captured video. [program] A program for displaying images captured by an imaging device on a display unit, A computer is used as a display control means to control the display unit to display an image of the region of interest included in the captured video. The display control means is When the area of ​​interest is within the field of view of the frame of the captured video, the image of the area of ​​interest included in that frame is displayed. A program characterized in that, when the area of ​​interest falls outside the field of view of the frame of the captured video, it displays an image of the area of ​​interest included in a frame prior to that frame. [Structure 8] A network system that displays images captured by an imaging device on a display unit, A spatial recognition means for recognizing space from the aforementioned captured video footage, A focus region arrangement means that maps the focus region included in the captured video image to the space recognized by the spatial recognition means, The system includes a display control means for controlling the display unit to display an image of the area of ​​interest, The display control means is When the area of ​​interest is within the field of view of the frame of the captured video, the image of the area of ​​interest included in that frame is displayed. A network system characterized in that, when the area of ​​interest falls outside the field of view of the frame of the captured video, the system displays an image of the area of ​​interest contained in a frame prior to that frame. [Explanation of symbols]

[0049] 101: HMD camera unit, 110: Remote support device, 111: Receiving unit, 112: Display unit, 113: Spatial recognition unit, 114: Focus area designation unit, 115: Focus area placement unit, 116: Display control unit, 117: Bus

Claims

1. An information processing device that displays captured video images captured by an imaging device on a display unit, A display control means that controls the display unit to display an image of the region of interest included in the captured video, The system includes an image processing means that performs a geometric transformation on the image of the region of interest so as to maintain the same field of view when the field of view of the imaging device changes, The display control means is If the area of ​​interest is within the field of view of the frame of the captured video, a live view image of the area of ​​interest included in that frame is displayed. An information processing device characterized in that, if the area of ​​interest is outside the field of view of the frame of the captured video, it displays a still image obtained by enlarging the image of the area of ​​interest included in the frame immediately preceding the frame in which the area of ​​interest is included within the field of view of the frame of the captured video.

2. A spatial recognition means for recognizing space from the aforementioned captured video footage, The information processing apparatus according to claim 1, further comprising: a focus region arrangement means for mapping the focus region to the space recognized by the spatial recognition means.

3. The information processing apparatus according to claim 1 or 2, further comprising means for specifying a region of interest based on the captured video footage.

4. There are multiple areas of interest as described above. The information processing apparatus according to claim 1 or 2, characterized in that the display control means displays a plurality of images of the area of ​​interest in a wipe-out manner, and displays a selected image from among the wipe-out images in a larger size.

5. The information processing apparatus according to claim 1 or 2, characterized in that the display control means indicates whether the image of the area of ​​interest is an image of the area of ​​interest within the field of view or an image of the area outside the field of view.

6. The information processing apparatus according to claim 1 or 2, characterized in that the imaging device is a digital camera mounted on a head-mounted display.

7. An information processing method for displaying captured images taken by an imaging device on a display unit, A display control step that controls the display unit to display an image of the region of interest included in the captured video, The image processing step includes performing a geometric transformation on the image of the region of interest so as to maintain the same field of view when the field of view of the imaging device changes. In the aforementioned display control step, If the area of ​​interest is within the field of view of the frame of the captured video, a live view image of the area of ​​interest included in that frame is displayed. An information processing method characterized in that, if the area of ​​interest is outside the field of view of the frame of the captured video, the method displays a still image obtained by enlarging the image of the area of ​​interest included in the frame immediately preceding the frame in which the area of ​​interest is included within the field of view of the frame of the captured video.

8. A program for causing a computer to function as an information processing device that displays captured images taken by an imaging device on a display unit, A display control process that controls the display unit to display an image of the region of interest included in the captured video, When the field of view of the imaging device changes, the computer is instructed to perform image processing that performs a geometric transformation on the image of the area of ​​interest so as to maintain the same field of view of the image of the area of ​​interest. In the aforementioned display control process, If the area of ​​interest is within the field of view of the frame of the captured video, a live view image of the area of ​​interest included in that frame is displayed. A program characterized in that, if the area of ​​interest is outside the field of view of the frame of the captured video, it displays a still image obtained by enlarging the image of the area of ​​interest included in the frame immediately preceding the frame in which the area of ​​interest is included within the field of view of the frame of the captured video.

Images