Display processing device, display processing method, and program

Abstract

[Problem] To provide a display processing device capable of displaying assistance information so that, at the time of photogrammetric photographing, an image can be photographed under an appropriate photographing condition while understanding the surrounding situation. [Solution] This display processing device is communicably connected to an imaging device and comprises: a detection unit that detects an object present in the line-of-sight direction of a user; a generation unit that generates space region information corresponding to the view angle of the imaging device and the region of the object; and a display unit that displays the space region information.

Description

Display processing device, display processing method, and program

[0001] The present invention relates to a technique for modeling three-dimensional space, and more particularly to a technique for displaying assist information when photographing an object from various angles.

[0002] There are known techniques for 3D modeling of an object using images captured from various angles. Specifically, there is a technique called photogrammetry, which creates a three-dimensional model by analyzing and integrating a set of images captured from various positions and orientations surrounding the object.

[0003] When checking the camera's angle of view, it is usually necessary to look through the viewfinder, but doing so makes it difficult to grasp the situation at your feet and around you. When taking photos for photogrammetry, the user takes photos while walking around the subject, so trying to check the angle of view while also checking the surrounding situation makes the shooting process cumbersome.

[0004] Patent Document 1 discloses a configuration in which the area that the camera is about to capture is displayed on a head-mounted display (HMD), making it possible to visually grasp the capture area even when there is a large angular difference between the optical axis and the user's line of sight.

[0005] JP 2016-201686 A

[0006] When generating a high-quality 3D model, it is desirable that the image set input into photogrammetry be images with little motion blur or depth blur, and taken at low sensitivity with little noise. In other words, the camera's shooting conditions require an appropriate aperture setting so that the exposure time is not too long or the sensitivity is not too high due to the aperture being closed too much.

[0007] However, the information displayed by the configuration of Patent Document 1 does not take into consideration the depth information in which the object is located, so it is not possible to correctly recognize the area in which the subject is located and then set appropriate shooting conditions.

[0008] An object of the present invention is to provide a display processing device that can display assist information so that an image can be captured under appropriate shooting conditions while understanding the surrounding situation when capturing images for photogrammetry.

[0009] A display processing device according to one aspect of the present invention is a display processing device that is communicatively connected to an imaging device, and is characterized by having a detection unit that detects an object present in the user's line of sight, a generation unit that generates spatial region information according to the angle of view of the imaging device and the region of the object, and a display unit that displays the spatial region information.

[0010] According to the present invention, it is possible to provide a display processing device that can display assist information so that an image can be captured under appropriate shooting conditions while understanding the surrounding situation when capturing images for photogrammetry.

[0011] FIG. 1 is a block diagram of a display processing system according to an embodiment of the present invention. FIG. 2 is a diagram for explaining a coordinate system. FIG. 3 is a flowchart for explaining the operation of the display processing system. FIG. 4 is a diagram for explaining image data acquired by a display device. FIG. 5 is a diagram for explaining assist information displayed on a display device. FIG. 6 is a diagram for explaining image data acquired by an imaging device. FIG. 7 is a diagram for explaining a spatial region determined based on the angle of view of an imaging device and the region of an object. FIG. 8 is a diagram for explaining photography for photogrammetry.

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same components, and redundant explanations will be omitted.

[0013] 1 is a block diagram of a display processing system 100 according to an embodiment of the present invention. The display processing system 100 is configured with a display device (display processing device) 110, an imaging device 120, and a network 130. The display device 110 and the imaging device 120 are connected via the network 130 so as to be able to communicate data with each other.

[0014] The display device 110 is a device that can be worn on the head, such as an HMD or a glasses-type display. In this embodiment, the display device 110 has a stereo camera configuration and can acquire an image for the left eye and an image for the right eye. The display device 110 includes a control unit 111, a display unit 112, a ROM 113, a RAM 114, an imaging unit 115, a gaze direction acquisition unit (acquisition unit) 116, an object detection unit (detection unit) 117, an image processing unit (generation unit) 118, and a position and orientation estimation unit 119.

[0015] The control unit 111 is, for example, a CPU, and controls the operation of each block of the display device 110 by reading out an operation program for each block of the display device 110 from the ROM 113, expanding it into the RAM 114, and executing it.

[0016] The display unit 112 is a non-transmissive display and is divided into a right eye and a left eye. An eyepiece is disposed between the user's eyes and the display unit 112. By displaying an image captured by the imaging unit 115 on the display unit 112, the user can observe the space in front of the user's eyes. By displaying images or information such as CG (Computer Graphics) for photography assistance on the image displayed on the display unit 112, the displayed image can be superimposed on the space the user is viewing through the display unit 112. Note that a transmissive display may be used as the display unit 112.

[0017] The ROM 113 is a rewritable non-volatile memory, and stores operation programs for each block of the display device 110, as well as parameters required for the operation of each block, captured image data, and the like.

[0018] The RAM 114 is a rewritable volatile memory, and is used as a temporary storage area for data output during the operation of each block included in the display device 110 .

[0019] The imaging unit 115 includes an imaging element such as a CCD or CMOS sensor, an optical system, and an A / D conversion circuit, and captures an image of a subject in front of the user wearing the device as digital image data and outputs it to the ROM 113.

[0020] The gaze direction acquisition unit 116 is, for example, a gaze detection module based on the corneal reflex method, and acquires the direction in which the user is gazing (user's gaze direction).

[0021] The object detection unit 117 detects an object that exists in the direction the user is gazing (gaze destination) and sets a gaze point based on information about the direction the user is gazing in that direction acquired by the gaze direction acquisition unit 116. The object detection unit 117 also outputs supplementary information about the detected object (subject feature amounts such as color and brightness histograms) to the RAM 114 to communicate this information to the imaging device 120.

[0022] The image processing unit 118 generates image data by applying various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the ROM 113. The image processing unit 118 also calculates, by coordinate transformation, in which area of ​​the image data obtained by the imaging unit 115 the spatial area captured by the imaging device 120 exists, and generates a CG image having a truncated quadrangular pyramid shape by connecting the boundaries of the calculated areas with line segments, and superimposes this on the image data.

[0023] Here, the coordinate system of this embodiment will be described with reference to FIG. 2 . FIG. 2 is a diagram for explaining the coordinate system. Reference numeral 201 denotes an object for which a three-dimensional model is to be created. Reference numeral 202 denotes a gaze point at which a user gazes via the display device 110. Reference numeral 203 denotes the origin of the world coordinate system, which is composed of the Xw, Yw, and Zw axes. Reference numeral 204 denotes the origin of the coordinate system of the image capture device 120, which is composed of the Xc, Yc, and Zc axes. In the following description, this will be referred to as the camera coordinate system. Reference numeral 205 denotes the imaging plane of the image capture device 120. Reference numeral 206 denotes the origin of the coordinate system for the right-eye camera of the display device 110, which is composed of the Xh, Yh, and Zh axes. In the following description, this will be referred to as the display coordinate system. Reference numeral 207 denotes the imaging plane for the right-eye camera of the display device 110. Furthermore, the dotted lines represent information for the left-eye camera, with 209 being the origin of the coordinate system and 210 being the imaging plane for the left-eye camera. In the following explanation, processing will be explained using image data for the right-eye camera. It is assumed that the imaging planes for the right-eye camera and the left-eye camera are positioned so that they coincide when shifted in the Xh direction. Object 208 is an object that exists in the background area relative to object 201, and is not a target for creating a 3D model.

[0024] 2, it is assumed that the user is not looking through the viewfinder of the imaging device 120 and is therefore unable to grasp the angle of view of the imaging device 120. In the following description, it is assumed that the angle of view acquired by the display device 110 is wider than the angle of view acquired by the imaging device 120.

[0025] The position and orientation estimation unit 119 calculates (estimates) the position and orientation in the world coordinate system of the display device 110. Note that it is assumed that the internal camera parameters of the display device 110 are known.

[0026] Here, the camera internal parameter K is defined by the focal length of the camera and the optical center coordinates, and is specifically expressed by the following matrix: It is also assumed that there is no lens distortion.

[0027]

[0028] Here, fx and fy are the focal lengths of the camera, and cx and cy are the optical center coordinates of the camera.

[0029] The imaging device 120 is a mirrorless camera, etc. The imaging device 120 includes a control unit 121, a ROM 122, a RAM 123, an imaging unit 124, an image processing unit 125, a position and orientation estimation unit 126, and an operation unit 127.

[0030] The control unit 121, ROM 122, and RAM 123 are similar to the control unit 111, ROM 113, and RAM 114, respectively, and therefore description thereof will be omitted.

[0031] The imaging unit 124 includes an optical system, an image sensor, and an A / D conversion circuit. The optical system includes, for example, a variable magnification lens that changes the focal length and a focus lens that adjusts the focus. The optical system also includes an aperture, which adjusts the aperture diameter of the optical system to adjust the amount of light during shooting. The optical image formed on the image sensor by the optical system is photoelectrically converted, the resulting analog image signal is subjected to A / D conversion processing, and the resulting digital image data is output to the ROM 122 for storage.

[0032] The image processing unit 125 applies various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the ROM 122 and outputs the image data to the ROM 122 .

[0033] The position and orientation estimation unit 126 calculates (estimates) the position and orientation of the image capture device 120 in the world coordinate system. Note that the internal camera parameters of the image capture device 120 are known.

[0034] The operation unit 127 is an aperture operation member or the like, and can change the shooting conditions of the imaging device 120 .

[0035] The following describes the assist display operation on the display device 110 when an image for creating a three-dimensional model of the object 201 is captured using the imaging device 120 in the situation shown in Fig. 2. Fig. 3 is a flowchart for explaining the operation of the display processing system 100.

[0036] Fig. 3A is a flowchart for explaining the operation of the display device 110. Each process shown in Fig. 3A is executed by the control unit 111 after a program recorded in the ROM 113 is loaded into the RAM 114.

[0037] In step S301, the gaze direction acquisition unit 116 acquires the direction in which the user is gazing. Based on information about the direction in which the user is gazing, the object detection unit 117 detects an object present at the gaze point in the image data acquired by the imaging unit 115 for the right eye, and sets a gaze point (xh, yh). The object detection unit 117 also outputs supplementary information about the detected object to the RAM 114 to convey it to the imaging device 120. FIG. 4 is a diagram illustrating image data 401 acquired by the imaging unit 115. As shown in FIG. 4, the display device 110 captures the object 201 at an angle of view that is viewed from diagonally above.

[0038] In step S302, the position and orientation estimation unit 119 estimates the position and orientation of the display device 110 by referring to the current image data and image data acquired before the current one. Specifically, the position and orientation are estimated from the image data group using a technique such as SfM (Structure from Motion). Note that the position and orientation may be estimated using an acceleration sensor or an angular velocity sensor. Alternatively, they may be estimated using a combination of these. The position and orientation estimated here is a camera extrinsic parameter T, which is composed of a rotation component and a translation component, and is expressed by the following matrix:

[0039]

[0040] Here, r11, r12, r13, r21, r22, r23, r31, r32, and r33 are rotation components of the camera, and t1, t2, and t3 are translation components of the camera.

[0041] The coordinates (u, v) in the image data acquired by the imaging unit 115 and the three-dimensional coordinates (X, Y, Z) in the world coordinate system can be converted using the following equation (1) using the camera internal parameters K and the camera external parameters T.

[0042]

[0043] Here, s is a coefficient representing the uncertainty of the scale.

[0044] In step S303, the position and orientation estimation unit 119 converts the gaze point (xh, yh) set in step S301 and the distance zh from the display device 110 to the gaze point into three-dimensional coordinates (Xh, Yh, Zh) in the world coordinate system. Note that the distance zh to the gaze point may be measured by stereo ranging using an image for the right eye and an image for the left eye.

[0045] From equation (1), the calculation formula for conversion is expressed by the following equation (2).

[0046]

[0047] In step S304, the three-dimensional coordinates (Xh, Yh, Zh) of the gaze point and additional information about the object are transmitted to the image capture device 120 as object information.

[0048] In step S305, it is determined whether or not the area information generated in step S316, which will be described later, has been received by the imaging device 120. If it is determined that the area information has been received, the processing of step S306 is executed, and if it is determined that the area information has not been received, the processing of this step is executed again.

[0049] In step S306, the image processing unit 118 generates information for generating spatial region information corresponding to region information, which is information regarding the depth direction in which the object exists. In this embodiment, the region information is three-dimensional coordinates (Xci, Yci, Zci) (i: 1 to 8) of eight points determined based on the region of the object (depth direction information) determined by the angle of view and depth of field of the image capture device 120, as described below. The image processing unit 118 converts the three-dimensional coordinates (Xci, Yci, Zci) of the eight points into coordinates of image data (vertex position information) on the display device 110 according to equation (1), and acquires them as information for generating spatial region information. Note that, since it is assumed that the position and orientation of the display device 110 change over time, the camera extrinsic parameters are updated each time.

[0050] In step S307, the image processing unit 118 generates spatial domain information using the information acquired in step S306. Here, the spatial domain information is information for understanding the content that the image capture device 120 is attempting to capture (the three-dimensional spatial domain captured by the image capture device 120). Specifically, the image processing unit 118 first generates image data by applying various image processing processes, such as white balance adjustment, color interpolation, and gamma processing, to the image data stored in the ROM 122. Next, the image processing unit 118 generates, as spatial domain information, a CG image in the shape of a truncated pyramid, connecting the coordinates acquired in step S306 with line segments. The spatial domain information is displayed by the display unit 112, superimposed on the image data. FIG. 5 is a diagram for explaining the spatial domain information as assist information displayed on the display device 110, showing the state in which the spatial domain information is superimposed on the image data. Only the object 201, for which a three-dimensional model is to be created, is surrounded by the CG image in the shape of a truncated pyramid, indicating that the image capture device 120 is facing in the desired direction and capturing the subject with an appropriate aperture setting. In this way, the user can understand what the image capture device 120 is trying to capture without looking through the viewfinder of the image capture device 120, allowing the user to capture images for photogrammetry while understanding the surrounding situation. It also makes it possible to simultaneously understand the depth of field in the depth direction, and to easily capture images of the subject without depth blur, which enables the generation of high-quality 3D models. The CG for the left eye can be generated by shifting the CG for the right eye by the amount of deviation of the imaging plane, and the shifted CG is superimposed on the image data for the left eye.

[0051] In step S308, it is determined whether or not shooting completion information has been received, which is transmitted in step S319 described later by the imaging device 120. If it is determined that shooting completion information has been received, this flow is ended, and if it is determined that shooting completion information has not been received, the processing of step S305 is executed.

[0052] Fig. 3B is a flowchart for explaining the operation of the imaging device 120. Each process shown in Fig. 3B is executed by the control unit 121 after a program recorded in the ROM 122 is loaded into the RAM 123.

[0053] In step S311, it is determined whether the object information transmitted in step S304 has been received by the display device 110. If it is determined that the object information has been received, the process of step S312 is executed, and if it is determined that the object information has not been received, the process of this step is executed again.

[0054] In step S312, the position and orientation estimation unit 126 estimates the position and orientation of the image capture device 120 using the same method as in step S302.

[0055] In step S313, the position and orientation estimation unit 126 converts the three-dimensional coordinates (Xh, Yh, Zh) of the point of gaze received in step S311 using equation (1) to convert them into coordinates in the image data of the image capture device 120. Fig. 6 is a diagram for explaining the image data acquired by the image capture device 120, showing the image data captured by the image capture device 120. The entire body of the object 201 is captured from the front, and an object 208 exists in the background.

[0056] In step S314, the imaging unit 124 drives the focus lens to focus on the object 201 at the gaze point. At this time, in order to improve the accuracy of focusing on the object, the object may be re-detected by referring to the additional information of the object received in step S311.

[0057] In step S315, the image capture unit 124 calculates the depth of field when the image capture device 120 focuses on the object 201. The depth of field is expressed as Z-Df to Z+Db using the front depth of field Df, the rear depth of field Db, and the focused object distance Z. Df is expressed as (r·Av·Z^2) / (f^2+r·Av·Z), and Db is expressed as (r·Av·Z^2) / (f^2-r·Av·Z). r is the permissible circle of confusion diameter, Av is the aperture value, and f is the focal length. The permissible circle of confusion diameter r is assumed to be twice the pixel pitch. FIG. 7 is a diagram illustrating a spatial region determined based on the angle of view of the image capture device 120 and the region of the object determined by the depth of field when the image capture device 120 focuses on the object 201. The spatial region cut out by the vertical angle of view Yφ and the front depth of field Df and rear depth of field Db at the focused subject distance Z is represented by diagonal lines. To generate a high-quality 3D model through photogrammetry, it is necessary to input an image in which the subject does not extend beyond the angle of view and is within the depth of field without depth blur. By communicating the spatial region represented by diagonal lines in Figure 7 to the user, it is possible to assist with photogrammetric photography. In Figure 7, the object 201 to be used for creating a 3D model is located within the diagonal line area, satisfying appropriate shooting conditions. Furthermore, an object 208, which is not the subject for creating a 3D model, is located outside the spatial region represented by diagonal lines in Figure 7.

[0058] In this embodiment, the allowable circle of confusion diameter r is set to twice the pixel pitch, but the present invention is not limited to this, and the diameter may be set to be coarser or finer depending on, for example, the number of polygons in the three-dimensional data to be created.

[0059] Furthermore, although the depth of field is set to Z-Df to Z+Db, the present invention is not limited to this, and the range may be set to a wider range, such as Z-2·Df to Z+2·Db, taking into account errors in the focal length f and aperture value Av that can be obtained from the camera.

[0060] Furthermore, in this embodiment, the object region is determined according to the depth of field when the image capture device 120 focuses on the object 201, but the present invention is not limited to this. For example, the object region may be determined according to the distance range in which the object 201 exists. In this case, depth direction information Z+ΔZ of the object can be calculated from the defocus value calculated for each pixel position in a phase difference image obtained from an image capture sensor in which all pixels are phase difference pixels. The depth direction information Z+ΔZ can be derived from the lens formula (1 / Z+1 / Z'=1 / f and 1 / (Z+ΔZ)+1 / (Z'+def)=1 / f), where f is the focal length of the lens. This allows the depth of the subject to be grasped with high precision, making it possible to set the aperture value more precisely.

[0061] In step S316, the position and orientation estimation unit 126 converts the coordinate positions of the eight points necessary to represent the spatial region represented by the diagonal lines in Fig. 7 into three-dimensional coordinates in the world coordinate system using equation (2). Here, the positions of the eight points are the vertex positions of the trapezoid represented by the diagonal lines in Fig. 7, and when the Xc direction is also taken into account, they are the vertex coordinate positions of a total of eight points of a quadrangular pyramid. The converted coordinates are transmitted to the display device 110 as region information.

[0062] In step S317, it is determined whether the aperture value has been changed. If it is determined that the aperture value has been changed, the process of step S315 is executed; if it is determined that the aperture value has not been changed, the process of step S318 is executed. With a constant exposure compensation setting, narrowing the aperture value results in a deeper depth of field, which in turn results in a longer exposure time or a higher ISO sensitivity. Longer exposure times increase the effects of camera shake and subject motion, while higher ISO sensitivity increases noise. Therefore, by adjusting the aperture on the operation unit 127 while viewing the CG image of a truncated pyramid rendered on the display unit 112, the user can find an aperture value that does not result in an unnecessarily deep depth of field. Alternatively, the imaging device 120 may automatically set an appropriate aperture value, and the user may simply confirm the result. In either case, photography can be performed with an appropriate aperture value setting for photogrammetry.

[0063] In step S318, exposure processing is performed by the imaging unit 124, and image data suitable for photogrammetry processed by the image processing unit 125 is recorded in the ROM 122.

[0064] In step S319, command information (photographing completion information) is transmitted to the display device 110 to inform it that the photographing process in step S318 has been completed.

[0065] When the image capturing is completed in step S318, the user moves to the next image capturing position and continues capturing images while changing the position and angle so as to surround the object 201, as shown in Fig. 8. Note that Fig. 8 is a bird's-eye view seen from above, and 801 to 810 indicate the position and orientation of the image capturing device 120.

[0066] It is preferable that the imaging device 120 be configured to automatically set the shooting parameters and perform the exposure operation. This allows the user to focus more attention on the situation around them and their surroundings, enabling safer shooting. In this case, it is preferable that the timing of starting the exposure operation be configured to monitor the amount of movement of the imaging device 120 and start exposure when it has moved more than a predetermined amount. This configuration makes it possible to capture an image set for photogrammetry with an appropriate storage capacity.

[0067] Furthermore, although photogrammetry has been mentioned as a means for creating a stereoscopic three-dimensional model, the present invention is not limited to this, and known means using, for example, neural rendering may also be used.

[0068] 7, if there is a region of the object 201 that extends beyond the angle of view of the image capture device 120 (if at least a portion of the object 201 is outside the angle of view of the image capture device 120), it is preferable to notify the user of this in order to encourage the user to change the angle of view. Specifically, the CG in FIG. 5 may be rendered in a different color (for example, the color of the surface or line where the extension occurs). Alternatively, a message such as "The subject is extending beyond the angle of view. Please rotate the camera left" may be displayed. Note that any method can be used as long as it can notify the user that there is a region of the object 201 that extends beyond the angle of view of the image capture device 120. For example, vibration or sound may be used.

[0069] Furthermore, there are cases where the angle of view of the image capture device 120 does not fit entirely within the field of view captured by the display device 110, such as when outstretched arms are used to capture an image of the top of the head of a stone statue on a person's stand from above. If such a situation is determined from the position and orientation estimation results of the display device 110 and the image capture device 120, it is preferable to configure the image data generated by the image processing unit 125 of the image capture device 120 to be displayed in an area that is less likely to obstruct the field of view, such as the four corners of the display unit 112. This makes it possible to perform desired photographing in photogrammetry even in situations where it is difficult for the user to grasp the subject scene captured by the image capture device 120.

[0070] Furthermore, the image processing unit 118 may generate a three-dimensional model from a set of multiple images captured by the imaging device 120 using photogrammetry or neural rendering technology, and display the model on the display unit 112. This allows the user to check the quality of the three-dimensional model on the spot and re-capture the image as necessary, thereby improving the efficiency of capturing images.

[0071] As described above, according to the configuration of this embodiment, when taking photographs for photogrammetry, it is possible to display assist information so that an image can be taken under appropriate photographing conditions while understanding the surrounding situation. [Other Embodiments] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0072] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

Claims

1. A display processing device communicatively connected to an imaging device, comprising: a detection unit that detects an object present in the user's line of sight; a generation unit that generates spatial domain information according to the angle of view of the imaging device and the area of ​​the object; and a display unit that displays the spatial domain information.

2. A display processing device according to claim 1, characterized in that the area of ​​the object is determined according to the depth of field when the imaging device focuses on the object.

3. The display processing device according to claim 1, wherein the area of ​​the object is determined according to the distance range in which the object exists.

4. The display processing device according to any one of claims 1 to 3, characterized in that the spatial domain information is CG having a quadrangular pyramid shape.

5. The display processing device according to claim 4, wherein the generation unit generates vertex position information of the truncated pyramid using the relationship between the positions and orientations of the imaging device and the display processing device.

6. The display processing device according to any one of claims 1 to 5, characterized in that, when at least a portion of the object is located outside the angle of view, a user is notified that at least a portion of the object is located outside the angle of view.

7. The display processing device according to claim 6, characterized in that the spatial domain information includes CG having a truncated pyramid shape, and when at least a portion of the object is located outside the angle of view, the display unit changes the color of the CG.

8. A display processing device according to claim 6, characterized in that, when at least a portion of the object is located outside the angle of view, the display unit displays a statement indicating that at least a portion of the object is located outside the angle of view.

9. A display processing device according to any one of claims 1 to 8, characterized in that the generation unit generates a three-dimensional model from multiple images acquired by the imaging device using photogrammetry or neural rendering technology.

10. A display processing device as described in any one of claims 1 to 9, characterized in that when there is no overlapping area between the angle of view of the imaging device and the angle of view of the display processing device, the display unit displays image data generated by the imaging device.

11. The display processing device according to any one of claims 1 to 10, further comprising an acquisition unit for acquiring the line of sight of the user.

12. A display processing method for a display processing device having a display unit and communicatively connected to an imaging device, comprising the steps of: detecting an object present in the direction of a user's line of sight; generating spatial domain information according to the angle of view of the imaging device and the area of ​​the object; and displaying the spatial domain information on the display unit.

13. A program for causing a computer to execute the display processing method according to claim 12.

Images