Display processing device, display processing method, and program

Abstract

To provide a display processing device capable of displaying assist information so that an image can be captured under appropriate shooting conditions while grasping the surrounding situation when capturing an image for photogrammetry.SOLUTION: A display processing device is communicatively connected to an imaging device, and includes a detection unit that detects an object present in a user's sight direction, a generation unit that generates spatial domain information according to the angle of view of the imaging device and the domain of the object, and a display unit that displays the spatial domain information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for modeling three-dimensional space, and more particularly to a technique for displaying assist information when photographing an object from various angles. [Background technology]

[0002] Conventionally, there is a known technology for 3D modeling of an object using images taken from various angles. Specifically, there is a technology called photogrammetry, which creates a solid 3D model by analyzing and integrating a set of images taken around the object while changing the position and orientation.

[0003] When checking the camera's angle of view, it is usually necessary to look through the viewfinder, but in that state it is difficult to grasp the situation at your feet and the surroundings. When taking photos for photogrammetry, the user takes photos while walking around the object, so if you try to check the surroundings while checking the angle of view, the shooting work becomes complicated.

[0004] Patent document 1 discloses a configuration that displays the area that a camera is about to capture on a head-mounted display (HMD), making it possible to visually grasp the capture area even if there is a large angular difference between the optical axis and the user's line of sight. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2016-201686 A Summary of the Invention [Problem to be solved by the invention]

[0006] When generating high-quality 3D models, it is desirable for the image set input to photogrammetry to be images with little motion blur or depth blur, and low noise, taken at low sensitivity. In other words, the camera shooting conditions require appropriate aperture settings to avoid excessively narrowing the aperture, which results in long exposure times or high sensitivity.

[0007] However, the information displayed by the configuration of Patent Document 1 does not take into consideration the depth information of the object, so it is not possible to correctly recognize the area in which the subject is located and set appropriate shooting conditions.

[0008] An object of the present invention is to provide a display processing device capable of displaying assist information so that an image can be captured under appropriate shooting conditions while grasping the surrounding situation when capturing images for photogrammetry. [Means for solving the problem]

[0009] A display processing device as one aspect of the present invention is a display processing device that is communicatively connected to an imaging device, and is characterized by having a detection unit that detects an object present in the user's line of sight, a generation unit that generates spatial domain information according to the angle of view of the imaging device and the area of ​​the object, and a display unit that displays the spatial domain information. Effect of the Invention

[0010] According to the present invention, it is possible to provide a display processing device capable of displaying assist information so that an image can be captured under appropriate shooting conditions while grasping the surrounding situation when capturing images for photogrammetry. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram of a display processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining a coordinate system. [Diagram 3]11 is a flowchart for explaining the operation of the display processing system. [Figure 4] 1 is a diagram for explaining image data acquired by a display device; [Diagram 5] 11A and 11B are diagrams for explaining assist information displayed on a display device. [Figure 6] FIG. 2 is a diagram for explaining image data acquired by an imaging device. [Figure 7] 11 is a diagram for explaining a spatial region that is determined based on the angle of view of an imaging device and the region of an object. FIG. [Figure 8] FIG. 1 is a diagram for explaining photography for photogrammetry. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to refer to the same components, and duplicated descriptions will be omitted.

[0013] 1 is a block diagram of a display processing system 100 according to an embodiment of the present invention. The display processing system 100 is composed of a display device (display processing device) 110, an imaging device 120, and a network 130. The display device 110 and the imaging device 120 are connected via the network 130 so as to be able to communicate data with each other.

[0014] The display device 110 is a device that can be worn on the head, such as an HMD or a glasses-type display. In this embodiment, the display device 110 has a stereo camera configuration and can acquire an image for the left eye and an image for the right eye. The display device 110 has a control unit 111, a display unit 112, a ROM 113, a RAM 114, an imaging unit 115, a gaze direction acquisition unit (acquisition unit) 116, an object detection unit (detection unit) 117, an image processing unit (generation unit) 118, and a position and orientation estimation unit 119.

[0015] The control unit 111 is, for example, a CPU, and controls the operation of each block of the display device 110 by reading out an operation program for each block of the display device 110 from a ROM 113, expanding it into a RAM 114, and executing it.

[0016] The display unit 112 is a non-transmissive display, and is divided into a right eye display and a left eye display. An eyepiece is disposed between the eye and the display unit 112. The image captured by the imaging unit 115 is displayed on the display unit 112, allowing the user to observe the space in front of the user. By displaying images and information such as CG (Computer Graphics) for shooting assistance on the image displayed on the display unit 112, the displayed image can be superimposed on the space the user is viewing through the display unit 112. Note that a transmissive display may be used as the display unit 112.

[0017] The ROM 113 is a rewritable non-volatile memory, and stores, in addition to operation programs for each block of the display device 110, parameters required for the operation of each block, captured image data, and the like.

[0018] The RAM 114 is a rewritable volatile memory, and is used as a temporary storage area for data output during the operation of each block included in the display device 110.

[0019] The imaging unit 115 includes an imaging element such as a CCD or CMOS sensor, an optical system, and an A / D conversion circuit, and obtains an image of a subject in front of the user wearing the device as digital image data and outputs it to the ROM 113.

[0020] The gaze direction acquisition unit 116 is, for example, a gaze detection module based on a corneal reflex method, and acquires the direction in which the user is gazing (the user's gaze direction).

[0021] The object detection unit 117 detects an object existing in the direction in which the user is gazing (the gaze destination) and sets a gaze point based on information on the direction in which the user is gazing acquired by the gaze direction acquisition unit 116. In addition, the object detection unit 117 outputs supplementary information (subject feature amount such as color or brightness histogram) of the detected object to the RAM 114 to convey it to the imaging device 120.

[0022] The image processing unit 118 generates image data by applying various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the ROM 113. The image processing unit 118 also calculates, by coordinate conversion, in which area of ​​the image data obtained by the imaging unit 115 the spatial area captured by the imaging device 120 exists, and generates a CG image having a truncated pyramid shape by connecting the boundaries of the calculated areas with line segments, and superimposes it on the image data.

[0023] Here, the coordinate system of this embodiment will be described with reference to FIG. 2. FIG. 2 is a diagram for explaining the coordinate system. 201 is an object for which a three-dimensional model is to be created. 202 is a gaze point at which a user gazes via the display device 110. 203 is the origin of the world coordinate system, which is composed of the Xw, Yw, and Zw axes. 204 is the origin of the coordinate system of the image capture device 120, which is composed of the Xc, Yc, and Zc axes. In the following description, this will be referred to as a camera coordinate system. 205 is the imaging plane of the image capture device 120. 206 is the origin of the coordinate system for the right eye camera in the display device 110, which is composed of the Xh, Yh, and Zh axes. In the following description, this will be referred to as a display coordinate system. 207 is the imaging plane for the right eye camera in the display device 110. Also, what is represented by a dotted line is information for the left eye camera, 209 is the origin of the coordinate system, and 210 is the imaging plane for the left eye camera. In the following explanation, the processing will be explained using image data for the right-eye camera. It is assumed that the image plane for the right-eye camera and the image plane for the left-eye camera are in a positional relationship that matches when shifted in the Xh direction. Object 208 is an object that exists in the background area relative to object 201, and is not a target for creating a 3D model.

[0024] 2, it is assumed that the user is not looking through the viewfinder of the imaging device 120 and is therefore unable to grasp the angle of view of the imaging device 120. In the following description, it is assumed that the angle of view acquired by the display device 110 is wider than the angle of view acquired by the imaging device 120.

[0025] The position and orientation estimation unit 119 calculates (estimates) the position and orientation in the world coordinate system of the display device 110. It is assumed that the internal parameters of the camera of the display device 110 are known.

[0026] Here, the camera internal parameter K is defined by the focal length of the camera and the optical center coordinates, and is specifically expressed by the following matrix. It is also assumed that there is no lens distortion.

[0027]

number

[0028] Here, fx and fy are the focal lengths of the camera, and cx and cy are the optical center coordinates of the camera.

[0029] The imaging device 120 is a mirrorless camera, etc. The imaging device 120 includes a control unit 121, a ROM 122, a RAM 123, an imaging unit 124, an image processing unit 125, a position and orientation estimation unit 126, and an operation unit 127.

[0030] The control unit 121, the ROM 122, and the RAM 123 are similar to the control unit 111, the ROM 113, and the RAM 114, respectively, and therefore the description thereof will be omitted.

[0031] The imaging unit 124 includes an optical system, an imaging element, and an A / D conversion circuit. The optical system includes, for example, a variable magnification lens that changes the focal length and a focus lens that adjusts the focus. The optical system also includes an aperture, and the aperture adjusts the aperture diameter of the optical system to adjust the amount of light during shooting. The optical image formed on the imaging element by the optical system is photoelectrically converted, the resulting analog image signal is subjected to A / D conversion processing, and the resulting digital image data is output to the ROM 122 for storage.

[0032] The image processing unit 125 applies various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the ROM 122, and outputs the image data to the ROM 122.

[0033] The position and orientation estimation unit 126 calculates (estimates) the position and orientation in the world coordinate system of the image capture device 120. Note that the camera internal parameters of the image capture device 120 are known.

[0034] The operation unit 127 is an aperture operation member or the like, and can change the shooting conditions of the imaging device 120.

[0035] Hereinafter, an assist display operation in the display device 110 will be described when an image for creating a three-dimensional model of the object 201 is captured using the imaging device 120 in the situation shown in Fig. 2. Fig. 3 is a flowchart for explaining the operation of the display processing system 100.

[0036] Fig. 3(a) is a flowchart for explaining the operation of the display device 110. Each process shown in Fig. 3(a) is executed by the control unit 111 after a program recorded in the ROM 113 is loaded into the RAM 114.

[0037] In step S301, the gaze direction acquisition unit 116 acquires the direction in which the user is gazing. The object detection unit 117 detects an object present at the gaze destination in the image data acquired by the imaging unit 115 for the right eye based on the information on the direction in which the user is gazing, and sets the gaze point (xh, yh). The object detection unit 117 also outputs supplementary information of the detected object to the RAM 114 in order to convey it to the imaging device 120. FIG. 4 is a diagram for explaining image data 401 acquired by the imaging unit 115. As shown in FIG. 4, the display device 110 captures the object 201 at an angle of view looking down diagonally from above.

[0038] In step S302, the position and orientation estimation unit 119 estimates the position and orientation of the display device 110 by referring to the current image data and image data acquired before the current one. Specifically, the position and orientation are estimated from the image data group using a technique such as SfM (Structure from Motion). The position and orientation may be estimated using an acceleration sensor or an angular velocity sensor. Alternatively, the position and orientation may be estimated by combining these. The position and orientation estimated here is the camera extrinsic parameter T, which is composed of a rotation component and a translation component, and is expressed by the following matrix.

[0039]

number

[0040] Here, r11, r12, r13, r21, r22, r23, r31, r32, and r33 are the rotation components of the camera, and t1, t2, and t3 are the translation components of the camera.

[0041] Coordinates (u, v) in the image data acquired by the imaging unit 115 and three-dimensional coordinates (X, Y, Z) in the world coordinate system can be converted by the following equation (1) using camera internal parameters K and camera external parameters T.

[0042]

number

[0043] (1) Here, s is a coefficient representing the uncertainty of the scale.

[0044] In step S303, the position and orientation estimation unit 119 converts the gaze point (xh, yh) set in step S301 and the distance zh from the display device 110 to the gaze point into three-dimensional coordinates (Xh, Yh, Zh) in the world coordinate system. Note that the distance zh to the gaze point may be measured by stereo ranging using an image for the right eye and an image for the left eye.

[0045] From equation (1), the calculation formula for conversion is expressed as the following equation (2).

[0046]

number

[0047] (2) In step S304, the three-dimensional coordinates (Xh, Yh, Zh) of the gaze point and additional information of the object are transmitted to the image capture device 120 as object information.

[0048] In step S305, it is determined whether or not the area information generated in step S316 described later has been received by the imaging device 120. If it is determined that the area information has been received, the process of step S306 is executed, and if it is determined that the area information has not been received, the process of this step is executed again.

[0049] In step S306, the image processing unit 118 generates information for generating spatial region information according to region information, which is information about the depth direction in which an object exists. In this embodiment, the region information is three-dimensional coordinates (Xci, Yci, Zci) (i: 1 to 8) of eight points determined based on the region (depth direction information) of the object determined by the angle of view and depth of field of the imaging device 120 as described below. The image processing unit 118 converts the three-dimensional coordinates (Xci, Yci, Zci) of the eight points into coordinates (vertex position information) of image data in the display device 110 according to formula (1) and acquires them as information for generating spatial region information. Note that since it is assumed that the position and orientation of the display device 110 change over time, the camera extrinsic parameters are updated each time.

[0050] In step S307, the image processing unit 118 generates spatial domain information using the information acquired in step S306. Here, the spatial domain information is information for grasping the contents that the imaging device 120 is trying to capture (the three-dimensional spatial domain captured by the imaging device 120). Specifically, the image processing unit 118 first generates image data by applying various image processes such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the ROM 122. Next, the image processing unit 118 generates a CG image in the shape of a quadrangular pyramid in which the coordinates acquired in step S306 are connected by lines as spatial domain information. The spatial domain information is displayed by the display unit 112 in a state where it is superimposed on the image data. FIG. 5 is a diagram for explaining the spatial domain information as assist information displayed on the display device 110, and shows a state where the spatial domain information is superimposed on the image data. Only the object 201, which is the object for which a three-dimensional model is to be created, is surrounded by the CG image in the shape of a quadrangular pyramid, and it is shown that the imaging device 120 faces in a desired direction and captures a subject with an appropriate aperture setting. In this way, the user can understand what the imaging device 120 is trying to capture without looking into the viewfinder of the imaging device 120, and can capture images for photogrammetry while understanding the surrounding situation. In addition, the depth of field in the depth direction can be simultaneously understood, and it becomes easy to capture an image of the object without depth blur, which enables the generation of a high-quality 3D model. The CG for the left eye can be generated by shifting the CG for the right eye by the amount of deviation of the imaging plane, and the shifted CG is superimposed on the image data for the left eye.

[0051] In step S308, it is determined whether shooting completion information transmitted in step S319 described later by the imaging device 120 has been received. If it is determined that shooting completion information has been received, this flow is ended, and if it is determined that shooting completion information has not been received, the process of step S305 is executed.

[0052] Fig. 3(b) is a flowchart for explaining the operation of the imaging device 120. Each process shown in Fig. 3(b) is executed by the control unit 121 after a program recorded in the ROM 122 is loaded into the RAM 123.

[0053] In step S311, it is determined whether the object information transmitted in step S304 has been received by the display device 110. If it is determined that the object information has been received, the process of step S312 is executed, and if it is determined that the object information has not been received, the process of this step is executed again.

[0054] In step S312, the position and orientation estimation unit 126 estimates the position and orientation of the image capture device 120 in the same manner as in step S302.

[0055] In step S313, the position and orientation estimation unit 126 converts the three-dimensional coordinates (Xh, Yh, Zh) of the gaze point received in step S311 using equation (1) to convert it into coordinates in the image data of the imaging device 120. Fig. 6 is a diagram for explaining the image data acquired by the imaging device 120, and shows the image data captured by the imaging device 120. The entire body of the object 201 is captured from the front, and an object 208 exists in the background.

[0056] In step S314, the imaging unit 124 drives the focus lens to focus on the object 201 at the gaze point. At this time, in order to improve the accuracy of focusing on the subject, the imaging unit 124 may redetect the subject by referring to the additional information of the object received in step S311.

[0057] In step S315, the imaging unit 124 calculates the depth of field when the imaging device 120 focuses on the object 201. The depth of field is expressed as Z-Df to Z+Db using the front depth of field Df, the rear depth of field Db, and the focused object distance Z. Df is expressed as (r·Av·Z^2) / (f^2+r·Av·Z), and Db is expressed as (r·Av·Z^2) / (f^2-r·Av·Z). r is the permissible circle of confusion diameter, Av is the aperture value, and f is the focal length. The permissible circle of confusion diameter r is set to twice the pixel pitch. FIG. 7 is a diagram for explaining a spatial region determined based on the angle of view of the imaging device 120 and the region of the object determined by the depth of field when the imaging device 120 focuses on the object 201. The spatial region cut out by the vertical angle of view Yφ and the front depth of field Df and the rear depth of field Db at the focused subject distance Z is represented by diagonal lines. In order to generate a high-quality 3D model through photogrammetry, it is necessary to input an image in which the subject does not go out of the angle of view and the subject is within the depth of field without depth blur. By conveying the spatial region represented by diagonal lines in FIG. 7 to the user, it is possible to assist in taking pictures for photogrammetry. In FIG. 7, the object 201 for which a 3D model is to be created is contained within the diagonal line area and satisfies appropriate shooting conditions. In addition, the object 208 for which a 3D model is not to be created is located outside the spatial region represented by diagonal lines in FIG. 7.

[0058] In this embodiment, the permissible circle of confusion diameter r is set to twice the pixel pitch, but the present invention is not limited to this. For example, the permissible circle of confusion diameter r may be set to be coarser or finer depending on the number of polygons in the three-dimensional data to be created.

[0059] In addition, although the depth of field is set to Z-Df to Z+Db, the present invention is not limited to this, and the range may be set with some margin, such as Z-2·Df to Z+2·Db, taking into account errors in the focal length f and aperture value Av that can be obtained from the camera.

[0060] In addition, in this embodiment, the area of ​​the object is determined according to the depth of field when the imaging device 120 focuses on the object 201, but the present invention is not limited to this. For example, the area of ​​the object may be determined according to the distance range in which the object 201 exists. In this case, the information Z+ΔZ of the object in the depth direction may be calculated from the defocus value calculated for each pixel position in the phase difference image obtained from the imaging sensor in which all pixels are phase difference pixels. The information Z+ΔZ of the depth direction can be derived from the lens formula (1 / Z+1 / Z'=1 / f and 1 / (Z+ΔZ)+1 / (Z'+def)=1 / f) when the focal length of the lens is f. This allows the depth of the subject to be grasped with fine accuracy, making it possible to set the aperture value more precisely.

[0061] In step S316, the position and orientation estimation unit 126 converts the coordinate positions of the eight points required to express the spatial region represented by the diagonal lines in Fig. 7 into three-dimensional coordinates in the world coordinate system using equation (2). Here, the positions of the eight points are the vertex positions of the trapezoid represented by the diagonal lines in Fig. 7, and when the Xc direction is further taken into account, they are the vertex coordinate positions of a total of eight points of a quadrangular pyramid. The converted coordinates are transmitted to the display device 110 as region information.

[0062] In step S317, it is determined whether the aperture value has been changed. If it is determined that the aperture value has been changed, the process of step S315 is executed, and if it is determined that the aperture value has not been changed, the process of step S318 is executed. When the exposure compensation setting is constant, the narrower the aperture value, the deeper the depth of field becomes, and the longer the exposure time becomes, or the higher the ISO sensitivity becomes. If the exposure time becomes longer, the effects of camera shake and subject motion become greater, and if the ISO sensitivity increases, the more noise becomes. Therefore, the user can find an aperture value that does not make the depth of field deeper than necessary by changing the aperture of the operation unit 127 while looking at the CG of a quadrangular pyramid shape drawn on the display unit 112. Note that the imaging device 120 may be configured to automatically set an appropriate aperture value, and the user may simply check the result. In any case, shooting can be performed with an appropriate aperture value setting for photogrammetry.

[0063] In step S318, exposure processing is performed by the imaging unit 124, and image data suitable for photogrammetry processed by the image processing unit 125 is recorded in the ROM 122.

[0064] In step S319, command information (image capture completion information) is transmitted to the display device 110 to inform it that the image capture process in step S318 has been completed.

[0065] When shooting is completed in step S318, the user moves to the next shooting position and continues shooting while changing the position and angle so as to surround the object 201, as shown in Fig. 8. Note that Fig. 8 is a bird's-eye view seen from above, and 801 to 810 indicate the position and orientation of the imaging device 120.

[0066] It is preferable that the imaging device 120 is configured to automatically set the shooting parameters and perform the exposure operation. This allows the user to focus more attention by understanding the situation around him / her and enables safe shooting. In this case, it is preferable that the timing of starting the exposure operation is determined by monitoring the amount of movement of the imaging device 120 and starting exposure when the amount of movement is greater than a predetermined amount. With this configuration, it is possible to capture an image set for photogrammetry with an appropriate recording capacity.

[0067] Furthermore, although photogrammetry has been mentioned as a means for creating a solid three-dimensional model, the present invention is not limited to this, and known means using, for example, neural rendering may also be used.

[0068] 7, if there is an area of ​​the object 201 that protrudes from the angle of view of the imaging device 120 (at least a part of the object 201 is outside the angle of view of the imaging device 120), it is preferable to notify the user of this in order to encourage the user to change the angle of view. Specifically, the color of the CG in FIG. 5 (for example, the color of the surface or line where the protrusion occurs) may be changed. Also, a message such as "The subject is protruding. Please rotate the camera to the left" may be displayed. Note that any method may be used as long as it is possible to notify the user that there is an area of ​​the object 201 that protrudes from the angle of view of the imaging device 120. For example, vibration or sound may be used.

[0069] Also, there are cases where the angle of view of the imaging device 120 does not fall within the field of view captured by the display device 110, such as when the user reaches out to capture the top of the head of a stone statue on a person stand from above. When such a situation is determined from the position and orientation estimation results of the display device 110 and the imaging device 120, it is preferable to configure the imaging device 120 to display image data generated by the image processing unit 125 in an area that is unlikely to obstruct the field of view, such as the four corners of the display unit 112. This makes it possible to perform desired shooting for photogrammetry even in situations where it is difficult for the user to grasp the subject scene captured by the imaging device 120.

[0070] Furthermore, the image processing unit 118 may generate a three-dimensional model from a set of multiple images captured by the imaging device 120 using photogrammetry or neural rendering technology, and display the model on the display unit 112. This allows the user to check the quality of the three-dimensional model on the spot and re-shoot the image as necessary, thereby improving the efficiency of shooting.

[0071] As described above, according to the configuration of this embodiment, when photographing for photogrammetry, it is possible to display assist information so that an image can be photographed under appropriate photographing conditions while grasping the surrounding situation. [Other Examples] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0072] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) A display processing device communicably connected to an imaging device, A detection unit that detects an object present in a line of sight of a user; a generation unit that generates spatial region information according to an angle of view of the imaging device and a region of the object; and a display unit for displaying the spatial domain information. (Configuration 2) 2. The display processing device according to configuration 1, wherein the area of ​​the object is determined according to a depth of field when the imaging device focuses on the object. (Configuration 3) 2. The display processing device according to configuration 1, wherein the area of ​​the object is determined according to a distance range in which the object exists. (Configuration 4) 4. The display processing device according to any one of configurations 1 to 3, wherein the spatial domain information is a CG image having a shape of a truncated quadrangular pyramid. (Configuration 5) 5. The display processing device according to configuration 4, wherein the generation unit generates vertex position information of the frustum of a quadrangular pyramid by using a relationship between the positions and orientations of the imaging device and the display processing device. (Configuration 6) A display processing device according to any one of configurations 1 to 5, characterized in that, when at least a portion of the object is located outside the angle of view, a user is notified that at least a portion of the object is located outside the angle of view. (Configuration 7) The spatial domain information includes a CG having a quadrangular pyramid shape, 7. The display processing device according to configuration 6, wherein the display unit changes a color of the CG when at least a part of the object is located outside the angle of view. (Configuration 8) The display processing device according to configuration 6, wherein, when at least a portion of the object is located outside the angle of view, the display unit displays a statement indicating that at least a portion of the object is located outside the angle of view. (Configuration 9) The display processing device according to any one of configurations 1 to 8, wherein the generation unit generates a three-dimensional model from a plurality of images acquired by the imaging device using photogrammetry or neural rendering technology. (Configuration 10) A display processing device described in any one of configurations 1 to 9, characterized in that when there is no overlapping area between the angle of view of the imaging device and the angle of view of the display processing device, the display unit displays image data generated by the imaging device. (Configuration 11) 11. The display processing device according to claim 1, further comprising a gaze detection unit that detects a gaze direction of the user. (Method 1) A display processing method for a display processing device including a display unit and communicably connected to an imaging device, comprising: Detecting an object present in a line of sight of a user; generating spatial region information according to an angle of view of the imaging device and a region of the object; and displaying the spatial domain information on the display unit. (Configuration 12) A program for causing a computer to execute the display processing method according to Method 1.

[0073] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0074] 110 Display device (display processing device) 112 Display section 117 Object detection unit (detection unit) 118 Image processing unit (generation unit)

Claims

1. A display processing device communicably connected to an imaging device, A detection unit that detects an object present in a line of sight of a user; a generation unit that generates spatial region information according to an angle of view of the imaging device and a region of the object; and a display unit for displaying the spatial domain information.

2. The display processing device according to claim 1 , wherein the area of ​​the object is determined according to a depth of field when the imaging device focuses on the object.

3. The display processing device according to claim 1 , wherein the area of ​​the object is determined according to a distance range in which the object exists.

4. 4. The display processing device according to claim 1, wherein the spatial domain information is CG data having a shape of a truncated quadrangular pyramid.

5. The display processing device according to claim 4 , wherein the generation unit generates vertex position information of the truncated pyramid by using a relationship between the positions and orientations of the imaging device and the display processing device.

6. 4 . The display processing device according to claim 1 , wherein, when at least a part of the object is located outside the angle of view, a user is notified that at least a part of the object is located outside the angle of view.

7. the spatial domain information includes CG having a shape of a quadrangular pyramid; The display processing device according to claim 6 , wherein, when at least a part of the object is located outside the angle of view, the display unit changes a color of the CG.

8. The display processing device according to claim 6 , wherein, when at least a part of the object is located outside the angle of view, the display unit displays a statement indicating that at least a part of the object is located outside the angle of view.

9. 4. The display processing device according to claim 1, wherein the generation unit generates a three-dimensional model from a plurality of images acquired by the imaging device by using photogrammetry or neural rendering technology.

10. 4. The display processing device according to claim 1, wherein when there is no overlapping area between the angle of view of the imaging device and the angle of view of the display processing device, the display unit displays image data generated by the imaging device.

11. The display processing device according to claim 1 , further comprising an acquisition unit that acquires a line-of-sight direction of the user.

12. A display processing method for a display processing device including a display unit and communicably connected to an imaging device, comprising: Detecting an object present in a line of sight of a user; generating spatial region information according to an angle of view of the imaging device and a region of the object; and displaying the spatial domain information on the display unit.

13. A program causing a computer to execute the display processing method according to claim 12.

Images