Display method and information processing apparatus

Abstract

To provide a display method capable of matching a virtual viewpoint with an actual viewpoint of a frame being an actual image while providing a degree of freedom to the selection of the virtual viewpoint in a viewer.SOLUTION: The display method includes specifying a first region around an actual viewpoint of a frame generated by capturing an image of an object from the actual viewpoint (S114A), setting, when a setting operation relating to a viewpoint is received from a user during display of a three dimensional model of the object, virtual viewpoints in the first region based on the setting operation, and displaying the three dimensional model viewed from the virtual viewpoints (S115A).SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present disclosure relates to a viewer control method and an information processing device. [Background technology]

[0002] There is a technique for estimating the camera position and orientation using a camera image as an input, and generating a three-dimensional model using the results (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 0141775 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional viewers that display three-dimensional models allow users to select any virtual viewpoint. However, there is a need to align the virtual viewpoint with the real viewpoint of a frame, which is a real image, rather than selecting a virtual viewpoint in the viewer. Therefore, an object of the present disclosure is to provide a viewer control method and an information processing device that allow users to align the virtual viewpoint with the real viewpoint of a frame, which is a real image, while allowing freedom in selecting a virtual viewpoint in the viewer. [Means for solving the problem]

[0005] A display method according to one embodiment of the present disclosure identifies a first area around a real viewpoint in a frame generated by photographing an object from the real viewpoint, and when a setting operation related to the viewpoint is received from a user while a three-dimensional model of the object is being displayed, sets a virtual viewpoint within the first area based on the setting operation, displays the three-dimensional model as seen from the virtual viewpoint, and controls the display of the three-dimensional model as seen from the real viewpoint if a first position based on the setting operation is outside the first area. [Effects of the Invention]

[0006] The present disclosure can provide a viewer control method and an information processing device that can align a virtual viewpoint with the real viewpoint of a frame that is a real image, while allowing a degree of freedom in selecting a virtual viewpoint in a viewer. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing the configuration of a three-dimensional model display system according to an embodiment. [Figure 2] FIG. 2 is a diagram for explaining the feature point matching process according to the embodiment. [Figure 3] FIG. 3 is a flowchart showing the operation of the display device according to the embodiment. [Figure 4] FIG. 4 is a diagram showing a display example of the display unit according to the embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a display after the viewpoint is changed according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of image search processing according to the embodiment. [Figure 7] FIG. 7 is a diagram showing a display example after image search according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a display after changing the viewpoint according to the embodiment. [Figure 9] FIG. 9 is a diagram showing a display example in which a plurality of frames are displayed according to the embodiment. [Figure 10] FIG. 10 is a diagram showing an example of frame selection by a user according to the embodiment. [Figure 11] FIG. 11 is a diagram showing a display example in which a real viewpoint is displayed according to the embodiment. [Figure 12] FIG. 12 is a diagram showing an example of displaying a plurality of three-dimensional models according to the embodiment. [Figure 13] FIG. 13 is a flowchart showing the operation of the modified example of the display device according to the embodiment. [Figure 14]FIG. 14 is a flowchart showing the operation of the modified example of the display device according to the embodiment. [Figure 15] FIG. 15 is a flowchart showing the operation of the modified example of the display device according to the embodiment. [Figure 16] FIG. 16 is a flowchart showing the operation of the modified example of the display device according to the embodiment. [Figure 17] FIG. 17 is a diagram showing a display example of the first region according to the embodiment. [Figure 18] FIG. 18 is a diagram showing a display example of the second area according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] A viewer control method according to one aspect of the present disclosure is a viewer control method for controlling a viewer, which identifies a first region around a real viewpoint in a frame generated by photographing an object from the real viewpoint, outputs information about the first region to the viewer, and the viewer displays a three-dimensional model of the object as seen from a virtual viewpoint within the first region.

[0009] According to this, the viewer control method can display a 3D model seen from a virtual viewpoint close to the real viewpoint in the viewer. Therefore, the viewer control method can align (bring closer) the virtual viewpoint to the real viewpoint while allowing the viewer some freedom in selecting the virtual viewpoint.

[0010] For example, the viewer control method may further include controlling the viewer so that the viewer does not display the three-dimensional model as seen from a virtual viewpoint in a second area other than the first area.

[0011] According to this, the viewer control method can prohibit the setting of a virtual viewpoint far from the real viewpoint.

[0012] For example, the real viewpoint may be represented in a first coordinate system and the three-dimensional model may be represented in a second coordinate system, and the viewer control method may further identify a second region in the first coordinate system around the real viewpoint and transform the second region into the first region in the second coordinate system.

[0013] According to this, the viewer control method can determine whether the virtual viewpoint is included in the first region even when the coordinate systems of the real viewpoint and the three-dimensional model are different.

[0014] For example, the viewer control method may further include acquiring a first position in the second coordinate system via an interface, and if the first position is within the first area, controlling the viewer so that the viewer displays the three-dimensional model as seen from the first position.

[0015] According to this, the viewer control method can display a 3D model in the viewer as seen from a virtual viewpoint that is close to the real viewpoint, and thus the viewer control method can align the virtual viewpoint with the real viewpoint while allowing the viewer to select a virtual viewpoint with a degree of freedom.

[0016] For example, the viewer control method may further include acquiring a first position in the second coordinate system via an interface, and controlling the viewer so that if the first position is outside the first region, the viewer displays warning information indicating that the three-dimensional model viewed from the first position will not be displayed.

[0017] According to this, the viewer control method can prohibit the setting of a virtual viewpoint specified by the user that is far from the real viewpoint.

[0018] For example, the viewer control method may further include controlling the viewer so as to display the three-dimensional model as seen from the real viewpoint when the first position is outside the first region.

[0019] According to this, when the virtual viewpoint specified by the user is far from the real viewpoint, for example, in the case of a user's incorrect operation, the viewer control method does not simply prohibit it, but can instead display a three-dimensional model as seen from the real viewpoint.

[0020] For example, the viewer control method may further identify an additional real viewpoint of an additional frame generated by photographing the object from an additional real viewpoint in the first coordinate system, obtain a first position in the second coordinate system via an interface, and, if the first position is closer to the real viewpoint than the additional real viewpoint, control the viewer to display the three-dimensional model viewed from the real viewpoint, and, if the first position is closer to the additional real viewpoint than the real viewpoint, control the viewer to display the three-dimensional model viewed from the additional real viewpoint.

[0021] According to this, when there are multiple frames, the viewer control method can display a 3D model viewed from a real viewpoint that is close to the virtual viewpoint specified by the user, thereby improving operability when there are limitations on the viewer's viewpoint selection.

[0022] For example, the viewer control method may further include controlling the viewer to display the first region.

[0023] This allows the viewer control method to support the user in setting an appropriate virtual viewpoint.

[0024] For example, the viewer control method may further control the viewer to display a second area other than the first area, and the three-dimensional model viewed from a virtual viewpoint within the second area may not be displayed in the viewer.

[0025] This allows the viewer control method to support the user in setting an appropriate virtual viewpoint.

[0026] For example, the first region may include a plurality of real viewpoints of a plurality of frames including the frame, which are generated by photographing the object from a plurality of real viewpoints including the real viewpoint, and the three-dimensional model may be generated by feature point matching using the plurality of frames.

[0027] According to this, the viewer control method can set a virtual viewpoint close to the actual viewpoint of the multiple frames used for feature point matching. Here, a 3D model viewed from a viewpoint close to the actual viewpoint of the multiple frames used for feature point matching is likely to be highly accurate. Also, a 3D model viewed from a viewpoint far from the actual viewpoint is likely to have data loss. Therefore, the viewer control method can display a highly accurate 3D model.

[0028] For example, the first area may be an area within a predetermined distance from the real viewpoint.

[0029] According to this, the viewer control method can display a three-dimensional model seen from a virtual viewpoint that is close to the real viewpoint in the viewer.

[0030] For example, the viewer control method may further include displaying the frame.

[0031] According to this, the viewer control method can display a 3D model viewed from a virtual viewpoint that is close to the actual viewpoint of the displayed frame. Therefore, the user can easily understand the correspondence between the frame and the 3D model, and can check the details of the object by referring to the frame even if the accuracy of the 3D model is low.

[0032] For example, the first region may include a plurality of first real viewpoints of a plurality of first frames including the frame, which are generated by photographing the object from a plurality of first real viewpoints including the real viewpoint, and the viewer control method may further acquire a specified viewpoint, which is a virtual viewpoint specified based on a user's operation, search for a second frame among the plurality of first real viewpoints that was photographed from a first real viewpoint near the specified viewpoint, and display the second frame.

[0033] According to this, the viewer control method can display a frame with a real viewpoint close to the viewpoint specified by the user. Therefore, the user can easily understand the correspondence between the frame and the 3D model, and can check the details of the object by referring to the frame even if the accuracy of the 3D model is low.

[0034] For example, the viewer control method may further include causing the viewer to display the first real viewpoint of the second frame.

[0035] This allows the viewer control method to improve operability for the user.

[0036] For example, in searching for the second frame, a plurality of second frames including the second frame that are taken from a plurality of second real viewpoints that are a plurality of first real viewpoints near the specified viewpoint among a plurality of first real viewpoints are searched for, and in displaying the second frames, the plurality of second frames are displayed, and the viewer may be controlled so that the viewpoint of the three-dimensional model displayed by the viewer becomes the second real viewpoint of a second frame selected from the plurality of second frames based on a user operation.

[0037] According to this, the viewer control method can display in the viewer a 3D model seen from the real viewpoint of a frame specified by the user among multiple frames with real viewpoints close to the viewpoint specified by the user, thereby improving user operability.

[0038] For example, the viewer control method may further cause the viewer to display a plurality of the three-dimensional models viewed from a plurality of viewpoints that have a predetermined relationship with the second real viewpoint of the selected second frame.

[0039] According to this, the viewer control method can display, in the viewer, a 3D model viewed from a plurality of viewpoints that are related to the actual viewpoint of the frame selected by the user, thereby improving user convenience.

[0040] For example, in searching for the second frame, a plurality of points included in the three-dimensional model viewed from the specified viewpoint may be projected onto each of the plurality of first frames, and the second frame may be determined based on the number of points projected onto each first frame.

[0041] According to this, the viewer control method can appropriately determine a frame of an actual viewpoint that is close to the viewpoint specified by the user.

[0042] Furthermore, an information processing device according to one embodiment of the present disclosure is an information processing device that controls a viewer, and includes a processor and a memory, wherein the processor uses the memory to identify a first area around a real viewpoint in a frame generated by photographing an object from the real viewpoint, outputs information about the first area to the viewer, and the viewer displays a three-dimensional model of the object as seen from a virtual viewpoint within the first area.

[0043] According to this, the information processing device can display a 3D model in the viewer as seen from a virtual viewpoint that is close to the real viewpoint. Therefore, the viewer control method can align the virtual viewpoint with the real viewpoint while allowing the viewer to select a virtual viewpoint with a degree of freedom.

[0044] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0045] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0046] (Embodiment) [1. Configuration] First, the configuration of a 3D model display system according to this embodiment will be described. Fig. 1 is a block diagram showing the configuration of a 3D model display system according to this embodiment. The 3D model display system is a system that displays a 3D model on a viewer, and includes a plurality of image capture devices 101, a 3D model generation device 102, and a display device 103.

[0047] A plurality of imaging devices 101 (cameras) capture images of a subject (object) from different viewpoints and output the captured frames to a three-dimensional model generation device 102. The captured frames are also called multi-viewpoint images. In other words, a frame is an image.

[0048] The multiple frames do not necessarily have to be captured by multiple image capturing devices 101, but may be captured by a single image capturing device 101 while moving. In other words, each of the multiple frames is captured (generated) by the image capturing device 101 from a viewpoint where at least one of the position and the orientation is different.

[0049] The three-dimensional model generation device 102 includes a camera position and orientation estimation unit 111 and a three-dimensional model generation unit 112. The camera position and orientation estimation unit 111 uses multiple frames obtained by multiple image capture devices 101 to generate camera position and orientation information indicating the position and orientation (also called viewpoint) of the image capture device 101 at the time of capturing a frame. Here, the orientation of the image capture device 101 indicates at least one of the image capture direction of the image capture device 101 and the tilt of the image capture device 101. The image capture direction of the image capture device 101 is the direction of the optical axis of the image capture device 101. The tilt of the image capture device 101 is the rotation angle around the optical axis of the image capture device 101 from a reference orientation.

[0050] Specifically, the camera position and orientation estimation unit 111 estimates camera parameters of the multiple image capture devices 101 based on multiple frames acquired from the multiple image capture devices 101. Here, the camera parameters are parameters that indicate the characteristics of the image capture devices 101, and include internal parameters such as the focal length and image center of the image capture device 101, and external parameters that indicate the position (more specifically, the three-dimensional position) and orientation of the image capture device 101.

[0051] There are no particular limitations on the method used by the camera position and orientation estimation unit 111 to estimate the positions and orientations of the image capture devices 101. The camera position and orientation estimation unit 111 may estimate the positions and orientations of the multiple image capture devices 101 using, for example, Visual-SLAM (Simultaneous Localization and Mapping), Structure-From-Motion, or ICP (Iterative Closest Point). Alternatively, the positions and orientations of the multiple image capture devices 101 may be input by a user.

[0052] Specifically, the camera position and orientation estimation unit 111 performs feature point matching processing on multiple frames captured by multiple image capture devices 101. That is, the camera position and orientation estimation unit 111 extracts features from the multiple frames and extracts sets of similar points that are similar between the multiple frames from the extracted multiple feature points. Next, the camera position and orientation estimation unit 111 estimates the position and orientation of the image capture device 101 using the extracted sets of similar points.

[0053] The three-dimensional model generation unit 112 generates a three-dimensional model of the subject based on multiple frames and camera position and orientation information (camera parameters). For example, the three-dimensional model generation unit 112 generates the three-dimensional model using MVS (multi-view stereo). First, the three-dimensional model generation unit 112 performs feature point matching processing. FIG. 2 is a diagram for explaining the feature point matching processing. As shown in FIG. 2, a similar point 162 on a frame 161 that is similar to a point 152 (pixel) on a frame 151 is searched for. Specifically, when a match is performed between the frame 161 and the point 152 in the frame 151 within a search range R1, an epipolar line 163 corresponding to a straight line L1 passing through the viewpoint V1 and the point 152 exists across the entire length of the frame 161. Note that the frame 151 is an image acquired at the viewpoint V1, and the frame 161 is an image acquired at the viewpoint V2. The straight line L1 coincides with the shooting direction of the image capturing device 101 located at the viewpoint V1. Point 152 corresponds to point 142 of subject 141 .

[0054] The three-dimensional model generation unit 112 calculates the normalized cross correlation (NCC) between small regions as N(I, J) for a combination of frame I and frame J, as shown in the following equation 1, and generates matching information indicating the results of matching between the frames.

[0055]

number

[0056] Note that Ixy and Jxy are pixel values ​​within the small regions of frame I and frame J.

number

number

[0057] Then, the three-dimensional model generating unit 112 generates a three-dimensional model by performing triangulation using the search results from the matching.

[0058] The display device 103 displays the 3D model on a viewer using the camera position and orientation information and the 3D model generated by the 3D model generation device 102. For example, the display device 103 is a tablet terminal, a smartphone, a personal computer, or the like.

[0059] Note that communication between the multiple image capture devices 101, the 3D model generation device 102, and the display device 103 may be performed by any method, such as wired communication or wireless communication. Furthermore, these communications may be performed directly between the devices, or may be performed indirectly via other communication devices or a server, etc. Furthermore, the transmission and reception of multiple frames, camera position and orientation information, and the 3D model may be performed in real time, or may be recorded once in an external storage device such as a memory or a cloud server, and then output from that external storage device to the 3D model generation device 102 or the display device 103.

[0060] Furthermore, the methods for generating the multiple frames, camera position and orientation information, and three-dimensional model are merely examples, and the multiple frames, camera position and orientation information, and three-dimensional model may be generated by any method, not limited to the above. For example, in addition to or instead of the multiple frames, a depth image (depth map) obtained by a depth sensor or the like, or three-dimensional information obtained by a laser measuring device such as a three-dimensional laser measuring device (LiDAR) or a millimeter-wave radar measuring device may be used.

[0061] The three-dimensional model also includes position information indicating multiple three-dimensional positions on the subject. For example, the three-dimensional model is a three-dimensional point cloud including multiple three-dimensional points indicating multiple three-dimensional positions. Each three-dimensional point may have attribute information such as color or reflectance. The three-dimensional model is not limited to a three-dimensional point cloud, and may also be a collection of three-dimensional meshes, etc.

[0062] The display device 103 includes an acquisition unit 121, a storage unit 122, a viewer control unit 123, a UI unit 124, and a display unit 125.

[0063] The acquisition unit 121 acquires (receives) a plurality of frames captured by a plurality of image capturing devices 101, camera position and orientation information, and a three-dimensional model from the three-dimensional model generation device 102. The acquisition unit 121 is, for example, a communication interface for communicating with the three-dimensional model generation device 102.

[0064] The storage unit 122 stores the multiple frames, camera position and orientation information, and three-dimensional model acquired by the acquisition unit 121. The storage unit 122 may also store processing results of processing units included in the display device 103. The storage unit 122 may also store, for example, a control program that causes a processing circuit (processor) to execute processing by each processing unit included in the display device 103. The storage unit 122 is realized by, for example, an HDD (Hard Disk Drive), a flash memory, or the like.

[0065] The UI unit 124 is a user interface that acquires user operations, such as a touch panel, a keyboard, and a mouse. The display unit 125 is a display that displays images. The viewer control unit 123 displays a viewer on the display unit 125 that displays a three-dimensional model viewed from a virtual viewpoint.

[0066] It should be noted that the viewer control unit 123, the UI unit 124, and the display unit 125 do not necessarily have to be included in a single device. The display device 103 may also include some or all of the processing units included in the three-dimensional model generation device 102.

[0067] [2. Operation] Next, the operation of the display device 103 will be described. Fig. 3 is a flowchart showing the operation of the display device 103. First, the acquisition unit 121 acquires a plurality of frames, camera position and orientation information, and a three-dimensional model from the three-dimensional model generation device 102 (S101). The storage unit 122 saves the acquired plurality of frames, camera position and orientation information, and three-dimensional model. Note that the storage unit 122 may store, as header information of a frame, information that identifies the imaging device 101 that captured the frame.

[0068] Next, the viewer control unit 123 displays the viewer 131, which displays a three-dimensional model seen from a certain viewpoint, on the display unit 125 (S102). Fig. 4 is a diagram showing an example of a display on the display unit 125.

[0069] Next, a virtual viewpoint is selected in response to a user operation on the UI unit 124 (S103). For example, the user selects the virtual viewpoint by tapping or sliding on the viewer 131. Specifically, the user performs operations such as scaling, rotation, and translation of the three-dimensional model. Note that the method of viewpoint manipulation by the user is not limited to these, and any known method may be used, such as operating an operation button or operation bar displayed on the display unit 125. FIG. 5 is a diagram showing an example of a display after the viewpoint has been changed.

[0070] Next, the viewer control unit 123 searches for a frame with a viewpoint close to the virtual viewpoint of the three-dimensional model displayed on the viewer (S104). For example, the image search is performed when the user taps the image search button 132 displayed on the display unit 125 as shown in FIG. 5. Note that the operation to start the image search is not limited to this, and any method may be used. Also, the image search may be performed automatically without any operation by the user. For example, the image search may be performed when the user has not performed an operation to change the viewpoint for a certain period of time.

[0071] Fig. 6 is a diagram showing an example of image search processing. The viewer control unit 123 compares multiple real viewpoints, which are the positions and orientations of the multiple image capture devices 101 indicated by the camera position and orientation information stored in the storage unit 122, with the current virtual viewpoint, and searches for a real viewpoint that is close to the virtual viewpoint. For example, in the example shown in Fig. 6, of real viewpoints A, B, and C, real viewpoint B, which is close to the virtual viewpoint, is selected.

[0072] Specifically, the viewer control unit 123 searches for a real viewpoint close to the virtual viewpoint based on the difference in position (e.g., xyz coordinates) between the virtual viewpoint and the real viewpoint and the difference in gaze direction (shooting direction) between the virtual viewpoint and the real viewpoint. For example, the viewer control unit 123 extracts, from a plurality of real viewpoints, a real viewpoint whose difference in position is smaller than a predetermined first threshold and whose difference in gaze direction is smaller than a predetermined second threshold. For example, Euclidean distance may be used as the difference in position and cosine similarity may be used as the difference in gaze direction. Alternatively, the viewer control unit 123 may narrow down the real viewpoints based on the difference in gaze direction and determine the difference in position for the real viewpoints after narrowing down. Alternatively, the viewer control unit 123 may compare a value obtained by weighting and adding the difference in position and the difference in gaze direction with a threshold, rather than comparing each of the difference in position and the difference in gaze direction with a threshold.

[0073] Note that the viewer control unit 123 may use only the difference in position, without using the difference in gaze direction. Here, the multiple frames are, for example, multiple frames used to generate a three-dimensional model of the subject, that is, multiple frames obtained by photographing the subject from multiple directions. Therefore, the multiple gaze directions of the multiple real viewpoints are generally directed toward the subject. Furthermore, since the selected virtual viewpoint also faces the direction of the three-dimensional model (subject), if the positions of the virtual viewpoint and the real viewpoint are close, it is highly likely that the gaze directions are also close. Therefore, a certain level of accuracy can be ensured even when only the positions are used.

[0074] Furthermore, in determining the position, different thresholds may be used for the depth direction (the direction connecting the viewpoint and the subject) and the up / down / left / right directions (directions perpendicular to the depth direction). For example, when comparing a case where the depth position (distance to the subject) is different from a case where the up / down / left / right directions are different, it is easier to compare the three-dimensional model and the frame when the depth position is different. Therefore, the threshold for the depth direction may be smaller than the threshold for the up / down / left / right directions. Note that the threshold for the depth direction may be larger than the threshold for the up / down / left / right directions. Furthermore, different thresholds may be used for the up / down direction and the left / right direction. Furthermore, the threshold settings for each of these directions may be switched depending on the subject or the purpose of use, or may be switched according to the user's settings.

[0075] As another method, the viewer control unit 123 may project the three-dimensional model seen from a virtual viewpoint, i.e., the multiple three-dimensional points that make up the three-dimensional model displayed in the viewer, onto multiple frames, and extract the frame with the largest number of projected three-dimensional points or the frame with the number of projected three-dimensional points greater than a threshold value as the frame of the real viewpoint that is closest to the virtual viewpoint.

[0076] The viewer control unit 123 displays the extracted real viewpoint frame on the display unit 125 (S105). Fig. 7 is a diagram showing an example of a display after image search. As shown in Fig. 7, a real viewpoint frame 134 that is close to the virtual viewpoint of the three-dimensional model displayed in the viewer 131 is displayed on the display unit 125.

[0077] Next, the viewer control unit 123 changes the virtual viewpoint of the three-dimensional model to the real viewpoint of the displayed frame (S106). That is, the viewer control unit 123 displays the three-dimensional model seen from the real viewpoint of the frame 134 on the viewer 131. For example, the viewpoint is changed when the user taps a viewpoint change button 133 displayed on the display unit 125 as shown in FIG. 7. Note that the viewpoint change operation is not limited to this, and any method may be used. Furthermore, the viewpoint may be changed automatically without any operation by the user.

[0078] 8 is a diagram showing an example of a display after the viewpoint has been changed. As shown in FIG. 8, the virtual viewpoint of the three-dimensional model displayed on the viewer 131 is changed to the real viewpoint of the frame 134.

[0079] Furthermore, the virtual viewpoint to be changed does not have to be completely identical to the real viewpoint, but may be a viewpoint close to the real viewpoint. Here, a close viewpoint is, for example, a viewpoint whose difference in position and line of sight from the real viewpoint is smaller than a predetermined threshold.

[0080] If the display processing of the three-dimensional model has not ended (No in S107), or if the virtual viewpoint is changed by the next user operation (S103), the processing from step S104 onwards is performed. That is, the processing of steps S103 to S106 is repeated. If the display processing of the three-dimensional model has ended (Yes in S107), for example, if the user performs an operation to end the processing, the viewer control unit 123 ends the processing.

[0081] As described above, the display device 103 according to this embodiment displays a frame from a real viewpoint that is close to the virtual viewpoint of the displayed three-dimensional model. This allows the user to check the detailed configuration of the subject by referring to the frame when, for example, the resolution of the three-dimensional model is insufficient. Furthermore, the display device 103 can align the viewpoint of the frame with that of the three-dimensional model. This allows the user to easily recognize the correspondence between the frame and the three-dimensional model. This improves user convenience.

[0082] [3. Modifications] In step S104, if there are multiple real viewpoints close to the virtual viewpoint, multiple frames may be displayed in step S105. Fig. 9 is a diagram showing a display example in this case. As shown in Fig. 9, the display unit 125 displays multiple frames 134A and 134B of multiple real viewpoints close to the virtual viewpoint.

[0083] In this case, in step S106, the user selects a frame corresponding to the real viewpoint to be set as the virtual viewpoint from among the multiple frames. Fig. 10 is a diagram showing an example of selection by the user. In the example shown in Fig. 10, for example, the user taps frame 134A to select frame 134A. Furthermore, a three-dimensional model viewed from the real viewpoint of frame 134A is displayed in viewer 131. Furthermore, in the state shown in Fig. 10, tapping frame 134B may change the viewpoint of the three-dimensional model displayed in viewer 131 to the real viewpoint of frame 134B.

[0084] Note that even in this case, the virtual viewpoint of the three-dimensional model may be changed automatically, rather than the user selecting frame 134A or 134B. For example, the three-dimensional model viewed from the real viewpoint of frame 134A and the three-dimensional model viewed from the real viewpoint of frame 134B may be displayed alternately at predetermined intervals. Alternatively, display unit 125 may display an additional viewer, and the three-dimensional model viewed from the real viewpoint of frame 134A and the three-dimensional model viewed from the real viewpoint of frame 134B may be displayed in viewer 131 and the additional viewer, respectively.

[0085] Furthermore, in addition to the three-dimensional model, the real viewpoint of frame 134 may be displayed on viewer 131. FIG. 11 is a diagram showing a display example in this case. As shown in FIG. 11, display unit 125 displays real viewpoint 135 of frame 134 on viewer 131. In this case, the user may select (e.g., tap) real viewpoint 135, thereby changing the viewpoint of the three-dimensional model displayed on the viewer to real viewpoint 135 (i.e., the viewpoint of frame 134). Note that when multiple frames 134A and 134B are displayed as in the example shown in FIG. 10, the real viewpoint corresponding to frame 134A and the real viewpoint corresponding to frame 134B may be displayed on viewer 131.

[0086] Furthermore, when a viewpoint change is performed in step S106, in addition to the three-dimensional model viewed from the real viewpoint of frame 134, multiple three-dimensional models viewed from multiple viewpoints that have a predetermined relationship with the real viewpoint may be displayed. FIG. 12 is a diagram showing a display example in this case. In the example shown in FIG. 12, in addition to the three-dimensional model viewed from the real viewpoint of frame 134, an enlarged image of the three-dimensional model and an image of the three-dimensional model that is reduced are displayed. Note that, without being limited to enlargement and reduction, a three-dimensional model obtained by rotating the three-dimensional model viewed from the real viewpoint of frame 134 in a predetermined direction or by translating it may also be displayed. Furthermore, an orthogonal image or multiple orthogonal images viewed from multiple directions (top, side, bottom, etc.) may be displayed.

[0087] In the above, the image search (S103) and viewpoint change (S106) are performed based on the user's operation, but they may also be performed automatically. Fig. 13 is a flowchart showing the processing of the display device 103 in this case.

[0088] Steps S111 to S113 are similar to, for example, steps S101 to S103 shown in Fig. 3. After a virtual viewpoint is selected (S113), the viewer control unit 123 determines whether a frame of a real viewpoint close to the selected virtual viewpoint exists (whether it is stored in the storage unit 122) (S114). Note that the method for determining a real viewpoint close to the virtual viewpoint is similar to, for example, the method used in step S104 described above.

[0089] If the first coordinate system of the real viewpoint indicated by the camera position and orientation information differs from the second coordinate system of the three-dimensional model and the virtual viewpoint, the viewer control unit 123 may convert the real viewpoint represented in the first coordinate system into the second coordinate system, and determine a real viewpoint that is close to the virtual viewpoint based on the real viewpoint in the second coordinate system and the virtual viewpoint in the second coordinate system. Alternatively, the viewer control unit 123 may convert the virtual viewpoint represented in the second coordinate system into the first coordinate system, and determine a real viewpoint that is close to the virtual viewpoint based on the virtual viewpoint in the first coordinate system and the real viewpoint in the first coordinate system.

[0090] If there is a frame of a real viewpoint close to the selected virtual viewpoint (Yes in S114), the viewer control unit 123 displays the three-dimensional model seen from the real viewpoint on the viewer 131 (S115). Furthermore, the viewer control unit 123 displays a frame captured from the real viewpoint on the display unit 125. Note that the viewer control unit 123 does not have to display the frame on the display unit 125. Furthermore, if there are multiple real viewpoints close to the virtual viewpoint, the viewer control unit 123 may, for example, display on the viewer 131 the three-dimensional model seen from the real viewpoint closest to the virtual viewpoint.

[0091] On the other hand, if there is no frame from a real viewpoint close to the selected virtual viewpoint (No in S114), the viewer control unit 123 displays a warning on the display unit 125 and does not display the three-dimensional model in the viewer 131 (S116). The warning indicates, for example, that the three-dimensional model will not be displayed because there is no frame captured from a real viewpoint close to the virtual viewpoint. Note that the viewer control unit 123 may display the three-dimensional model seen from the selected virtual viewpoint on the viewer 131, and may also display a warning on the display unit 125 indicating that there is no frame captured from a real viewpoint close to the virtual viewpoint.

[0092] If the display processing of the three-dimensional model has not ended (No in S117), or if the virtual viewpoint is changed by the next user operation (S113), the processing from step S114 onwards is performed. That is, the processing of steps S113 to S116 is repeated. If the display processing of the three-dimensional model has ended (Yes in S117), the viewer control unit 123 ends the processing.

[0093] By the above processing, a three-dimensional model is displayed when the virtual viewpoint is close to the real viewpoint, but is not displayed when the virtual viewpoint is far from the real viewpoint. In other words, the user's viewpoint operation is limited to virtual viewpoints close to the real viewpoint. This makes it possible to prevent the selection of a virtual viewpoint for which no corresponding frame exists. Furthermore, when a three-dimensional model is generated using multiple frames, a three-dimensional model viewed from a virtual viewpoint for which no corresponding frame exists, i.e., a virtual viewpoint far from the real viewpoint, may have low accuracy. Therefore, the above processing makes it possible to prevent a low-accuracy three-dimensional model from being displayed.

[0094] Furthermore, if there is a real viewpoint close to the virtual viewpoint, a three-dimensional model seen from the virtual viewpoint may be displayed in the viewer 131. Fig. 14 is a flowchart showing the processing of the display device 103 in this case. The processing shown in Fig. 14 differs from the processing shown in Fig. 13 in that step S115 is replaced with step S115A. If there is a real viewpoint close to the selected virtual viewpoint (Yes in S114), the viewer control unit 123 displays the three-dimensional model seen from the selected virtual viewpoint in the viewer 131 (S115A).

[0095] The above process limits the user's viewpoint operation to a virtual viewpoint that is close to the real viewpoint. This prevents the selection of a virtual viewpoint for which no corresponding frame exists. It also prevents a low-precision 3D model from being displayed. Furthermore, since it is possible to set a virtual viewpoint that does not completely match the real viewpoint, it is possible to ease the restrictions on viewpoint operation compared to the process shown in FIG. 13.

[0096] Furthermore, if there is no real viewpoint close to the virtual viewpoint, a three-dimensional model viewed from the real viewpoint may be displayed in the viewer 131. FIG. 15 is a flowchart showing the processing of the display device 103 in this case. The processing shown in FIG. 15 differs from the processing shown in FIG. 14 in that step S116 is replaced with step S116A. If there is no real viewpoint close to the selected virtual viewpoint (No in S114), the viewer control unit 123 displays the three-dimensional model viewed from the real viewpoint in the viewer 131 (S116A). For example, the viewer control unit 123 displays the three-dimensional model viewed from the real viewpoint closest to the selected virtual viewpoint in the viewer 131.

[0097] The above process limits the user's viewpoint operation to a virtual viewpoint that is close to the real viewpoint. This makes it possible to prevent the selection of a virtual viewpoint for which no corresponding frame exists. It also makes it possible to prevent a low-precision 3D model from being displayed. Furthermore, since cases where a 3D model is not displayed do not occur, user operability can be improved.

[0098] 13 to 15 may be combined. For example, if the difference in the position and / or line of sight direction between the virtual viewpoint and the real viewpoint is less than a first threshold, a three-dimensional model seen from the virtual viewpoint is displayed, if the difference is equal to or greater than the first threshold and less than a second threshold that is greater than the first threshold, a three-dimensional model seen from a real viewpoint (for example, a real viewpoint closest to the virtual viewpoint) is displayed, and if the difference is equal to or greater than the second threshold, a warning is displayed and the three-dimensional model may not be displayed.

[0099] Furthermore, the viewer control unit 123 may determine whether the selected virtual viewpoint is included in a first region near the real viewpoint in determining whether a real viewpoint close to the selected virtual viewpoint exists in step S114. Fig. 16 is a flowchart showing the processing of the display device 103 in this case. The processing shown in Fig. 16 differs from the processing shown in Fig. 14 in that step S114 is replaced with step S114A.

[0100] In this case, for example, the first region is determined at any timing after the multiple frames, the camera position and orientation information, and the 3D model are acquired. For example, the first region is a collection of 3D regions that are less than a predetermined distance from each of the multiple real viewpoints. Specifically, for example, the regions near each real viewpoint are determined using a method similar to the method used in step S104 described above, and the first region is composed of the multiple determined regions.

[0101] If the first coordinate system of the real viewpoint indicated by the camera position and orientation information is different from the second coordinate system of the 3D model and the virtual viewpoint, the viewer control unit 123 may determine the second region of the real viewpoint represented by the first coordinate system and convert the second region represented by the first coordinate system into the first region of the second coordinate system. This allows the viewer control unit 123 to determine whether the virtual viewpoint represented by the second coordinate system is included in the first region also represented by the second coordinate system.

[0102] Furthermore, the predetermined distance for defining the first region does not have to be uniform in multiple directions. For example, the distance in a first direction (e.g., a direction from the real viewpoint toward the three-dimensional model) may be greater than the distance in a second direction perpendicular to the first direction. Even if the virtual viewpoint is far from the real viewpoint in the direction from the real viewpoint toward the three-dimensional model, the user will hardly notice a difference between the display content of the three-dimensional model as seen from the virtual viewpoint and the display content of the frame captured from the real viewpoint.

[0103] 16, after a virtual viewpoint is selected (S113), the viewer control unit 123 determines whether the selected virtual viewpoint is included in the first region (S114A). If the selected virtual viewpoint is included in the first region (Yes in S114A), the viewer control unit 123 displays the three-dimensional model seen from the selected virtual viewpoint in the viewer 131 (S115A). On the other hand, if the selected virtual viewpoint is not included in the first region (No in S114A), a warning is displayed on the display unit 125, and the three-dimensional model is not displayed in the viewer 131 (S116).

[0104] In this case, the viewer control unit 123 may display a first area 171 on the viewer 131. Fig. 17 is a diagram showing a display example of the first area 171. This allows the user to grasp the first area 171 that can be selected as a virtual viewpoint, thereby improving user operability.

[0105] Alternatively, the viewer control unit 123 may display a second area other than the first area 171 in the viewer 131. Fig. 18 is a diagram showing a display example of the second area 172. This allows the user to know that the second area 172 cannot be selected as a virtual viewpoint, thereby improving user operability.

[0106] Furthermore, the process shown in FIG. 16 is an example in which modifications have been made to the process shown in FIG. 14, but similar modifications can also be applied to the examples shown in FIGS. 3, 13, 15, or other examples.

[0107] [4. Summary] As described above, the display device 103 (viewer control unit 123) according to this embodiment is a viewer control method for controlling the viewer 131, in which a first region (e.g., first region 171) around the real viewpoint of a frame generated by photographing an object from the real viewpoint is identified, and information relating to the first region is output to the viewer 131 (or the display unit 125), and the viewer 131 (or the display unit 125) displays a three-dimensional model of the object as seen from a virtual viewpoint within the first region (e.g., S115A in FIG. 16).

[0108] This allows the display device 103 to display a three-dimensional model viewed from a virtual viewpoint that is close to the real viewpoint on the viewer 131. Therefore, the display device 103 can align the virtual viewpoint with the real viewpoint while allowing the viewer 131 to select a virtual viewpoint with some degree of freedom.

[0109] For example, the information about the first region includes information indicating the center position of the first region and information indicating the distance from the center position to the outer edge of the first region. Alternatively, the information about the first region includes information about multiple positions indicating the outer edge of the first region. Note that the information about the first region is not limited to these pieces of information, and may be any information that can identify the first region.

[0110] For example, the display device 103 further controls the viewer 131 so that the viewer 131 does not display the three-dimensional model seen from a virtual viewpoint in a second area (e.g., second area 172) other than the first area. This allows the display device 103 to prohibit the setting of a virtual viewpoint far from the real viewpoint. The second area can also be represented by the same information as the first area.

[0111] For example, the actual viewpoint is represented in a first coordinate system, and the three-dimensional model is represented in a second coordinate system. The display device 103 further identifies a second region in the first coordinate system around the actual viewpoint, and converts the second region into a first region in the second coordinate system. Note that the second region can also be represented by the same information as the first region.

[0112] This allows the display device to determine whether the virtual viewpoint is included in the first area even when the coordinate systems of the real viewpoint and the three-dimensional model are different.

[0113] For example, the display device 103 further acquires a first position (virtual viewpoint) in the second coordinate system via an interface (e.g., UI unit 124), and if the first position is within the first area, controls the viewer 131 so that the viewer displays the three-dimensional model as seen from the first position.

[0114] This allows the display device 103 to display a three-dimensional model viewed from a virtual viewpoint that is close to the real viewpoint on the viewer 131. Therefore, the display device 103 can align the virtual viewpoint with the real viewpoint while allowing the viewer 131 to select a virtual viewpoint with some degree of freedom.

[0115] For example, the display device 103 further acquires a first position (virtual viewpoint) in the second coordinate system via an interface (e.g., the UI unit 124), and controls the viewer 131 so that the viewer 131 displays warning information if the first position is outside the first region (e.g., S116 in FIG. 16), where the warning information indicates that the three-dimensional model viewed from the first position will not be displayed. Note that the warning information may be displayed only when an operation input is being made to the UI unit 124. In this way, the display device 103 can prompt the user to input an appropriate position before the user's erroneous operation input is confirmed.

[0116] This allows the display device 103 to prohibit the setting of a virtual viewpoint specified by the user that is far from the real viewpoint.

[0117] For example, when the first position is outside the first region, the display device 103 further controls the viewer to display the three-dimensional model as seen from the real viewpoint (for example, S116A in FIG. 15).

[0118] According to this, when the virtual viewpoint specified by the user is far from the real viewpoint, for example, in the case of an erroneous operation by the user, the display device 103 can display a three-dimensional model seen from the real viewpoint instead of simply prohibiting it.

[0119] For example, the display device 103 further identifies an additional real viewpoint of an additional frame generated by photographing an object from the additional real viewpoint in the first coordinate system, acquires a first position in the second coordinate system via an interface (e.g., the UI unit 124), and controls the viewer 131 to display a three-dimensional model viewed from the real viewpoint if the first position is closer to the real viewpoint than the additional real viewpoint, and controls the viewer to display the three-dimensional model viewed from the additional real viewpoint if the first position is closer to the additional real viewpoint than the real viewpoint.In other words, the display device 103 controls the viewer 131 to display a three-dimensional model viewed from the real viewpoint closest to the first position among multiple real viewpoints of multiple frames.

[0120] This allows the display device 103 to display a three-dimensional model viewed from a real viewpoint that is close to the virtual viewpoint specified by the user when there are multiple frames, thereby improving operability when there are limitations on the viewpoint selection of the viewer 131.

[0121] 17, the display device 103 further controls the viewer 131 to display a first area 171. This allows the display device 103 to support the user in setting an appropriate virtual viewpoint.

[0122] 18, the display device 103 further controls the viewer 131 to display a second area 172 other than the first area 171, and the three-dimensional model viewed from the virtual viewpoint in the second area 172 is not displayed on the viewer. This allows the display device 103 to support the user in setting an appropriate virtual viewpoint.

[0123] For example, the first region includes a plurality of real viewpoints of a plurality of frames, including a frame, generated by photographing the object from each of the plurality of real viewpoints, and the three-dimensional model is generated by feature point matching using the plurality of frames.

[0124] This allows the display device 103 to set a virtual viewpoint close to the actual viewpoint of the multiple frames used for feature point matching. Here, a 3D model viewed from a viewpoint close to the actual viewpoint of the multiple frames used for feature point matching is likely to be highly accurate. Also, a 3D model viewed from a viewpoint far from the actual viewpoint is likely to have data loss. Therefore, the display device can display a highly accurate 3D model.

[0125] For example, the first region is a region within a predetermined distance from the real viewpoint. This allows the display device 103 to display on the viewer 131 a three-dimensional model seen from a virtual viewpoint that is close to the real viewpoint.

[0126] For example, the display device 103 further displays a frame 134. This allows the display device 103 to display a three-dimensional model viewed from a virtual viewpoint that is close to the actual viewpoint of the displayed frame 134. Therefore, the user can easily grasp the correspondence between the frame 134 and the three-dimensional model, and can therefore check the details of the object by referring to the frame even if the accuracy of the three-dimensional model is low.

[0127] For example, the first region includes a plurality of first real viewpoints of a plurality of first frames including a frame generated by photographing an object from each of a plurality of first real viewpoints including the real viewpoint. The display device 103 further acquires a designated viewpoint, which is a virtual viewpoint designated based on a user operation, searches for a second frame photographed from a first real viewpoint near the designated viewpoint among the plurality of first real viewpoints, and displays the second frame.

[0128] This allows the display device 103 to display a frame of a real viewpoint that is close to the viewpoint specified by the user. Therefore, the user can easily understand the correspondence between the frame and the three-dimensional model, and can check the details of the object by referring to the frame even if the accuracy of the three-dimensional model is low.

[0129] 11, the display device 103 further displays the first real viewpoint (real viewpoint 135) of the second frame on the viewer 131. This allows the display device 103 to improve the operability for the user.

[0130] For example, in searching for the second frame, a plurality of second frames including the second frame captured from a plurality of second real viewpoints that are a plurality of first real viewpoints near the specified viewpoint among a plurality of first real viewpoints are searched for, and in displaying the second frames, a plurality of second frames (for example, frames 134A and 134B shown in FIG. 9) are displayed, and the viewer 131 is controlled so that the viewpoint of the three-dimensional model displayed by the viewer 131 becomes the second real viewpoint of the second frame selected based on the user's operation from among the plurality of second frames (for example, FIG. 10).

[0131] This allows the display device 103 to display, on the viewer 131, a three-dimensional model viewed from the real viewpoint of a frame specified by the user, among a plurality of frames with real viewpoints close to the viewpoint specified by the user. This improves user operability.

[0132] 12, the display device 103 further displays, on the viewer 131, a plurality of three-dimensional models viewed from a plurality of viewpoints that have a predetermined relationship with the second real viewpoint of the selected second frame. This allows the display device 103 to display, on the viewer, a three-dimensional model viewed from a plurality of viewpoints that are related to the real viewpoint of the frame selected by the user. This improves user convenience.

[0133] For example, in searching for the second frame, the display device 103 projects a plurality of points included in the three-dimensional model seen from the specified viewpoint onto each of a plurality of first frames, and determines the second frame based on the number of points projected onto each first frame. This allows the display device 103 to appropriately determine a frame of a real viewpoint that is close to the viewpoint specified by the user.

[0134] The above processing may also be executed by an information processing device included in the display device 103 (for example, the viewer control unit 123).

[0135] For example, the display device or information processing device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0136] Although the display devices and the like according to the embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments.

[0137] Furthermore, each processing unit included in the display device according to the above-described embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0138] Furthermore, the integration is not limited to LSI, but may be realized by dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI fabrication, or reconfigurable processors, which allow the connections and settings of circuit cells within LSIs to be reconfigured, may also be used.

[0139] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0140] The present disclosure may also be realized as a viewer control method or the like executed by a display device or the like.

[0141] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0142] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Some of the steps may be executed simultaneously (in parallel) with other steps.

[0143] While the display devices according to one or more aspects have been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects. [Industrial Applicability]

[0144] The present disclosure can be applied to a display device that displays a viewer that displays a three-dimensional model. [Explanation of symbols]

[0145] 101 Imaging device 102 Three-dimensional model generation device 103 Display device 111 Camera position and orientation estimation unit 112 3D model generation unit 121 Acquisition Department 122 Storage section 123 Viewer control section 124 UI section 125 Display section 131 Viewers 132 Image search button 133 Viewpoint change button 134, 134A, 134B Frames 135 Real Viewpoint 141 Subject 142, 152 points 151, 161 frames 162 Similarities 163 Epipolar Lines 171 First area 172 Second area

Claims

1. A display method comprising: Identifying a first region around a real viewpoint in a frame generated by photographing an object from the real viewpoint; when a setting operation relating to a viewpoint is received from a user while the three-dimensional model of the object is being displayed, a virtual viewpoint is set within the first area based on the setting operation, and the three-dimensional model is displayed as seen from the virtual viewpoint; When the first position based on the setting operation is outside the first area, control is performed so that the three-dimensional model viewed from the real viewpoint is displayed. Display method.

2. The display method according to claim 1 , wherein the information about the first region includes a plurality of positions of the outer edge expressed in the coordinate system of the three-dimensional model, or a distance between a center position and the outer edge.

3. The display method according to claim 1 , wherein the first region is a region within a predetermined distance from the real viewpoint.

4. The display method according to claim 1 , wherein, when there are a plurality of real viewpoints, the display method displays the three-dimensional model as seen from the real viewpoint closest to the first position.

5. An information processing device comprising a processor and a memory, wherein the processor uses the memory to: Identifying a first region around a real viewpoint in a frame generated by photographing an object from the real viewpoint; when a setting operation relating to a viewpoint is received from a user while the three-dimensional model of the object is being displayed, a virtual viewpoint is set within the first area based on the setting operation, and the three-dimensional model is displayed as seen from the virtual viewpoint; When the first position based on the setting operation is outside the first area, control is performed so that the three-dimensional model viewed from the real viewpoint is displayed. Information processing device.

Images