Viewer control method and information processing device
By defining the area around the actual viewpoint in the viewer and outputting relevant information to display the virtual viewpoint, the problem of lack of freedom in virtual viewpoint selection is solved, achieving matching between the virtual viewpoint and the actual viewpoint and accurate display of the 3D model.
Patent Information
- Application Number
- CN202380032469.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-15
- Filing Date
- 2023-02-07
- Publication Date
- 2024-11-19
AI Technical Summary
In existing technologies, the selection of virtual viewpoints in the viewer lacks freedom and is difficult to match with the frame viewpoints of the actual image, resulting in inaccurate display of 3D models.
By defining a first region around the actual viewpoint, information related to that region is output to the viewer, displaying a 3D model viewed from a virtual viewpoint within that region, limiting the selection of virtual viewpoints far from the actual viewpoint, and assisting the user in setting an appropriate virtual viewpoint.
It enables free selection of virtual viewpoints in the viewer while ensuring the matching of virtual viewpoints with actual viewpoints, thereby improving the accuracy of 3D model display and user operability.
Smart Images

Figure CN118985010A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a viewer control method and an information processing device. Background Art
[0002] There is a technology that estimates the position and posture of a camera using a camera image as input and uses the result to generate a three-dimensional model (for example, see Patent Document 1).
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: International Publication No. 2021 / 0141775 Summary of the invention
[0006] Problems to be solved by the invention
[0007] In a conventional viewer that displays a three-dimensional model, it is possible to select an arbitrary virtual viewpoint. However, it is necessary to match the virtual viewpoint with the actual viewpoint of a frame that is an actual image without arbitrarily selecting the virtual viewpoint in the viewer. Therefore, an object of the present disclosure is to provide a viewer control method and an information processing device that can match the virtual viewpoint with the actual viewpoint of a frame that is an actual image while allowing freedom in the selection of the virtual viewpoint in the viewer.
[0008] Means for solving problems
[0009] A viewer control method according to one embodiment of the present disclosure is a viewer control method for controlling a viewer, wherein a first area around an actual viewpoint of a frame generated by photographing an object from the actual viewpoint is determined, information related to the first area is output to the viewer, and the viewer displays a three-dimensional model of the object observed from a virtual viewpoint within the first area.
[0010] Effects of the Invention
[0011] The present disclosure can provide a viewer control method and an information processing device capable of matching a virtual viewpoint with an actual viewpoint of a frame as an actual image while allowing freedom in selection of a virtual viewpoint in a viewer. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 This is a block diagram showing the structure of a three-dimensional model display system according to an embodiment.
[0013] Figure 2 It is a diagram for explaining the feature point matching process according to the embodiment.
[0014] Figure 3This is a flowchart showing the operation of the display device according to the embodiment.
[0015] Figure 4 It is a diagram showing a display example of the display unit according to the embodiment.
[0016] Figure 5 It is a diagram showing a display example after the viewpoint is changed according to the embodiment.
[0017] Figure 6 It is a diagram showing an example of image retrieval processing according to an embodiment.
[0018] Figure 7 It is a diagram showing a display example after image retrieval according to the embodiment.
[0019] Figure 8 It is a diagram showing a display example after the viewpoint is changed according to the embodiment.
[0020] Fig. 9 It is a diagram showing a display example of a case where a plurality of frames are displayed according to the embodiment.
[0021] Fig.10 It is a diagram showing an example of selection of a frame by a user according to the embodiment.
[0022] Fig.11 It is a diagram showing a display example of a case where an actual viewpoint is displayed according to the embodiment.
[0023] Fig.12 It is a diagram showing a display example of a plurality of three-dimensional models according to an embodiment.
[0024] Fig.13 This is a flowchart showing the operation of a modification example of the display device according to the embodiment.
[0025] Fig.14 This is a flowchart showing the operation of a modification example of the display device according to the embodiment.
[0026] Fig.15 This is a flowchart showing the operation of a modification example of the display device according to the embodiment.
[0027] Fig.16 This is a flowchart showing the operation of a modification example of the display device according to the embodiment.
[0028] Fig.17 It is a diagram showing a display example of the first area according to the embodiment.
[0029] Fig.18 It is a diagram showing a display example of the second area according to the embodiment. DETAILED DESCRIPTION
[0030] A viewer control method involved in one embodiment of the present disclosure is a viewer control method for controlling a viewer, wherein a first area around an actual viewpoint of a frame generated by photographing an object from the actual viewpoint is determined, information related to the first area is output to the viewer, and the viewer displays a three-dimensional model of the object observed from a virtual viewpoint within the first area.
[0031] Thus, the viewer control method can display a three-dimensional model observed from a virtual viewpoint close to the actual viewpoint on the viewer. Therefore, the viewer control method can make the virtual viewpoint match (close to) the actual viewpoint while allowing freedom in the selection of the virtual viewpoint in the viewer.
[0032] For example, the viewer control method may further control the viewer so that the viewer does not display the three-dimensional model observed from a virtual viewpoint in a second area outside the first area.
[0033] Thus, the viewer control method can prohibit setting a virtual viewpoint that is far away from the actual viewpoint.
[0034] For example, the actual viewpoint may be represented by a first coordinate system, the three-dimensional model may be represented by a second coordinate system, and the viewer control method may further determine a second area around the actual viewpoint in the first coordinate system, and convert the second area into the first area in the second coordinate system.
[0035] Thus, even when the actual viewpoint is different from the three-dimensional model coordinate system, the viewer control method can determine whether the virtual viewpoint is included in the first area.
[0036] For example, the viewer control method may further perform the following processing: acquiring a first position in the second coordinate system via an interface, and when the first position is within the first area, controlling the viewer so that the viewer displays the three-dimensional model observed from the first position.
[0037] Thus, the viewer control method can display a three-dimensional model observed from a virtual viewpoint close to the actual viewpoint on the viewer. Therefore, the viewer control method can match the virtual viewpoint with the actual viewpoint while allowing freedom in the selection of the virtual viewpoint in the viewer.
[0038] For example, the viewer control method may further perform the following processing: acquiring a first position in the second coordinate system via an interface, and when the first position is outside the first area, controlling the viewer so that the viewer displays a warning message, wherein the warning message indicates that the three-dimensional model observed from the first position is not displayed.
[0039] Thus, the viewer control method can prohibit setting a user-specified virtual viewpoint that is far from an actual viewpoint.
[0040] For example, the viewer control method may further control the viewer to display the three-dimensional model viewed from the actual viewpoint when the first position is outside the first area.
[0041] Therefore, when the virtual viewpoint designated by the user is far from the actual viewpoint, for example, in the case of a user's erroneous operation, the viewer control method can display the three-dimensional model observed from the actual viewpoint instead of simply prohibiting it.
[0042] For example, the viewer control method may further perform the following processing: determine the additional actual viewpoint of an additional frame generated by photographing the object from the additional actual viewpoint in the first coordinate system, obtain a first position in the second coordinate system via an interface, and when the first position is closer to the actual viewpoint than the additional actual viewpoint, control the viewer to display the three-dimensional model observed from the actual viewpoint; and when the first position is closer to the additional actual viewpoint than the actual viewpoint, control the viewer to display the three-dimensional model observed from the additional actual viewpoint.
[0043] Thus, when there are multiple frames, the viewer control method can display a three-dimensional model viewed from an actual viewpoint close to a virtual viewpoint designated by a user. Thus, when there are limitations on viewpoint selection in the viewer, operability can be improved.
[0044] For example, the viewer control method may further control the viewer to display the first area.
[0045] Thus, the viewer control method can assist the user in setting an appropriate virtual viewpoint.
[0046] For example, the viewer control method may further control the viewer to display a second area in addition to the first area, and the three-dimensional model viewed from a virtual viewpoint in the second area is not displayed on the viewer.
[0047] Thus, the viewer control method can assist the user in setting an appropriate virtual viewpoint.
[0048] For example, the first area may include a plurality of actual viewpoints including a plurality of frames generated by respectively photographing the object from a plurality of actual viewpoints including the actual viewpoint, and the three-dimensional model may be generated by matching feature points using the plurality of frames.
[0049] Thus, the viewer control method can set a virtual viewpoint close to the actual viewpoint of the plurality of frames used for feature point matching. Here, the three-dimensional model observed from a viewpoint close to the actual viewpoint of the plurality of frames used for feature point matching is likely to have high accuracy. In addition, the three-dimensional model observed from a viewpoint far from the actual viewpoint is likely to have data loss. Therefore, the display device can display a high-precision three-dimensional model.
[0050] For example, the first area may be an area within a predetermined distance from the actual viewpoint.
[0051] Thus, the viewer control method can display, on the viewer, a three-dimensional model observed from a virtual viewpoint close to an actual viewpoint.
[0052] For example, the viewer control method may further display the frame.
[0053] Thus, the viewer control method can display a three-dimensional model observed from a virtual viewpoint close to the actual viewpoint of the displayed frame. Therefore, the user can easily grasp the correspondence between the frame and the three-dimensional model, and can confirm the details of the object by referring to the frame even if the accuracy of the three-dimensional model is low.
[0054] For example, the first area may include multiple first actual viewpoints including multiple first frames of the frame generated by respectively photographing the object from multiple first actual viewpoints including the actual viewpoint, and the viewer control method further performs the following processing: obtaining a virtual viewpoint specified based on a user's operation, that is, a specified viewpoint, searching for a second frame shot from a first actual viewpoint near the specified viewpoint among the multiple first actual viewpoints, and displaying the second frame.
[0055] Thus, the viewer control method can display a frame of an actual viewpoint close to the viewpoint specified by the user. Therefore, the user can easily understand the correspondence between the frame and the three-dimensional model, and can confirm the details of the object by referring to the frame even if the accuracy of the three-dimensional model is low.
[0056] For example, the viewer control method may further cause the viewer to display the first actual viewpoint of the second frame.
[0057] Therefore, the viewer control method can improve the operability of the user.
[0058] For example, in the search for the second frame, a plurality of second frames including the second frame shot from a plurality of first actual viewpoints near the designated viewpoint among a plurality of first actual viewpoints, that is, a plurality of second actual viewpoints, are searched, and in the display of the second frame, the plurality of second frames are displayed, and the viewer is controlled so that the viewpoint of the three-dimensional model displayed by the viewer becomes the second actual viewpoint of the second frame selected based on the user's operation among the plurality of second frames.
[0059] Thus, the viewer control method can display on the viewer a three-dimensional model viewed from the actual viewpoint of a frame designated by the user among a plurality of frames of actual viewpoints near the viewpoint designated by the user. Thus, the operability of the user can be improved.
[0060] For example, the viewer control method may further cause the viewer to display a plurality of the three-dimensional models observed from a plurality of viewpoints having a predetermined relationship with the second actual viewpoint of the selected second frame.
[0061] Thus, the viewer control method can display, on the viewer, a three-dimensional model observed from a plurality of viewpoints associated with an actual viewpoint of a frame selected by a user, thereby improving user convenience.
[0062] For example, in the search for the second frame, a plurality of points included in the three-dimensional model observed from the designated viewpoint may be projected onto each of the plurality of first frames, and the second frame may be determined based on the number of points projected onto each first frame.
[0063] Thus, the viewer control method can appropriately determine a frame of an actual viewpoint close to the viewpoint designated by the user.
[0064] In addition, the information processing device according to an embodiment of the present disclosure is an information processing device for controlling a viewer, comprising: a processor; and a memory, wherein the processor uses the memory to perform the following processing: determining a first area around an actual viewpoint of a frame generated by photographing an object from the actual viewpoint, outputting information related to the first area to the viewer, and the viewer displays a three-dimensional model of the object observed from a virtual viewpoint within the first area.
[0065] Thus, the information processing device can display a three-dimensional model viewed from a virtual viewpoint close to an actual viewpoint on the viewer. Therefore, the viewer control method can match the virtual viewpoint with the actual viewpoint while allowing freedom in selecting the virtual viewpoint in the viewer.
[0066] In addition, these general or specific methods can be implemented by systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, and recording media.
[0067] Hereinafter, the embodiments are described in detail with reference to the accompanying drawings. In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, materials, components, configuration positions of components, connection methods, steps, order of steps, etc. shown in the following embodiments are examples and do not limit the present disclosure. In addition, among the components in the following embodiments, the components not described in the independent claims are described as arbitrary components.
[0068] (Implementation Method)
[0069] [1.Structure]
[0070] First, the configuration of the three-dimensional model display system according to the present embodiment will be described. Figure 1 1 is a block diagram showing the structure of the three-dimensional model display system of this embodiment. The three-dimensional model display system is a system for displaying a three-dimensional model on a viewer, and includes a plurality of imaging devices 101 , a three-dimensional model generating device 102 , and a display device 103 .
[0071] The plurality of imaging devices 101 (cameras) capture a subject (object) from different viewpoints, and output the captured frames to the three-dimensional model generation device 102. The captured frames are also referred to as multi-viewpoint images. In other words, a frame is an image.
[0072] The plurality of frames do not necessarily need to be captured by a plurality of imaging devices 101, and may be captured by a single imaging device 101 while moving. That is, the plurality of frames are captured (generated) by the imaging device 101 from at least one different viewpoint in position and posture.
[0073] The three-dimensional model generation device 102 has a camera position and posture estimation unit 111 and a three-dimensional model generation unit 112. The camera position and posture estimation unit 111 uses a plurality of frames obtained by a plurality of camera devices 101 to generate camera position and posture information indicating the position and posture (also referred to as viewpoint) of the camera device 101 when the frame was captured. Here, the posture of the camera device 101 indicates at least one of the shooting direction of the camera device 101 and the tilt of the camera device 101. The shooting direction of the camera device 101 is the direction of the optical axis of the camera device 101. The tilt of the camera device 101 refers to the rotation angle of the camera device 101 around the optical axis from the reference posture.
[0074] Specifically, the camera position and posture estimation unit 111 estimates the camera parameters of the plurality of cameras 101 based on the plurality of frames acquired from the plurality of cameras 101. Here, the camera parameters are parameters representing the characteristics of the camera 101, including internal parameters such as the focal length and image center of the camera 101 and external parameters representing the position (more specifically, the three-dimensional position) and posture of the camera 101.
[0075] In addition, the camera position and attitude estimation unit 111 estimates the position and attitude of the camera device 101. The camera position and attitude estimation unit 111 can use, for example, Visual-SLAM (Simultaneous Localization and Mapping), Structure-From-Motion, or ICP (Iterative Closest Point) to estimate the position and attitude of multiple camera devices 101. Alternatively, the position and attitude of multiple camera devices 101 can also be input by the user.
[0076] Specifically, the camera position and posture estimation unit 111 performs feature point matching processing on multiple frames captured by multiple camera devices 101. That is, the camera position and posture estimation unit 111 extracts features of multiple frames and extracts a group of similar points that are similar between multiple frames among the extracted multiple feature points. Then, the camera position and posture estimation unit 111 estimates the position and posture of the camera device 101 using the extracted group of similar points.
[0077] The 3D model generation unit 112 generates a 3D model of the subject based on multiple frames and camera position and posture information (camera parameters). For example, the 3D model generation unit 112 generates the 3D model using MVS (multi-view stereo). First, the 3D model generation unit 112 performs feature point matching processing. Figure 2 is a diagram for explaining feature point matching processing. Figure 2 As shown, a similar point 162 on frame 161 similar to point 152 (pixel) on frame 151 is searched. Specifically, regarding point 152 in frame 151, when matching with frame 161 is performed through search range R1, in frame 161, an epipolar line 163 corresponding to straight line L1 passing through viewpoint V1 and point 152 exists across the entire end of frame 161. In addition, frame 151 is an image obtained at viewpoint V1, and frame 161 is an image obtained at viewpoint V2. Straight line L1 coincides with the shooting direction of camera 101 located at viewpoint V1. Point 152 corresponds to point 142 of subject 141.
[0078] As shown in the following formula 1, the three-dimensional model generation unit 112 calculates the normalized cross correlation (NCC) between small regions in a combination of frame I and frame J as N(I, J), and generates matching information indicating the result of matching between frames.
[0079] [Formula 1]
[0080]
[0081] In addition, Ixy and Jxy are pixel values in small areas of frame I and frame J.
[0082] [Formula 2]
[0083]
[0084] as well as
[0085] [Formula 3]
[0086]
[0087] are the average values of the pixel values in the small areas of frame I and frame J respectively.
[0088] Then, the three-dimensional model generation unit 112 generates a three-dimensional model by performing triangulation using the search results in the matching.
[0089] The display device 103 displays the three-dimensional model on the viewer using the camera position and posture information and the three-dimensional model generated by the three-dimensional model generation device 102. For example, the display device 103 is a tablet terminal, a smartphone, or a personal computer.
[0090] In addition, the communication between the multiple camera devices 101 and the three-dimensional model generation device 102 and the display device 103 can be performed by any method such as wired communication or wireless communication. In addition, these communications can be performed directly between devices, or indirectly via other communication devices or servers. In addition, the sending and receiving of multiple frames, camera position and posture information, and three-dimensional models can be performed in real time, or can be recorded in an external storage device such as a memory or a cloud server, and then output from these external storage devices to the three-dimensional model generation device 102 or the display device 103.
[0091] In addition, the method for generating multiple frames, camera position and posture information, and a three-dimensional model is an example, and multiple frames, camera position and posture information, and a three-dimensional model are not limited to the above description, and can also be generated by any method. For example, in addition to or instead of multiple frames, a depth image (depth map) obtained by a depth sensor or the like, or three-dimensional information obtained by a laser measuring device such as a three-dimensional laser measuring device (LiDAR) or a millimeter-wave radar measuring device can be used.
[0092] In addition, the three-dimensional model includes position information representing multiple three-dimensional positions on the subject. For example, the three-dimensional model is a three-dimensional point group (point cloud) including multiple three-dimensional points representing multiple three-dimensional positions. In addition, each three-dimensional point may also have attribute information such as color or reflectivity. In addition, the three-dimensional model is not limited to a three-dimensional point group, but may also be a collection of three-dimensional grids, etc.
[0093] The display device 103 includes an acquisition section 121 , a storage section 122 , a viewer control section 123 , a UI section 124 , and a display section 125 .
[0094] The acquisition unit 121 acquires (receives) a plurality of frames captured by the plurality of imaging devices 101, camera position and posture information, and a three-dimensional model from the three-dimensional model generation device 102. The acquisition unit 121 is, for example, a communication interface for communicating with the three-dimensional model generation device 102.
[0095] The storage unit 122 stores a plurality of frames, camera position and posture information, and a three-dimensional model acquired by the acquisition unit 121. In addition, the storage unit 122 may also store processing results of the processing unit of the display device 103. In addition, the storage unit 122 may also store, for example, a control program for causing the processing circuit (processor) to execute the processing of each processing unit of the display device 103. The storage unit 122 is implemented, for example, by a HDD (Hard Disk Drive), a flash memory, or the like.
[0096] The UI unit 124 is a user interface for acquiring user operations, such as a touch panel, a keyboard, a mouse, etc. The display unit 125 is a display for displaying images. The viewer control unit 123 displays a viewer on the display unit 125, and the viewer displays the three-dimensional model when the three-dimensional model is viewed from a virtual viewpoint.
[0097] In addition, the viewer control unit 123 , the UI unit 124 , and the display unit 125 may not be included in a single device. In addition, the display device 103 may have a part or all of the processing units included in the three-dimensional model generation device 102 .
[0098] [2.Action]
[0099] Next, the operation of the display device 103 will be described. Figure 31 is a flowchart showing the operation of the display device 103. First, the acquisition unit 121 acquires a plurality of frames, camera position and posture information, and a three-dimensional model from the three-dimensional model generation device 102 (S101). The storage unit 122 stores the acquired plurality of frames, camera position and posture information, and three-dimensional model. In addition, the storage unit 122 may store information for identifying the camera 101 that captured the frame as frame header information.
[0100] Next, the viewer control unit 123 displays the viewer 131 on the display unit 125 , and the viewer 131 displays a three-dimensional model viewed from a certain viewpoint ( S102 ). Figure 4 It is a diagram showing a display example of the display unit 125.
[0101] Next, a virtual viewpoint is selected according to the user's operation on the UI unit 124 (S103). For example, the user selects a virtual viewpoint by tapping or sliding on the viewer 131. Specifically, the three-dimensional model is enlarged or reduced, rotated, and parallelized. In addition, the user's viewpoint operation method is not limited to this, and any known method such as operating an operation button or an operation bar displayed on the display unit 125 can also be used. Figure 5 A diagram showing a display example after the viewpoint is changed.
[0102] Next, the viewer control unit 123 searches for a frame of a viewpoint close to the virtual viewpoint of the three-dimensional model displayed on the viewer (S104). Figure 5 The image search button 132 shown in the display unit 125 is used to search for images. In addition, the operation of starting the image search is not limited to this, and any method can be used. In addition, the image search can be automatically performed without the user's operation. For example, the image search can be performed when the user does not perform the operation of changing the viewpoint for a certain period of time.
[0103] Figure 6 1 is a diagram showing an example of image retrieval processing. The viewer control unit 123 compares the positions and postures of the plurality of camera devices 101 represented by the camera position and posture information stored in the storage unit 122, that is, the plurality of actual viewpoints with the current virtual viewpoint, and retrieves an actual viewpoint close to the virtual viewpoint. Figure 6 In the example shown, the actual viewpoint B that is close to the virtual viewpoint is selected among the actual viewpoints A, B, and C.
[0104] Specifically, the viewer control unit 123 retrieves an actual viewpoint close to the virtual viewpoint based on the difference in position (e.g., xyz coordinates) between the virtual viewpoint and the actual viewpoint and the difference in the line of sight direction (shooting direction) between the virtual viewpoint and the actual viewpoint. For example, the viewer control unit 123 extracts an actual viewpoint whose difference in position is less than a predetermined first threshold and whose difference in line of sight direction is less than a predetermined second threshold from a plurality of actual viewpoints. For example, the Euclidean distance may be used as the difference in position, and the cosine similarity may be used as the difference in line of sight direction. In addition, the viewer control unit 123 may filter the actual viewpoints by the difference in line of sight direction, and may determine the difference in position for the filtered actual viewpoints. In addition, the viewer control unit 123 may not compare the difference in position and the difference in line of sight direction with the thresholds separately, but may compare the value obtained by weighted addition of the difference in position and the difference in line of sight direction with the threshold.
[0105] In addition, the viewer control unit 123 may use only the difference in position instead of the difference in line of sight direction. Here, the multiple frames are, for example, multiple frames used to generate a three-dimensional model of the subject, and are multiple frames obtained by photographing the subject from multiple directions. Therefore, the multiple line of sight directions of multiple actual viewpoints are roughly in the direction of the subject. In addition, since the selected virtual viewpoint is also in the direction of the three-dimensional model (subject), when the position of the virtual viewpoint is close to that of the actual viewpoint, there is a high possibility that the line of sight directions are also close. Therefore, even if only the position is used, a certain degree of accuracy can be ensured.
[0106] In addition, in the determination of the position, different thresholds may be used in the depth direction (the direction connecting the viewpoint and the subject) and in the up, down, left, and right directions (directions orthogonal to the depth direction). For example, in the case where the depth position (the distance to the subject) is different and in the case where the up, down, left, and right directions are different, it is easier to compare the three-dimensional model and the frame in the case where the depth position is different. Therefore, the threshold in the depth direction may be smaller than the threshold in the up, down, left, and right directions. In addition, the threshold in the depth direction may be larger than the threshold in the up, down, left, and right directions. Furthermore, different thresholds may be used in the up, down, left, and right directions. In addition, the setting of the threshold for each of these directions may be switched according to the subject or the intended use, or may be switched according to the user's settings.
[0107] In addition, as another method, the viewer control unit 123 may project a three-dimensional model observed from a virtual viewpoint, that is, a plurality of three-dimensional points constituting the three-dimensional model displayed on the viewer, onto each of a plurality of frames, and extract a frame having the largest number of projected three-dimensional points or a frame having a number of projected three-dimensional points greater than a threshold as a frame of an actual viewpoint close to the virtual viewpoint.
[0108] The viewer control unit 123 displays the extracted frame of the actual viewpoint on the display unit 125 ( S105 ). Figure 7 is a diagram showing an example of display after image retrieval. Figure 7 As shown, a frame 134 of an actual viewpoint close to the virtual viewpoint of the three-dimensional model displayed on the viewer 131 is displayed on the display unit 125 .
[0109] Next, the viewer control unit 123 changes the virtual viewpoint of the three-dimensional model to the actual viewpoint of the displayed frame (S106). That is, the viewer control unit 123 displays the three-dimensional model observed from the actual viewpoint of the frame 134 on the viewer 131. For example, by the user tapping Figure 7 The viewpoint can be changed by pressing the viewpoint change button 133 displayed on the display unit 125. The viewpoint change operation is not limited to this, and any method can be used. In addition, the viewpoint change can also be performed automatically without the user's operation.
[0110] Figure 8 is a diagram showing a display example after the viewpoint is changed. Figure 8 As shown, the virtual viewpoint of the three-dimensional model displayed on the viewer 131 is changed to the actual viewpoint of the frame 134.
[0111] In addition, the changed virtual viewpoint may not be completely consistent with the actual viewpoint, but may be a viewpoint near the actual viewpoint. Here, the nearby viewpoint is, for example, a viewpoint whose position difference and line of sight direction difference with the actual viewpoint are less than a predetermined threshold.
[0112] If the display process of the three-dimensional model is not completed (S107: No), if the virtual viewpoint is changed by the next operation of the user (S103), the process after step S104 is performed. That is, the process of steps S103 to S106 is repeated. If the display process of the three-dimensional model is completed (S107: Yes), for example, if the user performs an operation to end the process, the viewer control unit 123 ends the process.
[0113] As described above, the display device 103 of the present embodiment displays a frame of an actual viewpoint close to the virtual viewpoint of the displayed three-dimensional model. Thus, when the resolution of the three-dimensional model is insufficient, the user can confirm the detailed structure of the subject by referring to the frame. In addition, the display device 103 can combine the frame with the viewpoint of the three-dimensional model. Thus, the user can easily recognize the corresponding relationship between the frame and the three-dimensional model. Therefore, the convenience of the user can be improved.
[0114] [3. Modifications]
[0115] Furthermore, in step S104 , in the case where there are a plurality of actual viewpoints close to the virtual viewpoint, a plurality of frames may be displayed in step S105 . Fig. 9 is a diagram showing an example display of this situation. Fig. 9 As shown, the display unit 125 displays a plurality of frames 134A and 134B of a plurality of actual viewpoints close to the virtual viewpoints.
[0116] Also in this case, in step S106 , the user selects a frame corresponding to the actual viewpoint set for the virtual viewpoint from among the plurality of frames. Fig.10 is a diagram showing an example of user selection. Fig.10 In the example shown, for example, the user selects frame 134A by tapping frame 134A. In addition, the three-dimensional model observed from the actual viewpoint of frame 134A is displayed on viewer 131. Fig.10 In the illustrated state, by tapping the frame 134B, the viewpoint of the three-dimensional model displayed on the viewer 131 can be changed to the actual viewpoint of the frame 134B.
[0117] In this case, the virtual viewpoint of the three-dimensional model may be automatically changed instead of the user selecting the frame 134A or 134B. For example, the three-dimensional model observed from the actual viewpoint of the frame 134A and the three-dimensional model observed from the actual viewpoint of the frame 134B may be displayed alternately at a predetermined interval. Alternatively, the display unit 125 may display an additional viewer, and the three-dimensional model observed from the actual viewpoint of the frame 134A and the three-dimensional model observed from the actual viewpoint of the frame 134B may be displayed on the viewer 131 and the additional viewer, respectively.
[0118] Furthermore, in addition to the three-dimensional model, the actual viewpoint of the frame 134 may be displayed on the viewer 131 . Fig.11 is a diagram showing an example display of this situation. Fig.11 As shown in FIG. 1 , the display unit 125 displays the actual viewpoint 135 of the frame 134 on the viewer 131. In this case, the user can change the viewpoint of the three-dimensional model displayed on the viewer to the actual viewpoint 135 (i.e., the viewpoint of the frame 134) by selecting (e.g., tapping) the actual viewpoint 135. Fig.10 When a plurality of frames 134A and 134B are displayed as in the illustrated example, the actual viewpoint corresponding to the frame 134A and the actual viewpoint corresponding to the frame 134B may be displayed on the viewer 131 .
[0119] Furthermore, in step S106 , when the viewpoint is changed, in addition to the three-dimensional model viewed from the actual viewpoint of the frame 134 , a plurality of three-dimensional models viewed from a plurality of viewpoints having a predetermined relationship with the actual viewpoint may be displayed. Fig.12 FIG. 4 is a diagram showing an example of a display of this situation. Fig.12In the example shown, in addition to displaying the three-dimensional model observed from the actual viewpoint of the frame 134, an image of the three-dimensional model enlarged and an image of the three-dimensional model reduced are displayed. In addition, without limitation to enlargement and reduction, a three-dimensional model observed from the actual viewpoint of the frame 134 may be displayed which is rotated in a predetermined direction or which is parallel moved. In addition, an image displayed orthogonally or multiple images displayed orthogonally from multiple directions (upward, horizontal, downward, etc.) may be displayed.
[0120] In the above description, image retrieval ( S103 ) and viewpoint change ( S106 ) are performed based on user operations, but they may be performed automatically. Fig.13 This is a flowchart showing the processing of the display device 103 in this case.
[0121] Steps S111 to S113 are, for example, Figure 3 The steps S101 to S103 shown are the same. After the virtual viewpoint is selected (S113), the viewer control unit 123 determines whether there is a frame of the actual viewpoint close to the selected virtual viewpoint (whether it is stored in the storage unit 122) (S114). In addition, the method of determining the actual viewpoint close to the virtual viewpoint is the same as the method used in the above-mentioned step S104, for example.
[0122] In addition, when the first coordinate system of the actual viewpoint represented by the camera position and posture information is different from the second coordinate system of the three-dimensional model and the virtual viewpoint, the viewer control unit 123 may also convert the actual viewpoint represented by the first coordinate system into the second coordinate system, and determine the actual viewpoint close to the virtual viewpoint based on the actual viewpoint of the second coordinate system and the virtual viewpoint of the second coordinate system. Alternatively, the viewer control unit 123 may also transform the virtual viewpoint represented by the second coordinate system into the first coordinate system, and determine the actual viewpoint close to the virtual viewpoint based on the virtual viewpoint of the first coordinate system and the actual viewpoint of the first coordinate system.
[0123] If there is a frame of an actual viewpoint close to the selected virtual viewpoint (S114: "Yes"), the viewer control unit 123 displays the three-dimensional model observed from the actual viewpoint on the viewer 131 (S115). In addition, the viewer control unit 123 displays the frame taken from the actual viewpoint on the display unit 125. In addition, the viewer control unit 123 may not display the frame on the display unit 125. In addition, in the case where there are multiple actual viewpoints close to the virtual viewpoint, the viewer control unit 123 may display the three-dimensional model observed from the actual viewpoint closest to the virtual viewpoint on the viewer 131, for example.
[0124] On the other hand, if there is no frame of the actual viewpoint close to the selected virtual viewpoint (S114: "No"), the viewer control unit 123 displays a warning on the display unit 125 and does not display the three-dimensional model on the viewer 131 (S116). The warning indicates, for example, that the three-dimensional model is not displayed because there is no frame shot from the actual viewpoint close to the virtual viewpoint. In addition, the viewer control unit 123 may display the three-dimensional model observed from the selected virtual viewpoint on the viewer 131, while displaying a warning on the display unit 125 indicating that there is no frame shot from the actual viewpoint close to the virtual viewpoint.
[0125] If the display process of the three-dimensional model has not been completed (S117: No), if the virtual viewpoint is changed by the next operation of the user (S113), the process after step S114 is performed. That is, the process of steps S113 to S116 is repeated. If the display process of the three-dimensional model is completed (S117: Yes), the viewer control unit 123 ends the process.
[0126] Through the above processing, the three-dimensional model is displayed when the virtual viewpoint is close to the actual viewpoint, but the three-dimensional model is not displayed when the virtual viewpoint is far from the actual viewpoint. That is, the user's viewpoint operation is limited to a virtual viewpoint close to the actual viewpoint. As a result, it is possible to prohibit the selection of a virtual viewpoint for which there is no corresponding frame. In addition, when a three-dimensional model is generated using multiple frames, the accuracy of the three-dimensional model observed from a virtual viewpoint for which there is no corresponding frame, that is, a virtual viewpoint far from the actual viewpoint, may be low. Therefore, through the above processing, it is possible to suppress the display of a three-dimensional model with low accuracy.
[0127] In addition, in the case where there is an actual viewpoint close to the virtual viewpoint, a three-dimensional model observed from the virtual viewpoint may be displayed on the viewer 131 . Fig.14 This is a flowchart showing the processing of the display device 103 in this case. Fig.14 The processing shown is similar to Fig.13 The processing shown is different in that step S115 is replaced by step S115A. If there is an actual viewpoint close to the selected virtual viewpoint (S114: Yes), the viewer control unit 123 displays the three-dimensional model viewed from the selected virtual viewpoint on the viewer 131 (S115A).
[0128] Through the above processing, the user's viewpoint operation is limited to a virtual viewpoint close to the actual viewpoint. This can prevent the selection of a virtual viewpoint for which there is no corresponding frame. In addition, the display of a low-precision three-dimensional model can be suppressed. Furthermore, since a virtual viewpoint that is not completely consistent with the actual viewpoint can be set, Fig.13 Compared with the processing shown, the restrictions on viewpoint operation can be relaxed.
[0129] Furthermore, in the case where there is no actual viewpoint close to the virtual viewpoint, a three-dimensional model observed from the actual viewpoint may be displayed on the viewer 131 . Fig.15 This is a flowchart showing the processing of the display device 103 in this case. Fig.15 The processing shown is similar to Fig.14 The processing shown is different in that step S116 is replaced by step S116A. In the case where there is no actual viewpoint close to the selected virtual viewpoint (S11: "No"), the viewer control unit 123 displays the three-dimensional model observed from the actual viewpoint on the viewer 131 (S116A). For example, the viewer control unit 123 displays the three-dimensional model observed from the actual viewpoint closest to the selected virtual viewpoint on the viewer 131.
[0130] Through the above processing, the user's viewpoint operation is limited to a virtual viewpoint close to the actual viewpoint. As a result, it is possible to prohibit the selection of a virtual viewpoint for which there is no corresponding frame. In addition, it is possible to suppress the display of a low-precision three-dimensional model. Furthermore, since the situation where the three-dimensional model is not displayed does not occur, the user's operability can be improved.
[0131] In addition, you can also combine Figure 13 to Figure 15 For example, when the difference between the position and / or the sight direction of the virtual viewpoint and the actual viewpoint is less than a first threshold, the three-dimensional model observed from the virtual viewpoint can be displayed; when the difference is greater than or equal to the first threshold and less than a second threshold that is greater than the first threshold, the three-dimensional model observed from the actual viewpoint (e.g., the actual viewpoint closest to the virtual viewpoint) can be displayed; when the difference is greater than or equal to the second threshold, a warning is displayed and the three-dimensional model is not displayed.
[0132] Furthermore, as the determination in step S114 whether there is an actual viewpoint close to the selected virtual viewpoint, the viewer control section 123 may determine whether the selected virtual viewpoint is included in the first area near the actual viewpoint. Fig.16 This is a flowchart showing the processing of the display device 103 in this case. Fig.16 The processing shown is similar to Fig.14 The processing shown is different in that step S114 is replaced with step S114A.
[0133] In this case, for example, the first area is determined at any timing after acquiring multiple frames, camera position and posture information, and a three-dimensional model. For example, the first area is a set of three-dimensional areas that are less than a specified distance from each of the multiple actual viewpoints. Specifically, for example, the area near each actual viewpoint is determined by the same method as the method used in the above-mentioned step S104, and the first area is composed of the determined multiple areas.
[0134] In addition, when the first coordinate system of the actual viewpoint represented by the camera position and posture information is different from the second coordinate system of the three-dimensional model and the virtual viewpoint, the viewer control unit 123 may also determine the second area of the actual viewpoint represented by the first coordinate system and convert the second area represented by the first coordinate system into the first area of the second coordinate system. Therefore, the viewer control unit 123 can determine whether the virtual viewpoint represented by the second coordinate system is included in the first area represented by the second coordinate system.
[0135] In addition, the prescribed distance for specifying the first area may also be non-uniform in multiple directions. For example, the distance in a first direction (e.g., the direction from the actual viewpoint toward the three-dimensional model) may be greater than the distance in a second direction orthogonal to the first direction. Even if the position of the virtual viewpoint deviates from the actual viewpoint in the direction from the actual viewpoint toward the three-dimensional model, it is difficult for the user to feel the difference between the display content of the three-dimensional model observed from the virtual viewpoint and the display content of the frame captured from the actual viewpoint.
[0136] like Fig.16 As shown, after selecting a virtual viewpoint (S113), the viewer control unit 123 determines whether the selected virtual viewpoint is included in the first area (S114A). If the selected virtual viewpoint is included in the first area (S114A: "Yes"), the viewer control unit 123 displays the three-dimensional model observed from the selected virtual viewpoint on the viewer 131 (S115A). On the other hand, in the case where the selected virtual viewpoint is not included in the first area (S114A: "No"), a warning is displayed on the display unit 125, and the three-dimensional model is not displayed on the viewer 131 (S116).
[0137] In addition, in this case, the viewer control section 123 may display the first area 171 on the viewer 131 . Fig.17 2 is a diagram showing a display example of the first region 171. Thus, the user can understand the first region 171 that can be selected as a virtual viewpoint, and thus the user's operability can be improved.
[0138] In addition, the viewer control section 123 may display a second area other than the first area 171 on the viewer 131 . Fig.18 2 is a diagram showing a display example of the second area 172. As a result, the user can understand the second area 172 that cannot be selected as a virtual viewpoint, and thus the user's operability can be improved.
[0139] in addition, Fig.16 The treatment shown is for Fig.14 The processing shown is an example of applying changes, but for Figure 3 , Fig.13 , Fig.15 Or other examples can also apply the same changes.
[0140] [4. Conclusion]
[0141] As described above, the display device 103 (viewer control unit 123) of the present embodiment is a viewer control method for controlling the viewer 131, determining a first area (for example, the first area 171) around an actual viewpoint of a frame generated by photographing an object from the actual viewpoint, outputting information related to the first area to the viewer 131 (or the display unit 125), and the viewer 131 (or the display unit 125) displays a three-dimensional model of the object observed from a virtual viewpoint in the first area (for example, the first area 171). Fig.16 S115A).
[0142] Thus, the display device 103 can display the three-dimensional model viewed from a virtual viewpoint close to the actual viewpoint on the viewer 131. Therefore, the display device 103 can match the virtual viewpoint with the actual viewpoint while providing freedom in selecting the virtual viewpoint in the viewer 131.
[0143] For example, the information related to the first area includes information indicating the center position of the first area and information indicating the distance from the center position to the outer edge of the first area. Alternatively, the information related to the first area includes information indicating multiple positions of the outer edge of the first area. In addition, the information related to the first area is not limited to this information, as long as it can identify the first area.
[0144] For example, the display device 103 further controls the viewer 131 so that the viewer 131 does not display the three-dimensional model observed from the virtual viewpoint in the second area (for example, the second area 172) other than the first area. In this way, the display device 103 can prohibit the setting of the virtual viewpoint far from the actual viewpoint. In addition, the second area can also be represented by the same information as the first area.
[0145] For example, the actual viewpoint is represented by the first coordinate system, and the three-dimensional model is represented by the second coordinate system. The display device 103 further determines a second area around the actual viewpoint in the first coordinate system, and converts the second area into the first area in the second coordinate system. In addition, the second area can also be represented by the same information as the first area.
[0146] Thus, even when the actual viewpoint and the three-dimensional model coordinate system are different, the display device can determine whether the virtual viewpoint is included in the first area.
[0147] For example, the display device 103 also obtains the first position (virtual viewpoint) in the second coordinate system via an interface (such as the UI unit 124), and when the first position is within the first area, controls the viewer 131 so that the viewer displays the three-dimensional model observed from the first position.
[0148] Thus, the display device 103 can display the three-dimensional model viewed from a virtual viewpoint close to the actual viewpoint on the viewer 131. Therefore, the display device 103 can match the virtual viewpoint with the actual viewpoint while providing freedom in selecting the virtual viewpoint in the viewer 131.
[0149] For example, the display device 103 further obtains the first position (virtual viewpoint) in the second coordinate system via the interface (e.g., the UI unit 124), and controls the viewer 131 to display a warning message (e.g., Fig.16 In S116), the warning information indicates that the three-dimensional model observed from the first position is not displayed. In addition, the warning information can also be displayed only when an operation input is made to the UI unit 124. Thus, the display device 103 can prompt the user to make an appropriate position input before the user's erroneous operation input is confirmed.
[0150] This allows the display device 103 to prohibit the setting of a virtual viewpoint designated by the user that is far from the actual viewpoint.
[0151] For example, when the first position is outside the first area, the display device 103 further controls the viewer to display the three-dimensional model observed from the actual viewpoint (eg, Fig.15 S116A).
[0152] Thus, when the virtual viewpoint designated by the user is far from the actual viewpoint, for example, in the case of a user's erroneous operation, the display device 103 can display the three-dimensional model observed from the actual viewpoint instead of simply prohibiting it.
[0153] For example, the display device 103 further determines an additional actual viewpoint of an additional frame generated by photographing an object from an additional actual viewpoint in the first coordinate system, obtains a first position in the second coordinate system via an interface (e.g., the UI unit 124), and controls the viewer 131 to display the three-dimensional model observed from the actual viewpoint when the first position is closer to the actual viewpoint than the additional actual viewpoint, and controls the viewer to display the three-dimensional model observed from the additional actual viewpoint when the first position is closer to the additional actual viewpoint than the actual viewpoint, that is, the display device 103 controls the viewer 131 to display the three-dimensional model observed from the actual viewpoint closest to the first position among multiple actual viewpoints of multiple frames.
[0154] Thus, the display device 103 can display a three-dimensional model viewed from an actual viewpoint close to a virtual viewpoint designated by the user when there are a plurality of frames. Therefore, when the viewpoint selection of the viewer 131 is limited, operability can be improved.
[0155] For example, Fig.17 As shown, the display device 103 further controls the viewer 131 to display the first area 171. Thus, the display device 103 can assist the user in setting an appropriate virtual viewpoint.
[0156] For example, Fig.18 As shown, the display device 103 further controls the viewer 131 to display the second area 172 outside the first area 171, and the three-dimensional model observed from the virtual viewpoint in the second area 172 is not displayed on the viewer. Thus, the display device 103 can assist the user in setting an appropriate virtual viewpoint.
[0157] For example, the first area includes a plurality of actual viewpoints including a plurality of frames generated by photographing the object from a plurality of actual viewpoints including the actual viewpoint, respectively, and the three-dimensional model is generated by matching feature points using the plurality of frames.
[0158] Thus, the display device 103 can set a virtual viewpoint close to the actual viewpoint of the multiple frames used for feature point matching. Here, the three-dimensional model observed from a viewpoint close to the actual viewpoint of the multiple frames used for feature point matching is likely to have high accuracy. In addition, the three-dimensional model observed from a viewpoint far from the actual viewpoint is likely to have data defects. Therefore, the display device can display a high-precision three-dimensional model.
[0159] For example, the first area is an area within a predetermined distance from the actual viewpoint. Thus, the display device 103 can display the three-dimensional model viewed from a virtual viewpoint close to the actual viewpoint on the viewer 131 .
[0160] For example, the display device 103 also displays the frame 134. Thus, the display device 103 can display the three-dimensional model observed from a virtual viewpoint close to the actual viewpoint of the displayed frame 134. Therefore, the user can easily understand the correspondence between the frame 134 and the three-dimensional model, and can confirm the details of the target object by referring to the frame even when the accuracy of the three-dimensional model is low.
[0161] For example, the first area includes a plurality of first actual viewpoints including a plurality of first frames of a frame generated by photographing an object from a plurality of first actual viewpoints including an actual viewpoint. The display device 103 also obtains a virtual viewpoint designated based on a user's operation, that is, a designated viewpoint, searches for a second frame photographed from a first actual viewpoint near the designated viewpoint among the plurality of first actual viewpoints, and displays the second frame.
[0162] Thus, the display device 103 can display a frame of an actual viewpoint close to the viewpoint designated by the user. Therefore, the user can easily understand the correspondence between the frame and the 3D model, and can confirm the details of the object by referring to the frame even if the accuracy of the 3D model is low.
[0163] For example, Fig.11 As shown, the display device 103 also causes the viewer 131 to display the first actual viewpoint (actual viewpoint 135) of the second frame. Thus, the display device 103 can improve the operability of the user.
[0164] For example, in the search for the second frame, a plurality of second frames including the second frame shot from a plurality of first actual viewpoints near a designated viewpoint among the plurality of first actual viewpoints are searched, and in the display of the second frame, the plurality of second frames (e.g. Fig. 9 The viewer 131 is controlled so that the viewpoint of the three-dimensional model displayed by the viewer 131 becomes the second actual viewpoint of the second frame selected based on the user's operation among the plurality of second frames (for example, Fig.10 ).
[0165] Thus, the display device 103 can display the three-dimensional model observed from the actual viewpoint of the frame designated by the user among a plurality of frames of the actual viewpoint close to the viewpoint designated by the user on the viewer 131. Thus, the operability for the user can be improved.
[0166] For example, Fig.12 As shown, the display device 103 further causes the viewer 131 to display a plurality of three-dimensional models observed from a plurality of viewpoints having a predetermined relationship with the second actual viewpoint of the selected second frame. Thus, the display device 103 can display the three-dimensional models observed from a plurality of viewpoints having a correlation with the actual viewpoint of the frame selected by the user on the viewer. Therefore, the convenience of the user can be improved.
[0167] For example, in the search for the second frame, the display device 103 projects a plurality of points included in the three-dimensional model observed from the designated viewpoint onto a plurality of first frames, respectively, and determines the second frame based on the number of points projected onto each first frame. Thus, the display device 103 can appropriately determine the frame of the actual viewpoint close to the viewpoint designated by the user.
[0168] Furthermore, the above-described processing may be performed by an information processing device (for example, the viewer control section 123 ) included in the display device 103 .
[0169] For example, a display device or an information processing device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.
[0170] As mentioned above, although the display device etc. which concern on embodiment of this disclosure are demonstrated, this disclosure is not limited to this embodiment.
[0171] In addition, each processing unit included in the display device and the like of the above-mentioned embodiment is typically implemented as an LSI, which is an integrated circuit. These may be implemented as a single chip individually, or may be implemented as a single chip in a manner that a part or all of them are included.
[0172] In addition, the integrated circuit is not limited to LSI, and can also be realized by a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI.
[0173] In addition, in each of the above embodiments, each component may be formed by dedicated hardware, or implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.
[0174] Furthermore, the present disclosure may be implemented as a viewer control method or the like performed by a display device or the like.
[0175] In addition, the division of the functional blocks in the block diagram is an example, and multiple functional blocks can also be implemented as one functional block, or one functional block can be divided into multiple blocks, or a part of the function can be moved to other functional blocks. In addition, a single hardware or software can process the functions of multiple functional blocks with similar functions in parallel or time-sharing.
[0176] In addition, the order of executing each step in the flowchart is an illustrative step for specifically explaining the present disclosure, and may be an order other than the above. In addition, part of the above steps may be executed simultaneously (in parallel) with other steps.
[0177] In the above, one or more display devices and the like are described based on the embodiments, but the present disclosure is not limited to the embodiments. As long as it does not deviate from the main purpose of the present disclosure, various modifications that a person skilled in the art may think of may be implemented in the present embodiment, and the method of combining the constituent elements of different embodiments may also be included in the scope of one or more embodiments.
[0178] Industrial Applicability
[0179] The present disclosure can be applied to a display device or the like that displays a viewer for displaying a three-dimensional model.
[0180] Description of symbols
[0181] 101 Camera Device
[0182] 102 Three-dimensional model generation device
[0183] 103 Display device
[0184] 111 Camera Position and Attitude Estimation Unit
[0185] 112 3D Model Generation Department
[0186] 121 Acquisition Department
[0187] 122 Storage
[0188] 123 Viewer Control
[0189] 124 UI Department
[0190] 125 Display
[0191] 131 Viewer
[0192] 132 Image search button
[0193] 133 Viewpoint change button
[0194] 134, 134A, 134B frames
[0195] 135 Actual Viewpoint
[0196] 141 Subject
[0197] 142, 152 points
[0198] 151, 161 frames
[0199] 162 Similarities
[0200] 163 Epipolar Line
[0201] 171 First Area
[0202] 172 Second Area
Claims
1. A viewer control method, controlling a viewer, wherein: determining a first area around an actual viewpoint of a frame generated by photographing an object from the actual viewpoint, outputting information related to the first area to the viewer, The viewer displays a three-dimensional model of the object viewed from a virtual viewpoint within the first area.
2. The viewer control method according to claim 1, wherein: The viewer control method further controls the viewer so that the viewer does not display the three-dimensional model observed from a virtual viewpoint in a second area outside the first area.
3. The viewer control method according to claim 1, wherein: The actual viewpoint is represented by a first coordinate system, The three-dimensional model is represented by a second coordinate system, The viewer control method further determines a second area around the actual viewpoint in the first coordinate system, and converts the second area into the first area in the second coordinate system.
4. The viewer control method according to claim 3, wherein: The viewer control method further performs the following processing: acquiring the first position in the second coordinate system via the interface, In a case where the first position is within the first area, the viewer is controlled so that the viewer displays the three-dimensional model viewed from the first position.
5. The viewer control method according to claim 3, wherein: The viewer control method further performs the following processing: acquiring the first position in the second coordinate system via the interface, In a case where the first position is outside the first area, the viewer is controlled so that the viewer displays warning information indicating that the three-dimensional model viewed from the first position is not displayed.
6. The viewer control method according to claim 4, wherein: The viewer control method further controls the viewer to display the three-dimensional model observed from the actual viewpoint when the first position is outside the first area.
7. The viewer control method according to claim 3, wherein: The viewer control method further performs the following processing: determining an additional actual viewpoint of an additional frame generated by photographing the object from the additional actual viewpoint in the first coordinate system, acquiring the first position in the second coordinate system via the interface, controlling the viewer to display the three-dimensional model observed from the actual viewpoint if the first position is closer to the actual viewpoint than the additional actual viewpoint, In a case where the first position is closer to the additional actual viewpoint than the actual viewpoint, the viewer is controlled to display the three-dimensional model observed from the additional actual viewpoint.
8. The viewer control method according to claim 1, wherein: The viewer control method further controls the viewer to display the first area.
9. The viewer control method according to claim 1, wherein: The viewer control method further controls the viewer to display a second area in addition to the first area, The three-dimensional model observed from a virtual viewpoint within the second area is not displayed on the viewer.
10. The viewer control method according to claim 1, wherein: the first area includes a plurality of actual viewpoints including a plurality of frames of the frame generated by photographing the object from a plurality of actual viewpoints including the actual viewpoint, respectively, The three-dimensional model is generated by matching feature points using the plurality of frames.
11. The viewer control method according to claim 1, wherein: The first area is an area within a predetermined distance from the actual viewpoint.
12. The viewer control method according to claim 1, wherein: The viewer control method further displays the frame.
13. The viewer control method according to claim 1, wherein: the first area includes a plurality of first actual viewpoints including a plurality of first frames of the frame generated by photographing the object from a plurality of first actual viewpoints including the actual viewpoint, respectively, The viewer control method further performs the following processing: Obtaining a virtual viewpoint designated based on a user's operation, that is, a designated viewpoint, searching for a second frame shot from a first actual viewpoint near the designated viewpoint among the plurality of first actual viewpoints, The second frame is displayed.
14. The viewer control method according to claim 13, wherein: The viewer control method further causes the viewer to display the first actual viewpoint of the second frame.
15. The viewer control method according to claim 13, wherein: In searching for the second frame, a plurality of second frames including the second frame shot from a plurality of first actual viewpoints near the designated viewpoint among a plurality of first actual viewpoints, that is, a plurality of second actual viewpoints, are searched. In displaying the second frame, displaying the plurality of second frames, The viewer is controlled so that the viewpoint of the three-dimensional model displayed by the viewer becomes a second actual viewpoint of a second frame selected based on a user's operation among the plurality of second frames.
16. The viewer control method according to claim 15, wherein: The viewer control method further causes the viewer to display a plurality of the three-dimensional models observed from a plurality of viewpoints having a predetermined relationship with the second actual viewpoint of the selected second frame.
17. The viewer control method according to claim 13, wherein: In searching for the second frame, a plurality of points included in the three-dimensional model viewed from the designated viewpoint are projected onto each of the plurality of first frames, and the second frame is determined based on the number of points projected onto each first frame.
18. An information processing device for controlling a viewer, comprising: Processor; and Memory, The processor uses the memory to perform the following processing: determining a first area around an actual viewpoint of a frame generated by photographing an object from the actual viewpoint, outputting information related to the first area to the viewer, The viewer displays a three-dimensional model of the object viewed from a virtual viewpoint within the first area.
