Terminal, information processing apparatus, display method, and information processing method

Abstract

To provide a terminal or the like capable of appropriately displaying a three-dimensional model.SOLUTION: A terminal 100 is provided, comprising: a first sensor 101 which sequentially acquires first sensor information; a second sensor 102 which sequentially acquires second sensor information; and a circuit 103 which is connected to the first sensor and the second sensor. The circuit 103 transmits, to an information processing apparatus, the first sensor information acquired at a first time by the first sensor 101, acquires from the information processing apparatus a first terminal position at the first time of the terminal estimated by the information processing apparatus on the basis of the first sensor information acquired at the first time, and a three-dimensional model generated by the information processing apparatus on the basis of the first sensor information acquired at the first time, estimates a second terminal position at a second time on the basis of the first terminal position and the second sensor information, the second time being the time when estimation of the second terminal position is executed, and displays the acquired three-dimensional model on the basis of the second terminal position.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a terminal, an information processing apparatus, a display method, and an information processing method.

Background Art

[0002] Patent Document 1 discloses a technique for estimating the position and orientation of a plurality of imaging devices (i.e., cameras) by performing feature point matching on a plurality of images obtained by the plurality of imaging devices.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a terminal or the like that can appropriately display a three-dimensional model.

Means for Solving the Problems

[0005] A terminal according to one aspect of the present disclosure is a terminal, comprising: a first sensor that sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal; a second sensor that sequentially acquires second sensor information for estimating the amount of change in the position and orientation of the terminal; and a circuit connected to the first sensor and the second sensor. The circuit transmits the first sensor information acquired by the first sensor at a first time to an information processing device, and acquires from the information processing device the first terminal position of the terminal at the first time estimated by the information processing device based on the first sensor information acquired at the first time, and a three-dimensional model generated by the information processing device based on the first sensor information acquired at the first time. Based on the first terminal position and the second sensor information, a second terminal position at a second time is estimated, where the second time is the time when the estimation of the second terminal position is being executed. Based on the second terminal position, the acquired three-dimensional model is displayed.

[0006] An information processing device according to one aspect of the present disclosure includes a memory and a circuit connected to the memory. The circuit acquires from the terminal the first sensor information acquired at a first time among the first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal, which is sequentially acquired by a first sensor provided in the terminal. Based on the first sensor information acquired at the first time, the first terminal position of the terminal at the first time is estimated, and a three-dimensional model is generated based on the first sensor information acquired at the first time. The first terminal position, the three-dimensional model, and time information for specifying the first time are transmitted to the terminal.

[0007] Note that these general or specific aspects may be implemented by a system, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a method, a system, a device, an integrated circuit, a computer program, and a non-transitory recording medium.

Advantages of the Invention

[0008] A terminal or the like in the present disclosure can appropriately display a three-dimensional model.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0010] (Findings on which the present disclosure is based) The present inventors have found that the following problems occur with the conventional system described in the "Background Art" section.

[0011] In a technique such as Patent Document 1, it is disclosed that shooting may be performed so as to generate a multi-viewpoint image including a plurality of images with different viewpoints in one imaging device while moving the one imaging device.

[0012] By the way, the present inventors generate a three-dimensional model in real time based on a multi-viewpoint image obtained while moving one terminal as in the prior art, and superimpose the three-dimensional model on the image being shot, so that the user can easily visually recognize the location where the three-dimensional model is generated in the space of the shooting target. It has been found that there is a need for such a device.

[0013] However, the terminal often has insufficient processing power. Therefore, in order to reduce the processing load, it is conceivable to pass a plurality of images to an information processing device with sufficient processing power and let the information processing device generate a three-dimensional model with a large processing load. In this way, if the device for taking images and the device for generating the three-dimensional model are separate devices, it is necessary to communicate between the devices. When the terminal transmits the captured image to the information processing device in real time and acquires and displays the three-dimensional model generated by the information processing device, a time lag occurs from when the image is captured until the generated three-dimensional model is displayed. For this reason, there is a high possibility that the terminal moves during the time lag. If the terminal attempts to directly display the three-dimensional model generated based on a plurality of images captured at a previous time for the image captured at the current time, since the viewpoint at the time of capturing the image and the viewpoint at the time of capturing the image that is the basis of the three-dimensional model are different, there is a high possibility that the three-dimensional model is displayed at a position shifted from the object in the image.

[0014] Therefore, the inventors have found a terminal or the like that can appropriately display a three-dimensional model.

[0015] A terminal according to a first aspect of the present disclosure is a terminal, comprising: a first sensor that sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal; a second sensor that sequentially acquires second sensor information for estimating the amount of change in the position and orientation of the terminal; and a circuit connected to the first sensor and the second sensor, wherein the circuit transmits the first sensor information acquired by the first sensor at a first time to an information processing device, and acquires from the information processing device the first terminal position of the terminal at the first time estimated by the information processing device based on the first sensor information acquired at the first time and the three-dimensional model generated by the information processing device based on the first sensor information acquired at the first time, estimates a second terminal position at a second time based on the first terminal position and the second sensor information, the second time being the time when the estimation of the second terminal position is being executed, and displays the acquired three-dimensional model based on the second terminal position.

[0016] According to this, the terminal estimates a second terminal position, which is the position of the terminal at a second time, based on a first terminal position, which is the position of the terminal at a first time, and second sensor information, and displays a three-dimensional model based on the estimated second terminal position. For this reason, for example, a three-dimensional model with the second terminal position as the viewpoint can be displayed. Therefore, the terminal can display a three-dimensional model corresponding to the second terminal position at the second time, and can appropriately display the three-dimensional model.

[0017] The terminal according to the second aspect of the present disclosure is the terminal according to the first aspect, wherein the circuit estimates the first terminal position based on the second sensor information in addition to the first sensor information.

[0018] According to this, since the terminal further estimates the first terminal position based on the second sensor information, the first terminal position can be estimated with higher accuracy.

[0019] The terminal according to the third aspect of the present disclosure is the terminal according to the first aspect or the second aspect, wherein the accuracy of the first terminal position is higher than the accuracy of the second terminal position.

[0020] According to this, since the terminal estimates the second terminal position based on the highly accurate first terminal position, the processing load can be reduced.

[0021] The terminal according to the fourth aspect of the present disclosure is the terminal according to any one of the first aspect to the third aspect, wherein in the estimation of the second terminal position, the circuit estimates a change amount of the position and orientation of the terminal from the first time to the second time based on the second sensor information acquired from the first time to the second time, and estimates the second terminal position based on the first terminal position and the change amount.

[0022] Therefore, the terminal can reduce the processing load for estimating the second terminal position, which is the position of the terminal at the second time, in order to estimate the second terminal position at the second time based on the amount of change in the position and orientation of the terminal from the first time to the second time with reference to the first terminal position.

[0023] The terminal according to the fifth aspect of the present disclosure is a terminal according to any one of the first aspect to the fourth aspect, and the second sensor information includes the acceleration of the terminal and the angular velocity of the terminal.

[0024] The terminal according to the sixth aspect of the present disclosure is a terminal according to any one of the first aspect to the fifth aspect, and the circuit further acquires time information for specifying the first time together with the first terminal position and the three-dimensional model, and estimates the second terminal position based on the time information.

[0025] Therefore, the terminal can easily specify the first time corresponding to the first terminal position and the three-dimensional model using the time information. Thus, the terminal can easily estimate, for example, the amount of change in the position and orientation of the terminal from the first time to the second time based on the second sensor information.

[0026] The terminal according to the seventh aspect of the present disclosure is a terminal according to any one of the first aspect to the sixth aspect, the first sensor information includes a plurality of images taken at different positions and orientations, the circuit acquires the images included in the first sensor information acquired at the second time by the first sensor, and superimposes and displays the three-dimensional model on the images based on the second terminal position.

[0027] Therefore, the terminal can superimpose and display the three-dimensional model when viewed from the same viewpoint as the viewpoint when the image was taken in order to display the three-dimensional model with the second terminal position as the viewpoint for the image taken at the second time. Thus, the terminal can display the three-dimensional model corresponding to the second terminal position at the second time and can appropriately display the three-dimensional model.

[0028] The terminal according to the eighth aspect of the present disclosure is the terminal according to the seventh aspect, wherein the three-dimensional model superimposed on the image is displayed in a manner different from the image.

[0029] Therefore, the terminal can display the image and the three-dimensional model in an easily distinguishable manner, so that the user can easily confirm the progress of the three-dimensional model generation by looking at the image and the three-dimensional model displayed on the terminal.

[0030] The information processing apparatus according to the ninth aspect of the present disclosure includes a memory and a circuit connected to the memory. The circuit is first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal, and among the first sensor information sequentially acquired by the first sensor included in the terminal, the first sensor information acquired at the first time is acquired from the terminal, the first terminal position of the terminal at the first time is estimated based on the first sensor information acquired at the first time, a three-dimensional model is generated based on the first sensor information acquired at the first time, and the first terminal position, the three-dimensional model, and time information for specifying the first time are transmitted to the terminal.

[0031] According to this, the information processing apparatus transmits to the terminal the first terminal position, the three-dimensional model, and time information for specifying the first time corresponding to the first terminal position and the three-dimensional model. Therefore, the terminal can easily specify the first time corresponding to the first terminal position and the three-dimensional model. Thus, the terminal can easily estimate, for example, the amount of change in the position and orientation of the terminal from the first time to the second time based on the second sensor information. That is, a three-dimensional model corresponding to the second terminal position at the second time can be displayed on the terminal, and the three-dimensional model can be appropriately displayed.

[0032] The display method according to the tenth aspect of the present disclosure is a display method executed by a terminal. It includes: transmitting to an information processing device the first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal, which is the first sensor information sequentially acquired by a first sensor provided in the terminal and acquired at a first time; acquiring from the information processing device the first terminal position of the terminal at the first time estimated by the information processing device based on the first sensor information acquired at the first time, and the three-dimensional model generated by the information processing device based on the first sensor information acquired at the first time; estimating a second terminal position at a second time based on the first terminal position and second sensor information for estimating the amount of change in the position and orientation of the terminal, which is the second sensor information sequentially acquired by a second sensor provided in the terminal, where the second time is when the estimation of the second terminal position is being executed; and displaying the acquired three-dimensional model based on the second terminal position.

[0033] According to this, the terminal estimates the second terminal position, which is the position of the terminal at the second time, based on the first terminal position, which is the position of the terminal at the first time, and the second sensor information, and displays the three-dimensional model based on the estimated second terminal position. Therefore, for example, a three-dimensional model with the second terminal position as the viewpoint can be displayed. Thus, the terminal can display a three-dimensional model corresponding to the second terminal position at the second time, and can appropriately display the three-dimensional model.

[0034] The information processing method according to the eleventh aspect of the present disclosure is an information processing method executed by an information processing device. It includes: acquiring from the terminal the first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal, which is the first sensor information sequentially acquired by a first sensor provided in the terminal and acquired at a first time; estimating the first terminal position of the terminal at the first time based on the first sensor information acquired at the first time; generating a three-dimensional model based on the first sensor information acquired at the first time; and transmitting the first terminal position, the three-dimensional model, and time information for specifying the first time to the terminal.

[0035] According to this, since the information processing apparatus transmits the time information for specifying a specific first time corresponding to the first terminal position and the three-dimensional model to the terminal together with the first terminal position and the three-dimensional model, the terminal can easily specify the first time corresponding to the first terminal position and the three-dimensional model. Therefore, the terminal can easily estimate, for example, the amount of change in the position and orientation of the terminal from the first time to the second time based on the second sensor information. That is, the terminal can display the three-dimensional model corresponding to the second terminal position at the second time, and the three-dimensional model can be appropriately displayed.

[0036] Note that these general or specific aspects may be realized by a system, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized by any combination of a method, a system, an apparatus, an integrated circuit, a computer program, and a non-transitory recording medium.

[0037] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, a more detailed description than necessary may be omitted. For example, a detailed description of well-known matters or a redundant description of substantially the same configuration may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art.

[0038] Note that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0039] (Embodiment) Hereinafter, embodiments will be described with reference to FIGS. 1 to 4.

[0040] [Configuration] FIG. 1 is a diagram for explaining an example of the configuration of a system according to an embodiment.

[0041] System 1 includes a terminal 100 and an information processing apparatus 200. In System 1, the terminal 100 senses an object 10 using sensors included therein, and the information processing apparatus 200 generates a three-dimensional model of the object 10 using the sensing result. The terminal 100 performs sensing, for example, while being moved by a user U1. The terminal 100 in the present disclosure displays the progress of the sensing while executing the sensing for generating a three-dimensional model of the object 10. The information processing apparatus 200 acquires the sensing result from the terminal 100, generates a three-dimensional model based on the sensing result, and transmits the three-dimensional model to the terminal 100 in order to cause the terminal 100 to display the progress.

[0042] The terminal 100 and the information processing apparatus 200 are communicably connected to each other via a communication network 50. The terminal 100 may be communicably connected to the communication network 50 via a wireless LAN (Local Area Network), or may be communicably connected via a mobile phone communication network. The terminal 100 is, for example, a smartphone, a tablet terminal, a PC (Personal Computer), or the like. The information processing apparatus 200 is communicably connected to the communication network 50 via a wired LAN (Local Area Network). Note that the information processing apparatus 200 may be communicably connected to the communication network 50 via a wireless LAN.

[0043] FIG. 2 is a block diagram showing an example of the configuration of the system according to the embodiment.

[0044] The configuration of the terminal 100 will be described.

[0045] The terminal 100 includes a first sensor 101, a second sensor 102, a circuit 103, a display unit 104, and a storage unit 105.

[0046] The first sensor 101 is a sensor that sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal 100. The first sensor 101 is, for example, a camera. The camera may be, for example, an RGB camera or a monochrome camera. The first sensor information is an image. The image may be, for example, a color image having RGB sub-pixels or a monochrome image. The first sensor information includes a plurality of images captured by the camera that is the first sensor 101 at different positions and orientations.

[0047] The second sensor 102 is a sensor that sequentially acquires second sensor information for estimating the amount of change in the position of the terminal 100 and the amount of change in the orientation. The second sensor 102 has, for example, an acceleration sensor that acquires the acceleration of the terminal 100 and an angular velocity sensor that acquires the angular velocity of the terminal 100. That is, the second sensor information includes the acceleration of the terminal 100 and the angular velocity of the terminal 100. The acceleration sensor is, for example, a three-axis acceleration sensor that acquires the acceleration in each of the three-axis directions of the terminal 100. The three-axis directions are the X-axis direction, the Y-axis direction, and the Z-axis direction that are orthogonal to each other. The angular velocity sensor is, for example, a three-axis angular velocity sensor that acquires the angular velocity around each axis in the three-axis directions of the terminal 100. The second sensor 102 may further have a geomagnetic sensor.

[0048] The circuit 103 has a communication unit 111, an estimation unit 112, and a display control unit 113.

[0049] The communication unit 111 exchanges information with the information processing device 200 via the communication network 50. Specifically, the communication unit 111 transmits the first sensor information sequentially acquired by the first sensor 101 to the information processing device 200. Also, the communication unit 111 may transmit the second sensor information sequentially acquired by the second sensor 102 to the information processing device 200. The communication unit 111 receives (acquires) the first terminal position and the three-dimensional model from the information processing device 200. The first terminal position is the position and orientation of the terminal 100 at the first time estimated by the information processing device 200 based on the first sensor information acquired at the first time. Note that the first terminal position may be estimated based on the first sensor information and the second sensor information acquired at the first time. The three-dimensional model is a three-dimensional model generated by the information processing device 200 based on the first sensor information acquired at the first time. Also, the communication unit 111 may receive (acquire) time information for specifying the first time together with the first terminal position and the three-dimensional model. The time information may be any information for specifying the first time, may be information indicating the first time, may be information for specifying the first sensor information corresponding to the first time, or may be information for specifying the second sensor information corresponding to the first time.

[0050] The estimation unit 112 estimates a second terminal position, which is the position and orientation of the terminal 100 at the second time, based on the first terminal position and the second sensor information acquired via the communication unit 111. The second time is the time when the estimation of the second terminal position is being executed. The second time is a time later than the first time. The estimation unit 112 identifies a first time, which is the time corresponding to the first terminal position and the three-dimensional model acquired together with the acquired time information, based on the acquired time information. Specifically, the estimation unit 112 estimates the amount of change in the position and orientation of the terminal 100 from the first time to the second time based on the second sensor information acquired by the second sensor 102 from the first time to the second time. Then, the estimation unit 112 estimates the second terminal position based on the first terminal position and the estimated amount of change. That is, the estimation unit 112 estimates the second terminal position, which is the position of the terminal 100 at the second time, by adding the amount of change in the position and orientation of the terminal 100 from the first time to the second time to the first terminal position, which is the position of the terminal 100 at the first time.

[0051] The display control unit 113 displays the three-dimensional model acquired via the communication unit 111 on the display unit 104 based on the second terminal position acquired via the communication unit 111. The display control unit 113 acquires an image included in the first sensor information acquired by the first sensor 101 at the second time, that is, an image taken at the second time.

[0052] The display control unit 113 superimposes the three-dimensional model on the image acquired at the second time and displays it on the display unit 104 based on the second terminal position estimated by the estimation unit 112. The display control unit 113 superimposes and displays on the display unit 104 a three-dimensional model that can be seen when the sensing direction of the first sensor 101 of the terminal 100 at the second terminal position is taken as the line-of-sight direction. The display control unit 113 identifies the rotation matrix and the translation vector in Equation 1 based on the second terminal position, and identifies the two-dimensional position (two-dimensional coordinates) on the image on which the three-dimensional model is to be superimposed using Equation 1 to which the identified rotation matrix and translation vector are applied. Then, the display control unit 113 superimposes the three-dimensional model at the identified two-dimensional position.

[0053] [Number]

[0054] Here, u and v indicate two-dimensional positions (two-dimensional coordinates) on the image. u indicates the vertical position, and v indicates the horizontal position. fx and fy indicate the focal points of the camera as the first sensor 101. fx and fy are constants. cx and cy indicate the centers of the camera as the first sensor 101. cx and cy are constants. X, Y, and Z indicate three-dimensional positions (three-dimensional coordinates) of the three-dimensional model to be superimposed. Equation 1 is, for example, an equation applied to each of a plurality of three-dimensional positions to specify the two-dimensional position on the image corresponding to each three-dimensional position. r11 to r33 indicate a rotation matrix (i.e., the pose of the camera) among the positions of the camera as the first sensor 101. t1 to t3 indicate a translation vector among the positions of the camera as the first sensor 101. The first terminal position and the second terminal position can be represented by three-dimensional positions using r11 to r33 and t1 to t3 of Equation 1. Note that the three-dimensional position can also be represented by other methods such as quaternion, but this is omitted.

[0055] In addition, the display control unit 113 displays the three-dimensional model in a mode different from the image. That is, the three-dimensional model superimposed on the image is displayed in a mode different from the image.

[0056] The display unit 104 displays at least one of the image and the three-dimensional model based on the control instruction from the display control unit 113. The display unit 104 is, for example, a display.

[0057] The storage unit 105 stores at least one of the first sensor information sequentially acquired by the first sensor 101 and the second sensor information sequentially acquired by the second sensor 102. The storage unit 105 stores the first sensor information together with the time when the first sensor information was acquired. The storage unit 105 stores the second sensor information together with the time when the second sensor information was acquired. The storage unit 105 may store the first terminal position and the three-dimensional model acquired from the information processing device 200. The storage unit 105 is, for example, a storage, a memory, etc.

[0058] Next, the configuration of the information processing apparatus 200 will be described.

[0059] The information processing apparatus 200 includes a circuit 201 and a storage unit 202.

[0060] The circuit 201 includes a communication unit 211, an estimation unit 212, and a generation unit 213.

[0061] The communication unit 211 exchanges information with the terminal 100 via the communication network 50. Specifically, the communication unit 211 receives (acquires) first sensor information from the terminal 100. For example, the communication unit 211 acquires the first sensor information acquired at the first time from the terminal 100. Further, the communication unit 211 may further receive (acquire) second sensor information from the terminal 100. In addition, the communication unit 211 transmits the first terminal position, the three-dimensional model, and the time information to the terminal 100. The first terminal position is the position of the terminal 100 at the first time estimated by the information processing apparatus 200 based on the first sensor information acquired at the first time. The three-dimensional model is a three-dimensional model generated by the information processing apparatus 200 based on the first sensor information acquired at the first time. The time information is information for specifying the first time.

[0062] The estimation unit 212 estimates the first terminal position of the terminal 100 at the first time based on the first sensor information acquired at the first time. For the sake of simplicity of explanation, first, an estimation method using only the first sensor information will be described. The estimation unit 212 identifies one or more feature points in each of the plurality of images included in the first sensor information. Then, the estimation unit 212 matches, for example, one or more feature points included in one of two images among the plurality of images with one or more feature points included in the other image, and identifies a pair of one or more feature points (hereinafter referred to as a feature point pair). The two images include the first image acquired at the first time. The estimation unit 212 identifies a three-dimensional position (three-dimensional point) corresponding to the feature point pair for each of the one or more feature point pairs of the two images. The estimation unit 212 estimates the first terminal position, which is the position of the terminal 100 at the first time when the first image was acquired, and the position of the terminal 100 when the other image was acquired, based on the one or more three-dimensional positions. For example, SfM (Structure from Motion) is used for the position estimation by the estimation unit 212. When the second sensor information is not used for the estimation of the first terminal position, the communication unit 211 may be designed not to receive the second sensor information from the terminal 100.

[0063] In this way, the estimation unit 212 uses one or more three-dimensional positions obtained by feature point matching using a plurality of images to identify the position of the terminal 100 (the position of the first sensor 101) when each image was taken. The accuracy of the first terminal position estimated by the estimation unit 212 is higher than the accuracy of the second terminal position estimated by the estimation unit 112 of the terminal 100. This is because the second terminal position is calculated based only on acceleration and angular velocity, and errors are likely to accumulate.

[0064] Incidentally, the estimation unit 212 may estimate the first terminal position of the terminal 100 at the first time based on the first sensor information and the second sensor information. For example, in addition to the first sensor information, the estimation unit 212 may identify, from the second sensor information, the difference between the first terminal position at the first time when the first image was captured and the other terminal positions when other images were captured, and use it for the estimation of the first terminal position and other terminal positions. In this way, by using the second sensor information including information such as acceleration and angular velocity for the estimation of the first terminal position and the like, an initial value for specifying the feature point pair using the first sensor information can be given, and the processing speed and accuracy can be improved.

[0065] The generation unit 213 generates a three-dimensional model of the object 10 based on a plurality of images included in the first sensor information. The generation unit 213 may generate a three-dimensional model of the object 10 based on the first terminal position estimated by the estimation unit 212 in addition to the plurality of images included in the first sensor information. The generation unit 213 may generate a three-dimensional model by adding three-dimensional points obtained based on a plurality of images to the sparse three-dimensional points generated in the position estimation by the estimation unit 212. Note that the three-dimensional model may be composed of a plurality of three-dimensional points, may be composed of a mesh based on a plurality of three-dimensional points, or may be composed of a NeRF model.

[0066] The storage unit 202 stores the first sensor information acquired by the communication unit 211, the second sensor information acquired by the communication unit 211, the first terminal position estimated by the estimation unit 212, and the three-dimensional model generated by the generation unit 213. The storage unit 202 is, for example, a storage, a memory, or the like.

[0067] [Operation] The operation of the system 1 configured as described above will be described.

[0068] FIG. 3 is a sequence diagram showing an example of the operation of the system.

[0069] When the operation of the terminal 100 starts, it acquires first sensor information and second sensor information (S11). For example, the terminal 100 acquires the first sensor information and the second sensor information in a first period. The first period is, for example, 0.1 seconds. The first period is not limited to 0.1 seconds and may be less than 0.1 seconds or may be 0.2 seconds or more. Also, the period for acquiring the first sensor information and the period for acquiring the second sensor information are not necessarily the same as each other and may be different from each other. For example, the terminal 100 may acquire the first sensor information 10 times per second and acquire the second sensor information 200 times per second, so that the second sensor information is acquired at a shorter period than the first sensor information.

[0070] The terminal 100 transmits the first sensor information and the second sensor information to the information processing apparatus 200 via the communication network 50 (S12).

[0071] When the information processing apparatus 200 receives the first sensor information and the second sensor information, it estimates a first terminal position based on the first sensor information and the second sensor information (S13). The information processing apparatus 200 may estimate the first terminal position every time it receives the first sensor information and the second sensor information. In this case, the information processing apparatus 200 estimates the first terminal position in the first period.

[0072] The information processing apparatus 200 generates a three-dimensional model based on the first sensor information (S14). For example, the information processing apparatus 200 generates the three-dimensional model in a second period. The second period is longer than the first period. The second period is, for example, 1 second. The second period is not limited to 1 second and may be less than 1 second or may be 2 seconds or may be 3 seconds or more. The second period may be the same period as the first period.

[0073] The information processing device 200 transmits the estimated first terminal position, the generated three-dimensional model, and the time information to the terminal 100 via the communication network 50 (S15). The time information is information for specifying the first time. The first time is the time when the first sensor information and the second sensor information, which are the basis for the estimation of the first terminal position and the generation of the three-dimensional model, were acquired.

[0074] The terminal 100 acquires the first sensor information acquired at the second time (current time) and the second sensor information acquired between the first time specified based on the time information and the second time (S16). The terminal 100 may store the received first terminal position, three-dimensional model, and time information in the storage unit 105.

[0075] The terminal 100 estimates the second terminal position, which is the position of the terminal 100 at the second time, by adding the amount of change in the position and orientation of the terminal 100 from the first time to the second time to the first terminal position, which is the position of the terminal 100 at the first time (S17).

[0076] The terminal 100 displays the acquired three-dimensional model based on the second terminal position (S18). Specifically, the terminal 100 superimposes and displays the three-dimensional model on the first sensor information (image) acquired in step S16 based on the second terminal position.

[0077] FIG. 4 is a diagram showing an example of displaying a three-dimensional model.

[0078] As shown in FIG. 4, the terminal 100 displays a UI (User Interface) 300 in which a three-dimensional model is superimposed on the image captured at the second time (current time) based on the second terminal position. Specifically, the terminal 100 superimposes and displays, on the image, the three-dimensional model viewed from the same viewpoint as the viewpoint from which the image was captured, on the object that is the origin of the three-dimensional model in the image. In FIG. 4, since the three-dimensional model is composed of a plurality of three-dimensional points, it is displayed as dots. In this way, the three-dimensional model is displayed in a manner different from the image so as to be distinguishable from the image.

[0079] The terminal 100 and the information processing apparatus 200 repeat the processes of steps S11 to S18 in the third cycle. The third cycle may be the same as or different from the second cycle. The information processing apparatus 200 may not repeat the estimation of the first terminal position and the generation of the three-dimensional model in the same cycle, and may execute only the estimation of the first terminal position. For example, since the second cycle for generating the three-dimensional model is shorter than the first cycle for estimating the first terminal position, only the estimation of the first terminal position may be executed. In this case, the terminal 100 and the information processing apparatus 200 execute the operations shown in FIG. 5. FIG. 5 is a sequence diagram showing another example of the operations of the system.

[0080] The terminal 100 acquires the first sensor information and the second sensor information (S21). Step S21 is the same as step S11.

[0081] The terminal 100 transmits the first sensor information and the second sensor information to the information processing apparatus 200 via the communication network 50 (S22). Step S22 is the same as step S12.

[0082] When the information processing apparatus 200 receives the first sensor information and the second sensor information, it estimates the first terminal position based on the first sensor information and the second sensor information (S23). Step S23 is the same as step S13.

[0083] The terminal 100 acquires the first sensor information acquired at the third time (current time) and the second sensor information acquired between the first time and the third time (S24). Here, the first time is the time specified based on the time information acquired in step S15 when the process shown in FIG. 3 was performed last time. In step S24, the terminal 100 may specify the first time by reading out the latest time information among the time information stored in the storage unit 105. For example, the third time is a time after the second time.

[0084] The terminal 100 estimates the second terminal position, which is the position of the terminal 100 at the third time, by adding the amount of change in the position and orientation of the terminal 100 from the first time to the third time to the first terminal position, which is the position of the terminal 100 at the first time (S25).

[0085] Based on the second terminal position, the terminal 100 displays the three-dimensional model acquired in step S15 when the process shown in FIG. 3 was executed last time (S26). Specifically, the terminal 100 superimposes and displays the three-dimensional model on the first sensor information (image) acquired in step S24 based on the second terminal position.

[0086] Note that after step S23, the information processing device 200 may transmit the estimated first terminal position and the time information indicating the first time to the terminal 100. In this case, in step S24, the terminal 100 acquires the first sensor information acquired at the third time (current time) and the second sensor information acquired between the first time and the third time based on the time information acquired at this time.

[0087] [Problems and effects, etc.] Problems that occur when the present invention is not used will be described. As shown in FIG. 5, after the terminal 100 acquires the first sensor information and the second sensor information at the first time t1, until the second time t2 when the three-dimensional model generated based on the first sensor information is displayed based on the second terminal position, the user scans the object 10 while moving the terminal 100, so the position and orientation of the terminal 100 change. The delay time T from the first time t1 to the second time t2 is the sum of the communication time for the terminal 100 to transmit the first sensor information and the second sensor information to the information processing apparatus 200, the processing time required for the information processing apparatus 200 to estimate the first terminal position, the processing time required for the information processing apparatus 200 to generate the three-dimensional model, and the communication time for the information processing apparatus 200 to transmit the first terminal position and the three-dimensional model to the terminal 100. Thus, since the position and orientation of the terminal 100 change at the delay time T, there is a high possibility that it is different from the first terminal position at the first time. Therefore, if the terminal 100 attempts to directly display the three-dimensional model at the first time with respect to the image captured at the second time, since the viewpoint at the time of capturing the image is different from the viewpoint at the time of capturing the image that is the source of the three-dimensional model, there is a high possibility that the three-dimensional model is displayed at a position shifted from the object in the image.

[0088] The effects when using the present invention will be described. The terminal 100 according to the present embodiment includes a first sensor 101, a second sensor 102, and a circuit 103. The first sensor 101 sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal 100. The second sensor 102 sequentially acquires second sensor information for estimating the amount of change in the position and orientation of the terminal 100. The circuit 103 is connected to the first sensor 101 and the second sensor 102. The circuit 103 transmits the first sensor information acquired by the first sensor 101 at the first time to the information processing device 200. The circuit 103 acquires from the information processing device 200 the first terminal position of the terminal 100 at the first time estimated by the information processing device 200 based on the first sensor information acquired at the first time, and the three-dimensional model generated by the information processing device 200 based on the first sensor information acquired at the first time. The circuit 103 estimates the second terminal position at the second time based on the first terminal position and the second sensor information. Here, the second time is the time when the estimation of the second terminal position is being executed. The circuit 103 displays the acquired three-dimensional model based on the second terminal position.

[0089] According to this, based on the first terminal position, which is the position of the terminal 100 at the first time, and the second sensor information, the terminal 100 estimates the second terminal position, which is the position of the terminal 100 at the second time, and based on the estimated second terminal position, since the three-dimensional model is to be displayed, a three-dimensional model with the second terminal position as the viewpoint can be displayed. Therefore, the terminal 100 can display a three-dimensional model corresponding to the second terminal position at the second time, and can appropriately display the three-dimensional model.

[0090] In the terminal 100 according to the present embodiment, the circuit 103 estimates the first terminal position based on the second sensor information in addition to the first sensor information.

[0091] According to this, since the terminal 100 further estimates the first terminal position based on the second sensor information, the first terminal position can be estimated more accurately.

[0092] In the terminal 100 according to the present embodiment, the accuracy of the first terminal position is higher than that of the second terminal position.

[0093] According to this, since the terminal 100 estimates the second terminal position based on the highly accurate first terminal position, the processing load can be reduced.

[0094] In the terminal 100 according to the present embodiment, in the estimation of the second terminal position, the circuit 103 estimates the amount of change in the position and orientation of the terminal 100 from the first time to the second time based on the second sensor information acquired from the first time to the second time, and estimates the second terminal position based on the first terminal position and the amount of change.

[0095] Therefore, the terminal 100 estimates the second terminal position at the second time based on the amount of change in the position and orientation of the terminal 100 from the first time to the second time with the first terminal position as a reference, so that the processing load for estimating the second terminal position, which is the position of the terminal 100 at the second time, can be reduced. Also, since the terminal 100 estimates the second terminal position according to the amount of change in the position from the first time to the second time with the highly accurate first terminal position as a reference, the second terminal position can be estimated with the error minimized.

[0096] In the terminal 100 according to the present embodiment, the circuit 103 further acquires time information for specifying the first time together with the first terminal position and the three-dimensional model. Then, the circuit 103 estimates the second terminal position based on the time information.

[0097] Therefore, the terminal 100 can easily specify the first time corresponding to the first terminal position and the three-dimensional model using the time information. Thus, the terminal 100 can easily estimate the amount of change in the position and orientation of the terminal from the first time to the second time based on the second sensor information.

[0098] In the terminal 100 according to the present embodiment, the first sensor information includes a plurality of images captured at different positions and postures. The circuit 103 acquires the images included in the first sensor information acquired at the second time by the first sensor, and based on the second terminal position, superimposes and displays a three-dimensional model on the image.

[0099] Therefore, in order to display a three-dimensional model with the second terminal position as the viewpoint for the image captured at the second time, the terminal 100 can superimpose and display the three-dimensional model when viewed from the same viewpoint as the viewpoint when the image was captured. Thus, the terminal 100 can display a three-dimensional model corresponding to the second terminal position at the second time, and can appropriately display the three-dimensional model.

[0100] In the terminal 100 according to the present embodiment, the three-dimensional model superimposed on the image is displayed in a manner different from the image.

[0101] Therefore, since the terminal 100 can display the image and the three-dimensional model in an easily distinguishable manner, the user can easily confirm the progress of the three-dimensional model generation by viewing the image and the three-dimensional model displayed on the terminal 100.

[0102] Further, the information processing apparatus 200 according to the present embodiment includes a storage unit 202 (memory) and a circuit 201 connected to the storage unit 202. The circuit 201 acquires, from the terminal 100, the first sensor information acquired at the first time among the first sensor information that is sequentially acquired by the first sensor included in the terminal 100 for generating a three-dimensional model and estimating the position and posture of the terminal 100. The circuit 201 estimates the first terminal position of the terminal 100 at the first time based on the first sensor information acquired at the first time. The circuit 201 generates a three-dimensional model based on the first sensor information acquired at the first time. The circuit 201 transmits the first terminal position, the three-dimensional model, and time information for specifying the first time to the terminal 100.

[0103] According to this, the information processing apparatus 200 transmits, together with the first terminal position and the three-dimensional model, time information for specifying a specific first time corresponding to the first terminal position and the three-dimensional model to the terminal 100. Therefore, the terminal 100 can easily specify the first time corresponding to the first terminal position and the three-dimensional model. Thus, the terminal 100 can easily estimate, for example, the amount of change in the position and orientation of the terminal 100 from the first time to the second time based on the second sensor information. That is, a three-dimensional model corresponding to the second terminal position at the second time can be displayed on the terminal 100, and the three-dimensional model can be appropriately displayed.

[0104] [Modification Example] (1) In the above embodiment, the first sensor 101 provided in the terminal 100 is assumed to be a camera, but it is not limited to this, and it may be a distance sensor. The distance sensor measures the distance between the distance sensor and an object, for example, by emitting an electromagnetic wave, receiving the electromagnetic wave reflected by the object, and measuring the arrival time of the electromagnetic wave reflected by the object after emitting the electromagnetic wave. The electromagnetic wave includes, for example, sound waves, light, radio waves, etc. The distance sensor is, for example, a LiDAR, a ToF sensor, etc. In this case, the information processing apparatus 200 generates a three-dimensional model using the distance information acquired by the distance sensor.

[0105] (2) In the above embodiment, the first sensor 101 provided in the terminal 100 may be a stereo camera. Thereby, the information processing apparatus 200 can generate a three-dimensional model at the first time using the two images taken at the first time.

[0106] (3) In the above embodiment, the terminal 100 superimposes the three-dimensional model on the image acquired at the second time and displays it on the display unit 104, but it is not limited to superimposing and displaying the three-dimensional model on the image. The terminal 100 may display the image acquired at the second time and the three-dimensional model in two independent areas of the display unit 104 without superimposing them. The terminal 100 may display only the three-dimensional model on the display unit 104.

[0107] (4) In the above-described embodiment, the terminal 100 is assumed to specify the first time based on the time information. However, it is not limited to this. The time from the first time to the second time becomes a fixed time if, for example, the communication speed by the communication network 50 and the processing speeds of various processes by the terminal 100 and the information processing apparatus 200 are stable and constant. Therefore, the terminal 100 may estimate the first time by subtracting the fixed time from the second time. Thereby, the terminal 100 can specify the first time without acquiring the time information from the information processing apparatus 200.

[0108] (5) In the above-described embodiment, the information processing apparatus 200 is assumed to transmit the time information to the terminal 100 together with the first terminal position and the three-dimensional model. However, it is not limited to this, and it is not always necessary to transmit the first terminal position and the three-dimensional model every time there is an opportunity to transmit them. For example, the information processing apparatus 200 may transmit the time information to the terminal 100 at a frequency of once every predetermined number of opportunities to transmit the first terminal position and the three-dimensional model. At this time, since the information processing apparatus 200 repeatedly transmits the first terminal position and the three-dimensional model to the terminal 100, for example, at the third period, the terminal 100 can specify the time corresponding to the first terminal position and the three-dimensional model by integrating the third period based on the time information. In this way, the terminal 100 can specify the first time without always acquiring the time information from the information processing apparatus 200 together with the first terminal position and the three-dimensional model.

[0109] [Others] Note that, in the above-described embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software for realizing the information processing apparatus and the like in the above-described embodiment is a program that causes a computer to execute each step included in the flowchart shown in the figure.

[0110] In addition, the following cases are also included in the present disclosure.

[0111] (1) Specifically, each of the above devices is a computer system composed of a microprocessor, ROM, RAM, hard disk unit, display unit, keyboard, mouse, etc. A computer program is stored in the RAM or the hard disk unit. When the microprocessor operates according to the computer program, each device achieves its function. Here, the computer program is composed of a plurality of instruction codes indicating instructions for the computer in order to achieve a predetermined function.

[0112] (2) Part or all of the components constituting each of the above devices may be composed of one system LSI (Large Scale Integration). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components on one chip, and specifically, is a computer system including a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. When the microprocessor operates according to the computer program, the system LSI achieves its function.

[0113] (3) Part or all of the components constituting each of the above devices may be composed of an IC card or a single module detachable from each device. The IC card or the module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or the module may include the above super multifunctional LSI. When the microprocessor operates according to the computer program, the IC card or the module achieves its function. This IC card or this module may have tamper resistance.

[0114] (4) The present disclosure may be the method described above. It may also be a computer program that realizes these methods by a computer, or a digital signal composed of the computer program.

[0115] In addition, the present disclosure may be recorded on a computer-readable recording medium such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. that records the computer program or the digital signal. It may also be the digital signal recorded on these recording media.

[0116] In addition, the present disclosure may transmit the computer program or the digital signal via a telecommunication line, a wireless or wired communication line, a network typified by the Internet, data broadcasting, etc.

[0117] In addition, the present disclosure may be a computer system including a microprocessor and a memory, where the memory stores the computer program, and the microprocessor operates according to the computer program.

[0118] In addition, it may be implemented by another independent computer system by recording and transferring the program or the digital signal to the recording medium, or by transferring the program or the digital signal via the network or the like.

[0119] (5) The above embodiments and the above modification examples may be combined respectively.

Industrial Applicability

[0120] The present disclosure is useful as a terminal or the like that can appropriately display a three-dimensional model.

Explanation of Signs

[0121] 1 System 10 Object 50 Communication Network 100 Terminal 101 First Sensor 102 Second Sensor 103 Circuit 104 Display Unit 105 Memory Unit 111 Communication Unit 112 Estimation Unit 113 Display Control Unit 200 Information Processing Device 201 Circuit 202 Memory Unit 211 Communication Unit 212 Estimation Unit 213 Generation Unit

Claims

1. A terminal, a first sensor that sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal, a second sensor that sequentially acquires second sensor information for estimating the amount of change in the position and orientation of the terminal, and a circuit connected to the first sensor and the second sensor, wherein the circuit transmits the first sensor information acquired by the first sensor at a first time to an information processing device, acquires from the information processing device the first terminal position of the terminal at the first time estimated by the information processing device based on the first sensor information acquired at the first time and the three-dimensional model generated by the information processing device based on the first sensor information acquired at the first time, estimates a second terminal position at a second time based on the first terminal position and the second sensor information, where the second time is the time when the estimation of the second terminal position is being executed, and displays the acquired three-dimensional model based on the second terminal position. A terminal.

2. The circuit estimates the first terminal position based on the second sensor information in addition to the first sensor information. The terminal according to claim 1.

3. The accuracy of the first terminal position is higher than the accuracy of the second terminal position. The terminal according to claim 1.

4. In the estimation of the second terminal position, the circuit estimates the amount of change in the position and orientation of the terminal from the first time to the second time based on the second sensor information acquired from the first time to the second time, and estimates the second terminal position based on the first terminal position and the amount of change. The terminal according to claim 1.

5. The second sensor information includes the acceleration of the terminal and the angular velocity of the terminal. The terminal according to any one of claims 1 to 4.

6. The circuit further acquires time information for specifying the first time together with the first terminal position and the three-dimensional model, and estimates the second terminal position based on the time information. The terminal according to any one of claims 1 to 4.

7. The first sensor information includes a plurality of images taken at different positions and orientations, and the circuit acquires the images included in the first sensor information acquired by the first sensor at the second time, and superimposes and displays the three-dimensional model on the images based on the second terminal position. The terminal according to any one of claims 1 to 4.

8. The three-dimensional model superimposed on the image is displayed in a manner different from the image The terminal according to claim 7

9. A memory, A circuit connected to the memory, and includes The circuit is First sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal, among the first sensor information sequentially acquired by the first sensor included in the terminal, the first sensor information acquired at the first time is acquired from the terminal, Estimate the first terminal position of the terminal at the first time based on the first sensor information acquired at the first time, Generate a three-dimensional model based on the first sensor information acquired at the first time, Transmit the first terminal position, the three-dimensional model, and time information for specifying the first time to the terminal An information processing device

10. A display method executed by a terminal, First sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal, among the first sensor information sequentially acquired by the first sensor included in the terminal, the first sensor information acquired at the first time is transmitted to an information processing device, The first terminal position of the terminal at the first time estimated by the information processing device based on the first sensor information acquired at the first time and the three-dimensional model generated by the information processing device based on the first sensor information acquired at the first time are acquired from the information processing device, Based on the first terminal position and second sensor information for estimating the amount of change in the position and orientation of the terminal, which is the second sensor information sequentially acquired by the second sensor included in the terminal, estimate the second terminal position at the second time, and the second time is when the estimation of the second terminal position is being executed, Based on the second terminal position, display the acquired three-dimensional model A display method

11. An information processing method executed by an information processing device, First sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal, among the first sensor information sequentially acquired by the first sensor included in the terminal, the first sensor information acquired at the first time is acquired from the terminal, Estimate the first terminal position of the terminal at the first time based on the first sensor information acquired at the first time, Generate a three-dimensional model based on the first sensor information acquired at the first time, Transmit the first terminal position, the three-dimensional model, and time information for specifying the first time to the terminal. Information processing method.

Images