Terminal, information processing device, display method, and information processing method

Abstract

Provided is a terminal (100) comprising: a first sensor (101) for sequentially acquiring first sensor information; a second sensor (102) for sequentially acquiring second sensor information; and a circuit (103) connected to the first sensor and the second sensor. The circuit (103) transmits first sensor information acquired at a first time by the first sensor (101) to an information processing device, acquires from the information processing device a first terminal position at the first time of the terminal estimated by the information processing device on the basis of the first sensor information acquired at the first time and a three-dimensional model generated by the information processing device on the basis of the first sensor information acquired at the first time, estimates a second terminal position at a second time on the basis of the first terminal position and the second sensor information, the second time referring to a time when estimation of the second terminal position is being estimated, and displays the acquired three-dimensional model on the basis of the second terminal position.

Description

Terminal, information processing device, display method, and information processing method

[0001] The present disclosure relates to a terminal, an information processing device, a display method, and an information processing method.

[0002] Patent Document 1 discloses a technique for estimating the positions and orientations of a plurality of image capturing devices (i.e., cameras) by performing feature point matching on a plurality of images obtained by the plurality of image capturing devices.

[0003] International Publication No. 2020 / 153264

[0004] The present disclosure provides a terminal or the like that can appropriately display a three-dimensional model.

[0005] A terminal according to one aspect of the present disclosure is a terminal comprising: a first sensor that sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and attitude of the terminal; a second sensor that sequentially acquires second sensor information for estimating changes in the position and attitude of the terminal; and a circuit connected to the first sensor and the second sensor, wherein the circuit transmits the first sensor information acquired by the first sensor at a first time to an information processing device; acquires from the information processing device a first terminal position of the terminal at the first time estimated by the information processing device based on the first sensor information acquired at the first time and a three-dimensional model generated by the information processing device based on the first sensor information acquired at the first time; estimates a second terminal position at a second time based on the first terminal position and the second sensor information, the second time being the time when the estimation of the second terminal position is being performed; and displays the acquired three-dimensional model based on the second terminal position.

[0006] Moreover, an information processing device according to one aspect of the present disclosure includes a memory and a circuit connected to the memory, wherein the circuit acquires from the terminal first sensor information acquired at a first time, the first sensor information being used to generate a three-dimensional model and estimate the position and attitude of the terminal, the first sensor information being sequentially acquired by a first sensor provided in the terminal; estimating a first terminal position of the terminal at the first time based on the first sensor information acquired at the first time; generating a three-dimensional model based on the first sensor information acquired at the first time; and transmitting the first terminal position, the three-dimensional model, and time information for identifying the first time to the terminal.

[0007] These general or specific aspects may be realized as a system, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a method, a system, an apparatus, an integrated circuit, a computer program, and a non-transitory recording medium.

[0008] The terminal etc. in the present disclosure can appropriately display the three-dimensional model.

[0009] Fig. 1 is a diagram for explaining an example of the configuration of a system according to an embodiment. Fig. 2 is a block diagram showing an example of the configuration of a system according to an embodiment. Fig. 3 is a sequence diagram showing an example of operation by the system. Fig. 4 is a diagram showing an example of displaying a three-dimensional model. Fig. 5 is a sequence diagram showing another example of operation by the system. Fig. 6 is a diagram for explaining a delay when a three-dimensional model is displayed superimposed on an image.

[0010] (Findings that Form the Basis of the Present Disclosure) The present inventors have found that the following problems occur with the conventional systems described in the "Background Art" section.

[0011] A technology such as that disclosed in Patent Document 1 discloses that a single imaging device may be moved while capturing images to generate a multi-viewpoint image including multiple images with different viewpoints.

[0012] However, the inventors have discovered that there is a need for a device that generates a three-dimensional model in real time based on multi-view images obtained while moving a single terminal, as in conventional technology, and superimposes the three-dimensional model on the image being captured, allowing the user to easily see the location in the space being captured where the three-dimensional model was generated.

[0013] However, such terminals often lack sufficient processing capabilities. Therefore, in order to reduce the processing load, it is conceivable to transfer multiple images to an information processing device with sufficient processing capabilities and have the information processing device generate the 3D model, which requires a large processing load. In this way, when the device that captures the images and the device that generates the 3D model are separate devices, communication between the devices is necessary. When a terminal transmits captured images to an information processing device in real time and acquires and displays the 3D model generated by the information processing device, a time lag occurs between capturing the images and displaying the generated 3D model. Therefore, there is a high possibility that the terminal will move during this time lag. If the terminal attempts to display a 3D model generated based on multiple images captured at a previous time on an image captured at the current time, the viewpoint at the time the image was captured differs from the viewpoint at the time the image was captured, and therefore the 3D model is likely to be displayed at a position offset from the object in the image.

[0014] Therefore, the present inventors have come up with a terminal or the like that can appropriately display a three-dimensional model.

[0015] A terminal according to a first aspect of the present disclosure is a terminal comprising: a first sensor that sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and attitude of the terminal; a second sensor that sequentially acquires second sensor information for estimating changes in the position and attitude of the terminal; and a circuit connected to the first sensor and the second sensor, wherein the circuit transmits the first sensor information acquired by the first sensor at a first time to an information processing device; acquires from the information processing device a first terminal position of the terminal at the first time estimated by the information processing device based on the first sensor information acquired at the first time and a three-dimensional model generated by the information processing device based on the first sensor information acquired at the first time; estimates a second terminal position at a second time based on the first terminal position and the second sensor information, the second time being the time when the estimation of the second terminal position is being performed; and displays the acquired three-dimensional model based on the second terminal position.

[0016] According to this, the terminal estimates the second terminal position, which is the terminal's position at the second time, based on the first terminal position, which is the terminal's position at the first time, and the second sensor information, and displays a three-dimensional model based on the estimated second terminal position. Therefore, for example, it is possible to display a three-dimensional model with the second terminal position as the viewpoint. Therefore, the terminal can display a three-dimensional model according to the second terminal position at the second time, and can display the three-dimensional model appropriately.

[0017] A terminal according to a second aspect of the present disclosure is the terminal according to the first aspect, wherein the circuit estimates the position of the first terminal based on the second sensor information in addition to the first sensor information.

[0018] According to this, the terminal further estimates the first terminal position based on the second sensor information, and therefore, the first terminal position can be estimated with higher accuracy.

[0019] A terminal according to a third aspect of the present disclosure is the terminal according to the first or second aspect, wherein the accuracy of the first terminal location is higher than the accuracy of the second terminal location.

[0020] According to this, the terminal estimates the second terminal position based on the highly accurate first terminal position, and therefore the processing load can be reduced.

[0021] A terminal according to a fourth aspect of the present disclosure is a terminal according to any one of the first to third aspects, wherein, in estimating the second terminal position, the circuit estimates the amount of change in the position and attitude of the terminal from the first time to the second time based on the second sensor information acquired from the first time to the second time, and estimates the second terminal position based on the first terminal position and the amount of change.

[0022] Therefore, the terminal estimates the second terminal position at the second time based on the amount of change in the terminal's position and attitude from the first time to the second time, using the first terminal position as a reference, thereby reducing the processing load involved in estimating the second terminal position, which is the terminal's position at the second time.

[0023] A terminal according to a fifth aspect of the present disclosure is a terminal according to any one of the first to fourth aspects, wherein the second sensor information includes an acceleration of the terminal and an angular velocity of the terminal.

[0024] A terminal according to a sixth aspect of the present disclosure is a terminal according to any one of the first to fifth aspects, wherein the circuit further acquires time information for identifying the first time together with the first terminal position and the three-dimensional model, and estimates the second terminal position based on the time information.

[0025] Therefore, the terminal can easily identify the first time to which the first terminal position and the three-dimensional model correspond using the time information, and can easily estimate, for example, the amount of change in the position and attitude of the terminal from the first time to the second time based on the second sensor information.

[0026] A terminal according to a seventh aspect of the present disclosure is a terminal according to any one of the first to sixth aspects, wherein the first sensor information includes a plurality of images taken at different positions and postures, and the circuit acquires an image included in the first sensor information acquired by the first sensor at the second time, and displays the three-dimensional model superimposed on the image based on the second terminal position.

[0027] Therefore, the terminal displays a three-dimensional model with the second terminal position as the viewpoint for the image captured at the second time, and can superimpose and display the three-dimensional model as viewed from the same viewpoint as when the image was captured. Thus, the terminal can display a three-dimensional model according to the second terminal position at the second time, and can appropriately display the three-dimensional model.

[0028] A terminal according to an eighth aspect of the present disclosure is the terminal according to the seventh aspect, wherein the three-dimensional model superimposed on the image is displayed in a manner different from that of the image.

[0029] As a result, the terminal can display images and three-dimensional models in a way that makes them easy to distinguish, allowing the user to easily check the progress of three-dimensional model generation by looking at the images and three-dimensional models displayed on the terminal.

[0030] An information processing device according to a ninth aspect of the present disclosure includes a memory and a circuit connected to the memory, wherein the circuit acquires from the terminal first sensor information acquired at a first time, the first sensor information being used to generate a three-dimensional model and estimate the position and attitude of the terminal, the first sensor information being sequentially acquired by a first sensor provided in the terminal; estimating a first terminal position of the terminal at the first time based on the first sensor information acquired at the first time; generating a three-dimensional model based on the first sensor information acquired at the first time; and transmitting the first terminal position, the three-dimensional model, and time information for identifying the first time to the terminal.

[0031] According to this, the information processing device transmits to the terminal the first terminal position, the three-dimensional model, and time information for identifying the first time corresponding to the first terminal position and the three-dimensional model, so that the terminal can easily identify the first time corresponding to the first terminal position and the three-dimensional model. Therefore, the terminal can easily estimate, for example, the amount of change in the position and attitude of the terminal from the first time to the second time based on the second sensor information. In other words, the terminal can display a three-dimensional model corresponding to the second terminal position at the second time, and the three-dimensional model can be displayed appropriately.

[0032] A display method according to a tenth aspect of the present disclosure is a display method executed by a terminal, the display method including: transmitting, to an information processing device, first sensor information for generating a three-dimensional model and estimating the position and attitude of the terminal, the first sensor information being sequentially acquired by a first sensor provided in the terminal and acquired at a first time; acquiring from the information processing device a first terminal position of the terminal at the first time estimated by the information processing device based on the first sensor information acquired at the first time, and a three-dimensional model generated by the information processing device based on the first sensor information acquired at the first time; estimating a second terminal position at a second time based on the first terminal position and second sensor information for estimating a change in the position and attitude of the terminal, the second time being when estimation of the second terminal position is being performed; and displaying the acquired three-dimensional model based on the second terminal position.

[0033] According to this, the terminal estimates the second terminal position, which is the terminal's position at the second time, based on the first terminal position, which is the terminal's position at the first time, and the second sensor information, and displays a three-dimensional model based on the estimated second terminal position. Therefore, for example, it is possible to display a three-dimensional model with the second terminal position as the viewpoint. Therefore, the terminal can display a three-dimensional model according to the second terminal position at the second time, and can display the three-dimensional model appropriately.

[0034] An information processing method according to an eleventh aspect of the present disclosure is an information processing method executed by an information processing device, which acquires first sensor information for generating a three-dimensional model and estimating the position and attitude of a terminal from the terminal, the first sensor information being acquired at a first time from the terminal, estimating a first terminal position of the terminal at the first time based on the first sensor information acquired at the first time, generating a three-dimensional model based on the first sensor information acquired at the first time, and transmitting the first terminal position, the three-dimensional model, and time information for identifying the first time to the terminal.

[0035] According to this, the information processing device transmits to the terminal the first terminal position, the three-dimensional model, and time information for identifying the first time corresponding to the first terminal position and the three-dimensional model, so that the terminal can easily identify the first time corresponding to the first terminal position and the three-dimensional model. Therefore, the terminal can easily estimate, for example, the amount of change in the position and attitude of the terminal from the first time to the second time based on the second sensor information. In other words, the terminal can display a three-dimensional model corresponding to the second terminal position at the second time, and the three-dimensional model can be displayed appropriately.

[0036] These general or specific aspects may be realized as a system, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a method, a system, an apparatus, an integrated circuit, a computer program, and a non-transitory recording medium.

[0037] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of well-known matters or redundant explanation of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0038] The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0039] (Embodiment) Hereinafter, an embodiment will be described with reference to FIGS.

[0040] [Configuration] FIG. 1 is a diagram illustrating an example of the configuration of a system according to an embodiment.

[0041] The system 1 includes a terminal 100 and an information processing device 200. In the system 1, sensing of an object 10 is performed using a sensor included in the terminal 100, and the information processing device 200 generates a three-dimensional model of the object 10 using the sensing results. The terminal 100 performs sensing while being moved by, for example, a user U1. The terminal 100 in the present disclosure displays the progress of sensing while performing sensing for generating the three-dimensional model of the object 10. The information processing device 200 generates a three-dimensional model based on the sensing results while acquiring sensing results from the terminal 100, and transmits the three-dimensional model to the terminal 100 so that the terminal 100 can display the progress.

[0042] The terminal 100 and the information processing device 200 are communicatively connected to each other via a communication network 50. The terminal 100 may be communicatively connected to the communication network 50 via a wireless local area network (LAN) or via a mobile phone communication network. The terminal 100 is, for example, a smartphone, a tablet terminal, a personal computer (PC), or the like. The information processing device 200 is communicatively connected to the communication network 50 via a wired local area network (LAN). Note that the information processing device 200 may also be communicatively connected to the communication network 50 via a wireless LAN.

[0043] FIG. 2 is a block diagram illustrating an example of a configuration of a system according to an embodiment.

[0044] The configuration of the terminal 100 will be described.

[0045] The terminal 100 includes a first sensor 101 , a second sensor 102 , a circuit 103 , a display unit 104 , and a memory unit 105 .

[0046] The first sensor 101 is a sensor that sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal 100. The first sensor 101 is, for example, a camera. The camera may be, for example, an RGB camera or a monochrome camera. The first sensor information is an image. The image may be, for example, a color image having RGB subpixels or a monochrome image. The first sensor information includes multiple images captured by the camera that is the first sensor 101 at different positions and orientations.

[0047] The second sensor 102 is a sensor that sequentially acquires second sensor information for estimating the amount of change in the position and the amount of change in the attitude of the terminal 100. The second sensor 102 has, for example, an acceleration sensor that acquires the acceleration of the terminal 100 and an angular velocity sensor that acquires the angular velocity of the terminal 100. In other words, the second sensor information includes the acceleration of the terminal 100 and the angular velocity of the terminal 100. The acceleration sensor is, for example, a triaxial acceleration sensor that acquires the acceleration in each of the three axial directions of the terminal 100. The three axial directions are the X-axis direction, the Y-axis direction, and the Z-axis direction, which are orthogonal to each other. The angular velocity sensor is, for example, a triaxial angular velocity sensor that acquires the angular velocity around each axis of the three axial directions of the terminal 100. The second sensor 102 may further include a geomagnetic sensor.

[0048] The circuit 103 includes a communication unit 111 , an estimation unit 112 , and a display control unit 113 .

[0049] The communication unit 111 exchanges information with the information processing device 200 via the communication network 50. Specifically, the communication unit 111 transmits first sensor information sequentially acquired by the first sensor 101 to the information processing device 200. The communication unit 111 may also transmit second sensor information sequentially acquired by the second sensor 102 to the information processing device 200. The communication unit 111 receives (acquires) a first terminal position and a three-dimensional model from the information processing device 200. The first terminal position is the position and orientation of the terminal 100 at a first time, estimated by the information processing device 200 based on the first sensor information acquired at a first time. Note that the first terminal position may be estimated based on the first sensor information and second sensor information acquired at the first time. The three-dimensional model is a three-dimensional model generated by the information processing device 200 based on the first sensor information acquired at the first time. Furthermore, the communication unit 111 may receive (acquire) time information for identifying the first time together with the first terminal position and the three-dimensional model. The time information may be information for identifying the first time, and may be information indicating the first time, information for identifying first sensor information corresponding to the first time, or information for identifying second sensor information corresponding to the first time.

[0050] The estimation unit 112 estimates a second terminal position, which is the position and attitude of the terminal 100 at a second time, based on the first terminal position and second sensor information acquired via the communication unit 111. The second time is the time when estimation of the second terminal position is being performed. The second time is a time later than the first time. The estimation unit 112 identifies a first time, which is the time corresponding to the first terminal position and the three-dimensional model acquired together with the time information, based on the acquired time information. Specifically, the estimation unit 112 estimates the amount of change in the position and attitude of the terminal 100 from the first time to the second time, based on the second sensor information acquired by the second sensor 102 from the first time to the second time. Then, the estimation unit 112 estimates the second terminal position based on the first terminal position and the estimated amount of change. In other words, the estimation unit 112 estimates the second terminal position, which is the position of the terminal 100 at the second time, by adding the change in the position and attitude of the terminal 100 from the first time to the second time to the first terminal position, which is the position of the terminal 100 at the first time.

[0051] The display control unit 113 displays the three-dimensional model acquired via the communication unit 111 on the display unit 104 based on the second terminal position acquired via the communication unit 111. The display control unit 113 acquires an image included in the first sensor information acquired by the first sensor 101 at the second time, that is, an image captured at the second time.

[0052] Based on the second terminal position estimated by the estimation unit 112, the display control unit 113 superimposes a three-dimensional model on an image acquired at a second time and displays the superimposed three-dimensional model on the display unit 104. The display control unit 113 superimposes on the image a three-dimensional model that is visible when the sensing direction of the first sensor 101 of the terminal 100 at the second terminal position is set as the line of sight direction, and displays the superimposed three-dimensional model on the display unit 104. The display control unit 113 identifies a rotation matrix and a translation vector in Equation 1 based on the second terminal position, and identifies a two-dimensional position (two-dimensional coordinates) on the image at which to superimpose the three-dimensional model using Equation 1 to which the identified rotation matrix and translation vector are applied. The display control unit 113 then superimposes the three-dimensional model at the identified two-dimensional position.

[0053]

[0054] Here, u and v indicate two-dimensional positions (two-dimensional coordinates) on the image. u indicates a vertical position, and v indicates a horizontal position. fx and fy indicate the focus of the camera serving as the first sensor 101. fx and fy are constants. cx and cy indicate the center of the camera serving as the first sensor 101. cx and cy are constants. X, Y, and Z indicate the three-dimensional position (three-dimensional coordinates) of the three-dimensional model to be superimposed and displayed. Equation 1 is, for example, an equation applied to each of multiple three-dimensional positions to identify the two-dimensional position on the image corresponding to each three-dimensional position. r11 to r33 indicate the rotation matrix (i.e., the camera attitude) of the position of the camera serving as the first sensor 101. t1 to t3 indicate the translation vector of the position of the camera serving as the first sensor 101. The first terminal position and the second terminal position can be expressed as three-dimensional positions using r11 to r33 and t1 to t3 in Equation 1. It should be noted that three-dimensional positions can also be expressed using other methods such as quaternions, but this will not be explained here.

[0055] Furthermore, the display control unit 113 displays the three-dimensional model in a manner different from that of the image, that is, the three-dimensional model superimposed on the image is displayed in a manner different from that of the image.

[0056] The display unit 104 displays at least one of an image and a three-dimensional model based on a control instruction from the display control unit 113. The display unit 104 is, for example, a display.

[0057] The storage unit 105 stores at least one of first sensor information sequentially acquired by the first sensor 101 and second sensor information sequentially acquired by the second sensor 102. The storage unit 105 stores the first sensor information together with the time at which the first sensor information was acquired. The storage unit 105 stores the second sensor information together with the time at which the second sensor information was acquired. The storage unit 105 may store the first terminal position and the three-dimensional model acquired from the information processing device 200. The storage unit 105 is, for example, a storage or a memory.

[0058] Next, the configuration of the information processing device 200 will be described.

[0059] The information processing device 200 includes a circuit 201 and a storage unit 202 .

[0060] The circuit 201 includes a communication unit 211 , an estimation unit 212 , and a generation unit 213 .

[0061] The communication unit 211 exchanges information with the terminal 100 via the communication network 50. Specifically, the communication unit 211 receives (acquires) first sensor information from the terminal 100. For example, the communication unit 211 acquires the first sensor information acquired at a first time from the terminal 100. The communication unit 211 may further receive (acquire) second sensor information from the terminal 100. The communication unit 211 also transmits the first terminal position, the three-dimensional model, and time information to the terminal 100. The first terminal position is the position of the terminal 100 at the first time estimated by the information processing device 200 based on the first sensor information acquired at the first time. The three-dimensional model is a three-dimensional model generated by the information processing device 200 based on the first sensor information acquired at the first time. The time information is information for identifying the first time.

[0062] The estimation unit 212 estimates the first terminal position of the terminal 100 at a first time based on first sensor information acquired at a first time. For simplicity, an estimation method using only the first sensor information will be described first. The estimation unit 212 identifies one or more feature points in each of a plurality of images included in the first sensor information. Then, the estimation unit 212, for example, matches one or more feature points included in one of two images among the plurality of images with one or more feature points included in the other, thereby identifying one or more pairs of feature points (hereinafter referred to as feature point pairs). The two images include a first image acquired at a first time. For each of one or more feature point pairs in the two images, the estimation unit 212 identifies a three-dimensional position (three-dimensional point) corresponding to the feature point pair. Based on the one or more three-dimensional positions, the estimation unit 212 estimates the first terminal position, which is the position of the terminal 100 at the first time when the first image was acquired, and the position of the terminal 100 when the other image was acquired. For example, structure from motion (SfM) is used for estimating the position by the estimation unit 212. Note that if the second sensor information is not used for estimating the first terminal position, the communication unit 211 may be designed not to receive the second sensor information from the terminal 100.

[0063] In this way, the estimation unit 212 uses one or more three-dimensional positions obtained by feature point matching using multiple images to identify the position of the terminal 100 (the position of the first sensor 101) when each image was captured. The accuracy of the first terminal position estimated by the estimation unit 212 is higher than the accuracy of the second terminal position estimated by the estimation unit 112 of the terminal 100. This is because the second terminal position is calculated based only on acceleration and angular velocity, and errors are likely to accumulate.

[0064] The estimation unit 212 may estimate the first terminal position of the terminal 100 at the first time based on the first sensor information and the second sensor information. For example, the estimation unit 212 may identify, based on the second sensor information in addition to the first sensor information, a difference between the first terminal position at the first time when the first image was captured and the other terminal position when the other image was captured, from the second sensor information, and use this difference to estimate the first terminal position and the other terminal positions. In this way, by using the second sensor information including information such as acceleration and angular velocity to estimate the first terminal position, etc., initial values ​​for identifying feature point pairs using the first sensor information can be provided, thereby enabling faster and more accurate processing.

[0065] The generation unit 213 generates a three-dimensional model of the object 10 based on the multiple images included in the first sensor information. The generation unit 213 may generate the three-dimensional model of the object 10 based on the first terminal position estimated by the estimation unit 212 in addition to the multiple images included in the first sensor information. The generation unit 213 may generate the three-dimensional model by adding three-dimensional points obtained based on the multiple images to sparse three-dimensional points generated in position estimation by the estimation unit 212. Note that the three-dimensional model may be configured by multiple three-dimensional points, may be configured by a mesh based on the multiple three-dimensional points, or may be configured by an NeRF model.

[0066] The storage unit 202 stores the first sensor information acquired by the communication unit 211, the second sensor information acquired by the communication unit 211, the first terminal position estimated by the estimation unit 212, and the three-dimensional model generated by the generation unit 213. The storage unit 202 is, for example, a storage or a memory.

[0067] [Operation] The operation of the system 1 configured as above will be described.

[0068] FIG. 3 is a sequence diagram showing an example of an operation performed by the system.

[0069] When the terminal 100 starts operation, it acquires first sensor information and second sensor information (S11). The terminal 100 acquires the first sensor information and the second sensor information at a first cycle, for example. The first cycle is, for example, 0.1 seconds. The first cycle is not limited to 0.1 seconds and may be shorter than 0.1 seconds or longer than 0.2 seconds. Furthermore, the cycles for acquiring the first sensor information and the second sensor information do not need to be the same and may be different from each other. The terminal 100 may acquire the second sensor information at a cycle shorter than the first sensor information, for example, acquiring the first sensor information 10 times per second and acquiring the second sensor information 200 times per second.

[0070] The terminal 100 transmits the first sensor information and the second sensor information to the information processing device 200 via the communication network 50 (S12).

[0071] When the information processing device 200 receives the first sensor information and the second sensor information, it estimates the first terminal position based on the first sensor information and the second sensor information (S13). The information processing device 200 may estimate the first terminal position every time it receives the first sensor information and the second sensor information. In this case, the information processing device 200 estimates the first terminal position at a first cycle.

[0072] The information processing device 200 generates a three-dimensional model based on the first sensor information (S14). The information processing device 200 generates the three-dimensional model at a second period, for example. The second period is longer than the first period. The second period is, for example, one second. The second period is not limited to one second, and may be less than one second, two seconds, or three seconds or more. The second period may be the same period as the first period.

[0073] The information processing device 200 transmits the estimated first terminal position, the generated three-dimensional model, and time information to the terminal 100 via the communication network 50 (S15). The time information is information for identifying the first time. The first time is the time when the first sensor information and second sensor information, which were used to estimate the first terminal position and generate the three-dimensional model, were acquired.

[0074] The terminal 100 acquires the first sensor information acquired at the second time (current time) and the second sensor information acquired between the first time and the second time specified based on the time information (S16). The terminal 100 may store the received first terminal position, the three-dimensional model, and the time information in the storage unit 105.

[0075] The terminal 100 estimates the second terminal position, which is the position of the terminal 100 at the second time, by adding the change in the position and attitude of the terminal 100 from the first time to the second time to the first terminal position, which is the position of the terminal 100 at the first time (S17).

[0076] The terminal 100 displays the acquired three-dimensional model based on the second terminal position (S18). Specifically, the terminal 100 displays the three-dimensional model by superimposing it on the first sensor information (image) acquired in step S16 based on the second terminal position.

[0077] FIG. 4 is a diagram showing an example of a display of a three-dimensional model.

[0078] As shown in Fig. 4, the terminal 100 displays a UI (User Interface) 300 in which a three-dimensional model is superimposed on an image captured at a second time (current time) based on the position of the second terminal. Specifically, the terminal 100 displays a three-dimensional model as viewed from the same viewpoint as the viewpoint at which the image was captured, superimposed on the object in the image that is the basis for the three-dimensional model. In Fig. 4, the three-dimensional model is displayed as dots because it is made up of multiple three-dimensional points. In this way, the three-dimensional model is displayed in a manner different from the image so as to be distinguishable from the image.

[0079] The terminal 100 and the information processing device 200 repeat the processes of steps S11 to S18 in a third cycle. The third cycle may be the same as or different from the second cycle. The information processing device 200 may not repeat the estimation of the first terminal position and the generation of the three-dimensional model in the same cycle, and may only execute the estimation of the first terminal position. For example, because the second cycle for generating the three-dimensional model is shorter than the first cycle for estimating the first terminal position, only the estimation of the first terminal position may be executed. In this case, the terminal 100 and the information processing device 200 execute the operation shown in FIG. 5. FIG. 5 is a sequence diagram showing another example of the operation by the system.

[0080] The terminal 100 acquires the first sensor information and the second sensor information (S21). Step S21 is the same as step S11.

[0081] The terminal 100 transmits the first sensor information and the second sensor information to the information processing device 200 via the communication network 50 (S22). Step S22 is the same as step S12.

[0082] Upon receiving the first sensor information and the second sensor information, the information processing device 200 estimates the position of the first terminal based on the first sensor information and the second sensor information (S23). Step S23 is the same as step S13.

[0083] The terminal 100 acquires the first sensor information acquired at the third time (the current time) and the second sensor information acquired between the first time and the third time (S24). Here, the first time is the time identified based on the time information acquired in step S15 the previous time the process shown in FIG. 3 was performed. In step S24, the terminal 100 may identify the first time by reading the most recent time information among the time information stored in the storage unit 105. For example, the third time is a time later than the second time.

[0084] The terminal 100 estimates the second terminal position, which is the position of the terminal 100 at the third time, by adding the change in the position and attitude of the terminal 100 from the first time to the third time to the first terminal position, which is the position of the terminal 100 at the first time (S25).

[0085] 3 was previously executed (S26). Specifically, the terminal 100 displays the three-dimensional model superimposed on the first sensor information (image) acquired in step S24 based on the second terminal position.

[0086] After step S23, the information processing device 200 may transmit the estimated first terminal position and time information indicating the first time to the terminal 100. In this case, in step S24, the terminal 100 acquires the first sensor information acquired at the third time (current time) and the second sensor information acquired between the first time and the third time based on the time information acquired at this time.

[0087] [Issues and Effects, etc.] A description will be given of a problem that occurs when the present invention is not used. As shown in FIG. 5 , from when the terminal 100 acquires first sensor information and second sensor information at a first time t1 to when the terminal 100 displays a three-dimensional model generated based on the first sensor information based on the second terminal position at a second time t2, the user moves the terminal 100 to scan the object 10, causing the position and orientation of the terminal 100 to change. Note that the delay time T from the first time t1 to the second time t2 is the sum of the communication time required for the terminal 100 to transmit the first sensor information and the second sensor information to the information processing device 200, the processing time required for the information processing device 200 to estimate the first terminal position, the processing time required for the information processing device 200 to generate the three-dimensional model, and the communication time required for the information processing device 200 to transmit the first terminal position and the three-dimensional model to the terminal 100. As such, the position and orientation of the terminal 100 change during the delay time T, and therefore the terminal 100 is likely to differ from the first terminal position at the first time. Therefore, if terminal 100 attempts to display the three-dimensional model at the first time as is for the image taken at the second time, there is a high possibility that the three-dimensional model will be displayed at a position that is offset from the object in the image, because the viewpoint when the image was taken is different from the viewpoint when the image that served as the basis for the three-dimensional model was taken.

[0088] The effects of using the present invention will be described. The terminal 100 according to this embodiment includes a first sensor 101, a second sensor 102, and a circuit 103. The first sensor 101 sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal 100. The second sensor 102 sequentially acquires second sensor information for estimating changes in the position and orientation of the terminal 100. The circuit 103 is connected to the first sensor 101 and the second sensor 102. The circuit 103 transmits the first sensor information acquired by the first sensor 101 at a first time to the information processing device 200. The circuit 103 acquires from the information processing device 200 the first terminal position of the terminal 100 at the first time estimated by the information processing device 200 based on the first sensor information acquired at the first time, and the three-dimensional model generated by the information processing device 200 based on the first sensor information acquired at the first time. The circuit 103 estimates a second terminal position at a second time based on the first terminal position and the second sensor information, where the second time is the time at which the second terminal position estimation is being performed. The circuit 103 displays the obtained three-dimensional model based on the second terminal position.

[0089] According to this, the terminal 100 estimates the second terminal position, which is the position of the terminal 100 at the second time, based on the first terminal position, which is the position of the terminal 100 at the first time, and the second sensor information, and displays a three-dimensional model based on the estimated second terminal position, so that it is possible to display a three-dimensional model with the second terminal position as the viewpoint. Thus, the terminal 100 can display a three-dimensional model according to the second terminal position at the second time, and can appropriately display the three-dimensional model.

[0090] In the terminal 100 according to this embodiment, the circuit 103 estimates the first terminal position based on the second sensor information in addition to the first sensor information.

[0091] According to this, the terminal 100 further estimates the first terminal position based on the second sensor information, and therefore, can estimate the first terminal position with higher accuracy.

[0092] In terminal 100 according to this embodiment, the accuracy of the first terminal position is higher than the accuracy of the second terminal position.

[0093] This allows the terminal 100 to estimate the second terminal position based on the highly accurate first terminal position, thereby reducing the processing load.

[0094] In the terminal 100 according to this embodiment, in estimating the second terminal position, the circuit 103 estimates the amount of change in the position and attitude of the terminal 100 from the first time to the second time based on the second sensor information acquired from the first time to the second time, and estimates the second terminal position based on the first terminal position and the amount of change.

[0095] Therefore, since terminal 100 estimates the second terminal position at the second time based on the amount of change in the position and attitude of terminal 100 from the first time to the second time using the first terminal position as a reference, it is possible to reduce the processing load involved in estimating the second terminal position, which is the position of terminal 100 at the second time. Furthermore, since terminal 100 estimates the second terminal position in accordance with the amount of change in position from the first time to the second time using the highly accurate first terminal position as a reference, it is possible to estimate the second terminal position with minimal error.

[0096] In the terminal 100 according to the present embodiment, the circuit 103 further acquires time information for identifying the first time together with the first terminal position and the three-dimensional model, and then estimates the second terminal position based on the time information.

[0097] Therefore, the terminal 100 can easily identify the first time to which the first terminal position and the three-dimensional model correspond, using the time information, and can easily estimate the amount of change in the position and attitude of the terminal from the first time to the second time based on the second sensor information.

[0098] In the terminal 100 according to the present embodiment, the first sensor information includes a plurality of images captured at different positions and orientations. The circuit 103 acquires an image included in the first sensor information acquired by the first sensor at a second time, and displays a three-dimensional model superimposed on the image based on the second terminal position.

[0099] Therefore, since terminal 100 displays a three-dimensional model with the second terminal position as the viewpoint for the image captured at the second time, it is possible to superimpose and display the three-dimensional model as viewed from the same viewpoint as the viewpoint when the image was captured. Thus, terminal 100 can display a three-dimensional model according to the second terminal position at the second time, and can appropriately display the three-dimensional model.

[0100] In the terminal 100 according to the present embodiment, the three-dimensional model superimposed on the image is displayed in a manner different from the image itself.

[0101] Therefore, since terminal 100 can display images and three-dimensional models in a manner that makes it easy to distinguish between them, users can easily check the progress of three-dimensional model generation by looking at the images and three-dimensional models displayed on terminal 100.

[0102] Moreover, the information processing device 200 according to this embodiment includes a storage unit 202 (memory) and a circuit 201 connected to the storage unit 202. The circuit 201 acquires, from the terminal 100, first sensor information for generating a three-dimensional model and estimating the position and attitude of the terminal 100, the first sensor information being acquired at a first time among the first sensor information sequentially acquired by a first sensor included in the terminal 100. The circuit 201 estimates the first terminal position of the terminal 100 at the first time based on the first sensor information acquired at the first time. The circuit 201 generates a three-dimensional model based on the first sensor information acquired at the first time. The circuit 201 transmits the first terminal position, the three-dimensional model, and time information for identifying the first time to the terminal 100.

[0103] According to this, the information processing device 200 transmits to the terminal 100 the first terminal position, the three-dimensional model, and time information for identifying the first time corresponding to the first terminal position and the three-dimensional model, so that the terminal 100 can easily identify the first time corresponding to the first terminal position and the three-dimensional model. Therefore, the terminal 100 can easily estimate the amount of change in the position and attitude of the terminal 100 from the first time to the second time based on the second sensor information, for example. In other words, the terminal 100 can display a three-dimensional model corresponding to the second terminal position at the second time, and the three-dimensional model can be displayed appropriately.

[0104] [Modifications] (1) In the above embodiment, the first sensor 101 included in the terminal 100 is a camera. However, the present invention is not limited to this and may be a distance sensor. The distance sensor measures the distance between the distance sensor and an object by, for example, emitting electromagnetic waves, receiving the electromagnetic waves reflected by the object, and measuring the time it takes for the electromagnetic waves to reach the object after being emitted. Examples of electromagnetic waves include sound waves, light, and radio waves. Examples of the distance sensor include a LiDAR sensor and a Time of Flight (ToF) sensor. In this case, the information processing device 200 generates a three-dimensional model using distance information acquired by the distance sensor.

[0105] (2) In the above embodiment, the first sensor 101 included in the terminal 100 may be a stereo camera. This allows the information processing device 200 to generate a three-dimensional model at the first time point using two images captured at the first time point.

[0106] (3) In the above embodiment, terminal 100 superimposes a three-dimensional model on the image acquired at the second time and displays it on display unit 104. However, the present invention is not limited to superimposing a three-dimensional model on an image. Terminal 100 may display the image acquired at the second time and the three-dimensional model in two separate areas of display unit 104 without superimposing them. Terminal 100 may also display only the three-dimensional model on display unit 104.

[0107] (4) In the above embodiment, the terminal 100 identifies the first time based on time information, but this is not limited to this. The time from the first time to the second time may be a fixed time if, for example, the communication speed of the communication network 50 and the processing speed of various processes performed by the terminal 100 and the information processing device 200 are stable and constant. Therefore, the terminal 100 may estimate the first time by subtracting the fixed time from the second time. This allows the terminal 100 to identify the first time without acquiring time information from the information processing device 200.

[0108] (5) In the above embodiment, information processing device 200 transmits time information to terminal 100 along with the first terminal position and the three-dimensional model. However, this is not limited thereto, and the information processing device 200 does not necessarily have to transmit the time information at every opportunity to transmit the first terminal position and the three-dimensional model. For example, information processing device 200 may transmit time information to terminal 100 once every predetermined number of opportunities to transmit the first terminal position and the three-dimensional model. In this case, information processing device 200 repeatedly transmits the first terminal position and the three-dimensional model to terminal 100, for example, at a third cycle. Therefore, between the time information transmission to terminal 100 and the next time the information processing device 200 transmits the time information to terminal 100, terminal 100 can identify the time to which the first terminal position and the three-dimensional model correspond by integrating the third cycle based on the time information. In this way, terminal 100 can identify the first time without constantly acquiring time information from information processing device 200 along with the first terminal position and the three-dimensional model.

[0109] [Others] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Here, the software that realizes the information processing device of the above embodiments is a program that causes a computer to execute each step included in the flowcharts shown in the figures.

[0110] The following cases are also included in this disclosure:

[0111] (1) Specifically, each of the above devices is a computer system consisting of a microprocessor, ROM, RAM, hard disk unit, display unit, keyboard, mouse, etc. A computer program is stored in the RAM or hard disk unit. Each device achieves its function when the microprocessor operates in accordance with the computer program. Here, a computer program is composed of a combination of multiple instruction codes that indicate commands to a computer to achieve a specified function.

[0112] (2) Some or all of the components constituting each of the above devices may be configured as a single system LSI (Large Scale Integration). A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple components on a single chip, and specifically, is a computer system configured to include a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating in accordance with the computer program.

[0113] (3) Some or all of the components constituting each of the above devices may be configured as an IC card or a standalone module that can be attached to each device. The IC card or module may be a computer system configured with a microprocessor, ROM, RAM, etc. The IC card or module may include the above-mentioned ultra-multifunctional LSI. The IC card or module achieves its functions when the microprocessor operates in accordance with a computer program. The IC card or module may be tamper-resistant.

[0114] (4) The present disclosure may be embodied as the above-described methods, a computer program for implementing these methods on a computer, or a digital signal comprising the computer program.

[0115] The present disclosure may also be a computer program or a digital signal recorded on a computer-readable recording medium, such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. Alternatively, the present disclosure may be a digital signal recorded on such a recording medium.

[0116] Furthermore, the present disclosure may also be applied to transmitting the computer program or the digital signal via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, or the like.

[0117] The present disclosure may also be a computer system including a microprocessor and a memory, wherein the memory stores the computer program, and the microprocessor operates in accordance with the computer program.

[0118] The program or the digital signal may also be implemented by another independent computer system by recording it on the recording medium and transferring it, or by transferring it via the network or the like.

[0119] (5) The above-described embodiments and modifications may be combined with each other.

[0120] The present disclosure is useful as a terminal or the like that can appropriately display a three-dimensional model.

[0121] REFERENCE SIGNS LIST 1 System 10 Object 50 Communication network 100 Terminal 101 First sensor 102 Second sensor 103 Circuit 104 Display unit 105 Storage unit 111 Communication unit 112 Estimation unit 113 Display control unit 200 Information processing device 201 Circuit 202 Storage unit 211 Communication unit 212 Estimation unit 213 Generation unit

Claims

1. A terminal, comprising: a first sensor that sequentially acquires first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal; a second sensor that sequentially acquires second sensor information for estimating the amount of change in the position and orientation of the terminal; and a circuit connected to the first sensor and the second sensor, wherein the circuit transmits the first sensor information acquired by the first sensor at a first time to an information processing device, acquires from the information processing device the first terminal position of the terminal at the first time estimated by the information processing device based on the first sensor information acquired at the first time and the three-dimensional model generated by the information processing device based on the first sensor information acquired at the first time, estimates a second terminal position at a second time based on the first terminal position and the second sensor information, where the second time is the time when the estimation of the second terminal position is being executed, and displays the acquired three-dimensional model based on the second terminal position.

2. The terminal according to claim 1, wherein the circuit estimates the first terminal position based on the second sensor information in addition to the first sensor information.

3. The terminal according to claim 1, wherein the accuracy of the first terminal position is higher than the accuracy of the second terminal position.

4. The terminal according to claim 1, wherein in the estimation of the second terminal position, the circuit estimates the amount of change in the position and orientation of the terminal from the first time to the second time based on the second sensor information acquired from the first time to the second time, and estimates the second terminal position based on the first terminal position and the amount of change.

5. The terminal according to any one of claims 1 to 4, wherein the second sensor information includes the acceleration of the terminal and the angular velocity of the terminal.

6. The terminal according to any one of claims 1 to 4, wherein the circuit further acquires time information for specifying the first time together with the first terminal position and the three-dimensional model, and estimates the second terminal position based on the time information.

7. The first sensor information includes a plurality of images captured at different positions and postures, and the circuit obtains the images included in the first sensor information acquired by the first sensor at the second time, and based on the second terminal position, superimposes and displays the three-dimensional model on the image. The terminal according to any one of claims 1 to 4.

8. The three-dimensional model superimposed on the image is displayed in a manner different from the image. The terminal according to claim 7.

9. An information processing apparatus comprising a memory and a circuit connected to the memory, wherein the circuit obtains, from the terminal, first sensor information acquired at a first time among the first sensor information sequentially acquired by a first sensor provided in the terminal, which is for generating a three-dimensional model and estimating the position and posture of the terminal, estimates a first terminal position of the terminal at the first time based on the first sensor information acquired at the first time, generates a three-dimensional model based on the first sensor information acquired at the first time, and transmits the first terminal position, the three-dimensional model, and time information for specifying the first time to the terminal.

10. A display method executed by a terminal, which transmits, to an information processing apparatus, first sensor information acquired at a first time among the first sensor information sequentially acquired by a first sensor provided in the terminal, which is for generating a three-dimensional model and estimating the position and posture of the terminal, obtains from the information processing apparatus a first terminal position of the terminal at the first time estimated by the information processing apparatus based on the first sensor information acquired at the first time and a three-dimensional model generated by the information processing apparatus based on the first sensor information acquired at the first time, estimates a second terminal position at a second time based on the first terminal position and second sensor information for estimating a change amount of the position and posture of the terminal, which is the second sensor information sequentially acquired by a second sensor provided in the terminal, where the second time is when the estimation of the second terminal position is being executed, and based on the second terminal position, displays the obtained three-dimensional model.

11. An information processing method executed by an information processing apparatus, the method comprising: obtaining, from the terminal, first sensor information acquired sequentially by a first sensor included in the terminal, which is first sensor information for generating a three-dimensional model and estimating the position and orientation of the terminal, and which is the first sensor information acquired at a first time; estimating, based on the first sensor information acquired at the first time, a first terminal position of the terminal at the first time; generating a three-dimensional model based on the first sensor information acquired at the first time; and transmitting the first terminal position, the three-dimensional model, and time information for specifying the first time to the terminal.

Images