Terminal device, object administrative server, and terminal device control method

Abstract

This terminal device comprises: an acquisition unit that acquires first spatial structure data indicating a spatial structure of a real space and second spatial structure data indicating a spatial structure around an anchor serving as a reference when a virtual object is disposed in the real space; a reception unit that receives, on a screen displayed on a display device, an operation of a user who adjusts relative positions between a first image indicating the spatial structure determined by the first spatial structure data and a second image indicating the spatial structure around the anchor determined by the second space structure data and the anchor; and a position determination unit that determines a position of the anchor in the spatial structure determined by the first spatial structure data, on the basis of a user operation received by the reception unit.

Description

Terminal device, object management server, and terminal device control method

[0001] The present invention relates to a terminal device, an object management server, and a method for controlling a terminal device.

[0002] Patent Literature 1 discloses a technology for generating spatial data in which anchors that serve as references for the positions of virtual objects are mapped onto an image that represents a real-world environment captured by a user device.

[0003] Japanese Patent Application Laid-Open No. 2019-185759

[0004] However, there are cases where the first application that determines the spatial structure of the real space and the second application that places the anchor are different from each other. The first application determines the three-dimensional spatial structure by measuring the real space in three dimensions. On the other hand, the second application obtains a simple three-dimensional spatial structure. When using such different applications, it is necessary to align the simple spatial structure in which the anchor is placed with the high-precision spatial structure required to control the virtual object. However, conventional techniques have had a problem in that alignment between the simple airport structure and the high-precision spatial structure is not possible.

[0005] The present disclosure aims to provide a terminal device that determines the position of an anchor in a spatial structure for controlling a virtual object by performing registration between two types of spatial structures.

[0006] A terminal device according to the present disclosure includes an acquisition unit that acquires first spatial structure data indicating a spatial structure of a real space and second spatial structure data indicating a spatial structure around an anchor that serves as a reference when placing a virtual object in the real space; a reception unit that receives, on a screen displayed on a display device, a user's operation to adjust the relative positions of a first image indicating the spatial structure determined by the first spatial structure data, the spatial structure around the anchor determined by the second spatial structure data, and the second image indicating the anchor; and a position determination unit that determines the position of the anchor in the spatial structure determined by the first spatial structure data based on the user's operation received by the reception unit.

[0007] According to the present disclosure, by performing registration between two types of spatial structures, it is possible to determine the position of an anchor in a spatial structure for controlling a virtual object.

[0008] 1 is a block diagram showing the overall configuration of an information processing system 1 according to an embodiment. A block diagram showing the configuration of a terminal device 10 according to an embodiment. An explanatory diagram showing an example of a captured image Gk. An explanatory diagram showing the contour lines of a spatial structure included in first spatial structure data Ds1. An explanatory diagram showing an example of an image Gk1. An explanatory diagram showing an example of a first image G1. An explanatory diagram showing an example of a second image G2. An explanatory diagram showing an example of a screen displayed on a display device 17 by a position determination unit 116. An explanatory diagram showing an example of a screen displayed on a display device 17 by a position determination unit 116. An explanatory diagram showing an example of an image Gk2. A block diagram showing an example of the configuration of an object management server 30. An explanatory diagram showing an example of the data structure of a database DB. A flowchart showing the contents of an anchor placement process. A flowchart showing the contents of a position determination process. A flowchart showing the contents of an object placement process.

[0009] 1: Embodiment An information processing system 1 will be described below.

[0010] 1.1: Configuration of the Embodiment 1.1.1: Overall Configuration Fig. 1 is a block diagram showing the overall configuration of an information processing system 1. As shown in Fig. 1, the information processing system 1 includes a terminal device 10 used by a developer, an anchor management server 20 that manages anchors P (see Fig. 3C), an object management server 30 that manages virtual objects VO, and a user device 40 used by a user of the service. The information processing system 1 shown in Fig. 1 includes one terminal device 10. However, the information processing system 1 may include multiple terminal devices 10. Furthermore, the information processing system 1 may include multiple user devices 40.

[0011] In the information processing system 1, the terminal device 10, the anchor management server 20, the object management server 30, and the user device 40 communicate with each other via a communication network NW.

[0012] The terminal device 10 is configured, for example, by a smartphone or a tablet terminal. The terminal device 10 is used by a developer of a service that provides users with an augmented reality space or a mixed reality space. The developer virtually places a virtual object VO used in the service at a position in real space. The developer is a user of the terminal device 10. The terminal device 10 provides an image Gk2 (see FIG. 3H ) that assists the developer in placing the virtual object VO. The image Gk2 includes an anchor image Gak. The anchor image Gak indicates an X-axis (roll axis), a Y-axis (pitch axis), and a Z-axis (yaw axis) that are orthogonal to each other, and an anchor P is placed at the origin of each axis. The virtual object VO is placed at a relative position based on the anchor P shown in FIG. 3H . The orientation of the virtual object VO is determined by a yaw angle, a roll angle, and a pitch angle. The yaw angle is a rotation angle around the Z-axis. The roll angle is a rotation angle around the X-axis. The pitch angle is a rotation angle around the Z-axis. The virtual object image VOx shown in FIG. 3H is an image showing the virtual object VO in two dimensions.

[0013] When data relating to an anchor P is uploaded from the terminal device 10, the anchor management server 20 issues an anchor ID corresponding to the anchor P. The anchor ID is identification data that uniquely identifies the anchor P.

[0014] The anchor management server 20 stores the anchor ID in association with data related to the anchor P. The data related to the anchor P may include first spatial structure data Ds1 that indicates, in three dimensions, the spatial structure of the real space including the anchor P. The anchor ID is transmitted from the anchor management server 20 to the terminal device 10.

[0015] The terminal device 10 uploads virtual object data Dv indicating a virtual object VO, an anchor ID, position data Dp indicating the position of the anchor P, and additional data Da related to the anchor P to the object management server 30. The object management server 30 manages the virtual object data Dv, the anchor ID, the position data Dp, and the additional data Da in association with each other. The additional data Da includes image data Dk2 indicating an image Gk2 and second spatial structure data Ds2. The second spatial structure data Ds2 indicates the spatial structure around the anchor P. The second spatial structure data Ds2 may be in any data format as long as it can represent a three-dimensional spatial structure. Examples of data formats include the GL Transmission Format Binary (GLB) format, obj format, fbx format, and stl format. GLB represents the GL Transmission Format (glTF) in binary format. GLB is primarily used when creating content that runs on a web browser.

[0016] The user device 40 obtains virtual object data Dv and an anchor ID by accessing the object management server 30. The user device 40 transmits image data Dg captured by the user device 40 and the obtained anchor ID to the anchor management server 20, thereby obtaining self-position data from the anchor management server 20 indicating the position of the user device 40 relative to the position of the anchor P as the origin and the direction in which the user device 40 faces the outside world. The user device 40 is equipped with a display device and an inertial sensor. Based on the self-position data and output data from the inertial sensor, the user device 40 displays on the display device an image in which the virtual object VO is superimposed on real space.

[0017] 2: Configuration of the Terminal Device As shown in Fig. 2, the terminal device 10 includes a processing device 11, a storage device 12, a measuring device 13, an imaging device 14, a communication device 16, a display device 17, and a detection device 18. The elements of the terminal device 10 are connected to each other by one or more buses for communicating data. Note that the term "device" in this specification may be interpreted as other terms such as circuit, device, or unit.

[0018] The processing device 11 is a processor that controls the entire terminal device 10. The processing device 11 is configured using, for example, one or more chips. The processing device 11 is also configured using, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing device 11 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 11 executes various processes in parallel or sequentially.

[0019] The storage device 12 is a recording medium that can be read and written by the processing device 11. The storage device 12 also stores a control program PR1 executed by the processing device 11, a plurality of programs including an object adjustment application A1 and a space adjustment application A2, and various data including virtual object data Dv. The storage device 12 functions as a work area for the processing device 11. The virtual object data Dv is data that represents a virtual object VO in three dimensions.

[0020] The object adjustment application A1 is responsible for placing and confirming the anchor P, previewing the virtual object VO, and adjusting the position of the virtual object VO. Confirming the anchor P means that the developer confirms where the anchor P is placed by displaying the anchor P on the display device 17. In the process of placing the anchor P, the object adjustment application A1 uploads data related to the anchor P to the anchor management server 20. Furthermore, the object adjustment application A1 acquires an anchor ID from the anchor management server 20.

[0021] The space adjustment application A2 has a function of editing the anchor P placed by the object adjustment application A1 and the simple space structure around the anchor P.

[0022] The measuring device 13 measures the real space in three dimensions under the control of the processing device 11, and outputs measurement data Dm indicating the measurement results to the processing device 11. The measuring device 13 may be configured with, for example, a LiDEAR or ToF sensor. One or more physical objects, such as a floor, a wall, and a table, exist in the real space. In this embodiment, the measuring device 13 is included in the terminal device 10. However, the measuring device 13 does not have to be included in the terminal device 10. When the measuring device 13 is an external device of the terminal device 10, the terminal device 10 and the measuring device 13 perform wired communication or wireless communication.

[0023] The measurement data Dm indicates the distances from a reference point in real space to multiple points located on the surface of one or more physical objects present in the real space. The reference point is the point where the measurement device 13 is located in real space. The processing device 11 determines first spatial structure data Ds1 indicating the spatial structure of the real space based on the measurement data Dm. The one or more physical objects present in the real space include, for example, floors, walls, and tables present in the real space. For example, if the spatial structure of the elevator hall shown in FIG. 3A is measured in three dimensions using the measurement device 13, the first spatial structure data Ds1 includes at least the contours of the floors, walls, and elevator doors. FIG. 3B is an explanatory diagram showing the contours of the spatial structure included in the first spatial structure data Ds1. The dotted lines shown in FIG. 3B represent the contours of the floors, walls, and elevator doors.

[0024] The imaging device 14 outputs image data Dg representing an imaged image Gk obtained by capturing an image of the outside world. The imaging device 14 also includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an analog imaging signal by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified analog imaging signal into image data Dg, which is a digital signal. The converted image data Dg is output to the processing device 11. For example, consider a case where a virtual object VO is placed in an elevator hall. When a developer captures an image of the elevator hall using the terminal device 10, for example, an imaged image Gk shown in FIG. 3A is captured.

[0025] When placing an anchor P, the developer uses the imaging device 14 to capture images of the area around the anchor P from multiple directions. As a result, image data Dg captured from multiple directions is imported into the processing device 11. The processing device 11 determines second spatial structure data Ds2 that indicates the spatial structure around the anchor P based on the image data Dg.

[0026] The first spatial structure data Ds1 is determined based on measurement data Dm output from a measurement device 13 that measures distance. Therefore, the first spatial structure data Ds1 is more accurate than the second spatial structure data Ds2 that is determined based on image data Dg. By using the first spatial structure data Ds1, the user device 40 can accurately represent, for example, the appearance of a ball virtual object VO hitting a wall and bouncing back. On the other hand, the second spatial structure data Ds2 simply represents the spatial structure around the anchor P. By using the second spatial structure data Ds2, the terminal device 10 can provide the developer with a GUI for placing the anchor P.

[0027] The input device 15 accepts operations by the developer. The input device 15 outputs operation data Do corresponding to the developer's operation to the processing device 11. The input device 15 is configured, for example, by a touch panel. The developer can use the input device 15 to specify the position of an anchor P on the captured image Gk. For example, assume that the developer specifies point X on the captured image Gk shown in FIG. 3A. In this case, the input device 15 outputs operation data Do indicating the position of point X. For example, if the input device 15 is configured by a touch panel with 1,024 horizontal pixels and 768 vertical pixels, the position of point X is specified by coordinates on the touch panel. In the case of the captured image Gk shown in FIG. 3A, the horizontal direction of the captured image Gk is the vertical direction, and the vertical direction is the horizontal direction. In the following description, the operation data Do specifying the position of the anchor P is referred to as operation data Dox. When the processing device 11 receives the operation data Dox, it displays the image Gk1 shown in FIG. 3C on the display device 17. The image Gk1 is obtained by superimposing an anchor image Gak on the captured image Gk shown in FIG. 3A.

[0028] The input device 15 is used when aligning the spatial structure indicated by the first spatial structure data Ds1 with the spatial structure around the anchor P indicated by the second spatial structure data Ds2. Specifically, the input device 15 outputs operation data Do indicating a user operation for adjusting the relative positions of the first image G1 and the second image G2 on the screen displayed on the display device 17 to the processing device 11. In the following description, the operation data Do indicating a user operation for adjusting the relative positions of the first image G1 and the second image G2 will be referred to as operation data Doy.

[0029] The first image G1 shows a spatial structure determined by the first spatial structure data Ds1. Fig. 3D is an example of the first image G1. The first image G1 shown in Fig. 3D is determined based on the spatial structure of the first spatial structure data Ds1 shown in Fig. 3B.

[0030] The second image G2 shows the anchor P and the spatial structure around the anchor P determined by the second spatial structure data Ds2. The second image G2 shown in Fig. 3E includes an anchor image Gak.

[0031] The communication device 16 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 16 may also be referred to as, for example, a network device, a network controller, a network card, or a communication module. The communication device 16 may include a connector for wired connection, and the object management server 30 corresponding to the connector may include a circuit. The communication device 16 may also include a wireless communication interface. Examples of connectors and interface circuits for wired connection include products that comply with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0032] The display device 17 is a device for displaying images. The display device 17 displays various images under the control of the processing device 11.

[0033] The detection device 18 detects the state of the terminal device 10. The detection device 18 is, for example, an inertial sensor such as an acceleration sensor that detects acceleration and a gyro sensor that detects angular acceleration, and a geomagnetic sensor that detects orientation. The acceleration sensor detects acceleration along orthogonal X-, Y-, and Z-axes. The gyro sensor detects angular acceleration around the X-, Y-, and Z-axes as central axes of rotation. The detection device 18 can generate attitude data indicating the attitude of the terminal device 10 based on the output data of the gyro sensor. The detection device 18 also outputs to the processing device 11 the attitude data indicating the attitude of the terminal device 10, motion data related to the movement of the terminal device 10, and orientation data indicating the orientation of the terminal device 10. The motion data includes acceleration data indicating acceleration along each of the three axes and angular acceleration data indicating angular acceleration along each of the three axes. The attitude data, motion data, orientation data, etc. are included in the detection data Dd output from the detection device 18.

[0034] In the above configuration, the processing device 11 reads out the control program PR1, the object adjustment application A1, and the space adjustment application A2 from the storage device 12. The processing device 11 executes the read out control program PR1, the object adjustment application A1, and the space adjustment application A2, thereby functioning as a first structure determination unit 111, a second structure determination unit 112, an acquisition unit 113, a reception unit 114, a position determination unit 116, an object placement unit 117, and a communication control unit 118.

[0035] The first structure determination unit 111 determines first spatial structure data Ds1 based on the measurement data Dm. As described above, the measurement data Dm indicates distances from the measurement device 13 to multiple points located on the surfaces of one or more physical objects that exist in real space. The first structure determination unit 111 determines first spatial structure data Ds1 that represents the spatial structure of real space in three dimensions based on the distances.

[0036] The second structure determination unit 112 determines the second spatial structure data Ds2 based on the image data Dg indicating the captured image Gk around the anchor P. The range of the real space captured by the imaging device 14 does not necessarily coincide with the range of the real space indicated by the first spatial structure data Ds1. The spatial structure indicated by the first spatial structure data Ds1 only needs to include the spatial structure around the anchor P in the real space captured by the imaging device 14.

[0037] The second structure determination unit 112 determines the second spatial structure data Ds2 by, for example, executing a first process and a second process. In the first process, the second structure determination unit 112 detects one or more planes in the spatial structure around the anchor P. In the second process, the second structure determination unit 112 determines the second spatial structure data Ds2 by combining the one or more detected planes.

[0038] In the first process, the second structure determination unit 112 analyzes image data Dg obtained by, for example, capturing images of real space while moving the terminal device 10. There are various methods for detecting planes. In one example of a plane detection method, in a first step, feature points are extracted from the captured image Gk for each frame. In a second step, changes in the positions of the feature points between frames are tracked. In a third step, planes are detected by comparing changes in the position and direction of the image capture device 14 obtained from the detection data Dd with changes in the positions of the feature points. The detection data Dd is output from the detection device 18.

[0039] The acquisition unit 113 acquires the first spatial structure data Ds1 and the second spatial structure data Ds2.

[0040] The receiving unit 114 receives operation data Doy indicating a user operation for adjusting the relative positions of a first image G1 indicating the spatial structure determined by the first spatial structure data Ds1 and a second image G2 indicating the spatial structure around the anchor P determined by the second spatial structure data Ds2 and the anchor P on the screen displayed on the display device 17. The receiving unit 114 also receives operation data Dox for specifying, as a point on the captured image Gk, the anchor P that serves as a reference for the position of a virtual object VO that is virtually placed in real space.

[0041] The anchor placement unit 115 generates a second image G2 in which an anchor P is placed in the spatial structure indicated by the second spatial structure data Ds2, based on the operation data Dox and the second spatial structure data Ds2. The anchor placement unit 115 displays, for example, the second image G2 shown in FIG. 3E on the display device 17. This display allows the developer to confirm how the anchor P is placed in real space. As described above, the spatial structure indicated by the second spatial structure data Ds2 includes one or more planes. The anchor placement unit 115 places the anchor P on one of the one or more planes. More specifically, the anchor placement unit 115 places the anchor P at a position on the plane closest to the position of point X indicated by the operation data Dox.

[0042] The position determination unit 116 determines position data Dp indicating the position of the anchor P in the spatial structure determined by the first spatial structure data Ds1. The anchor placement unit 115 places the anchor P on the spatial structure indicated by the second spatial structure data Ds2. However, the second spatial structure data Ds2 merely indicates the spatial structure around the anchor P in a simplified manner. To control the virtual object VO in an augmented reality space or a mixed reality space, it is necessary to determine the position of the anchor P in the spatial structure determined by the first spatial structure data Ds1. As described above, the first spatial structure data Ds1 is determined by three-dimensionally measuring the spatial structure of the real space using the measurement device 13. On the other hand, the second spatial structure data Ds2 is determined based on the image data Dg. As such, the first spatial structure data Ds1 and the second spatial structure data Ds2 have different algorithms for determination. Therefore, it is necessary to align the spatial structure indicated by the first spatial structure data Ds1 and the spatial structure indicated by the second spatial structure data Ds2.

[0043] The position determination unit 116 aligns the spatial structure indicated by the first spatial structure data Ds1 with the spatial structure around the anchor P in the second spatial structure data Ds2, based on the operation data Doy received by the reception unit 114. Specifically, the position determination unit 116 prompts the developer to perform manual alignment by displaying the first image G1 and the second image G2 on the display device 17. Figures 3F and 3G show screens displayed on the display device 17 by the position determination unit 116.

[0044] In the screen shown in FIG. 3F, the floor surface of the first image G1 and the floor surface of the second image G2 are spaced apart. The developer operates the input device 15 to adjust the positions of the first image G1 and the second image G2 on the screen of the display device 17. As a result of this adjustment, the screen shown in FIG. 3G is displayed on the display device 17. The developer can confirm that the position of the floor surface of the first image G1 and the position of the floor surface of the second image G2 are approximately the same on the screen shown in FIG. 3G. When the developer presses button B shown in FIG. 3G, the receiving unit 114 receives operation data Do indicating that button B has been pressed. Triggered by the receiving unit 114 receiving this operation data Do, the position determining unit 116 determines the coordinates of the anchor P in the spatial structure indicated by the first spatial structure data Ds1 of the anchor P. The coordinates of the anchor P are position data Dp indicating the position of the anchor P.

[0045] The object placement unit 117 reads out virtual object data Dv from the storage device 12 based on the operation data Do. The object placement unit 117 displays a virtual object image VOx representing the virtual object VO on the display device 17 based on the read virtual object data Dv. The virtual object image VOx is an image that represents the virtual object VO in two dimensions. The object placement unit 117 places the virtual object image VOx at a position specified by the developer based on the operation data Do output from the input device 15.

[0046] The object placement unit 117 displays, for example, the image Gk2 shown in FIG. 3F on the display device 17. The image Gk2 has an anchor image Gak and a virtual object image VOx superimposed on the captured image Gk shown in FIG. 3A. When placing the virtual object VO, the developer can place the virtual object VO at coordinates (x, y, z) with the position of the anchor P as the origin by inputting x, y, and z coordinates into the input device 15. Alternatively, the developer may place the virtual object VO at coordinates (x, y, z) with the position of the anchor P as the origin by operating a touch panel serving as the input device 15 to adjust the position of the virtual object image VOx on the screen.

[0047] The object placement unit 117 determines the relative coordinate data Dr based on the coordinates (x, y, z) input by the developer using the input device 15 to place the virtual object VO. Alternatively, the object placement unit 117 determines the relative coordinate data Dr based on the result of an operation on the touch panel.

[0048] The communication control unit 118 transmits the first spatial structure data Ds1 and the additional data Da to the object management server 30 via the communication device 16 .

[0049] 1.1.3: Configuration of Object Management Server Fig. 4 is a block diagram showing an example configuration of the object management server 30. The object management server 30 comprises a processing device 31, a storage device 32, a communication device 33, a display device 34, and an input device 35. The elements of the object management server 30 are connected to each other by one or more buses for communicating data.

[0050] The processing device 31 is a processor that controls the entire object management server 30. The processing device 11 is configured, for example, using one or more chips. The processing device 11 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, registers, etc. Some or all of the functions of the processing device 11 may be realized by hardware such as a DSP, ASIC, PLD, FPGA, etc. The processing device 11 executes various processes in parallel or sequentially.

[0051] The storage device 32 is a recording medium that can be read from and written to by the processing device 31. The storage device 32 also stores a database DB and multiple programs including a control program PR2 executed by the processing device 31. The storage device 32 also functions as a work area for the processing device 31.

[0052] The database DB is used to manage virtual objects VO that are virtually placed in real space by the developer.

[0053] The communication device 33 shown in Fig. 4 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 33 is an example of a first communication device. The communication device 33 is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 33 may also include a wireless communication interface.

[0054] The display device 34 is a device that displays images and characters. The input device 35 includes, for example, a keyboard, a touch pad, a touch panel, or a pointing device such as a mouse.

[0055] The processing device 31 reads and executes the control program PR2 from the storage device 32. As a result, the processing device 31 functions as an acquisition unit 311 and a management unit 312.

[0056] The acquisition unit 311 acquires the virtual object data Dv, the anchor ID, the relative coordinate data Dr, the additional data Da, and the first spatial structure data Ds1 from the terminal device 10 via the communication device 33.

[0057] The management unit 312 manages the virtual object data Dv, the anchor ID, the relative coordinate data Dr, the additional data Da, and the first spatial structure data Ds1 in association with one another. Specifically, the management unit 312 manages a database DB.

[0058] 5 is an explanatory diagram showing an example of the data structure of the database DB. The database DB has multiple records. One record associates a virtual object ID, virtual object data Dv, anchor ID, relative coordinate data Dr, additional data Da, and first spatial structure data Ds1. The virtual object ID is identification data that uniquely identifies the virtual object VO.

[0059] 2: Operation of the embodiment The following describes the anchor placement process, position determination process, and object placement process in the terminal device 10. The anchor placement process and position determination process are processes related to the space adjustment application A2. The object placement process is processes related to the object adjustment application A1.

[0060] 1.2.1: Anchor Placement Processing FIG. 6 is a flowchart showing the contents of the anchor placement processing.

[0061] In step S10, the processing device 11 acquires image data Dg. The image data Dg represents a captured image Gk obtained by a developer by capturing an image of the environment in which the virtual object VO is to be placed. The processing device 11 displays the captured image Gk on the display device 17. For example, the display device 17 displays the captured image Gk shown in FIG. 3A.

[0062] In step S11, the processing device 11 accepts an operation to place an anchor P. Specifically, the processing device 11 accepts operation data Dox that specifies the position of the anchor P. For example, assume that the captured image Gk shown in FIG. 3A is displayed on the display device 17. When the developer specifies a point X by tapping or the like, operation data Dox that indicates the position of point X is output from the input device 15.

[0063] In step S12, the processing device 11 places an anchor P based on the operation data Dox and the second spatial structure data Ds2. Specifically, the processing device 11 places the anchor P at a position on a plane that is closest to the position of point X indicated by the operation data Dox, among one or more planes specified by the second spatial structure data Ds2. The processing device 11 displays, on the display device 17, the second image G2 in which the anchor P is placed, as shown in Fig. 3E, for example.

[0064] In step S13, the processing device 11 acquires an anchor ID from the anchor management server 20. More specifically, the processing device 11 transmits an anchor registration request to the anchor management server 20 via the communication device 16. The anchor registration request includes data related to the anchor P. Upon receiving the anchor registration request, the anchor management server 20 issues an anchor ID for uniquely identifying the anchor P. The anchor management server 20 returns an anchor registration response including the issued anchor ID to the terminal device 10. The processing device 11 acquires the anchor ID by extracting the anchor ID included in the anchor registration response.

[0065] In the above anchor placement process, the processing device 11 functions as follows: The processing device 11 functions as a reception unit 114 in steps S10 and S11. The processing device 11 functions as an anchor placement unit 115 in step S12. The processing device 11 functions as a communication control unit 118 in step S13.

[0066] 1.2.2: Position Determination Processing FIG. 7 is a flowchart showing the contents of the position determination processing.

[0067] In step S20, the processing device 11 acquires the first spatial structure data Ds1 and the second spatial structure data Ds2.

[0068] In step S21, the processing device 11 displays a support image for alignment on the display device 17. The support image includes a first image G1 based on the first spatial structure data Ds1 and a second image G2 based on the second spatial structure data Ds2. For example, the screen shown in FIG. 3F is displayed on the display device 17. The developer operates the input device 15 to adjust the positions of the first image G1 and the second image G2 on the screen of the display device 17.

[0069] In step S22, the processing device 11 receives operation data Doy, which indicates a user operation for adjusting the relative positions of the first image G1 and the second image G2.

[0070] In step S23, the processing device 11 displays a support image corresponding to the operation data Doy on the display device 17. For example, when the developer moves the second image G2 shown in FIG. 3F so that the floor and walls of the second image G2 match the floor and walls of the first image G1, the support image shown in FIG. 3G is displayed on the display device 17.

[0071] In step S24, the processing device 11 determines whether or not the alignment is complete. Specifically, the processing device 11 makes this determination based on whether or not operation data Do indicating that button B shown in FIG. 3G has been pressed has been received. If the determination result in step S24 is negative, the processing device 11 returns the process to step S22. If the determination result in step S24 is positive, the processing device 11 proceeds to step S25.

[0072] In step S25, the processing device 11 determines position data Dp. The position data Dp indicates the position of the anchor P on the spatial structure indicated by the first spatial structure data Ds1. Specifically, the processing device 11 determines the position data Dp by converting the position of the anchor P in the first image G1 at the time when alignment is completed into a position on the spatial structure.

[0073] In step S26, the processing device 11 transmits the anchor ID and the position data Dp to the anchor management server 20. The anchor management server 20 manages the anchor ID and the position data Dp in association with each other.

[0074] In the above position determination process, the processing device 11 functions as follows: In step S20, the processing device 11 functions as the acquisition unit 113. In steps S21 to S25, the processing device 11 functions as the position determination unit 116. In step S26, the processing device 11 functions as the communication control unit 118.

[0075] 1.2.3: Object Placement Processing FIG. 8 is a flowchart showing the contents of the object placement processing.

[0076] In step S30 , the processing device 11 reads the virtual object data Dv from the storage device 12 .

[0077] In step S31, the processing device 11 displays the virtual object image VOx on the display device 17 based on the virtual object data Dv.

[0078] In step S32, the processing device 11 receives operation data Do output from the input device 15. The developer operates the input device 15 to adjust the arrangement of the virtual object image VOx.

[0079] In step S33, the processing device 11 displays the virtual object image VOx at a position according to the operation data Do. For example, the image Gk2 shown in FIG.

[0080] In step S34, the processing device 11 determines whether or not the placement of the virtual object VO is complete. Specifically, the processing device 11 makes this determination based on whether or not operation data Do indicating the completion of placement has been received. If the determination result in step S34 is negative, the processing device 11 returns the process to step S32. If the determination result in step S34 is positive, the processing device 11 proceeds to step S35.

[0081] In step S35, the processing device 11 determines relative coordinate data Dr. The relative coordinate data Dr indicates the position of the virtual object VO with the anchor P as the origin in the spatial structure indicated by the first spatial structure data Ds1. The processing device 11 determines the relative coordinate data Dr based on relative coordinates input by the developer using the input device 15, for example.

[0082] In step S35, the processing device 11 transmits the anchor ID, the relative coordinate data Dr, the virtual object data Dv, and the additional data Da to the object management server 30. The object management server 30 associates these pieces of data with the anchor ID and stores them in the database DB.

[0083] In the above object placement process, the processing device 11 functions as follows: In step S30, the processing device 11 functions as an acquisition unit 113. In step S31 and steps S33 to S35, the processing device 11 functions as an object placement unit 117. In step S32, the processing device 11 functions as a reception unit 114.

[0084] 1.3: Effects of the embodiment According to the above description, the terminal device 10 includes an acquisition unit 113 that acquires first spatial structure data Ds1 that indicates the spatial structure of real space and second spatial structure data Ds2 that indicates the spatial structure around an anchor P that serves as a reference when placing a virtual object VO in real space; a reception unit 114 that receives, on a screen displayed on the display device 17, a user (developer) operation to adjust the relative positions of a first image G1 that indicates the spatial structure determined by the first spatial structure data Ds1 and the second image G2 that indicates the spatial structure around the anchor P that is determined by the second spatial structure data Ds2; and a position determination unit 116 that determines the position of the anchor P in the spatial structure determined by the first spatial structure data Ds1 based on the user operation received by the reception unit 114.

[0085] The terminal device 10 has the above configuration, and therefore can align the spatial structure based on the first spatial structure data Ds1 with the spatial structure based on the second spatial structure data Ds2. As a result, the terminal device 10 can determine the position of the anchor P expressed in the spatial structure based on the first spatial structure data Ds1.

[0086] The accuracy of the spatial structure indicated by the first spatial structure data Ds1 is higher than the accuracy of the spatial structure around the anchor P indicated by the second spatial structure data Ds2. Therefore, the terminal device 10 can represent the position of the anchor P using a highly accurate spatial structure.

[0087] The terminal device 10 includes a first structure determination unit 111 that determines first spatial structure data Ds1 based on the distances from a reference point in real space to multiple points located on the surface of one or more physical objects that exist in the real space, and a second structure determination unit 112 that determines second spatial structure data Ds2 based on image data Dg that shows an captured image Gk around the anchor P.

[0088] The algorithm for determining the first spatial structure data Ds1 and the algorithm for determining the second spatial structure data Ds2 are different from each other. The terminal device 10 can align the two spatial structures determined by the different algorithms.

[0089] In the terminal device 10, the second structure determination unit 112 detects one or more planes based on the image data Dg and determines the second spatial structure data Ds2 by combining the detected one or more planes. The terminal device 10 also includes an anchor placement unit 115 that places an anchor P on one of the detected one or more planes.

[0090] The terminal device 10 has the above configuration, and thus can place an anchor P on a plane included in the spatial structure based on the second spatial structure data Ds2. The second image G2 is an image showing the anchor P and the spatial structure around the anchor P. When aligning the first image G1 and the second image G2, the user can easily align them using the plane as a reference. Therefore, the terminal device 10 can accurately grasp where the anchor P placed on the plane of the second image G2 is located on the plane of the first image G1. As a result, the terminal device 10 can accurately determine the position of the anchor P in the spatial structure determined by the first spatial structure data Ds1, compared to when the anchor P is not placed on the plane of the spatial structure based on the second spatial structure data Ds2.

[0091] The position determination unit 116 of the terminal device 10 determines position data Dp indicating the position of the anchor P in the spatial structure determined by the first spatial structure data Ds1. The terminal device 10 also includes a communication control unit 118 that transmits the second spatial structure data Ds2 and the position data Dp to the object management server 30 that manages the virtual object VO.

[0092] The terminal device 10 includes the communication control unit 118, and therefore can manage the second spatial structure data Ds2 and the position data Dp in the object management server 30.

[0093] The object management server 30 also includes first spatial structure data Ds1 indicating the spatial structure of the real space, second spatial structure data Ds2 indicating the spatial structure around an anchor P that serves as a reference when placing a virtual object VO in the real space, an acquisition unit 311 that acquires position data Dp indicating the position of the anchor P in the spatial structure determined by the first spatial structure data Ds1, and virtual object data Dv indicating the virtual object VO, and a management unit 312 that associates and manages the first spatial structure data Ds1, the second spatial structure data Ds2, the position data Dp, and the virtual object data Dv.

[0094] The object management server 30 has the above configuration, and therefore can associate the second spatial structure data Ds2 used when placing the anchor P with the virtual object data Dv, the position data Dp, and the first spatial structure data. As a result, the object management server 30 can manage the situation in which the user places the anchor P and the virtual object VO in real space using the terminal device 10.

[0095] 3: Modifications The present disclosure is not limited to the above-described exemplary embodiments. Specific modifications are exemplified below. Two or more modifications selected from the following examples may be combined.

[0096] 3.1: Modification 1 In the above-described embodiment, the terminal device 10 executes the object adjustment application A1 to determine a simple spatial structure around the anchor P based on the detection data Dd and image data Dg output from the detection device 18. In this determination process, the terminal device 10 may generate a feature point map indicating feature points obtained from the captured image Gk and transmit the generated feature point map to the anchor management server 20. The anchor management server 20 manages the feature point map in association with the anchor ID. When the anchor management server 20 receives a captured image from the user device 40, it may extract feature points from the received captured image and compare the extracted feature points with the feature point map to estimate the position and direction of the user device 40, and return the estimated position and direction to the terminal device 10.

[0097] 3.2: Modification 2 In the above-described embodiment, the terminal device 10 transmitted a registration request including the virtual object data Dv to the object management server 30. However, the present disclosure is not limited to this. The terminal device 10 may transmit the virtual object data Dv to the object management server 30 in advance. In this case, the terminal device 10 transmits a registration request to the object management server 30 that does not include the virtual object data Dv.

[0098] 3.3: Modification 3 In the above-described embodiment, the position determination unit 116 aligned the spatial structure represented by the first spatial structure data Ds1 with the spatial structure represented by the second spatial structure data Ds2 based on the operation data Doy. However, the present disclosure is not limited to this, and alignment may be performed automatically rather than manually by a developer. The position determination unit 116 may also perform alignment based on a comparison result between one or more surfaces included in the spatial structure represented by the first spatial structure data Ds1 and one or more surfaces included in the spatial structure represented by the second spatial structure data Ds2. More specifically, the position determination unit 116 may perform alignment by associating the boundaries of the surfaces included in the spatial structure represented by the second spatial structure data Ds2 with the boundaries of the surfaces included in the spatial structure represented by the first spatial structure data Ds1.

[0099] 4: Others (1) In the above-described embodiment, the storage device 12 and the storage device 32 are exemplified by ROM and RAM, but they may also be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network NW via a telecommunications line.

[0100] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0101] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0102] (4) In the above-described embodiment, the determination may be made based on a value (0 or 1) represented using one bit, a Boolean value (true or false), or a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0103] (5) The order of the exemplary procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0104] (6) Each function illustrated in Figures 1 to 8 is realized by any combination of hardware and / or software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. A functional block may be realized by combining software with the single device or the multiple devices.

[0105] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0106] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0107] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0108] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0109] (10) The terminal device 10 and the user device 40 may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term. In this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0110] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0111] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0112] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching a table, database, or other data structure), and ascertaining something that is considered to be a "determining." Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and so on. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0113] (14) In the above embodiments, when the terms "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or," as used in this disclosure, is not intended to be an exclusive or.

[0114] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include the nouns following these articles being plural.

[0115] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0116] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0117] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0118] 1...information processing system, 10...terminal device, 11, 31...processing device, 14...imaging device, 16, 33...communication device, 17...display device, 111...first structure determination unit, 112...first structure determination unit, 113...acquisition unit, 114...reception unit, 115...anchor placement unit, 116...position determination unit, 117...object placement unit, 118...communication control unit, 311...acquisition unit, 312...management unit, Dg...image data, Ds1...first spatial structure data, Ds2...second spatial structure data, Dv...virtual object data, Gk...captured image, G1...first image, G2...second image, P...anchor.

Claims

1. An acquisition unit that acquires first spatial structure data indicating the spatial structure of the real space and second spatial structure data indicating the spatial structure around an anchor that serves as a reference when placing a virtual object in the real space; a reception unit that receives an operation of a user for adjusting a relative position between a first image indicating the spatial structure determined by the first spatial structure data and a second image indicating the spatial structure around the anchor and the anchor determined by the second spatial structure data on a screen displayed on a display device; and a positioning unit that determines the position of the anchor in the spatial structure determined by the first spatial structure data based on the operation of the user received by the reception unit. A terminal device comprising:

2. The terminal device according to claim 1, wherein the accuracy of the spatial structure indicated by the first spatial structure data is higher than the accuracy of the spatial structure around the anchor indicated by the second spatial structure data.

3. A first structure determination unit that determines the first spatial structure data based on each distance from a reference point in the real space to a plurality of points located on the surface of one or more physical objects existing in the real space; and a second structure determination unit that determines the second spatial structure data based on image data indicating a captured image around the anchor. The terminal device according to claim 1, further comprising:

4. The second structure determination unit further comprises: an anchor placement unit that detects one or more planes based on the image data, determines the second spatial structure data by combining the detected one or more planes, and places the anchor on one of the detected one or more planes. The terminal device according to claim 3.

5. The positioning unit determines position data indicating the position of the anchor in the spatial structure determined by the first spatial structure data, and further comprises a communication control unit that transmits the second spatial structure data and the position data to an object management server that manages the virtual object. The terminal device according to claim 3.

6. An object management server comprising: an acquisition unit that acquires first spatial structure data indicating a spatial structure of a real space, second spatial structure data indicating a spatial structure around an anchor that serves as a reference when placing a virtual object in the real space, position data indicating the position of the anchor in the spatial structure determined by the first spatial structure data, and virtual object data indicating the virtual object; and a management unit that manages the first spatial structure data, the second spatial structure data, the position data, and the virtual object data in association with each other.

7. A control method for a terminal device, the method comprising: acquiring first spatial structure data indicating a spatial structure of a real space and second spatial structure data indicating a spatial structure around an anchor that serves as a reference when placing a virtual object in the real space; receiving an operation by a user to adjust a relative position between a first image indicating the spatial structure determined by the first spatial structure data and a second image indicating the spatial structure around the anchor and the anchor determined by the second spatial structure data on a screen displayed on a display device; and determining position data indicating the position of the anchor in the spatial structure determined by the first spatial structure data based on the adjusted relative position between the first image and the second image.

Images