Information processing device

Abstract

This information processing apparatus comprises: an acquisition unit that acquires imaging information indicating images captured by an imaging device, imaging position information indicating the position of the imaging device, and temporary position information indicating the position of a wearable device; an identification unit that identifies a person wearing the wearable device from one or more person images included in the captured images on the basis of the imaging information and the temporary position information; a calculation unit that calculates, by using the imaging position information, self-position information indicating the position of the wearable device worn by the person identified by the identification unit; and an output unit that outputs the self-position information to the wearable device. The accuracy of the imaging position information is higher than that of the temporary position information.

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] In MR (Mixed Reality) technology, the real environment perceived by the user is augmented by a computer. Using this technology, it is possible, for example, to precisely overlay a virtual space on the real space perceived through MR glasses worn by the user.

[0003] In MR glasses, in order to precisely overlay a virtual space on a real space, it is necessary to acquire the accurate self-position of the MR glasses.

[0004] Regarding technology related to the overlay of real space and virtual space, for example, Patent Document 1 discloses a technology for replacing position coordinates in virtual space with latitude, longitude, and altitude information in real space. Specifically, in the technology described in Patent Document 1, a measurement processing device acquires position coordinates in real space of a measurement point measured as the position of the position specifying device from a position specifying device that measures latitude, longitude, and altitude. The measurement processing device also photographs the position specifying device. Furthermore, the measurement processing device generates a correspondence table by correlating the position coordinates in virtual space of the photographed position specifying device with the position coordinates in real space of the measurement point. Furthermore, the measurement processing device converts the coordinates of 3D point cloud data in virtual space into position coordinates in real space using a predetermined conversion formula generated using the correspondence table. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6928217 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology disclosed in Patent Document 1, if the accuracy of the latitude, longitude, and altitude measured by the positioning device as its own position is not high, the measurement processing device cannot accurately convert the coordinates of the 3D point cloud data in the virtual space into coordinates in the real space. Therefore, when the technology disclosed in Patent Document 1 is applied to MR glasses, the real space and the virtual space in the MR glasses do not precisely overlap. For example, a virtual object displayed on the MR glasses may be displayed at a position different from the position intended by the provider of the virtual space service.

[0007] Therefore, an object of the present invention is to provide an information processing device that can acquire a more accurate self-location when MR glasses acquire the self-location. [Means for solving the problem]

[0008] An information processing device according to a preferred embodiment of the present invention comprises an acquisition unit that acquires imaging information indicating an image captured by an imaging device, imaging position information indicating the position of the imaging device, and provisional position information indicating the position of a wearable device; an identification unit that identifies a person wearing the wearable device from one or more person images included in the captured image based on the imaging information and the provisional position information; a calculation unit that calculates self-position information indicating the position of the wearable device worn by the person identified by the identification unit by using the imaging position information; and an output unit that supplies the self-position information to the wearable device, wherein the accuracy of the imaging position information is higher than the accuracy of the provisional position information. [Effects of the Invention]

[0009] According to the present invention, when the MR glasses acquire their own position, it is possible to acquire a more accurate self-position. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing the overall configuration of an information processing system 1 according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of the configuration of a wearable device 20 according to a first embodiment. [Figure 3] 1 is a perspective view showing the appearance of MR glasses 50 according to a first embodiment. [Figure 4] 1 is a schematic diagram of a virtual space VS provided to a user U1 by using MR glasses 50. FIG. [Figure 5] 1 is a schematic diagram of a virtual space VS provided to a user U1 by using MR glasses 50. FIG. [Figure 6] 1 is a schematic diagram of a virtual space VS provided to a user U1 by using MR glasses 50. FIG. [Figure 7] FIG. 2 is a block diagram showing an example of the configuration of MR glasses 50. [Figure 8] FIG. 2 is a block diagram showing an example of the configuration of a terminal device 40 according to the first embodiment. [Figure 9] 1 is a block diagram showing an example of the arrangement of an information processing device 10 according to a first embodiment. [Figure 10] 3 is a flowchart showing the operation of the information processing system 1 according to the first embodiment. [Figure 11] FIG. 10 is a diagram showing the overall configuration of an information processing system 1A according to a second embodiment. [Figure 12] FIG. 10 is a block diagram showing an example of the configuration of a wearable device 20A according to a second embodiment. [Figure 13] FIG. 10 is a block diagram showing an example of the arrangement of an information processing device 10A according to a second embodiment. [Figure 14] FIG. 10 is a block diagram showing an example of the configuration of a terminal device 40A according to the second embodiment. [Figure 15] 10 is a flowchart showing the operation of an information processing system 1A according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1: First embodiment Hereinafter, the configuration of an information processing system 1 including an information processing device 10 according to a first embodiment of the present invention will be described with reference to FIGS.

[0012] 1.1: Configuration of the first embodiment 1.1.1: Overall structure FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to a first embodiment of the present invention. The information processing system 1 is a system that provides a virtual space to a user U1 wearing MR glasses 50 (described later) using MR technology. Here, MR technology refers to a technology that enables a user wearing a device such as a see-through head-mounted display to experience a combined world of the real world and the virtual world by precisely superimposing the coordinate space of the real world and the coordinate space of the virtual world. MR technology includes AR (Augmented Reality) technology and VR (Virtual Reality) technology.

[0013] The information processing system 1 includes an information processing device 10, a wearable device 20, and an imaging device 30. The information processing device 10, the wearable device 20, and the imaging device 30 are communicably connected to each other via a communication network NET. Note that in FIG. 1, two wearable devices 20, a wearable device 20-1 and a wearable device 20-2, are shown. However, this number is merely an example, and the information processing system 1 can include any number of wearable devices 20.

[0014] The wearable device 20 is a device worn by a user to perceive a virtual space. FIG. 2 is a block diagram showing an example configuration of the wearable device 20. The wearable device 20 includes a terminal device 40 and MR glasses 50. The terminal device 40 and the MR glasses 50 are connected to each other so that they can communicate with each other. The terminal device 40 is connected to the information processing device 10 so that they can communicate with each other, thereby acquiring image information to be displayed on the MR glasses 50 from the information processing device 10. Furthermore, the terminal device 40 displays the image information acquired from the information processing device 10 on the MR glasses 50.

[0015] The terminal device 40 is a device for displaying virtual objects arranged in a virtual space on the MR glasses 50 worn on the user's head. The virtual space is, for example, a celestial sphere. Examples of the virtual objects include virtual objects representing data such as still images, videos, 3DCG models, HTML files, and text files, as well as virtual objects representing applications. Examples of text files include memos, source code, diaries, and recipes. Examples of applications include browsers, applications for using SNS, and applications for generating document files. The terminal device 40 is preferably, for example, a mobile terminal device such as a smartphone or a tablet.

[0016] The MR glasses 50 are a see-through wearable display worn on the user's head. The MR glasses 50 display virtual objects on display panels provided in each of the lenses for both eyes under the control of the terminal device 40. In the present invention, the position of the MR glasses 50 is the position of the wearable device 20.

[0017] Returning to the explanation of FIG. 1 , the information processing device 10 provides various data and cloud services to the terminal device 40 via the communication network NET. The information processing device 10 also calculates self-location information indicating the absolute position of the wearable device 20 in a world coordinate system. Furthermore, the information processing device 10 outputs the calculated self-location information to the wearable device 20. Note that the world coordinate system in this embodiment is a coordinate system that represents the entire space including the wearable device 20 and the imaging device 30.

[0018] The imaging device 30 captures an image of a person wearing the wearable device 20. Specifically, the imaging device 30 is, for example, a stereo camera, and is capable of acquiring, in addition to an image of the person wearing the wearable device 20, vector information indicating the distance and direction from the imaging device 30 to the imaging target. More specifically, as an example, vector information indicating the distance and direction from the imaging device 30 to the pixel constituting the captured image captured by the imaging device 30 is associated with the pixel. The captured image captured by the imaging device 30, the distance from the imaging device 30 to the imaging target, and the direction from the imaging device 30 to the imaging target are supplied to the information processing device 10 via a communication network NET. The installation position of the imaging device 30 is fixed. The imaging information supplied from the imaging device 30 indicating the captured image and distance information is an example of second imaging information, which will be described later.

[0019] 1.1.2: MR Glasses Configuration FIG. 3 is a perspective view showing the appearance of the MR glasses 50. As shown in FIG. 3, the appearance of the MR glasses 50 is similar to that of ordinary eyeglasses, and includes temples 91 and 92, a bridge 93, body parts 94 and 95, and lenses 59L and 59R. An imaging device 56 is provided on the bridge 93. The imaging device 56 captures an image of the outside world and then outputs imaging information indicating the captured image. The imaging information indicating the captured image output from the imaging device 56 is an example of first imaging information, which will be described later.

[0020] Each of the lenses 59L and 59R is provided with a half mirror. The body 94 is provided with a liquid crystal panel or organic EL panel for the left eye (hereinafter collectively referred to as a display panel) and an optical member that guides light emitted from the display panel for the left eye to the lens 59L. The half mirror provided in the lens 59L transmits external light to guide it to the left eye and reflects the light guided by the optical member so that it enters the left eye. The body 95 is provided with a display panel for the right eye and an optical member that guides light emitted from the display panel for the right eye to the lens 59R. The half mirror provided in the lens 59R transmits external light to guide it to the right eye and reflects the light guided by the optical member so that it enters the right eye.

[0021] The display 58, which will be described later, includes a lens 59L, a display panel for the left eye, and an optical member for the left eye, as well as a lens 59R, a display panel for the right eye, and an optical member for the right eye.

[0022] In the above configuration, the user can observe the image displayed on the display panel in a see-through state, superimposed on the outside world. Furthermore, the MR glasses 50 display the image for the left eye on the left eye display panel and the image for the right eye on the right eye display panel, among the binocular images with parallax, so that the user U1 can perceive the displayed image as if it has depth and a three-dimensional effect.

[0023] 4 to 6 are schematic diagrams of a virtual space VS provided to a user U1 by using MR glasses 50. As shown in FIG. 4, virtual objects VO1 to VO5 representing various contents such as a browser, cloud services, images, and videos are arranged in the virtual space VS. By moving around a public space while wearing the MR glasses 50 displaying the virtual objects VO1 to VO5 arranged in the virtual space VS, the user U1 can experience the virtual space VS as a private space in the public space. Consequently, the user U1 can act in the public space while receiving the benefits provided by the virtual objects VO1 to VO5 arranged in the virtual space VS.

[0024] 5, it is also possible for multiple users U1 to U3 to share a virtual space VS. By sharing the virtual space VS among multiple users U1 to U3, the multiple users U1 to U3 can share one or more virtual objects VO, and can communicate with each other via the shared virtual objects VO.

[0025] 6, as an example, a text box X notifying the user of the status of e-mail reception and the schedule for the day is superimposed on a landscape image W as real space and displayed on the MR glasses 50. In this embodiment, by using MR technology, it is possible to display the text box X in an area of the landscape image W that is free of obstacles, i.e., in the example shown in FIG. 6, in the area of the blue sky.

[0026] 7 is a block diagram showing an example configuration of the MR glasses 50. The MR glasses 50 include a processing device 51, a storage device 52, a gaze detection device 53, a GPS device 54, a motion detection device 55, an imaging device 56, a communication device 57, and a display 58. The elements of the MR glasses 50 are connected to each other by one or more buses for communicating information. Note that the term "device" in this specification may be replaced with other terms such as circuit, device, or unit.

[0027] The processing device 51 is a processor that controls the entire MR glasses 50, and is configured using, for example, one or more chips. The processing device 51 is configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 51 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). The processing device 51 executes various processes in parallel or sequentially.

[0028] The storage device 52 is a recording medium that can be read and written by the processing device 51, and stores a plurality of programs including the control program PR1 that the processing device 51 executes.

[0029] After detecting the gaze of the user U1, the gaze detection device 53 supplies gaze information indicating the direction of the gaze of the user U1 based on the detection result to the processing device 51 described below. Any method may be used as a method for the gaze detection device 53 to detect the gaze. For example, the gaze detection device 53 may detect the gaze information based on the positions of the inner corners of the eyes and the irises.

[0030] The GPS device 54 receives radio waves from multiple satellites and generates first position information from the received radio waves. The position information indicates the position of the MR glasses 50. The first position information may be in any format as long as it can identify the position. The first position information indicates, for example, the latitude and longitude of the MR glasses 50. Note that the first position information may indicate the altitude in addition to the latitude and longitude of the MR glasses 50. As an example, the first position information is obtained from the GPS device 54. However, the MR glasses 50 may obtain the first position information by any method. The obtained first position information is supplied to the processing device 51 as provisional position information indicating the position of the wearable device 20.

[0031] After detecting the movement of the MR glasses 50, the movement detection device 55 supplies movement information indicating the movement to the processing device 51. The movement detection device 55 includes inertial sensors such as an acceleration sensor that detects acceleration and a gyro sensor that detects angular acceleration. The acceleration sensor detects acceleration on orthogonal X-, Y-, and Z-axes. The gyro sensor detects angular acceleration around the X-, Y-, and Z-axes as the central axes of rotation. The movement detection device 55 can generate attitude information indicating the attitude of the MR glasses 50 based on the output information of the gyro sensor. The movement data includes acceleration data indicating the acceleration on each of the three axes and angular acceleration data indicating the angular acceleration on each of the three axes.

[0032] The imaging device 56 outputs first imaging information indicating an image obtained by capturing an image of the external world. The imaging device 56 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an analog imaging signal by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified analog imaging signal into first imaging information, which is a digital signal. The converted first imaging information is supplied to the processing device 51. The first imaging information output to the processing device 51 is supplied to the terminal device 40 via the communication device 57. The terminal device 40 recognizes various gestures of the user U1 based on the first imaging information and then controls the terminal device 40 in accordance with the recognized gestures. That is, the imaging device 56 functions as an input device for inputting instructions from the user U1, like a pointing device or touch panel.

[0033] The communication device 57 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 57 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 57 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 57 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0034] The display 58 is a device that displays images. The display 58 displays various images under the control of the processing device 51. As described above, the display 58 includes the lens 59L, a display panel for the left eye, and an optical member for the left eye, as well as the lens 59R, a display panel for the right eye, and an optical member for the right eye. As the display panel, various display panels such as a liquid crystal display panel and an organic EL display panel are suitably used.

[0035] The processing device 51 functions as an acquisition unit 511 and a display control unit 512 by, for example, reading out a control program PR1 from the storage device 52 and executing it.

[0036] The acquisition unit 511 acquires a control signal from the terminal device 40. More specifically, the acquisition unit 511 acquires a control signal for controlling display on the MR glasses 50, which is generated by a display control unit 415 (described later) included in the terminal device 40.

[0037] Furthermore, the acquisition unit 511 acquires gaze information input from the gaze detection device 53, position information input from the GPS device 54, movement information input from the movement detection device 55, and first imaging information input from the imaging device 56. Then, the acquisition unit 511 supplies the acquired gaze information, first position information, movement information, and imaging information to the communication device 57. The communication device 57 transmits the gaze information, first position information, movement information, and imaging information acquired from the acquisition unit 511 to the terminal device 40. Furthermore, the acquisition unit 511 stores the acquired gaze information, first position information, movement information, and imaging information in the storage device 52.

[0038] Furthermore, the acquisition unit 511 acquires second position information as self-position information indicating the position of the wearable device 20 from the terminal device 40 by using the communication device 57. Furthermore, the acquisition unit 511 stores the acquired second position information in the storage device 52. Here, the second position information is more accurate than the first position information. Specifically, the first position information is GPS information acquired from the GPS device 54 as described above. A general-purpose GPS device may have an error of about 100 m. On the other hand, the second position information is self-position information acquired by the MR glasses 50 from the information processing device 10 via the terminal device 40. As will be described later, the self-position information is calculated using relative position information from the imaging device 30 to the MR glasses 50. Since the relative position information is information measured by the imaging device 30 as a stereo camera, for example, when the distance from the imaging device 30 to the imaging target is about 100 m, an error of at most about 1 m occurs.

[0039] The display control unit 512 controls the display on the display 58 based on the control signal from the terminal device 40 acquired by the acquisition unit 511 .

[0040] 1.1.3: Terminal device configuration 8 is a block diagram showing an example configuration of the terminal device 40. The terminal device 40 includes a processing device 41, a storage device 42, a communication device 43, a display 44, an input device 45, and an inertial sensor 46. The elements of the terminal device 40 are connected to each other by one or more buses for communicating information.

[0041] The processing device 41 is a processor that controls the entire terminal device 40, and is configured using, for example, one or more chips. The processing device 41 is configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 41 may be realized by hardware such as a DSP, an ASIC, a PLD, and an FPGA. The processing device 41 executes various processes in parallel or sequentially.

[0042] The storage device 42 is a recording medium that can be read and written by the processing device 41, and stores a plurality of programs including the control program PR2 that the processing device 41 executes.

[0043] The communication device 43 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 43 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 43 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 43 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0044] The display 44 is a device that displays images and text information. The display 44 displays various images under the control of the processing device 41. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display 44.

[0045] The input device 45 receives operations from a user U1 wearing the MR glasses 50 on their head. For example, the input device 45 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 45 includes a touch panel, it may also serve as the display 44.

[0046] The inertial sensor 46 is a sensor that detects inertial force. The inertial sensor 46 includes, for example, one or more of an acceleration sensor, an angular velocity sensor, and a gyro sensor. The processing device 41 detects the attitude of the terminal device 40 based on the output information of the inertial sensor 46. Furthermore, the processing device 41 accepts the selection of a virtual object VO, the input of text, and the input of instructions in the spherical virtual space VS based on the attitude of the terminal device 40. For example, the user U1 operates the input device 45 while pointing the central axis of the terminal device 40 toward a predetermined area in the virtual space VS, thereby selecting a virtual object VO to be placed in the predetermined area. The operation by the user U1 on the input device 45 is, for example, a double tap. By operating the terminal device 40 in this way, the user U1 can select a virtual object VO without looking at the input device 45 of the terminal device 40.

[0047] The processing device 41 functions as an acquisition unit 411, an action recognition unit 412, an identification unit 413, an output unit 414, and a display control unit 415 by reading and executing the control program PR2 from the storage device 42.

[0048] The acquisition unit 411 acquires instruction information corresponding to the movement of the user U1 wearing the MR glasses 50 on his / her head. Here, the movement of the user U1 is, for example, an input by the user U1 to the terminal device 40 using the input device 45. More specifically, the movement of the user U1 may be pressing a specific part of the terminal device 40 as the input device 45. Alternatively, the movement of the user U1 may be an operation using the terminal device 40 as a portable controller. In these cases, the instruction information is, for example, information for specifying a specific virtual object VO, information for changing the display of the specific virtual object VO, and, if the specific virtual object VO is an application, information for launching the application.

[0049] Alternatively, the action of the user U1 may be the user U1 looking at the MR glasses 50. When the action of the user U1 is the visual action, the instruction information is the viewpoint of the user U1 on the MR glasses 50. In this case, the instruction information is transmitted from the MR glasses 50 to the terminal device 40.

[0050] Alternatively, the action of the user U1 may be a gesture of the user U1. As will be described later, the action recognition unit 412 recognizes various gestures of the user U1. The acquisition unit 411 may acquire instruction information corresponding to the various gestures of the user U1.

[0051] Moreover, the acquisition unit 411 uses the communication device 43 to acquire the first imaging information and the first position information as the provisional position information from the MR glasses 50.

[0052] Furthermore, the acquisition unit 411 acquires second position information as self-position information indicating the position of the wearable device 20 from the information processing device 10 using the communication device 43. When the terminal device 40 causes the MR glasses 50 to display a virtual object VO, the self-position information acquired by the acquisition unit 411 is used to precisely align the virtual space in which the virtual object VO is displayed with the real space visually recognized by the user through the MR glasses 50.

[0053] The action recognition unit 412 recognizes various gestures of the user U1 based on the imaging information obtained from the MR glasses 50. More specifically, as described above, the imaging device 56 provided in the MR glasses 50 outputs imaging information obtained by imaging the outside world. If the imaging data includes a part of the body of the user U1 wearing the MR glasses 50 on his / her head, the action recognition unit 412 recognizes various gestures of the user U1 based on the imaging data obtained from the MR glasses 50.

[0054] The identifying unit 413 identifies one virtual object VO from among the multiple virtual objects VO arranged in the virtual space VS, based on the instruction information acquired by the acquiring unit 411.

[0055] When the instruction information acquired by the acquisition unit 411 is information for changing the display of one virtual object VO identified by the identification unit 413 or information for activating the one virtual object VO, the output unit 414 supplies the instruction information to the information processing device 10. Based on the instruction information acquired from the terminal device 40, the information processing device 10 changes the display of the one identified virtual object VO or activates the one identified virtual object VO.

[0056] Furthermore, the output unit 414 transmits the first imaging information acquired from the MR glasses 50 by the acquisition unit 411 to the information processing device 10. Furthermore, the output unit 414 transmits the first position information, which is acquired from the MR glasses 50 by the acquisition unit 411, as temporary position information of the wearable device 20 to the information processing device 10.

[0057] Furthermore, the output unit 414 supplies second position information, which is acquired from the information processing device 10 by the acquisition unit 411 and serves as self-position information of the wearable device 20, to the MR glasses 50 included in the wearable device 20. More specifically, the information processing device 10 calculates second position information with higher accuracy than the first position information by a method described below. The acquisition unit 411 acquires the second position information from the information processing device 10. The output unit 414 supplies the second position information to the MR glasses 50 included in the wearable device 20.

[0058] The display control unit 415 causes the MR glasses 50 as a display device to display a plurality of virtual objects VO arranged in the virtual space VS. More specifically, the display control unit 415 generates image data to be displayed on the MR glasses 50, and then transmits the generated image data to the MR glasses 50 via the communication device 13.

[0059] 8, the terminal device 40 may include a GPS device similar to the GPS device 54 included in the MR glasses 50. In this case, the MR glasses 50 may not include the GPS device 54.

[0060] 1.1.4: Configuration of information processing device 9 is a block diagram showing an example of the configuration of the information processing device 10. The information processing device 10 includes a processing device 11, a storage device 12, a communication device 13, a display 14, and an input device 15. The elements of the information processing device 10 are connected to each other by one or more buses for communicating information.

[0061] The processing device 11 is a processor that controls the entire information processing device 10, and is configured using, for example, one or more chips. The processing device 11 is configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 11 may be realized by hardware such as a DSP, an ASIC, a PLD, and an FPGA. The processing device 11 executes various processes in parallel or sequentially.

[0062] The memory device 12 is a recording medium that can be read and written by the processing device 11, and stores multiple programs including the control program PR3 executed by the processing device 11, the first learning model LM1, the second learning model LM2, the third learning model LM3, map data MD, and imaging position information indicating the absolute position of the imaging device 30.

[0063] The first learning model LM1 is a learning model used by the later-described specifying unit 112 to extract, from a captured image, a person image included in the captured image.

[0064] The first learning model LM1 is generated by learning training data in a learning phase. The training data used to generate the first learning model LM1 includes a plurality of pairs of captured images and human images included in the captured images.

[0065] The first learning model LM1 is generated by a device external to the information processing device 10. In particular, the first learning model LM1 is preferably generated by a server (not shown). In this case, the information processing device 10 acquires the first learning model LM1 from the server (not shown) via the communication network NET.

[0066] The second learning model LM2 is a learning model used by the calculation unit 113 (described later) to further extract an image of the MR glasses 50 from the person image of one person identified by the identification unit 112.

[0067] The second learning model LM2 is generated by learning training data in the learning phase. The training data used to generate the second learning model LM2 includes a plurality of pairs of captured images and images of the MR glasses 50 included in the captured images.

[0068] The second learning model LM2 is generated by a device external to the information processing device 10. In particular, the second learning model LM2 is preferably generated by a server (not shown). In this case, the information processing device 10 acquires the second learning model LM2 from the server (not shown) via the communication network NET.

[0069] The third learning model LM3 is a learning model used by the calculation unit 113 (described later) to extract, from a captured image, a building image included in the captured image.

[0070] The third learning model LM3 is generated by learning training data in the learning phase. The training data used to generate the third learning model LM3 includes a plurality of pairs of captured images and building images included in the captured images.

[0071] The third learning model LM3 is generated by a device external to the information processing device 10. In particular, the third learning model LM3 is preferably generated by a server (not shown). In this case, the information processing device 10 acquires the third learning model LM3 from the server (not shown) via the communication network NET.

[0072] The map data MD includes captured images of an object captured in advance using a stereo camera or the like that can simultaneously capture images of the object from multiple different directions, feature quantities of feature points included in the captured images, and absolute location information associated with the feature points. In particular, the captured images include images of buildings specific to the land. The absolute location information associated with the feature points is global location information as well as location information in the world coordinate system.

[0073] The communication device 13 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0074] The display 14 is a device that displays images and text information. The display 14 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL display panel are suitably used as the display 14.

[0075] The input device 15 is a device that accepts operations by an administrator of the information processing system 1. For example, the input device 15 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 15 includes a touch panel, it may also serve as the display 14.

[0076] The processing device 11 functions as an acquisition unit 111, an identification unit 112, a calculation unit 113, a comparison unit 114, and an output unit 115, for example, by reading and executing a control program PR3 from the storage device 12.

[0077] The acquisition unit 111 acquires, from the terminal device 40, first imaging information indicating an image captured by the imaging device 56 provided in the MR glasses 50. More specifically, the acquisition unit 111 acquires, from the terminal device 40, first imaging information indicating an image of the outside world of the MR glasses 50 captured by the imaging device 56. The acquisition unit 111 also acquires, from the terminal device 40, first position information as provisional position information indicating the position of the wearable device 20.

[0078] Furthermore, the acquisition unit 111 acquires imaging position information indicating the absolute position of the imaging device 30 in the world coordinate system. More specifically, the acquisition unit 111 may acquire the imaging position information from the imaging device 30. Alternatively, the acquisition unit 111 may acquire the imaging position information stored in the storage device 12. Note that the accuracy of the imaging position information is higher than the accuracy of the provisional position information.

[0079] Furthermore, the acquisition unit 111 acquires second imaging information indicating a captured image from the imaging device 30. More specifically, the acquisition unit 111 acquires second imaging information indicating a captured image including a person wearing the wearable device 20 from the imaging device 30. As described above, since the imaging device 30 is, for example, a stereo camera, the second imaging information includes distance information from the imaging device 30 to the imaging target.

[0080] The identification unit 112 uses the first learning model LM1 to extract one or more person images from the captured image indicated by the second imaging information. Furthermore, based on the first location information as provisional location information acquired by the acquisition unit 111, the identification unit 112 identifies one person wearing the wearable device 20 corresponding to the provisional location information from among the people indicated by the one or more extracted person images.

[0081] More specifically, as described above, the second imaging information includes vector information from the imaging device 30 to the imaging target. Each pixel included in one or more person images is associated with vector information corresponding to that pixel. The identification unit 112 identifies a person included in a person image having vector information closest to the first position information as provisional position information as a single person wearing the wearable device 20 corresponding to the provisional position information.

[0082] Furthermore, the identification unit 112 uses the second learning model LM2 to extract an image of the MR glasses 50 from the person image of the identified person.

[0083] The calculation unit 113 calculates third position information as self-position information of the wearable device 20 based on the first imaging information acquired by the acquisition unit 111. For example, the calculation unit 113 uses the third learning model LM3 to extract an image of a building specific to the land on which the wearable device 20 is located, from the captured image shown by the first imaging information. The calculation unit 113 also calculates the position of the extracted building by referring to the map data MD stored in the storage device 12. Furthermore, the calculation unit 113 calculates the distance and direction from the MR glasses 50 to the building based on the image of the building extracted from the first imaging information. Finally, the calculation unit 113 calculates third position information as self-position information of the wearable device 20 based on the position of the building and the distance and direction from the MR glasses 50 to the building.

[0084] The calculated third position information is normally transmitted to the terminal device 40 by the output unit 115, which will be described later.

[0085] Furthermore, the calculation unit 113 calculates self-location accuracy information based on the first imaging information acquired by the acquisition unit 111.

[0086] The accuracy of the self-location information calculated based on the first imaging information decreases as the number of feature amounts included in the captured image captured by the imaging device 56 of the MR glasses 50, for example, the number of feature points included in the image of buildings specific to the land on which the wearable device 20 is located included in the captured image, decreases. Therefore, the calculation unit 113 calculates the self-location accuracy information based on the feature amounts included in the captured image represented by the first imaging information. Alternatively, the calculation unit 113 may match the image of the building included in the captured image represented by the first imaging information with the image of the building included in the map data MD, and calculate the self-location accuracy information according to the degree of matching. Alternatively, the calculation unit 113 may match the feature points included in the captured image represented by the first imaging information with the feature points included in the map data MD, and calculate the self-location accuracy information according to the degree of matching.

[0087] Furthermore, the calculation unit 113 uses the imaging position information to calculate second position information as self-position information indicating the position of the wearable device 20 worn by one person identified by the identification unit 112. More specifically, the calculation unit 113 uses the second learning model LM2 to extract an image of a portion occupied by the MR glasses 50 worn by one person from the person image of the one person identified by the identification unit 112. Furthermore, the calculation unit 113 calculates relative position information of the MR glasses 50 worn by the one person, with the position of the imaging device 30 as the reference, based on a plurality of vector information linked to each pixel of the image of the extracted portion occupied by the MR glasses 50. The calculation unit 113 may use, among the plurality of vector information linked to pixels of the image of the portion occupied by the MR glasses 50, vector information having the shortest distance from the imaging device 30 as relative position information indicating the relative position of the wearable device 20 worn by the one person. Alternatively, the calculation unit 113 may use, among a plurality of pieces of vector information linked to pixels of the image of the portion occupied by the MR glasses 50, vector information that is farthest from the imaging device 30 as relative position information indicating the relative position of the wearable device 20 worn by the person. Then, the calculation unit 113 adds the relative position information of the wearable device 20 to the imaging position information indicating the absolute position of the imaging device 30, thereby calculating self-position information indicating the absolute position of the wearable device 20 as second position information.

[0088] The comparison unit 114 compares the self-location accuracy information calculated by the calculation unit 113 with a threshold. In particular, if the self-location accuracy information is less than the threshold, that is, if the accuracy of the third location information is lower than the threshold, the output unit 115, which will be described later, outputs a request signal to the terminal device 40 requesting the first location information as provisional location information of the wearable device 20.

[0089] In normal operation, the output unit 115 uses the communication device 13 to supply the terminal device 40 with third position information as self-position information of the wearable device 20 calculated by the calculation unit 113 based on the first imaging information.

[0090] Furthermore, the output unit 115 is triggered by the acquisition unit 111 acquiring the provisional position information from the terminal device 40, and transmits an imaging instruction signal to the imaging device 30 using the communication device 13. When the imaging device 30 acquires the imaging instruction signal from the information processing device 10, it captures an image of the person wearing the wearable device 20.

[0091] Furthermore, the output unit 115 uses the communication device 13 to transmit to the terminal device 40 second position information as self-position information of the wearable device 20 calculated by the calculation unit 113 based on the imaging position information.

[0092] 1.2: Operation of the First Embodiment 10 is a flowchart showing the operation of the information processing system 1 according to the first embodiment. Hereinafter, the operation of the information processing system 1 will be described with reference to FIG.

[0093] In step S101, the processing device 41 provided in the terminal device 40 functions as the output unit 414 to transmit the first imaging information acquired from the MR glasses 50 to the information processing device 10.

[0094] In step S102, the processing device 11 included in the information processing device 10 functions as the calculation unit 113 to calculate self-location accuracy information based on the first imaging information acquired from the terminal device 40.

[0095] In step S103, the processing device 11 included in the information processing device 10 functions as the comparison unit 114 to compare the calculated self-location accuracy information with a threshold. If the self-location accuracy information is equal to or greater than the threshold, that is, if the determination result in step S103 is true, the information processing device 10 ends all processing. If the self-location accuracy information is less than the threshold, that is, if the determination result in step S103 is false, the information processing device 10 executes processing in step S104.

[0096] In step S104, the processing device 11 included in the information processing device 10 functions as the output unit 115 to transmit a request signal to the terminal device 40 requesting first location information as provisional location information of the wearable device 20.

[0097] In step S105, the processing device 41 included in the terminal device 40 functions as the output unit 414 to transmit the first position information as the provisional position information of the wearable device 20 acquired from the MR glasses 50 to the information processing device 10.

[0098] In step S106, the processing device 11 included in the information processing device 10 functions as the output unit 115 to transmit an image capture instruction signal to the image capture device 30.

[0099] In step S107, the imaging device 30 captures an image of a person wearing the wearable device 20 including the terminal device 40 and the MR glasses 50.

[0100] In step S108, the imaging device 30 transmits second imaging information indicating the captured image captured by the imaging device 30 to the information processing device 10.

[0101] In step S109, the processing device 11 included in the information processing device 10 functions as the identification unit 112 to extract one or more person images from the captured image indicated by the second imaging information. Furthermore, the processing device 11 functions as the identification unit 112 to identify one person wearing the wearable device 20 corresponding to the provisional position information from among the people indicated by the one or more extracted person images, based on the first position information as the provisional position information.

[0102] In step S110, the processing device 11 provided in the information processing device 10 functions as a calculation unit 113, and uses the imaging position information to calculate self-position information indicating the position of the wearable device 20 worn by the person identified in step S109.

[0103] In step S111, the processing device 11 included in the information processing device 10 functions as the output unit 115, thereby transmitting the self-location information calculated in step S110 to the terminal device 40 using the communication device 13. Thereafter, the processing device 41 included in the terminal device 40 functions as the output unit 414, thereby transmitting the self-location information acquired from the information processing device 10 to the MR glasses 50.

[0104] 1.3: Effects of the First Embodiment According to the above description, the information processing device 10 includes an acquisition unit 111, an identification unit 112, a calculation unit 113, and an output unit 115. The acquisition unit 111 acquires imaging information indicating an image captured by the imaging device 30, imaging position information indicating the position of the imaging device 30, and tentative position information indicating the position of the wearable device 20. The identification unit 112 identifies a person wearing the wearable device 20 from one or more person images included in the captured image based on the imaging information and the tentative position information. The calculation unit 113 uses the imaging position information to calculate self-position information indicating the position of the wearable device 20 worn by the person identified by the identification unit 112. The output unit 115 supplies the self-position information to the wearable device 20. Here, the accuracy of the imaging position information is higher than the accuracy of the tentative position information.

[0105] By using the above-described configuration of the information processing device 10, the MR glasses 50 can acquire a more accurate self-location when acquiring their own location. In particular, the calculation unit 113 can calculate the self-location information with higher accuracy by calculating the self-location information using imaging position information with higher accuracy than the provisional position information that can be acquired by the MR glasses 50 being equipped with the GPS device 54.

[0106] Furthermore, the imaging position information indicates the absolute position of the imaging device 30. The self-position information indicates the absolute position of the wearable device 20. The calculation unit 113 calculates relative position information indicating the relative position of the wearable device 20 based on the imaging information, with the position of the imaging device 30 indicated by the imaging position information as a reference. Furthermore, the calculation unit 113 calculates the self-position information based on the imaging position information and the relative position information.

[0107] By using the above-mentioned configuration, the information processing device 10 can allow the calculation unit 113 to calculate self-position information, which is the absolute position of the wearable device 20, from imaging position information, which is the absolute position of the imaging device 30, and relative position information, which indicates the relative position of the wearable device 20 based on the position of the imaging device 30.

[0108] 2: Second embodiment Hereinafter, with reference to Figures 11 to 15, a configuration of an information processing system 1A including an information processing device 10A according to a second embodiment of the present invention will be described. In the following description, for the sake of simplicity, the same reference numerals will be used for the same components as in the first embodiment, and a description of their functions may be omitted. In addition, in the following description, for the sake of simplicity, differences between the second embodiment and the first embodiment will be mainly described.

[0109] 2.1: Configuration of the first embodiment 2.1.1: Overall structure 11 is a diagram showing the overall configuration of an information processing system 1A according to a second embodiment of the present invention. The information processing system 1A differs from the information processing system 1 according to the first embodiment in that it includes an information processing device 10A instead of the information processing device 10 and a wearable device 20A instead of the wearable device 20. Furthermore, the imaging device 30 simultaneously captures images of a first person wearing a first wearable device 20A-1 and a second person wearing a second wearable device 20A-2.

[0110] 12 is a block diagram showing an example of the configuration of the wearable device 20 A. The wearable device 20 A differs from the wearable device 20 according to the first embodiment in that it includes a terminal device 40 A instead of the terminal device 40.

[0111] 2.1.2: Configuration of information processing device 13 is a block diagram showing an example configuration of an information processing device 10A. The information processing device 10A differs from the information processing device 10 according to the first embodiment in that it includes a processing device 11A instead of the processing device 11. The processing device 11A differs from the processing device 11 according to the first embodiment in that it includes an acquisition unit 111A instead of the acquisition unit 111, an identification unit 112A instead of the identification unit 112, and a calculation unit 113A instead of the calculation unit 113. In addition, the storage device 12 stores a control program PR3A instead of the control program PR3 according to the first embodiment.

[0112] Unlike the acquisition unit 111, the acquisition unit 111A does not necessarily acquire the first location information as the provisional location information.

[0113] The identification unit 112A extracts multiple person images from the captured image indicated by the second imaging information, and then identifies a first person wearing the first wearable device 20A-1 and a second person wearing the second wearable device 20A-2 from the people represented by the extracted multiple person images. The identification unit 112A also extracts an image of the MR glasses 50-1 from the person image of the first person. The identification unit 112A also extracts an image of the MR glasses 50-2 from the person image of the second person.

[0114] The calculation unit 113A calculates self-position information indicating the first position of the first wearable device 20A-1 and the second position of the second wearable device 20A-2 using the imaging position information. As a result, the self-position information transmitted by the output unit 115 to the terminal device 40A-1 included in the first wearable device 20A-1 and the terminal device 40A-2 included in the second wearable device 20A-2 indicates both the first position of the first wearable device 20A-1 and the second position of the second wearable device 20A-2.

[0115] 2.1.3: Terminal Device Configuration 13 is a block diagram showing an example of the configuration of the terminal device 40A. The terminal device 40A differs from the terminal device 40 according to the first embodiment in that it includes a processing device 41A instead of the processing device 41. The processing device 41A further includes a selection unit 416 in addition to the components included in the processing device 41 according to the first embodiment. Furthermore, the storage device 42 stores a control program PR2A instead of the control program PR2 according to the first embodiment.

[0116] The selection unit 416 selects one position from the first position and the second position indicated by the self-position information acquired from the information processing device 10A, based on the position indicated by the provisional position information stored in the storage device 42. More specifically, the selection unit 416 selects the self-position information indicating the position, out of the first position and the second position, that is closer to the position indicated by the provisional position information.

[0117] When the terminal device 40A is the terminal device 40A-1, the first provisional position information is stored as the provisional position information in the storage device 42. Furthermore, the selection unit 416 selects the self-position information indicating the first position as the position closer to the first provisional position information from among the first position and the second position.

[0118] When the terminal device 40A is the terminal device 40A-2, the second provisional position information is stored as the provisional position information in the storage device 42. The selection unit 416 selects the self-position information indicating the second position as the position closer to the second provisional position information from among the first position and the second position.

[0119] 2.2: Operation of the second embodiment 15 is a flowchart showing the operation of the information processing system 1A according to the second embodiment. Hereinafter, the operation of the information processing system 1A will be described with reference to FIG.

[0120] In step S201, the processing device 41A included in the terminal device 40A-1 functions as the output unit 414 to transmit the first imaging information acquired from the MR glasses 50-1 to the information processing device 10.

[0121] In step S202, the processing device 41A included in the terminal device 40A-2 functions as the output unit 414 to transmit the first imaging information acquired from the MR glasses 50-2 to the information processing device 10.

[0122] In step S203, the processing device 11A included in the information processing device 10A functions as the calculation unit 113A to calculate self-location accuracy information based on the first imaging information acquired from the terminal device 40A-1. Also, the processing device 11A included in the information processing device 10A functions as the calculation unit 113A to calculate self-location accuracy information based on the first imaging information acquired from the terminal device 40A-2.

[0123] In step S204, the processing device 11A included in the information processing device 10A functions as the comparison unit 114 to compare the calculated self-location accuracy information with a threshold. If both pieces of self-location accuracy information calculated in step S203 are equal to or greater than the threshold, that is, if the determination result in step S204 is true, the terminal device 40A-1 ends all processing. If any of the self-location accuracy information calculated in step S203 is less than the threshold, that is, if the determination result in step S204 is false, the information processing device 10A executes the processing of step S205.

[0124] In step S205, the processing device 11A included in the information processing device 10A functions as the output unit 115 to transmit an image capture instruction signal to the image capture device 30.

[0125] In step S206, the imaging device 30 simultaneously captures an image of a person wearing a first wearable device 20A-1 including a terminal device 40A-1 and MR glasses 50-1, and a person wearing a second wearable device 20A-2 including a terminal device 40A-2 and MR glasses 50-2.

[0126] In step S207, the imaging device 30 outputs second imaging information indicating the captured image captured by itself to the information processing device 10A.

[0127] In step S208, the processing device 11A included in the information processing device 10A functions as the identification unit 112A to extract multiple person images from the captured image indicated by the second imaging information. Furthermore, the processing device 11A functions as the identification unit 112A to identify a first person wearing the first wearable device 20A-1 and a second person wearing the second wearable device 20A-2 from the people indicated by the extracted multiple person images.

[0128] In step S209, the processing device 11A provided in the information processing device 10A functions as a calculation unit 113A, and uses the imaging position information to calculate self-position information indicating a first position, which is the absolute position of the first wearable device 20A-1 worn by the first person identified in step S208, and a second position, which is the absolute position of the second wearable device 20A-2 worn by the second person.

[0129] In step S210, the processing device 11A included in the information processing device 10A functions as the output unit 115, and thereby uses the communication device 13 to transmit the self-location information calculated in step S209 to the terminal device 40A-1.

[0130] In step S211, the processing device 11A included in the information processing device 10A functions as the output unit 115, and thereby uses the communication device 13 to transmit the self-location information calculated in step S209 to the terminal device 40A-2.

[0131] In step S212, the processing device 41A included in the terminal device 40A-1 functions as the selection unit 416, thereby selecting self-position information indicating the first position as the position closer to the first tentative position information from the first position and the second position. Thereafter, the processing device 41A included in the terminal device 40A-1 functions as the output unit 414, thereby supplying the self-position information selected by the selection unit 416 to the MR glasses 50-1.

[0132] In step S213, the processing device 41A included in the terminal device 40A-2 functions as the selection unit 416 to select self-position information indicating the second position as the position closer to the second tentative position information from the first position and the second position. Thereafter, the processing device 41A included in the terminal device 40A-2 functions as the output unit 414 to supply the self-position information selected by the selection unit 416 to the MR glasses 50-2.

[0133] 2.3: Advantages of the Second Embodiment As described above, the information processing device 10A includes an acquisition unit 111, an identification unit 112A, a calculation unit 113A, and an output unit 115. The acquisition unit 111 acquires imaging information indicating an image captured by the imaging device 30 and imaging position information indicating the position of the imaging device 30. The identification unit 112A identifies a first person wearing the first wearable device 20A-1 and a second person wearing the second wearable device 20A-2, both of which are included in the captured image, based on the imaging information. The calculation unit 113A calculates self-position information indicating a first position of the first wearable device 20A-1 and a second position of the second wearable device 20A-2, using the imaging position information. The output unit 115 supplies the self-position information to the first wearable device 20A-1 and the second wearable device 20A-2. The first wearable device 20A-1 stores first provisional position information indicating a first position, and the second wearable device 20A-2 stores second provisional position information indicating a second position. The first wearable device 20A-1 determines the location of the first wearable device 20A-1 by selecting one of the first and second locations indicated by the self-location information based on the location indicated by the first provisional location information. The second wearable device 20A-2 determines the location of the second wearable device 20A-2 by selecting one of the first location and the second location indicated by the self-location information based on the location indicated by the second provisional location information. Here, the accuracy of the imaging position information is higher than the accuracy of the provisional position information.

[0134] By using the above configuration in the information processing device 10A, the MR glasses 50-1 and 50-2 can acquire more accurate self-locations when acquiring their own locations. In particular, the calculation unit 113A can calculate more accurate self-location information by calculating self-location information using imaging position information that is more accurate than the provisional position information that can be acquired by the first wearable device 20A-1 including the MR glasses 50-1 and the second wearable device 20A-2 including the MR glasses 50-2 being equipped with the GPS device 54.

[0135] Furthermore, the imaging position information indicates the absolute position of the imaging device 30. The self-position information indicates the absolute positions of the first wearable device 20A-1 and the second wearable device 20A-2. The calculation unit 113A further calculates, based on the imaging information, relative position information indicating the relative positions of the first wearable device 20A-1 and the second wearable device 20A-2 with the position of the imaging device 30 indicated by the imaging position information as a reference. Furthermore, the calculation unit 113A calculates the self-position information based on the imaging position information and the relative position information.

[0136] By using the above configuration, the information processing device 10A can allow the calculation unit 113A to calculate self-position information, which is the absolute position of the first wearable device 20A-1 and the second wearable device 20A-2, from imaging position information, which is the absolute position of the imaging device 30, and relative position information, which indicates the relative positions of the first wearable device 20A-1 and the second wearable device 20A-2 based on the position of the imaging device 30.

[0137] 3: Variation The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected arbitrarily from the following examples may be combined.

[0138] 3.1: Variation 1 In the information processing system 1 according to the first embodiment and the information processing system 1A according to the second embodiment, the imaging device 30 is a stereo camera, for example. However, the imaging device 30 is not limited to a stereo camera as long as it can acquire vector information indicating the distance and direction from the imaging device 30 to the imaging target. For example, the imaging device 30 may be configured to include a projection device that projects light from the imaging device 30 onto the imaging target and a distance measuring device that calculates the distance to the imaging target based on the phase of the light reflected from the imaging target. Alternatively, the imaging device 30 may acquire distance information indicating the distance to the imaging target by image processing the captured image captured by the imaging device 30. Alternatively, the imaging device 30 may have other depth estimation functions.

[0139] 3.2: Variation 2 In the information processing system 1 according to the first embodiment, the imaging device 30 captures an image of a person wearing a wearable device 20. In addition, in the information processing system 1A according to the second embodiment, the imaging device 30 simultaneously captures an image of a first person wearing a first wearable device 20A-1 and a second person wearing a second wearable device 20A-2. However, an imaging device other than the imaging device 30 may capture an image of the person wearing the wearable device 20, or the first person wearing the first wearable device 20A-1 and the second person wearing the second wearable device 20A-2. For example, in the information processing system 1 according to the first embodiment, an imaging device 56 provided in a second wearable device 20 different from the wearable device 20 may capture an image of the person wearing the wearable device 20. Similarly, in the information processing system 1A according to the second embodiment, an imaging device 56 provided in a third wearable device 20A-3 different from the first wearable device 20A-1 and the second wearable device 20A-2 may simultaneously capture images of a first person wearing the first wearable device 20A-1 and a second person wearing the second wearable device 20A-2. The information processing system 1 according to the first embodiment and the information processing system 1A according to the second embodiment have the above-described configuration, making it possible to capture images of a person wearing the wearable device 20 or the wearable device 20A without having to install an imaging device 30 separate from the wearable device 20 or the wearable device 20A.

[0140] 3.3: Variation 3 In the information processing system 1 according to the first embodiment, the calculation unit 113 included in the information processing device 10 calculates third position information as self-position information of the wearable device 20 based on the first imaging information acquired by the acquisition unit 111 from the MR glasses 50. Furthermore, the comparison unit 114 included in the information processing device 10 compares the self-position accuracy information acquired from the information processing device 10 by the acquisition unit 111 with a threshold value. The same applies to the information processing system 1A according to the second embodiment. However, the calculation unit 113 and the comparison unit 114 may be included in the terminal device 40 or the terminal device 40A, rather than in the information processing device 10 or the information processing device 10A.

[0141] 3.4: Variation 4 In the information processing system 1 according to the first embodiment, a calculation unit 113 included in the information processing device 10 calculates relative position information from the imaging device 30 to the wearable device 20. Similarly, in the information processing system 1A according to the second embodiment, a calculation unit 113A included in the information processing device 10 calculates relative position information from the imaging device 30 to the first wearable device 20A-1 and the second wearable device 20A-2. However, the imaging device 30 may calculate this relative position information instead of the information processing device 10 and the information processing device 10A.

[0142] 3.5: Variation 5 In the information processing system 1 according to the first embodiment, the terminal device 40 and the MR glasses 50 are realized as separate entities. Similarly, in the information processing system 1A according to the second embodiment, the terminal device 40A and the MR glasses 50 are realized as separate entities. However, the method of realizing the terminal device 40 or the terminal device 40A and the MR glasses 50 in the embodiments of the present invention is not limited to this. For example, the MR glasses 50 may have the same functions as the terminal device 40 or the terminal device 40A, so that the terminal device 40 or the 40A and the MR glasses 50 are realized in a single housing.

[0143] 4:Other (1) In the above-described embodiment, the storage devices 12, 42, and 52 are exemplified by ROM and RAM, but may be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media. The programs may also be transmitted from a network via telecommunications lines. The programs may also be transmitted from a communications network NET via telecommunications lines.

[0144] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0145] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0146] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values (e.g., comparison with a predetermined value).

[0147] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0148] (6) Each function illustrated in Figures 1 to 15 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically coupled, or may be realized by using two or more devices that are physically or logically separated and connected directly or indirectly (for example, by wire, wirelessly, etc.). A functional block may also be realized by combining software with the one device or the multiple devices.

[0149] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, and functions, regardless of whether they are called software, firmware, middleware, microcode, or hardware description language, or by other names.

[0150] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), these wired and / or wireless technologies are included within the definition of transmission media.

[0151] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0152] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or corresponding other information.

[0153] (10) In the above-described embodiments, the information processing device 10, the information processing device 10A, the imaging device 30, the terminal device 40, and the terminal device 40A may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term. Furthermore, in the present disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0154] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0155] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0156] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0157] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0158] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

[0159] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0160] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0161] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0162] 1, 1A...information processing system, 10, 10A...information processing device, 11, 11A...processing device, 12...storage device, 13...communication device, 14...display, 15...input device, 20, 20A...wearable device, 30...imaging device, 40, 40A...terminal device, 41, 41A...processing device, 42...storage device, 43...communication device, 44...display, 45...input device, 46...inertial sensor, 50...MR glasses, 51...processing device, 52...storage device, 53...gaze detection device, 54...GPS device, 55...motion detection device, 56...imaging device, 57...communication Reception device, 58...display, 59L, 59R...lenses, 91...temples, 93...bridge, 94, 95...torso, 111, 111A...acquisition unit, 112, 112A...identification unit, 113, 113A...calculation unit, 114...comparison unit, 115...output unit, 411...acquisition unit, 412...motion recognition unit, 413...identification unit, 414...output unit, 415...display control unit, 416...selection unit, 511...acquisition unit, 512...display control unit, LM1...learning model, PR1 to PR3...control programs, U1 to U3...user, VO, VO1 to VO5...virtual object

Claims

1. an acquisition unit that acquires imaging information indicating an image captured by an imaging device, imaging position information indicating a position of the imaging device, and temporary position information indicating a position of the wearable device; an identification unit that identifies a person wearing the wearable device from people represented by one or more person images included in the captured image based on the imaging information and the provisional position information; a calculation unit that calculates self-location information indicating a location of the wearable device worn by the person identified by the identification unit by using the imaging location information; an output unit that supplies the self-location information to the wearable device; The accuracy of the imaging position information is higher than the accuracy of the provisional position information. Information processing device.

2. The calculation unit further calculating, based on the imaging information, relative position information indicating a relative position of the wearable device with respect to a position of the imaging device indicated by the imaging position information; Calculating the self-position information based on the imaging position information and the relative position information; the imaging position information indicates an absolute position of the imaging device, and the self-position information indicates an absolute position of the wearable device; The information processing device according to claim 1 .

3. the wearable device is a first wearable device; The information processing device according to claim 1 , wherein the imaging device is attached to a second wearable device other than the first wearable device.

4. an acquisition unit that acquires imaging information indicating an image captured by an imaging device and imaging position information indicating a position of the imaging device; an identification unit that identifies a first person wearing a first wearable device and a second person wearing a second wearable device and included in the captured image based on the imaging information; a calculation unit that calculates self-position information indicating a first position of the first wearable device and a second position of the second wearable device by using the imaging position information; an output unit that supplies the self-location information to the first wearable device and the second wearable device; the first wearable device, First provisional position information indicating the first position is stored; the second wearable device second provisional position information indicating the second position is stored; the first wearable device specifies a location of the first wearable device by selecting one of the first location and the second location indicated by the self-location information based on the first location indicated by the first tentative location information; the second wearable device specifies a location of the second wearable device by selecting one location from the first location and the second location indicated by the self-location information based on the second location indicated by the second tentative location information; the accuracy of the imaging position information is higher than the accuracy of the first provisional position information and the accuracy of the second provisional position information; Information processing device.

5. The calculation unit further calculating, based on the imaging information, relative position information indicating the relative positions of the first wearable device and the second wearable device relative to the position of the imaging device indicated by the imaging position information; Calculating the self-position information based on the imaging position information and the relative position information; the imaging position information indicates an absolute position of the imaging device, and the self-position information indicates absolute positions of the first wearable device and the second wearable device; The information processing device according to claim 4 .

6. The information processing device according to claim 4 , wherein the imaging device is attached to a third wearable device other than the first wearable device and the second wearable device.

Images