Terminal device, information processing system, information processing method, and computer program

Abstract

To estimate a position and attitude of a terminal device with high accuracy.SOLUTION: A terminal device that is held by a user and moves with the user includes an imaging unit, a marker information acquisition unit, an image association information acquisition unit, a relative relation determination unit, and a terminal position and attitude estimation unit. The marker information acquisition unit obtains marker position and attitude information indicating a position and attitude of a marker portion of a moving body on a three-dimensional point group map. The image association information acquisition unit obtains image association information for associating an image in which the marker portion is captured with the relative position and relative attitude with respect to the marker portion. The relative relation determination unit determines relative relation information indicating the relative position and relative attitude of the terminal device with respect to the marker portion, on the basis of, both the image of the marker portion which is captured by the imaging unit and the image association information. The terminal position and attitude estimation unit estimates the position and attitude of the terminal device on the three-dimensional point group map, on the basis of, the marker position and attitude information and the relative relation information.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The technology disclosed in this specification relates to a terminal device, an information processing system, an information processing method, and a computer program.

Background Art

[0002] In recent years, the technology of Augmented Reality (AR), which extends the real environment perceived by humans by a computer, has been spreading. For example, in a terminal device (such as a smartphone) held by a user and moving with the user, AR-type information presentation that superimposes a virtual image (computer graphics) on an image of the real environment generated by an imaging unit and displays it on a display unit is used in forms such as walking navigation, sightseeing guidance, maintenance inspection guidance, location-based games, and the like.

[0003] In AR-type information presentation, based on the position and orientation of the terminal device, information on where the imaging unit of the terminal device is looking at what distance and angle is specified, and based on this information, the type, size, position, etc. of the virtual image to be displayed on the display unit are determined. Therefore, in AR-type information presentation, in order to display a virtual image that matches the real environment, it is required to accurately estimate the position and orientation of the terminal device (more specifically, the imaging unit of the terminal device, the same applies hereinafter).

[0004] Conventionally, a method of estimating the position and orientation of a terminal device by receiving signals from GNSS (Global Navigation Satellite System) such as GPS, GLONASS, Galileo, and QZSS by the terminal device is known. Also, a method is known in which markers with known positions and orientations in the real environment are installed, the imaging unit of the terminal device images the markers, information on the relative position and orientation of the terminal device with respect to the markers is specified, and based on this information, the position and orientation of the terminal device in the real environment are estimated (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2021-4894 [Summary of the Invention] [Problems to be Solved by the Invention]

[0006] In a method of estimating the position and orientation of a terminal device using GNSS, there are problems that the estimation error is relatively large (for example, on the order of the number of errors m), and it is difficult to use indoors because the signal cannot be received well indoors. Also, in a method of estimating the position and orientation of a terminal device using markers, there is a problem that it is necessary to install a large number of markers in advance whose positions and orientations are known.

[0007] Note that it is also possible to mount on the terminal device sensors (for example, LiDAR) and a three-dimensional point cloud map database used for self-position and orientation estimation by an autonomous mobile body, and use them to estimate the position and orientation of the terminal device. However, when such sensors and databases are mounted on a terminal device held by a user and moving with the user, there are problems that the terminal device becomes larger and heavier, and the storage capacity of the terminal device becomes insufficient.

[0008] Thus, conventionally, there has been a problem that it is difficult to estimate the position and orientation of a terminal device with high accuracy indoors and outdoors while suppressing the increase in size and weight of the terminal device and the shortage of storage capacity, and without the need to install a large number of markers in advance.

[0009] Note that such problems are not limited to the estimation of the position and orientation of a terminal device for AR-type information presentation, but are common problems in the estimation of the position and orientation of terminal devices for other uses.

[0010] This specification discloses a technology capable of solving the above-described problems. [Means for Solving the Problems]

[0011] The technology disclosed in this specification can be realized, for example, in the following forms.

[0012] (1) The terminal device disclosed in this specification is a device held by a user and moving with the user. The terminal device includes an imaging unit, a marker information acquisition unit, an image correspondence information acquisition unit, a relative relationship specifying unit, and a terminal position and orientation estimation unit. The marker information acquisition unit is a moving body that moves while specifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is assigned, and has a marker unit that functions as a marker. For the moving body, marker position and orientation information indicating the position and orientation of the marker unit on the three-dimensional point cloud map is acquired. The image correspondence information acquisition unit acquires image correspondence information that associates an image in which the marker unit is imaged with the relative position and relative orientation with respect to the marker unit. The relative relationship specifying unit specifies relative relationship information indicating the relative position and relative orientation of the terminal device with respect to the marker unit based on the image of the marker unit imaged by the imaging unit and the image correspondence information. The terminal position and orientation estimation unit estimates the position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information.

[0013] Thus, in this terminal device, based on the marker position and orientation information indicating the position and orientation of the marker unit of the moving body on the three-dimensional point cloud map and the relative relationship information indicating the relative position and relative orientation of the terminal device with respect to the marker unit, the position and orientation of the terminal device on the three-dimensional point cloud map can be estimated. Therefore, according to this terminal device, as long as the marker unit of the moving body is at a position where it can be imaged by the imaging unit, it can be seamless regardless of whether it is indoors or outdoors, and it is not necessary to install a large number of markers with known positions and orientations in advance. Furthermore, without mounting sensors other than the imaging unit (for example, LiDAR) or a three-dimensional point cloud map database on the terminal device, the position and orientation of the terminal device can be estimated with high accuracy.

[0014] (2) In the above terminal device, it may further include a tracking unit that tracks changes in the position and orientation of the terminal device from the time when the position and orientation of the terminal device are estimated, and an estimated result update unit that updates the estimated result of the position and orientation of the terminal device based on the changes. According to this terminal device, once the position and orientation (initial position and initial orientation) of the terminal device on the three-dimensional point cloud map are estimated based on the marker position and orientation information and the relative relationship information, regardless of whether the marker portion of the moving body can be imaged by the imaging unit, the terminal device alone can update the estimated result of the position and orientation of the terminal device. At this time, since the estimation accuracy of the initial position and initial orientation is high, the estimation accuracy of the position and orientation of the terminal device can be maintained above a certain level even after subsequent updates. Therefore, according to this terminal device, for a terminal device that moves independently of the moving body, the position and orientation of the terminal device can be continuously estimated with high accuracy.

[0015] (3) In the above terminal device, the tracking unit may be configured to track changes in the position and orientation of the terminal device using the image captured by the imaging unit. According to this terminal device, the position and orientation of the terminal device can be continuously estimated with high accuracy without the need for sensors other than the imaging unit.

[0016] (4) In the above terminal device, it may further include a display unit and an augmented reality processing unit that superimposes a virtual image that changes according to the estimated result of the position and orientation of the terminal device on the real image captured by the imaging unit and displays it on the display unit. According to this terminal device, a virtual image can be displayed based on the position and orientation of the terminal device estimated with high accuracy, and high-precision AR-type information presentation can be realized.

[0017] (5) The information processing system disclosed in this specification includes the above terminal device and the above mobile body. The terminal device further includes an estimation result transmission unit that transmits the estimation result of the position and orientation of the terminal device directly or via another device to the mobile body. The mobile body may be configured to include a specific process execution unit that executes a specific process using the estimation result of the position and orientation of the terminal device. According to this information processing system, a specific process can be executed using the position and orientation of the terminal device on the three-dimensional point cloud map estimated with high accuracy, and the accuracy of the specific process can be improved.

[0018] (6) In the above information processing system, the specific process may be configured to include a hazard notification process that notifies the terminal device of the presence of a hazard in the vicinity of the terminal device. According to this information processing system, the presence of a hazard can be notified from the mobile body to the terminal device using the position and orientation of the terminal device on the three-dimensional point cloud map estimated with high accuracy, and high-precision hazard notification can be realized.

[0019] (7) In the above information processing system, the mobile body may be configured to be an autonomous mobile body that moves autonomously using the surrounding detection information by sensors and the three-dimensional point cloud map. According to this information processing system, by moving the marker unit without requiring a human driving operation, it is possible to realize the estimation of the position and orientation of the terminal device with high accuracy using the moving marker unit in a wide range.

[0020] (8) In the above information processing system, a server device is further included. The mobile body transmits the marker position and orientation information to the server device, and the marker information acquisition unit of the terminal device acquires the marker position and orientation information via the server device. According to this information processing system, it is possible to realize the estimation of the position and orientation of the terminal device with high accuracy using the marker unit of the mobile body even in a terminal device that cannot communicate directly with the mobile body.

[0021] Note that the technology disclosed in this specification can be implemented in various forms. For example, it can be implemented in the form of a terminal device, an information processing system, an information processing method, a computer program for realizing those methods, a non-transitory recording medium recording the computer program, and the like.

Brief Description of Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Mode for Carrying Out the Invention

[0023] A. First Embodiment: A-1. Configuration of Information Processing System 10: FIG. 1 is an explanatory diagram schematically showing the configuration of the information processing system 10 in the first embodiment. The information processing system 10 of the present embodiment highly accurately estimates the position and orientation (direction) of the terminal device 100 (more specifically, the imaging unit 154 described later, the same applies hereinafter) on the three-dimensional point cloud map, and based on the estimated position and orientation of the terminal device 100, it is a system for executing AR-type information presentation in the terminal device 100. In the present embodiment, AR-type information presentation for performing guidance display indoors and outdoors of a shopping mall will be described as an example.

[0024] The information processing system 10 includes terminal devices 100 each used by a plurality of users U, a plurality of autonomous mobile bodies 200, and a server device 300. Each device constituting the information processing system 10 is communicably connected to each other via a communication network NET.

[0025] The terminal device 100 is a device held by the user U and moves with the user U. For example, it is a handheld device such as a smartphone or a tablet-type terminal, or a head-mounted image display device (Head Mounted Display, HMD), etc. Hereinafter, an example using a smartphone as the terminal device 100 will be described.

[0026] FIG. 2 is a block diagram schematically showing the configuration of the terminal device 100 in the first embodiment. The terminal device 100 includes a control unit 110, a storage unit 120, a display unit 151, an operation input unit 152, an audio output unit 153, an imaging unit 154, and a communication unit 155. These units are communicably connected to each other via a bus 190.

[0027] The display unit 151 of the terminal device 100 is composed of, for example, a liquid crystal display, an organic EL display, etc., and displays various images and information. The operation input unit 152 is composed of, for example, buttons, a microphone, etc., and receives operations and instructions from the user U. Note that if the display unit 151 includes a touch panel, it may function as the operation input unit 152. The audio output unit 153 is composed of, for example, a speaker, and outputs (plays back) audio. The imaging unit 154 is composed of, for example, a camera, and captures a subject to generate a still image or a moving image representing the subject. In the present embodiment, the imaging unit 154 is a camera (rear camera) in which a lens is disposed on the surface (rear surface) opposite to the display surface of the display unit 151 in the terminal device 100. The communication unit 155 is a communication interface that communicates with other devices by a predetermined communication method via the communication network NET.

[0028] The storage unit 120 of the terminal device 100 is composed of, for example, a ROM, a RAM, etc., stores various programs and data, and is used as a working area or a temporary storage area for data when executing various programs. For example, the storage unit 120 stores an augmented reality processing program CP11 for performing the AR type information presentation process described later. The augmented reality processing program CP11 is provided, for example, stored in a computer-readable recording medium (not shown) such as a CD-ROM, a DVD-ROM, a USB memory, etc., or is downloaded from an external device via the communication network NET and installed in the terminal device 100 to be stored in the storage unit 120.

[0029] Also, when the augmented reality processing program CP11 is installed, or during the AR type information presentation process described later, marker image information MI, marker position and orientation information LI, image correspondence information II, and relative relationship information CI are stored in the storage unit 120. The contents of these information will be described in accordance with the description of the AR type information presentation process described later.

[0030] The control unit 110 of the terminal device 100 is composed of, for example, a CPU or the like, and controls the operation of the terminal device 100 by executing a computer program read from the storage unit 120. For example, the control unit 110 functions as an augmented reality processing unit 111 that performs AR-type information presentation processing by reading and executing the augmented reality processing program CP11 from the storage unit 120. The augmented reality processing unit 111 includes a terminal position and orientation estimation unit 112, and the terminal position and orientation estimation unit 112 includes a marker information acquisition unit 113, an image correspondence information acquisition unit 114, a relative relationship specification unit 115, a tracking unit 116, and an estimation result update unit 117. The functions of these units will be described in accordance with the description of the AR-type information presentation processing below.

[0031] The autonomous mobile body 200 is a mobile body that travels autonomously without the need for a driving operation by a person. The autonomous mobile body 200 is, for example, an autonomous driving vehicle or an autonomous mobile robot. In the present embodiment, an example in which an autonomous mobile robot that transports luggage in a shopping mall is used as the autonomous mobile body 200 will be described. The autonomous mobile body 200 is an example of the mobile body in the claims.

[0032] FIG. 3 is a block diagram schematically showing the configuration of the autonomous mobile body 200 in the first embodiment. The autonomous mobile body 200 includes a control unit 210, a storage unit 220, a sensor 230, a driving unit 250, a communication unit 255, and a marker unit 260.

[0033] The sensor 230 of the autonomous mobile body 200 is composed of, for example, LiDAR, RADAR, visible light camera, far-infrared camera, ultrasonic sensor, etc., and recognizes the state (position, posture, shape, moving speed, etc.) of the surrounding environment (other vehicles, pedestrians, bicycles, road obstacles, street trees, road steps, etc.) of the autonomous mobile body 200 and the state of the autonomous mobile body 200 itself. The drive unit 250 is composed of, for example, a motor, an engine, a steering, a brake, etc., and performs driving operations (acceleration, steering, braking, etc.) of the autonomous mobile body 200. The communication unit 255 is a communication interface that communicates with other devices in a predetermined communication method via the communication network NET. Further, the marker unit 260 is a part that functions as a marker. In the present embodiment, the marker unit 260 is a predetermined planar image attached to the side surface of the autonomous mobile body 200.

[0034] The storage unit 220 of the autonomous mobile body 200 is composed of, for example, ROM, RAM, hard disk drive (HDD), etc., and stores various programs and data, and is also used as a work area and a temporary storage area for data when executing various programs. For example, the storage unit 220 stores an autonomous movement control program CP21, a position and attitude information providing program CP22, and a three-dimensional point cloud map database MD. The three-dimensional point cloud map database MD includes a high-precision three-dimensional point cloud map composed of point clouds to which information indicating the three-dimensional absolute position (latitude, longitude, altitude) of roads, passages, street trees, guardrails, buildings, etc. is attached. The three-dimensional point cloud map included in the three-dimensional point cloud map database MD covers the entire movable range of the autonomous mobile body 200. Hereinafter, the coordinate system of the three-dimensional point cloud map is also referred to as the "world coordinate system", and the position and attitude on the three-dimensional point cloud map are also referred to as the "absolute position" and the "absolute attitude".

[0035] The control unit 210 of the autonomous mobile body 200 is composed of, for example, a CPU or the like, and controls the operation of the autonomous mobile body 200 by executing a computer program read from the storage unit 220. For example, the control unit 210 functions as an autonomous movement control unit 212 that controls the autonomous driving of the autonomous mobile body 200 by reading and executing the autonomous movement control program CP21 from the storage unit 220. The autonomous movement control unit 212 sets a movement route based on a command issued by the administrator. Further, the autonomous movement control unit 212 estimates the position (absolute position) and orientation (absolute orientation) of the autonomous mobile body 200 (more specifically, the sensor 230 of the autonomous mobile body 200, the same applies hereinafter) on the three-dimensional point cloud map by performing matching between various signals output from the sensor 230 and the three-dimensional point cloud map included in the three-dimensional point cloud map database MD. Furthermore, the autonomous movement control unit 212 detects and recognizes the state of the surrounding environment based on various signals output from the sensor 230. The autonomous movement control unit 212 controls the driving operation by the driving unit 250 based on this information, and drives the autonomous mobile body 200 along the set movement route while avoiding collisions with surrounding objects.

[0036] In addition, the control unit 210 of the autonomous mobile body 200 functions as a position and orientation information providing unit 214 by reading and executing the position and orientation information providing program CP22 from the storage unit 220. The function of the position and orientation information providing unit 214 will be described in accordance with the description of the AR type information presentation process described later.

[0037] The server device 300 is a device for acquiring information from the terminal device 100 and the autonomous mobile body 200 and providing information to the terminal device 100 and the autonomous mobile body 200. FIG. 4 is a block diagram schematically showing the configuration of the server device 300 in the first embodiment. The server device 300 includes a control unit 310, a storage unit 320, a display unit 351, an operation input unit 352, and a communication unit 355. These units are connected to each other so as to be communicable via a bus 390.

[0038] The display unit 351 of the server device 300 is composed of, for example, a liquid crystal display, an organic EL display, etc., and displays various images and information. The operation input unit 352 is composed of, for example, a keyboard, a mouse, buttons, a microphone, etc., and receives operations and instructions from the administrator. The communication unit 355 is a communication interface that communicates with other devices in a predetermined communication method via the communication network NET.

[0039] The storage unit 320 of the server device 300 is composed of, for example, a ROM, a RAM, a hard disk drive (HDD), etc., stores various programs and data, and is used as a work area and a temporary storage area for data when executing various programs. For example, the storage unit 320 stores an augmented reality server program CP31 for performing the AR type information presentation process described later. The augmented reality server program CP31 is provided, for example, stored in a computer-readable recording medium (not shown) such as a CD-ROM, a DVD-ROM, a USB memory, etc., or is downloaded from an external device via the communication network NET and installed in the server device 300 to be stored in the storage unit 320.

[0040] Also, when the augmented reality server program CP31 is installed, or during the AR type information presentation process described later, a virtual image database ID and marker position and orientation information LI are stored in the storage unit 320 of the server device 300. In the virtual image database ID, a plurality of virtual images provided to the terminal device 100 in the AR type information presentation process described later are set in association with the position and orientation of the terminal device 100. In the present embodiment, the virtual image is an image for guiding display in the shopping mall (see FIGS. 6 and 7 described later). Also, the content of the marker position and orientation information LI will be described in accordance with the description of the AR type information presentation process described later.

[0041] The control unit 310 of the server device 300 is configured by, for example, a CPU or the like, and controls the operation of the server device 300 by executing a computer program read from the storage unit 320. For example, the control unit 310 functions as an augmented reality server processing unit 312 that performs augmented reality server processing for performing augmented reality (AR) type information presentation processing by reading and executing the augmented reality server program CP31 from the storage unit 320. The augmented reality server processing unit 312 includes a position and orientation server processing unit 314. The functions of these units will be described in accordance with the description of the AR type information presentation processing described later.

[0042] A-2. AR Type Information Presentation Processing: Next, the AR type information presentation processing executed in the information processing system 10 of the first embodiment will be described. The AR type information presentation processing of the present embodiment is processing for superimposing a virtual image on an image of the real environment captured by the imaging unit 154 in the terminal device 100 and displaying the superimposed image on the display unit 151. In the present embodiment, AR type information presentation for providing guidance indoors and outdoors in a shopping mall will be described as an example.

[0043] FIG. 5 is an explanatory diagram showing the flow of the AR type information presentation processing executed in the information processing system 10 of the first embodiment. FIGS. 6 and 7 are explanatory diagrams conceptually showing the AR type information presentation processing executed in the information processing system 10 of the first embodiment. FIG. 6 shows a scene where AR type information presentation is being performed outdoors in a shopping mall, and FIG. 7 shows a scene where AR type information presentation is being performed indoors in a shopping mall.

[0044] As shown in FIG. 5, when the process starts, the autonomous mobile body 200 starts autonomous movement outdoors and indoors in the shopping mall (S210). That is, the autonomous movement control unit 212 (FIG. 3) of the autonomous mobile body 200 autonomously runs the autonomous mobile body 200 while estimating the position (absolute position) and orientation (absolute orientation) of the autonomous mobile body 200 on the three-dimensional point cloud map included in the three-dimensional point cloud map database MD. Note that the autonomous movement control unit 212 estimates the position and orientation of the autonomous mobile body 200 on the three-dimensional point cloud map by performing matching between various signals output from the sensor 230 and the three-dimensional point cloud map, and the estimation accuracy is very high, for example, on the order of an error of several centimeters.

[0045] When the autonomous mobile body 200 starts autonomous movement, the position and orientation information providing unit 214 (FIG. 3) of the autonomous mobile body 200 transmits marker position and orientation information LI indicating the position and orientation of the marker unit 260 on the three-dimensional point cloud map to the server device 300 (S220). In the autonomous mobile body 200, information indicating the correspondence between the position and orientation of the autonomous mobile body 200 (of the sensor 230) grasped by the autonomous movement control unit 212 and the position and orientation of the marker unit 260 is stored in the storage unit 220 in advance. The position and orientation information providing unit 214 generates the marker position and orientation information LI by referring to the correspondence and converting the position and orientation of the autonomous mobile body 200 grasped by the autonomous movement control unit 212 into the position and orientation of the marker unit 260, and transmits the marker position and orientation information LI to the server device 300. The marker position and orientation information LI includes a mobile body ID for identifying the autonomous mobile body 200, a timestamp indicating the estimation time of the position and orientation, and information specifying the position and orientation (see FIG. 4).

[0046] In addition, the position and orientation information providing unit 214 of the autonomous mobile body 200 determines whether or not the autonomous movement of the autonomous mobile body 200 has ended (S230). If it is determined that the autonomous movement of the autonomous mobile body 200 has not ended (S230: NO), the transmission process of the marker position and orientation information LI (S220) is executed again. That is, during the execution of the autonomous movement by the autonomous mobile body 200, the transmission process of the marker position and orientation information LI (S220) is repeatedly executed periodically (for example, at about 1 Hz to 100 Hz). In this embodiment, a large number of autonomous mobile bodies 200 are executing such processing indoors and outdoors in the shopping mall. If it is determined that the autonomous movement of the autonomous mobile body 200 has ended (S230: YES), the autonomous mobile body 200 ends its operation.

[0047] In addition, the position and orientation server processing unit 314 (FIG. 4) of the server device 300 receives the marker position and orientation information LI transmitted from the autonomous mobile body 200 in S220, and stores the received marker position and orientation information LI in the storage unit 320 (S310). The position and orientation server processing unit 314 accumulates the marker position and orientation information LI transmitted from the plurality of autonomous mobile bodies 200 at all times in the storage unit 320.

[0048] Also, user U of the terminal device 100 instructs the execution of an application program (augmented reality processing program CP11) for AR-type information presentation both inside and outside the shopping mall, searches for the nearby autonomous mobile body 200, and holds the terminal device 100 so that the imaging unit 154 faces the direction of the marker unit 260 of the autonomous mobile body 200 (see FIGS. 6 and 7). Triggered by such an operation by user U, the terminal position and orientation estimation unit 112 (FIG. 2) of the terminal device 100 detects the image of the marker unit 260 from the image generated by the imaging unit 154 (S110). As described above, marker image information MI is stored in the storage unit 120 of the control unit 110. The marker image information MI is information for specifying the image used as the marker unit 260. The terminal position and orientation estimation unit 112 refers to the marker image information MI and detects the image of the marker unit 260 from the captured image by a known method such as pattern matching. In the example shown in FIG. 6, the marker unit 260 of the autonomous mobile body 200 moving near the entrance outside the shopping mall is captured by the imaging unit 154 of the terminal device 100 held by user U and is displayed on the display unit 151. Also, in the example shown in FIG. 7, the marker unit 260 of the autonomous mobile body 200 moving inside the shopping mall is captured by the imaging unit 154 of the terminal device 100 held by user U and is displayed on the display unit 151.

[0049] Also, the image correspondence information acquisition unit 114 of the terminal device 100 acquires the image correspondence information II from the storage unit 120, and the relative relationship specifying unit 115 of the terminal device 100 specifies relative relationship information CI indicating the relative position and relative orientation of the terminal device 100 (imaging unit 154 thereof) with respect to the marker unit 260 based on the image of the marker unit 260 captured by the imaging unit 154 and the image correspondence information II (S120).

[0050] More specifically, the image-corresponding information II is information that associates the image of the marker unit 260 with the relative position and relative orientation with respect to the marker unit 260. Depending on the relative position and relative orientation of the imaging unit 154 of the terminal device 100 with respect to the marker unit 260, the appearance of the marker unit 260 in the image captured by the imaging unit 154 changes. For example, if the distance between the imaging unit 154 and the marker unit 260 is relatively close, the marker unit 260 appears relatively large in the captured image. Conversely, if the distance between the imaging unit 154 and the marker unit 260 is relatively far, the marker unit 260 appears relatively small in the captured image. Also, the appearance of the marker unit 260 in the captured image is different between the state where the imaging unit 154 faces the marker unit 260 directly and the state where the imaging unit 154 is positioned obliquely with respect to the marker unit 260. The image-corresponding information II is information that associates such relative position and relative orientation of the imaging unit 154 of the terminal device 100 with respect to the marker unit 260 and the appearance of the marker unit 260 in the captured image. Therefore, the relative relationship specifying unit 115 can specify the relative position and relative orientation of the terminal device 100 (the imaging unit 154 thereof) with respect to the marker unit 260 based on the image of the marker unit 260 and the image-corresponding information II. In the present embodiment, the relative position and relative orientation of the terminal device 100 with respect to the marker unit 260 are expressed as the position and orientation of the marker unit 260 in the coordinate system used by the terminal device 100 (hereinafter referred to as the "user coordinate system"). The characterized relative relationship information CI is stored in the storage unit 120.

[0051] Further, the marker information acquisition unit 113 (Fig. 1) of the terminal device 100 requests the marker position and orientation information LI from the server device 300 and acquires the marker position and orientation information LI from the server device 300 (S130). In the present embodiment, the marker unit 260 of the autonomous mobile body 200 includes information (mobile body ID) for identifying the autonomous mobile body 200. Therefore, the marker information acquisition unit 113 designates the mobile body ID of the autonomous mobile body 200 whose relative position and relative orientation are specified in S120, and designates the specific time of the relative position and relative orientation, and requests the marker position and orientation information LI from the server device 300. The position and orientation server processing unit 314 (Fig. 4) of the server device 300 selects the marker position and orientation information LI corresponding to the designation of the mobile body ID and the time designation included in the request from the terminal device 100, and transmits it to the terminal device 100 (S320). The marker position and orientation information LI acquired by the terminal device 100 is stored in the storage unit 120.

[0052] Next, the terminal position and orientation estimation unit 112 (Fig. 2) of the terminal device 100 estimates the position (absolute position) and orientation (absolute orientation) of the terminal device 100 (imaging unit 154 thereof) on the three-dimensional point cloud map based on the marker position and orientation information LI acquired in S130 and the relative relationship information CI acquired in S120 (S140). As described above, the marker position and orientation information LI is information indicating the position (absolute position) and orientation (absolute orientation) of the marker unit 260 on the three-dimensional point cloud map, and the relative relationship information CI is information indicating the relative position and relative orientation of the terminal device 100 with respect to the marker unit 260. Therefore, the terminal position and orientation estimation unit 112 can estimate the position and orientation of the terminal device 100 on the three-dimensional point cloud map by performing coordinate transformation (transformation from the user coordinate system to the world coordinate system) using both pieces of information. The coordinate transformation used at this time will be described in detail later. Hereinafter, the point in time when the estimation in S140 is executed is referred to as the "tracking start point", and the position and orientation of the terminal device 100 estimated in S140 are referred to as the "initial position" and "initial orientation".

[0053] Thereafter, the tracking unit 116 (Fig. 2) of the terminal device 100 tracks the changes in the position and orientation of the terminal device 100 from the initial position and initial orientation at the tracking start point, and the estimation result update unit 117 updates the estimation results of the position and orientation of the terminal device 100 based on the changes (S150). The tracking of the changes in the position and orientation of the terminal device 100 by the tracking unit 116 (hereinafter also referred to as "tracking") is realized by a known method. For example, the tracking unit 116 uses Visual-SLAM, which is one of the SLAM (Simultaneous Localization and Mapping) technologies that simultaneously perform self-position estimation of a moving object and creation of an environmental map, to track the changes in the position and orientation of the terminal device 100. That is, the tracking unit 116 creates a map of the surrounding environment (a map in the user coordinate system) based on the images captured by the imaging unit 154, and estimates the amount of movement and the amount of orientation change of the terminal device 100 in the user coordinate system based on the transition of the feature amounts in the images captured by the imaging unit 154, and tracks the position and orientation of the imaging unit 154 on the surrounding environment map. In this way, the tracking by the tracking unit 116 can be continuously executed even if the marker unit 260 of the autonomous mobile body 200 does not exist within the imaging range of the imaging unit 154. Further, the estimation result update unit 117 updates the estimation results of the position and orientation of the imaging unit 154 of the terminal device 100 on the three-dimensional point cloud map by performing coordinate conversion (conversion from the user coordinate system to the world coordinate system) on the position and orientation of the imaging unit 154 on the surrounding environment map.

[0054] Here, the conversion from the user coordinate system to the world coordinate system will be described. Fig. 8 is an explanatory diagram showing a method of converting from the user coordinate system to the world coordinate system. As described above, the world coordinate system is the coordinate system of the three-dimensional point cloud map, and the user coordinate system is the coordinate system used by the terminal device 100. Further, hereinafter, the coordinate system with the position of the autonomous mobile body 200 as the origin is referred to as the "mobile body coordinate system", and the coordinate system with the position of the imaging unit 154 of the terminal device 100 as the origin is referred to as the "imaging unit coordinate system".

[0055] In column A of Fig. 8, the world coordinate system (origin: Ow ) The origin O of the moving body coordinate system in V W is shown, and in column B of FIG. 8, the origin O of the imaging unit coordinate system in the user coordinate system (origin: O U ) C U is shown, and in column C of FIG. 8, the origin O of the user coordinate system in the world coordinate system V U is shown, and in column C of FIG. 8, the origin O of the user coordinate system in the world coordinate system, the position and orientation of the user coordinate system, the position and orientation of the origin O of the imaging unit coordinate system U W and the position and orientation of the imaging unit coordinate system, and the position and orientation of the origin O of the moving body coordinate system C W are shown. The origin O of the user coordinate system V W is the position of the imaging unit 154 at the tracking start point where the estimation in S140 was executed. Therefore, at the tracking start point, the imaging unit coordinate system coincides with the user coordinate system. After the tracking start point, as the position and orientation of the imaging unit 154 of the terminal device 100 change (shown by the thin dashed line in FIG. 8), the origin position and orientation of the imaging unit coordinate system change from the position and orientation of the origin of the user coordinate system U .

[0056] As shown in the following formula (1), the linear transformation matrix T from the points in the user coordinate system to the world coordinate system OU w is the vector T from the origin O of the world coordinate system w to the origin O of the moving body coordinate system V W minus the vector T from the origin O of the user coordinate system in the world coordinate system OV W to the origin O of the moving body coordinate system U W can be obtained by subtracting V W the vector T O U V .

Number

[0057] Also, as shown in the following formula (2), the rotation transformation matrix R from the point in the user coordinate system to the world coordinate system OU w is the origin O of the world coordinate system w seen from the origin O of the moving body coordinate system V W can be obtained by the quaternion product of the rotation amount and the inverse quaternion of the rotation amount of the moving body coordinate system seen from the user coordinate system.

Number

[0058] By using these transformation matrices, at the tracking start point, the terminal position and orientation estimation unit 112 can estimate the position and orientation of the terminal device 100 on the three-dimensional point cloud map (S140). Also, after the tracking start point, the estimation result update unit 117 can update the estimation result of the position and orientation of the terminal device 100 on the three-dimensional point cloud map (S150). As a result, the terminal device 100 can continuously grasp the position and orientation of the terminal device 100 on the three-dimensional point cloud map (i.e., the absolute position and absolute orientation in the world coordinate system) after the tracking start point.

[0059] As shown in FIG. 5, after the tracking start point, the augmented reality processing unit 111 (FIG. 2) of the terminal device 100 requests a virtual image VI corresponding to the position and orientation of the terminal device 100 on the three-dimensional point cloud map from the server device 300, and acquires the virtual image VI from the server device 300 (S160). That is, the augmented reality processing unit 111 specifies the position and orientation of the terminal device 100 on the three-dimensional point cloud map and requests the virtual image VI from the server device 300. As described above, in the server device 300, a virtual image database ID (FIG. 4) in which a plurality of virtual images VI are set in association with the position and orientation of the terminal device 100 is constructed. The augmented reality server processing unit 312 of the server device 300 selects the virtual image VI corresponding to the position and orientation of the terminal device 100 included in the request from the terminal device 100 from the virtual image database ID, and transmits the virtual image VI to the terminal device 100 (S330).

[0060] The augmented reality processing unit 111 of the terminal device 100 superimposes the virtual image VI acquired in S160 on the real image RI captured by the imaging unit 154 and causes the display unit 151 to display it (S170). For example, in the example shown in FIG. 6, a virtual image VI for guidance display (such as "Main Entrance", "Parking", "Cafe", etc.) is displayed on the display unit 151 of the terminal device 100 held by the user U, superimposed on the real image RI representing the appearance near the entrance of the shopping mall. Also, in the example shown in FIG. 7, a virtual image VI for guidance display (such as "Downstair", "Ivent Space", "Food Court", etc.) is displayed on the display unit 151 of the terminal device 100 held by the user U, superimposed on the real image RI representing the inside of the building of the shopping mall. The user U can grasp the positions of the respective facilities in the shopping mall by referring to the virtual image VI displayed on the display unit 151.

[0061] After the tracking start point, the augmented reality processing unit 111 (Fig. 2) of the terminal device 100 monitors whether an image of the marker unit 260 is detected from the image generated by the imaging unit 154 (S180), and also monitors whether there is an end instruction for the AR type information presentation process (S190). When it is determined that the image of the marker unit 260 is not detected (S180: NO) and there is no end instruction (S190: NO), the terminal device 100 repeatedly executes the processes of S150 to S170 described above. That is, since the position and orientation of the terminal device 100 change every moment as the user U moves and changes their posture, the estimation result of the position and orientation of the terminal device 100 is updated (S150), the request / obtaining process of the virtual image VI (S160) is executed again according to the update result, and the displayed virtual image VI is updated (S170). As a result, even if the user U moves or changes their posture, an appropriate virtual image VI that matches the real image RI is displayed on the display unit 151 of the terminal device 100.

[0062] Also, when it is determined that the image of the marker unit 260 is detected in the captured image by the imaging unit 154 (S180: YES), the terminal device 100 re-executes the processes after S120 described above. That is, based on the image of the marker unit 260 and the image correspondence information II, the relative position and relative orientation of the terminal device 100 with respect to the marker unit 260 are specified (S120), the marker position and orientation information LI indicating the position and orientation of the marker unit 260 on the three-dimensional point cloud map is acquired (S130), and the position and orientation of the terminal device 100 on the three-dimensional point cloud map are estimated (S140). That is, the initial position and initial orientation of the terminal device 100 are updated. In the tracking by the terminal device 100 after the tracking start point, there is a possibility that the estimation error of the position and orientation of the terminal device 100 accumulates, but since the error is eliminated by updating the initial position and initial orientation of the terminal device 100, highly accurate estimation of the position and orientation of the terminal device 100 continuously is realized. Note that even after the autonomous movement of the autonomous mobile body 200 is completed (S230: YES), since the marker position and orientation information LI transmitted to the server device 300 immediately before the end of the autonomous movement can be used, the estimation or update of the initial position and initial orientation of the terminal device 100 using the marker position and orientation information LI is executable. Also, when the terminal device 100 fails in tracking or tracking becomes impossible halfway, the server device 300 may present the position of the autonomous mobile body 200 near the terminal device 100 or change the route of the autonomous mobile body 200 to move it near the terminal device 100. Thereafter, the processes after S150 (tracking, etc.) are executed in the same manner.

[0063] During the execution of the above-described processes by the terminal device 100, when it is determined that there is an instruction to end the AR type information presentation process (S190: YES), the augmented reality processing unit 111 of the terminal device 100 ends the AR type information presentation process. Similarly, also in the server device 300, when it is determined that there is an instruction to end the AR type information presentation process (S340: YES), the augmented reality server processing unit 312 of the server device 300 ends the AR type information presentation process.

[0064] A-3. Effects of the First Embodiment: As described above, the terminal device 100 that constitutes the information processing system 10 of the first embodiment is a device held by the user U and moves with the user U. The terminal device 100 includes an imaging unit 154, a marker information acquisition unit 113, an image correspondence information acquisition unit 114, a relative relationship specifying unit 115, and a terminal position and orientation estimation unit 112. The marker information acquisition unit 113 is an autonomous mobile body 200 that moves while specifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is assigned. The autonomous mobile body 200 has a marker unit 260 that functions as a marker. The marker information acquisition unit 113 acquires marker position and orientation information LI indicating the position and orientation of the marker unit 260 on the three-dimensional point cloud map. The image correspondence information acquisition unit 114 acquires image correspondence information II that associates an image in which the marker unit 260 is imaged with the relative position and relative orientation with respect to the marker unit 260. The relative relationship specifying unit 115 specifies relative relationship information CI indicating the relative position and relative orientation of the terminal device 100 with respect to the marker unit 260 based on the image of the marker unit 260 imaged by the imaging unit 154 and the image correspondence information II. The terminal position and orientation estimation unit 112 estimates the position and orientation of the terminal device 100 on the three-dimensional point cloud map based on the marker position and orientation information LI and the relative relationship information CI.

[0065] As described above, in the terminal device 100 of the present embodiment, based on the marker position and orientation information LI indicating the position and orientation of the marker unit 260 of the autonomous mobile body 200 on the three-dimensional point cloud map and the relative relationship information CI indicating the relative position and relative orientation of the terminal device 100 with respect to the marker unit 260, the position and orientation of the terminal device 100 on the three-dimensional point cloud map can be estimated. Therefore, according to the terminal device 100 of the present embodiment, as long as the marker unit 260 of the autonomous mobile body 200 is at a position where it can be imaged by the imaging unit 154, it can be seamless regardless of whether it is indoors or outdoors. Moreover, it is not necessary to install a large number of markers with known positions and orientations in advance. Furthermore, without mounting a sensor other than the imaging unit 154 (for example, LiDAR) or a three-dimensional point cloud map database on the terminal device 100, the position and orientation of the terminal device 100 can be estimated with high accuracy.

[0066] Further, the terminal device 100 of the present embodiment further includes a tracking unit 116 and an estimation result update unit 117. The tracking unit 116 tracks changes in the position and orientation of the terminal device 100 from the time point (tracking start point) when the position and orientation of the terminal device 100 are estimated. The estimation result update unit 117 updates the estimation result of the position and orientation of the terminal device 100 based on the changes. Therefore, according to the terminal device 100 of the present embodiment, once the position and orientation (initial position and initial orientation) of the terminal device 100 on the three-dimensional point cloud map are estimated based on the marker position and orientation information LI and the relative relationship information CI, regardless of whether the marker unit 260 of the autonomous mobile body 200 can be imaged by the imaging unit 154, the terminal device 100 alone can update the estimation result of the position and orientation of the terminal device 100. At this time, since the estimation accuracy of the initial position and initial orientation is high, the estimation accuracy of the position and orientation of the terminal device 100 is maintained above a certain level even after subsequent updates. Therefore, according to the terminal device 100 of the present embodiment, for the terminal device 100 that moves independently of the autonomous mobile body 200, the position and orientation of the terminal device 100 can be continuously estimated with high accuracy.

[0067] In the terminal device 100 of the present embodiment, the tracking unit 116 uses the image captured by the imaging unit 154 to track changes in the position and orientation of the terminal device 100. Therefore, in the terminal device 100 of the present embodiment, the position and orientation of the terminal device 100 can be continuously estimated with high accuracy without the need for sensors other than the imaging unit 154.

[0068] Further, the terminal device 100 of the present embodiment further includes a display unit 151 and an augmented reality processing unit 111. The augmented reality processing unit 111 superimposes a virtual image VI that changes according to the estimation result of the position and orientation of the terminal device 100 on the real image RI captured by the imaging unit 154 and displays it on the display unit 151. Therefore, according to the terminal device 100 of the present embodiment, based on the position and orientation of the terminal device 100 estimated with high accuracy, the virtual image VI can be displayed, and high-precision AR-type information presentation can be realized.

[0069] Further, the information processing system 10 of the present embodiment includes a terminal device 100 and an autonomous mobile body 200. The autonomous mobile body 200 is a mobile body that autonomously moves using the surrounding detection information by the sensor 230 and the three-dimensional point cloud map. Therefore, according to the information processing system 10 of the present embodiment, by moving the marker unit 260 without requiring a human driving operation, it is possible to estimate the position and orientation of the terminal device 100 with high accuracy using the moving marker unit 260 in a wide range.

[0070] Also, the information processing system 10 of the present embodiment further includes a server device 300. The autonomous mobile body 200 transmits the marker position and orientation information LI to the server device 300, and the marker information acquisition unit 113 of the terminal device 100 acquires the marker position and orientation information LI via the server device 300. Therefore, according to the information processing system 10 of the present embodiment, even in the terminal device 100 that cannot communicate directly with the autonomous mobile body 200, it is possible to estimate the position and orientation of the terminal device 100 with high accuracy using the marker unit 260 of the autonomous mobile body 200.

[0071] B. Second Embodiment: FIG. 9 is an explanatory diagram schematically showing the configuration of the information processing system 10a in the second embodiment, FIG. 10 is a block diagram schematically showing the configuration of the terminal device 100a in the second embodiment, FIG. 11 is a block diagram schematically showing the configuration of the autonomous mobile body 200a in the second embodiment, and FIG. 12 is a block diagram schematically showing the configuration of the server device 300a in the second embodiment. Hereinafter, among the configurations of the information processing system 10a, the terminal device 100a, the autonomous mobile body 200a, and the server device 300a in the second embodiment and the processing contents by each device, the same configurations and processing contents as those in the first embodiment described above are denoted by the same reference numerals, and the description thereof will be appropriately omitted.

[0072] The information processing system 10a of the second embodiment is a system that accurately estimates the position and orientation of the terminal device 100a on the 3D point cloud map, similar to the information processing system 10 of the first embodiment. However, instead of performing AR-type information presentation using the estimated position and orientation, it is a system for performing hazard notification. In this embodiment, a hazard notification that notifies the presence of a hazard close to the terminal device 100a at an intersection or the like with poor visibility on a public road will be described as an example.

[0073] As shown in FIG. 9, the information processing system 10a includes a plurality of terminal devices 100a, a plurality of autonomous mobile bodies 200a, and a server device 300a. In the second embodiment, the autonomous mobile body 200a is an autonomous driving vehicle that autonomously travels on a public road with a passenger on board.

[0074] As shown in FIG. 10, a hazard notification program CP12 for performing a hazard notification process described later is stored in the storage unit 120 of the terminal device 100a in the second embodiment. The control unit 110 of the terminal device 100a functions as a hazard notification unit 119 that performs the hazard notification process described later by reading and executing the hazard notification program CP12 from the storage unit 120. The hazard notification unit 119 includes a terminal position and orientation estimation unit 112a similar to that of the first embodiment. However, the terminal position and orientation estimation unit 112a of the second embodiment includes an estimation result transmission unit 118. Further, hazard information HI is stored in the storage unit 120 of the terminal device 100a during the hazard notification process described later.

[0075] As shown in FIG. 11, a hazard notification program CP23 for performing a hazard notification process described later is stored in the storage unit 220 of the autonomous mobile body 200a in the second embodiment. By reading out and executing the hazard notification program CP23 from the storage unit 220, the control unit 210 of the autonomous mobile body 200a functions as a hazard notification unit 216 that performs the hazard notification process described later. Further, terminal position and orientation information TI is stored in the storage unit 220 of the autonomous mobile body 200a during the hazard notification process described later. Note that the hazard notification process is an example of a specific process in the claims, and the hazard notification unit 216 is an example of a specific process execution unit in the claims.

[0076] As shown in FIG. 12, a hazard notification server program CP32 for performing a hazard notification process described later is stored in the storage unit 320 of the server device 300a in the second embodiment. By reading out and executing the hazard notification server program CP32 from the storage unit 320, the control unit 310 of the server device 300a functions as a hazard notification server processing unit 318 that performs the hazard notification process described later. The hazard notification server processing unit 318 includes a position and orientation server processing unit 314 similar to that in the first embodiment. Further, terminal position and orientation information TI and hazard information HI are stored in the storage unit 320 of the server device 300a during the hazard notification process described later.

[0077] Next, the hazard notification process executed in the information processing system 10a of the second embodiment will be described. The hazard notification process of the present embodiment is a process of notifying the terminal device 100a of the presence of a hazard approaching the terminal device 100a recognized by the autonomous mobile body 200a and causing the voice output unit 153 of the terminal device 100a to give an alarm. In the present embodiment, the hazard notification process of notifying the presence of a hazard approaching the terminal device 100a at an intersection or the like with poor visibility on a public road will be described as an example.

[0078] FIG. 13 is an explanatory diagram showing the flow of hazard notification processing executed in the information processing system 10a of the second embodiment. Further, FIG. 14 is an explanatory diagram conceptually showing the hazard notification processing executed in the information processing system 10a of the second embodiment. In the example shown in FIG. 14, at an intersection with poor visibility on a public road, the autonomous mobile body 200a is traveling within the imaging range of the imaging unit 154 of the terminal device 100a held by the user U. Also, there is a bicycle as a hazard Ha that cannot be visually recognized by the user U but can be recognized by the sensor 230 of the autonomous mobile body 200a in the shade of the building Bu, and the bicycle is about to jump out in front of the user U.

[0079] In the hazard notification processing shown in FIG. 13, the processing contents of S210 and S220 by the autonomous mobile body 200a, the processing contents of S310 and S320 by the server device 300a, and the processing contents from S110 to S140 by the terminal device 100a are the same as the processing contents of each step in the AR type information presentation processing of the first embodiment shown in FIG. 5. By these processes, the terminal device 100a can accurately estimate the position (initial position) and orientation (initial accuracy) of the terminal device 100a (of the imaging unit 154) on the three-dimensional point cloud map.

[0080] After that, the estimation result transmission unit 118 (Fig. 10) of the terminal device 100a transmits terminal position and orientation information TI indicating the estimation result of the position and orientation of the terminal device 100a on the 3D point cloud map to the autonomous mobile body 200a via the server device 300a. That is, the estimation result transmission unit 118 transmits the terminal position and orientation information TI toward the server device 300a (S142), the position and orientation server processing unit 314 (Fig. 12) of the server device 300a transmits the received terminal position and orientation information TI toward the autonomous mobile body 200a (S322), and the hazard notification unit 216 (Fig. 11) of the autonomous mobile body 200a receives the terminal position and orientation information TI (S222). The terminal position and orientation information TI includes a terminal ID for identifying the terminal device 100a, a timestamp indicating the estimation time of the position and orientation, and information specifying the position and orientation (see Fig. 12). In this embodiment, the terminal position and orientation information TI is transmitted for the autonomous mobile body 200a existing within a predetermined distance from the estimated position of the terminal device 100a (that is, existing near the terminal device 100a).

[0081] After transmitting the terminal position and orientation information TI, the terminal device 100a updates the estimation result of the position and orientation of the terminal device 100a based on the change in the position and orientation of the terminal device 100a from the initial position and initial orientation in the same manner as in the first embodiment (S150), and determines whether hazard information HI has been received (S152).

[0082] The hazard notification unit 216 (Fig. 11) of the autonomous mobile body 200a that has received the terminal position and orientation information TI specifies the position and orientation of the terminal device 100a on the 3D point cloud map based on the terminal position and orientation information TI, and searches for a hazard Ha around the terminal device 100a (S224). That is, the hazard notification unit 216 determines the presence or absence of a surrounding object that can be a hazard Ha to the terminal device 100a based on the relative position relationship and relative speed relationship between the surrounding object recognized by the sensor 230 and the terminal device 100a.

[0083] When the hazard notification unit 216 of the autonomous mobile body 200a detects a hazard Ha around the terminal device 100a (S224: YES), it creates hazard information HI that identifies the detected hazard Ha, and transmits the hazard information HI to the target autonomous mobile body 200a via the server device 300a. That is, the hazard notification unit 216 transmits the hazard information HI toward the server device 300a (S226), and the hazard notification server processing unit 318 (FIG. 12) of the server device 300a transmits the received hazard information HI toward the autonomous mobile body 200a (S324), and the hazard notification unit 119 (FIG. 10) of the terminal device 100a receives the hazard information HI (S152: YES). The hazard information HI is information indicating, for example, the position, speed, name, etc. of the hazard Ha. When the hazard Ha is not detected in the autonomous mobile body 200a (S224: NO), the process of S226 is skipped.

[0084] The hazard notification unit 119 of the terminal device 100a that has received the hazard information HI executes a notification process for notifying the user U of the presence of the hazard Ha (S154). For example, in the example of FIG. 14, a voice "attention!" is output from the voice output unit 153 of the terminal device 100a. By the notification process, the user U of the terminal device 100a can recognize the presence of the hazard Ha recognized by the autonomous mobile body 200a, although the user U himself / herself cannot visually recognize it. In the notification process, the direction of the hazard Ha (e.g., forward) and the name of the hazard Ha (e.g., bicycle) as seen from the terminal device 100a may be notified. Instead of, or together with, the voice notification, a notification by an image or a vibration may be executed. When the hazard information HI is not received in the terminal device 100a (S152: NO), the process of S154 is skipped. The subsequent process is the same as that of the first embodiment.

[0085] As described above, the information processing system 10a of the second embodiment includes the terminal device 100a and the autonomous mobile body 200a. The terminal device 100a includes an estimation result transmission unit 118 that transmits the estimation result of the position and orientation of the terminal device 100a to the autonomous mobile body 200a directly or via another device. The autonomous mobile body 200a includes a hazard notification unit 216 that executes a hazard notification process of notifying the terminal device 100a of the presence of the hazard Ha in the vicinity of the terminal device 100a by using the estimation result of the position and orientation of the terminal device 100a. Therefore, according to the information processing system 10a of the second embodiment, it is possible to notify the terminal device 100a of the presence of the hazard Ha from the autonomous mobile body 200a by using the position and orientation of the terminal device 100a on the three-dimensional point cloud map estimated with high accuracy, and to realize a highly accurate hazard notification.

[0086] C. Modification example: The technology disclosed in this specification is not limited to the above-described embodiments, and can be modified into various forms without departing from the gist thereof. For example, the following modifications are also possible.

[0087] The configuration of the information processing system 10 in the above embodiment is merely an example and can be variously modified. For example, in the above embodiment, the mobile body having the marker unit 260 is the autonomous mobile body 200 that autonomously moves using the surrounding detection information by the sensor 230 and the three-dimensional point cloud map. However, as long as it is a mobile body that moves while specifying its own position and orientation on the three-dimensional point cloud map, it does not necessarily have to be an autonomous mobile body, and it may be a mobile body that moves according to a driving operation by a person. Further, in the above embodiment, the autonomous mobile body 200 estimates its own position and orientation by matching the surrounding recognition result by the sensor 230 and the three-dimensional point cloud map. However, other methods (for example, methods using Visual-SLAM or GNSS) may be combined to estimate its own position and orientation with higher accuracy.

[0088] In the above embodiment, the autonomous mobile body 200 has an image attached to the side surface of the vehicle body as the marker portion 260. However, the form of the marker portion 260 is not limited to this. For example, the marker portion 260 may be a three-dimensional object, or the shape of the autonomous mobile body 200 itself may function as the marker portion 260.

[0089] The contents of the AR type information presentation process and the hazard notification process in the above embodiment are merely examples and can be variously changed. For example, in the above embodiment, the position and orientation information providing unit 214 of the autonomous mobile body 200 transmits the marker position and orientation information LI indicating the position and orientation of the marker portion 260 on the three-dimensional point cloud map to the server device 300 (S220 in FIG. 5), and the position and orientation server processing unit 314 of the server device 300 transmits the marker position and orientation information LI to the terminal device 100 (S320). However, the position and orientation information providing unit 214 of the autonomous mobile body 200 transmits information indicating the position and orientation of the autonomous mobile body 200 (sensor 230 thereof) on the three-dimensional point cloud map, instead of the marker position and orientation information LI, to the server device 300, and the position and orientation server processing unit 314 of the server device 300 generates the marker position and orientation information LI indicating the position and orientation of the marker portion 260 based on the information and transmits it to the terminal device 100. Also, in the above embodiment, the exchange of information such as the marker position and orientation information LI between the autonomous mobile body 200 and the terminal device 100 is executed via the server device 300, but the exchange of information may be directly executed between the autonomous mobile body 200 and the terminal device 100.

[0090] In the above embodiment, the tracking of the changes in the position and orientation of the terminal device 100 after the tracking start point is executed using Visual-SLAM. However, the tracking may be executed using other SLAMs such as LiDAR-SLAM, or may be executed using other known methods such as a method using GNSS.

[0091] In the above-described first embodiment, AR-type information presentation for providing guidance indoors and outdoors in a shopping mall was used as an example for explanation. However, the technology disclosed in this specification is similarly applicable to AR-type information presentation for other purposes (for example, walking navigation, tourist guidance, maintenance inspection guidance, location-based games, etc.).

[0092] In the above-described second embodiment, the autonomous mobile body 200a that has received the terminal position and orientation information TI executes a hazard notification process of notifying the terminal device 100a of the presence of a hazard Ha approaching the terminal device 100a using the estimated results of the position and orientation of the terminal device 100a. However, the autonomous mobile body 200a that has received the terminal position and orientation information TI may execute other processes (for example, precise guidance processing of the mobile body) using the estimated results of the position and orientation of the terminal device 100a. Conversely, when a hazard to the autonomous mobile body 200a is recognized by the imaging unit 154 of the terminal device 100a, the terminal device 100a may notify the autonomous mobile body 200a of the information of the hazard together with the terminal position and orientation information TI. Thereby, the terminal device 100a can supplement the recognition of the hazard by the autonomous mobile body 200a, and the running safety of the autonomous mobile body 200a can be improved.

[0093] In the above embodiment, a part of the configuration realized by hardware may be replaced with software, and conversely, a part of the configuration realized by software may be replaced with hardware.

Explanation of Reference Numerals

[0094] 10: Information Processing System 100: Terminal Device 110: Control Unit 111: Augmented Reality Processing Unit 112: Terminal Position and Orientation Estimation Unit 113: Marker Information Acquisition Unit 114: Image Corresponding Information Acquisition Unit 115: Relative Relationship Identification Unit 116: Tracking Unit 117: Estimation Result Update Unit 118: Estimation Result Transmission Unit 119: Hazard Notification Unit 120: Memory Unit 151: Display Unit 152: Operation Input Unit 153: Voice Output Unit 154: Imaging Unit 155: Communication Unit 190: Bus 200: Autonomous Mobile Body 210: Control Unit 212: Autonomous Movement Control Unit 214: Position and Orientation Information Provision Unit 216: Hazard Notification Unit 220: Memory Unit 230: Sensor 250: Driving Unit 255: Communication Unit 260: Marker Unit 300: Server Device 310: Control Unit 312: Augmented Reality Server Processing Unit 314: Position and Orientation Server Processing Unit 318: Hazard Notification Server Processing Unit 320: Memory Unit 351: Display Unit 352: Operation Input Unit 355: Communication Unit 390: Bus Ha: Hazard RI: Real Image U: User VI: Virtual Image

Claims

1. A terminal device held by a user and moving with the user, an imaging unit, a moving body that moves while specifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is assigned, and having a marker unit that functions as a marker, a marker information acquisition unit that acquires marker position and orientation information indicating the position and orientation of the marker unit on the three-dimensional point cloud map, an image correspondence information acquisition unit that acquires image correspondence information associating an image in which the marker unit is imaged with the relative position and relative orientation of the imaging unit with respect to the marker unit, a relative relationship specifying unit that specifies relative relationship information indicating the relative position and relative orientation of the terminal device with respect to the marker unit based on the image of the marker unit imaged by the imaging unit and the image correspondence information, a terminal position and orientation estimation unit that estimates the position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information, A terminal device comprising:

2. The terminal device according to claim 1, further comprising: a tracking unit that tracks changes in the position and orientation of the terminal device from the time when the position and orientation of the terminal device are estimated, an estimation result update unit that updates the estimation result of the position and orientation of the terminal device based on the change, A terminal device comprising:

3. The terminal device according to claim 2, wherein the tracking unit tracks changes in the position and orientation of the terminal device using an image captured by the imaging unit.

4. The terminal device according to any one of claims 1 to 3, further comprising: a display unit, an augmented reality processing unit that superimposes a virtual image that changes according to the estimation result of the position and orientation of the terminal device on a real image captured by the imaging unit and displays the superimposed image on the display unit, A terminal device comprising:

5. The terminal device according to any one of claims 1 to 4, the moving body, comprising, wherein the terminal device further comprises an estimation result transmission unit that transmits the estimation result of the position and orientation of the terminal device to the moving body directly or via another device, the moving body comprising a specific processing execution unit that executes specific processing using the estimation result of the position and orientation of the terminal device. An information processing system.

6. The information processing system according to claim 5, The information processing system, wherein the specific process includes a hazard notification process for notifying the terminal device of the presence of a hazard in the vicinity of the terminal device.

7. The information processing system according to claim 5 or claim 6, wherein the mobile body is an autonomous mobile body that moves autonomously using the surrounding detection information by sensors and the three-dimensional point cloud map.

8. The information processing system according to any one of claims 5 to 7, further comprising: a server device, wherein the mobile body transmits the marker position and orientation information to the server device, and the marker information acquisition unit of the terminal device acquires the marker position and orientation information via the server device.

9. An information processing method for estimating the position and orientation of a terminal device that has an imaging unit and is held by a user and moves with the user, for a mobile body that moves while specifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is assigned, and that has a marker unit that functions as a marker, a step of acquiring marker position and orientation information indicating the position and orientation of the marker unit on the three-dimensional point cloud map; a step of acquiring image correspondence information for associating an image in which the marker unit is imaged with the relative position and relative orientation of the imaging unit with respect to the marker unit; a step of specifying relative relationship information indicating the relative position and relative orientation of the terminal device with respect to the marker unit based on the image of the marker unit imaged by the imaging unit and the image correspondence information; a step of estimating the position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information. The information processing method comprising the steps.

10. A computer program for estimating the position and orientation of a terminal device that has an imaging unit and is held by a user and moves with the user, for the terminal device, a process of acquiring marker position and orientation information indicating the position and orientation of a marker unit on a three-dimensional point cloud map for a mobile body that moves while specifying its own position and orientation on the three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is assigned, and that has the marker unit that functions as a marker. A process of acquiring image correspondence information that associates an image of the marker portion that has been imaged with the relative position and relative orientation of the imaging unit with respect to the marker portion; A process of specifying relative relationship information indicating the relative position and relative orientation of the terminal device with respect to the marker portion based on the image of the marker portion imaged by the imaging unit and the image correspondence information; A process of estimating the position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information; A computer program that causes the above to be executed.

Images