Positioning system

Abstract

A relative position calculator (32) calculates a relative position and a relative attitude of a vehicle (1), on the basis of an image captured by a capturing device mounted in the vehicle (1). An absolute position calculator (34) extracts one of a plurality of markers (4) from the image captured by the capturing device, and calculates an absolute position and an absolute attitude of the vehicle (1), on the basis of the position and the attitude of the extracted one marker (4). A corrector (36) corrects the relative position and the relative attitude on the basis of the absolute position and the absolute attitude, and calculates a corrected position and a corrected attitude of the vehicle (1). The corrector (36), if the difference or ratio of an apparent height and width of the marker (4) in the image is less than or equal to a threshold value, calculates the corrected position and the corrected attitude without using the absolute position and the absolute attitude that are calculated on the basis of the position and the attitude of the marker (4). Otherwise, the corrector (36) calculates the corrected position and the corrected attitude using the absolute position and the absolute attitude that are calculated on the basis of the position and the attitude of the marker (4).

Description

[Technical field]

[0001] The present disclosure relates to a positioning device that measures the position of a moving body such as a vehicle, a moving body equipped with such a positioning device, and a positioning system including such a moving body. [Background technology]

[0002] When a vehicle for transporting luggage is used to move luggage within a predetermined area or between predetermined points, it may be necessary to measure and track the position of the vehicle. For example, Patent Document 1 discloses a luggage location management device that measures the position of the vehicle using positioning techniques such as GPS, wireless LAN positioning, and infrared positioning.

[0003] When positioning a vehicle moving outdoors, a positioning method using GPS is generally used. On the other hand, indoors, such as in a warehouse or a factory, radio waves from GPS satellites cannot be received, so the positioning method using GPS cannot be used. As an indoor positioning method, for example, there is one using a wireless signal such as UWB (ultra wide band), Wi-Fi, or BLE (Bluetooth (registered trademark) Low Energy). However, the positioning method using a wireless signal requires a large number of wireless transmitters for transmitting wireless signals to be installed within the moving range of the vehicle, so the initial introduction cost is high. In addition, as an indoor positioning method, there is one called PDR (Pedestrian Dead Reckoning). However, it is difficult for PDR to measure the position with high accuracy.

[0004] In order to measure and track the position of a moving object such as a vehicle with high accuracy without requiring a large number of wireless transmitters for transmitting wireless signals, there is a technology called Visual-SLAM (Visual Simultaneous Localization and Mapping), as disclosed in, for example, Non-Patent Document 1. According to Visual-SLAM, a moving object equipped with an image capture device moves while capturing images of its surroundings, and the amount of movement of the moving object is calculated based on the amount of movement of feature points in the multiple captured images. This makes it possible to estimate the current position of the moving object and generate a map based on the trajectory of the moving object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2011-219229 A [Non-patent literature]

[0006] [Non-Patent Document 1] R. Mur-Artal, et al., "ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras", IEEE Transactions on Robotics, Volume: 33, Issue: 5, Oct. 2017 Summary of the Invention [Problem to be solved by the invention]

[0007] In scenes where the background changes daily, such as in factories and warehouses, it is difficult to identify the current position using a map created in advance. In such cases, the position of a moving object obtained by Visual-SLAM is calculated as a relative position to a certain reference position (for example, the position where the moving object started moving), so the error increases cumulatively over time. Therefore, there is a demand for a positioning device that can measure the position of a moving object with smaller error than conventional devices using a camera.

[0008] An object of the present disclosure is to provide a positioning device that uses an imaging device to measure the position of a moving object, and that can measure the position of the moving object with a smaller error than conventional positioning devices. [Means for solving the problem]

[0009] According to one aspect of the present disclosure, a first calculator that calculates a first position and a first orientation of the moving body, the first position and the first orientation indicating a relative position and a relative orientation of the moving body with respect to a predetermined reference position and a reference orientation, based on a plurality of images captured by an image capturing device mounted on the moving body; A storage device that stores information on identifiers, positions, and attitudes of a plurality of visually identifiable markers arranged at predetermined positions, and information on a map including a path for the moving body; a second calculator that extracts one of the plurality of markers from an image captured by the image capture device, and calculates a second position and a second orientation of the moving object indicating the position and orientation of the moving object on the map based on the position and orientation of the one extracted marker; a corrector that corrects the first position and the first attitude based on the second position and the second attitude and calculates a corrected position and a corrected attitude of the moving body, The corrector calculates the corrected position and the corrected orientation by not using the second position and the second orientation calculated based on the position and orientation of the marker when a difference or ratio between the apparent height and width of the marker in the image is less than or equal to a first threshold, and by using the second position and the second orientation calculated based on the position and orientation of the marker when a difference or ratio between the apparent height and width of the marker in the image is greater than the first threshold.

[0010] These general and specific aspects may be realized by a system, a method, a computer program, or any combination of a system, a method, and a computer program. Effect of the Invention

[0011] According to one aspect of the present disclosure, the position and orientation of the moving body can be measured with high accuracy by correcting the first position and the first orientation based on the second position and the second orientation. Also, according to one aspect of the present disclosure, it is determined whether or not a recognized marker can be trusted, and only when the marker can be trusted, the position and orientation of the moving body are corrected based on the second position and the second orientation, thereby making it possible to measure the position and orientation of the moving body with high accuracy. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram showing a configuration of a vehicle 1 according to a first embodiment. [Diagram 2] 2 is a block diagram showing a configuration of a positioning system including the vehicle 1 of FIG. 1. [Diagram 3] 3 is a block diagram showing a configuration of a positioning device 12 in FIG. 2. [Figure 4] 2 is a map of a warehouse 100 including an aisle 101 along which the vehicle 1 of FIG. 1 travels. [Diagram 5] 5A and 5B are diagrams showing an example of the marker 4 in FIG. 4, in which (a) shows a front view of the marker 4 and (b) shows a top view of the marker 4. [Figure 6]4 is a table showing an example of marker information stored in the storage device 35 of FIG. 3. [Figure 7] 2 is a diagram showing an example of an image 40 captured by the image capturing device 11 of FIG. 1. FIG. [Figure 8] 3 is a flowchart showing a positioning process executed by the positioning device 12 of FIG. 2. [Figure 9] 9 is a flowchart showing a subroutine of step S2 (relative position calculation process) in FIG. 8. [Figure 10] FIG. 4 shows feature points extracted by image processor 31 in FIG. 3, where (a) shows feature points F1 and F2 extracted from image 40(n) at time n, and (b) shows feature points F1' and F2' extracted from image 40(n') at time n'. [Figure 11] 9 is a flowchart showing a subroutine of step S3 (absolute position calculation process) in FIG. 8. [Figure 12] 13 is a diagram showing the coordinates of the vertices of a marker 4 in the marker coordinate system. FIG. [Figure 13] 1. FIG. 4 is a diagram showing the coordinates of vertices of a marker 4 in an image 40A captured by the imaging device 11 of FIG. [Figure 14] 9 is a flowchart showing a subroutine of step S4 (correction process) in FIG. 8. [Figure 15] 15 is a diagram for explaining a correction movement amount calculated in step S31 of FIG. 14. FIG. [Figure 16] 15 is a diagram for explaining a correction rotation amount calculated in step S33 of FIG. 14. [Figure 17] 12 is a flowchart showing a subroutine of step S24 (marker evaluation process) of FIG. 11. [Figure 18] 2 is a diagram showing an exemplary arrangement of markers 4a-1 to 4a-12 photographed by the photographing device 11 of FIG. [Figure 19] 19 is a diagram showing an example of an image 40B obtained by photographing the markers 4a-1 to 4a-12 in FIG. 18 using the photographing device 11. FIG. [Figure 20]2 is a diagram showing an exemplary arrangement of markers 4b-1 to 4b-12 photographed by the photographing device 11 of FIG. [Figure 21] 21 is a diagram showing an example of an image 40C obtained by photographing the markers 4b-1 to 4b-12 in FIG. 20 by the photographing device 11. FIG. [Figure 22] 1. FIG. 2 is a diagram for explaining the fluctuation in the position of the photographing device 11 in the marker coordinate system caused by erroneous detection of the angle of the surface of the marker 4 relative to the optical axis of the photographing device 11 in FIG. [Diagram 23] 18 is a flowchart showing a subroutine of step S41 in FIG. 17. [Figure 24] 24 is a diagram for explaining the apparent height of the marker 4 in the image, relating to steps S51 and S52 in FIG. 23. FIG. [Diagram 25] FIG. 24 is a diagram for explaining the apparent width of the marker 4 in the image, relating to steps S51 and S52 in FIG. 23. [Figure 26] 24 is a diagram for explaining the apparent angle difference between adjacent corners of the markers 4 in the image, relating to step S53 in FIG. 23. FIG. [Figure 27] 1. FIG. 4 is a diagram showing an example of an image 40D including a marker 4a having a surface perpendicular to the optical axis of the image capturing device 11 of FIG. [Figure 28] 1. FIG. 4 is a diagram showing an example of an image 40E including a marker 4b having a surface parallel to the optical axis of the photographing device 11 of FIG. 1, where the difference between the height and width of the marker 4b in the image 40E is one pixel. [Figure 29] 2 is a diagram showing an example of an image 40F including a marker 4a having a surface perpendicular to the optical axis of the image capturing device 11 of FIG. 1. FIG. [Diagram 30] 1. FIG. 4 is a diagram showing an example of an image 40G including a marker 4b having a surface parallel to the optical axis of the photographing device 11 of FIG. 1, where the difference between the height and width of the marker 4b in the image 40G is 4 pixels. [Diagram 31] 1. FIG. 4 is a diagram showing an example of an image 40H including a marker 4a having a surface perpendicular to the optical axis of the image capturing device 11 of FIG. [Diagram 32]This is a figure showing an example of an image 40I including a marker 4b having a surface parallel to the optical axis of the photographing device 11 of Figure 1, where the angular difference between adjacent corners of the marker 4b in the image 40I is 35 degrees. [Diagram 33] 1. FIG. 4 is a diagram showing an example of an image 40J captured by the image capturing device 11 of FIG. [Diagram 34] 11 is a table including markers 4 detected by the image recognizer 33 of FIG. 3 and objects erroneously detected as markers 4. [Diagram 35] 1. FIG. 4 is a diagram showing an example of an image 40K including a marker 4 that is captured by the image capture device 11 of FIG. 1 and is suitable for calculating the absolute position and absolute attitude of a vehicle 1. [Diagram 36] 1. FIG. 4 is a diagram showing an example of an image 40L that is captured by the image capture device 11 of FIG. 1 and includes a marker 4 that is not suitable for calculating the absolute position and absolute attitude of the vehicle 1. [Figure 37] 1 passes substantially through the center of the surface of the marker 4 and is substantially perpendicular to the surface of the marker 4. FIG. [Figure 38] 4 is a table showing changes in the position of the image capturing device 11 in the marker coordinate system calculated by the position calculator 34 of FIG. 3. [Figure 39] 2 is a diagram showing an example of an image of a marker 4 captured by the image capturing device 11 of FIG. 1 and having a width suitable for calculating the absolute position and absolute attitude of a vehicle 1. FIG. [Diagram 40] 2 is a diagram showing an example of an image of a marker 4 captured by the image capturing device 11 of FIG. 1 and not having a width suitable for calculating the absolute position and absolute attitude of a vehicle 1. FIG. [Diagram 41] 18 is a graph showing an error in the absolute position of the vehicle 1 when the threshold value in step S44 of FIG. 17 is changed. [Diagram 42] 1 is a diagram for explaining the angle and distance at which an image of a marker 4 suitable for calculating the absolute position and absolute attitude of a vehicle 1 can be captured in a marker coordinate system. [Diagram 43]1 is a diagram for explaining the angle conditions when capturing an image of a marker 4 suitable for calculating the absolute position and absolute attitude of a vehicle 1. FIG. [Diagram 44] 44 is a diagram showing an example of an image 40M including a marker 4 captured by the imaging device 11 of FIG. 43. FIG. [Diagram 45] 10 is a diagram showing a trajectory 103 of a vehicle 1 calculated by executing a correction process according to a comparative example of the first embodiment. FIG. [Figure 46] FIG. 18 is a diagram showing a trajectory 104 of a vehicle 1 calculated by executing the marker evaluation process of FIG. [Figure 47] 3 is a diagram showing a first example of an image displayed on the display device 14 or 25 of FIG. 2. FIG. [Figure 48] 3 is a diagram showing a second example of an image displayed on the display device 14 or 25 of FIG. 2. FIG. [Figure 49] 2. FIG. 4 is a diagram showing a third example of an image displayed on the display device 14 or 25 of FIG. [Figure 50] FIG. 11 is a block diagram showing a configuration of a positioning device 12A according to a second embodiment. [Figure 51] FIG. 11 is a block diagram showing a configuration of a positioning device 12B according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following embodiments, similar components are denoted by the same reference numerals.

[0014] [First embodiment] First, a positioning device according to a first embodiment and a moving object equipped with the same will be described.

[0015] [Configuration of the first embodiment] [Overall configuration] FIG. 1 is a schematic diagram showing a configuration of a vehicle 1 according to a first embodiment. The vehicle 1 may be, for example, a forklift. The vehicle 1 includes a loading platform 1a on which luggage 3 is mounted. The vehicle 1 may further include a lifting mechanism 1b for loading and unloading the luggage 3 onto and from the loading platform 1a. The vehicle 1 also includes a console 1c that receives user operations such as forward movement, reverse movement, steering, and stopping. An imaging device 11 is installed on the body of the vehicle 1 so as to capture images of a predetermined direction (forward, backward, sideways, upward, and / or downward) relative to the vehicle 1.

[0016] Fig. 2 is a block diagram showing a configuration of a positioning system including the vehicle 1 of Fig. 1. The positioning system of Fig. 2 includes at least one vehicle 1 and a server device 2. Each vehicle 1 is equipped with a positioning device 12 that measures its position based on an image captured by an imaging device 11. The server device 2 acquires the positions of each vehicle 1 from the vehicles 1 and records the positions of each vehicle 1.

[0017] [Vehicle 1 Configuration] The vehicle 1 further includes an imaging device 11 , a positioning device 12 , a communication device 13 , a display device 14 , and a drive mechanism 15 .

[0018] The image capturing device 11 generates images of a subject in a predetermined orientation with respect to the vehicle 1 at predetermined time intervals while the vehicle 1 is moving. The image capturing device 11 includes, for example, at least one camera. The image capturing device 11 may capture still images at predetermined time intervals, or may extract frames at predetermined time intervals from a series of frames of a moving image. The image capturing device 11 sends the captured images to the positioning device 12. The image capturing device 11 assigns to each image a timestamp indicating the time when the image was captured.

[0019] The positioning device 12 calculates the position and attitude of the vehicle 1 based on the images captured by the image capture device 11. The positioning device 12 extracts feature points from the multiple images captured by the image capture device 11, associates the extracted feature points between the images, and calculates the relative position and relative attitude of the vehicle 1 with respect to a predetermined reference position and reference attitude based on the amount of change in the feature points between the images. The positioning device 12 also extracts one of multiple markers that are arranged at predetermined positions and are visually identifiable from the images captured by the image capture device 11, and calculates the absolute position and absolute attitude of the vehicle 1 in a pre-given map based on the one extracted marker. The positioning device 12 further corrects the relative position and relative attitude based on the absolute position and absolute attitude.

[0020] In this specification, the "attitude" of the vehicle 1 indicates, for example, the angle of the traveling direction of the vehicle 1 with respect to the coordinate axes of a predetermined coordinate system (the "world coordinate system" or "marker coordinate system" described later).

[0021] The communication device 13 includes a module such as Wi-Fi or Bluetooth and a control program for the module, and wirelessly communicates with the server device 2. The communication device 13 transmits the position and attitude of the vehicle 1 calculated by the positioning device 12 to the server device 2.

[0022] The display device 14 displays an image showing the position and attitude of the vehicle 1. The display device 14 may display the position of the vehicle 1 by superimposing it on a map. The display device 14 may also display an image captured by the imaging device 11. The display device 14 may also display information related to the operation of the vehicle 1 (e.g., an alarm, etc.).

[0023] The drive mechanism 15 includes an engine or a motor, a steering device, a braking device, and devices for controlling these devices of the vehicle 1. The drive mechanism 15 is controlled by a user via, for example, the console 1c.

[0024] [Server device 2 configuration] The server device 2 in FIG. 2 includes a processing device 21, a communication device 22, an input device 23, a storage device 24, and a display device 25. The processing device 21 is, for example, a general-purpose computer including a processor and a memory. The communication device 22 is communicatively connected to the communication device 13 of the vehicle 1. The input device 23 includes a keyboard, a pointing device, and the like. The storage device 24 records the position and attitude of the vehicle 1 received from the vehicle 1. The display device 25 displays the position and attitude of the vehicle 1 received from the vehicle 1. The processing device 21 acquires the positions and attitudes of the vehicles 1 from the vehicles 1 via the communication device 22, records the positions and attitudes of the vehicles 1 in the storage device 24, and displays the positions and attitudes of the vehicles 1 on the display device 25.

[0025] The display device 25 displays the position and attitude of the vehicle 1 calculated by the positioning device 12 of the vehicle 1. The processing device 21 may acquire a map of the movement range of the vehicle 1 (such as a warehouse or factory) in advance, and display the position and attitude of the vehicle 1 calculated by the positioning device 12 on the map by superimposing the map. Alternatively, the processing device 21 may generate a map by itself based on the movement route of the vehicle 1, and display the map on the display device 25. The display device 25 may also display an image captured by the imaging device 11 of the vehicle 1.

[0026] In this specification, the display device 14 is also referred to as a "first display device," and the display device 25 is also referred to as a "second display device."

[0027] [Configuration of positioning device 12] Fig. 3 is a block diagram showing the configuration of the positioning device 12 of Fig. 2. The positioning device 12 includes an image processor 31, a relative position calculator 32, an image recognizer 33, an absolute position calculator , a storage device 35, and a corrector .

[0028] The storage device 35 stores information on the identifiers, positions, and attitudes of a plurality of visually identifiable markers 4 arranged at predetermined positions, and information on a map (e.g., a map of a warehouse 100 described with reference to FIG. 4) including paths for the vehicle 1. The positions of the markers 4 may be represented as relative positions to a predetermined reference position and / or may be represented in association with the map.

[0029] 4 is a map of a warehouse 100 including an aisle 101 along which the vehicle 1 of FIG. 1 moves. The warehouse 100 includes structures such as a plurality of aisles 101 and a plurality of shelves 102. A plurality of markers 4 are pre-arranged at a plurality of predetermined positions in the warehouse 100. The vehicle 1 of FIG. 1 moves along the aisle 101 to transport luggage 3 from one shelf 102 to another shelf 102. The positions of the vehicle 1 and each marker 4 are represented by a world coordinate system (Xw, Yw, Zw) determined with respect to the entire warehouse 100.

[0030] FIG. 5 is a diagram showing an example of the marker 4 in FIG. 4, where (a) shows a front view of the marker 4 and (b) shows a top view of the marker 4. In the example of FIG. 5, the marker 4 is formed as a square flat plate. The marker 4 has a visually identifiable pattern that encodes the identifier of the marker 4 itself on one surface. In the example of FIG. 5, the marker 4 has a pattern consisting of 7×7 white or black square cells in a vertical and horizontal direction. The pattern of the marker 4 is further formed so that the attitude of the marker 4 itself can be detected from an image of the marker 4, such as a marker used in the field of augmented reality (also called an "AR marker"). Each marker 4 has a marker coordinate system (Xm, Ym, Zm) with an arbitrary point (for example, a center or one vertex) as the origin. In the lower part of FIG. 5 and other drawings, the front of the marker 4 (positive direction of the Zm axis) is indicated by an arrow at the center of the surface along the Xm-Ym plane.

[0031] FIG. 6 is a table showing an example of marker information stored in the storage device 35 of FIG. 3. In the example of FIG. 6, information of two markers 4 shown in FIG. 4 is shown. Each marker 4 has an identifier 001, 002. This identifier is encoded in the pattern of the marker 4. Each marker 4 has a predetermined coordinate in the world coordinate system (Xw, Yw, Zw). Each marker 4 is arranged in an orientation such that its front (positive direction of the Zm axis) has an angle θ (i.e., azimuth angle) with respect to the Xw axis in the Xw-Yw plane. The orientation of each marker 4 may be represented by an azimuth angle and an elevation angle. Each marker 4 has a square-shaped pattern and an actual size of 30 cm×30 cm. The size of the marker 4 indicates the size of the area of ​​the pattern and does not include margins around the pattern (not shown in FIG. 5).

[0032] The storage device 35 stores marker information including items such as those shown in Fig. 6 for all markers 4. The storage device 35 also stores map information for all passages 101 including their orientations, dimensions, and locations.

[0033] FIG. 7 is a diagram showing an example of an image 40 captured by the image capture device 11 of FIG. 1. The image 40 includes a plurality of feature points 41. The feature points 41 are points whose brightness values ​​or colors can be distinguished from surrounding pixels, and whose positions can be accurately determined. The feature points 41 are detected, for example, from vertices or edges of structures such as the passage 101 or shelves 102 along which the vehicle 1 moves, or patterns on the floor, wall, or ceiling. Furthermore, when the vehicle 1 passes near the marker 4, the image 40 includes the marker 4. The positions of the feature points 41 and the marker 4 in the image 40 are represented, for example, by an image coordinate system (Xi, Yi) with an arbitrary point (for example, the upper left corner) of the image 40 as the origin.

[0034] Referring again to FIG. 3, the image processor 31 extracts the coordinates of corresponding feature points from a plurality of images captured by the image capture device 11 at a plurality of times separated by a predetermined time. The relative position calculator 32 calculates the movement amount of the vehicle 1 based on the movement amount of the feature points in two temporally adjacent images. As a result, the relative position calculator 32 calculates the relative position and relative orientation of the vehicle 1 with respect to a predetermined reference position and reference orientation (for example, the position and orientation when the vehicle 1 starts moving) based on the coordinates of the feature points of the plurality of images. The relative position calculator 32 may calculate the relative position and relative orientation of the vehicle 1 using a known image processing and positioning technique such as Visual-SLAM or Visual-Odometry. The reference position and reference orientation are associated with map information stored in the storage device 35. The relative position calculator 32 also assigns the timestamp of the image (the latter of the two temporally adjacent images) associated with the calculation to the relative position and relative orientation.

[0035] The relative position calculator 32 may express the calculated position of the vehicle 1, for example, in Cartesian coordinates (XYZ coordinates). The relative position calculator 32 may calculate the speed and / or acceleration of the vehicle 1 based on the calculated position of the vehicle 1 and the time. The relative position calculator 32 may express the calculated attitude of the vehicle 1 in terms of roll (left / right tilt), pitch (front / back tilt), and yaw (rotation around an axis perpendicular to the floor surface (i.e., the Zw axis in FIG. 4)). This makes it possible to express not only the orientation of the vehicle 1 in a horizontal plane parallel to the ground, but also the tilt of the body of the vehicle 1 and the movement of the vehicle 1 in the height direction.

[0036] In this specification, the image processor 31 and the relative position calculator 32 are collectively referred to as a "first calculator." Furthermore, in this specification, the relative position and the relative orientation are also referred to as a "first position" and a "first orientation."

[0037] The image recognizer 33 extracts one of a plurality of visually identifiable markers 4 arranged at predetermined positions from the image captured by the image capture device 11. The absolute position calculator 34 calculates the absolute position and absolute attitude of the vehicle 1 indicating the position and attitude of the vehicle 1 in the map (i.e., the world coordinate system) based on the position and attitude of the extracted one marker 4, by referring to the information of the marker 4 and the map information stored in the storage device 35. The absolute position calculator 34 also assigns a timestamp of the image associated with these calculations to the absolute position and absolute attitude.

[0038] In this specification, the image recognizer 33 and the absolute position calculator 34 are collectively referred to as a "second calculator." Furthermore, in this specification, the absolute position and the absolute orientation are also referred to as a "second position" and a "second orientation."

[0039] The corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, and calculates a corrected position and corrected attitude of the vehicle 1. The corrector 36 synchronizes the absolute position and absolute attitude with the relative position and relative attitude based on the timestamps of the relative position and relative attitude and the timestamps of the absolute position and absolute attitude. The corrector 36 may, for example, consider the relative position and relative attitude and the absolute position and absolute attitude having the closest timestamps, which have a time difference smaller than a predetermined threshold, as being calculated from the same image.

[0040] At least some of the components 31 to 36 of the positioning device 12 may be integrated. For example, the image processor 31 and the image recognizer 33 may be integrated. Furthermore, the components 31 to 36 of the positioning device 12 may be implemented as dedicated circuits or as programs executed by a general-purpose processor.

[0041] [Operation of the first embodiment] As described above, the corrector 36 corrects the relative position and relative orientation calculated based on the feature points 41 using Visual-SLAM or the like, based on the absolute position and absolute orientation calculated based on the marker 4. This correction process is premised on the assumption that the absolute position and absolute orientation of the vehicle 1 are correct, and therefore, that the orientation of the marker 4 is correctly recognized from an image capturing the marker 4. However, an object other than the marker 4 may be erroneously detected as the marker 4. In addition, in order to correctly recognize the orientation of the marker 4, it is necessary to capture the marker 4 from an appropriate angle and an appropriate distance. If there is an error in the position and orientation of the marker 4 itself recognized from an image capturing the marker 4, an error also occurs in the absolute position and absolute orientation of the vehicle 1, and the accuracy of the position and / or orientation of the vehicle 1 calculated by the positioning device 12 may be reduced.

[0042] For this reason, in the positioning device 12 according to the first embodiment, the absolute position calculator 34 judges whether the marker 4 has been photographed from an appropriate angle and an appropriate distance, i.e., whether the recognized marker 4 is reliable, and sends the absolute position and absolute attitude of the vehicle 1 calculated based on the position and attitude of the marker 4 to the corrector 36 only when the marker 4 is reliable. As a result, the corrector 36 corrects the position and attitude of the vehicle 1 based on the absolute position and absolute attitude only when the marker 4 is reliable. This allows the position and attitude of the vehicle 1 to be correctly corrected based on the position and attitude of the correctly recognized marker 4, and therefore the position and attitude of the vehicle 1 to be measured with high accuracy.

[0043] The operation of the positioning device 12 will now be described in detail.

[0044] [Overall positioning process] FIG. 8 is a flowchart showing the positioning process executed by the positioning device 12 of FIG.

[0045] In step S1, the positioning device 12 acquires an image captured by the imaging device 11. In step S2, the image processor 31 and the relative position calculator 32 execute a relative position calculation process to calculate the relative position and the relative attitude of the vehicle 1. In step S3, the image recognizer 33 and the absolute position calculator 34 execute an absolute position calculation process to calculate the absolute position and the absolute attitude of the vehicle 1. Steps S2 and S3 may be executed in parallel as shown in FIG. 8 or may be executed sequentially. In step S4, the corrector 36 executes a correction process to correct the relative position and the relative attitude based on the absolute position and the absolute attitude, and calculates the corrected position and the corrected attitude of the vehicle 1. In step S5, the corrector 36 outputs the corrected position and the corrected attitude of the vehicle 1 to the communication device 13 and the display device 14.

[0046] [Relative position calculation process] FIG. 9 is a flowchart showing a subroutine of step S2 (relative position calculation process) in FIG.

[0047] In step S11, the image processor 31 acquires first and second images taken at first and second times, respectively, that are separated by a predetermined time (eg, frames adjacent in time).

[0048] In step S12, the image processor 31 detects feature points from the first image. In order to detect feature points from the image, for example, an image processing technique such as FAST (Features from Accelerated Segment Test) may be used.

[0049] In step S13, the image processor 31 detects feature points from the second image that correspond to the feature points from the first image. In order to detect corresponding feature points between images, a known image processing technique such as a KLT (Kanade-Lucas-Tomasi) tracker may be used.

[0050] FIG. 10 is a diagram showing feature points extracted by the image processor 31 of FIG. 3. FIG. 10(a) shows feature points F1 and F2 extracted from an image 40(n) at time n. FIG. 10(b) shows feature points F1' and F2' extracted from an image 40(n') at time n'. In the image coordinate system of the image 40(n) of FIG. 10(a), the feature point F1 has coordinates (xi1, yi1), and the feature point F2 has coordinates (xi2, yi2). In the image coordinate system of the image 40(n') of FIG. 10(b), the feature point F1' has coordinates (xi1', yi1'), and the feature point F2' has coordinates (xi2', yi2'). The feature points F1' and F2' of FIG. 10(b) correspond to the feature points F1 and F2 of FIG. 10(a), respectively.

[0051] 9, the image processor 31 acquires a set of coordinates of corresponding feature points in the first and second images. For example, the image processor 31 acquires a set of coordinates (xi1, yi1, xi1', yi1') of feature points F1 and F1', and acquires a set of coordinates (xi2, yi2, xi2', yi2') of feature points F2 and F2'.

[0052] In step S15, the relative position calculator 32 calculates a fundamental matrix E consisting of 3×3 elements, for example, using a five-point algorithm, based on the coordinates of the feature points acquired in step S14.

[0053] In step S16, the relative position calculator 32 performs singular value decomposition on the fundamental matrix E to calculate a rotation matrix R and a translation vector t that represent the movement of the vehicle 1 between the times when the first and second images were captured. The rotation matrix R indicates the change in attitude of the vehicle 1 between the times when the first and second images were captured. The translation vector t indicates the change in position of the vehicle 1 between the times when the first and second images were captured.

[0054] The calculation of the rotation matrix R and the translation vector t is formulated, for example, as follows:

[0055] The fundamental matrix E is obtained by singular value decomposition as E=UΣV THere, Σ is a diagonal matrix Σ consisting of 3 × 3 elements, and U and V are orthogonal matrices consisting of 3 × 3 elements.

[0056] The rotation matrix R is calculated by using the following matrix W, which consists of 3 × 3 elements: R = UW -1 V T It is calculated as follows:

[0057]

number

[0058] In addition, to calculate the translation vector t, we use a 3×3 matrix T=VWΣV T The matrix T satisfies E=TR and is expressed by the following equation.

[0059]

number

[0060] The translation vector t is expressed as t=(t x ,t y ,t z ) T It is expressed as:

[0061] In step S17, the relative position calculator 32 calculates the relative position and relative attitude of the vehicle 1. When the vehicle 1 has a relative position t(n-1) and a relative attitude R(n-1) at the immediately preceding time n-1, the relative position t(n) of the vehicle 1 at the current time n is expressed by t(n)=t(n-1)+tR(n-1) using the translation vector t calculated in step S16. Also, at the current time n, the relative attitude R(n) of the vehicle 1 is expressed by R(n)=RR(n-1) using the rotation matrix R calculated in step S16. In this way, the relative position calculator 32 cumulatively adds a plurality of translation vectors and cumulatively multiplies a plurality of rotation matrices to calculate the relative position and relative attitude of the vehicle 1 with respect to a predetermined reference position and reference attitude. The relative position calculator 32 sends the calculated relative position and relative attitude of the vehicle 1 to the corrector.

[0062] [Absolute position calculation process] FIG. 11 is a flowchart showing a subroutine of step S3 (absolute position calculation process) in FIG.

[0063] In step S21, the image recognizer 33 detects the marker 4 from the image. Here, the image recognizer 33 detects the coordinates of the four vertices (corners) of the quadrangular marker 4 in the image coordinate system, and also decodes the pattern of the marker 4 to obtain the identifier of the marker 4. Note that the image recognizer 33 may detect the coordinates of several predetermined points instead of the four vertices of the marker 4.

[0064] 12 is a diagram showing the coordinates of the vertices of the marker 4 in the marker coordinate system. In the marker coordinate system (Xm, Ym, Zm), the four vertices of the marker 4 have coordinates (xm0, ym0, zm0), (xm1, ym1, zm1), (xm2, ym2, zm2), and (xm3, ym3, zm3), respectively. Since the dimensions of the marker 4 are known, the coordinates of the four vertices of the marker 4 in the marker coordinate system are also known. For example, if the top left vertex of marker 4 in Figure 12 is the origin of the marker coordinate system (Xm, Ym, Zm) and marker 4 has an actual size of 30 cm x 30 cm, the vertices of marker 4 have, for example, coordinates (xm0, ym0, zm0) = (0, 0, 0), (xm1, ym1, zm1) = (0.3, 0, 0), (xm2, ym2, zm2) = (0, 0.3, 0), and (xm3, ym3, zm3) = (0.3, 0.3, 0).

[0065] Fig. 13 is a diagram showing the coordinates of the vertices of the marker 4 in the image 40A captured by the imaging device 11 in Fig. 1. In the image coordinate system (Xi, Yi), the four vertices of the marker 4 have coordinates (xi0, yi0), (xi1, yi1), (xi2, yi2), and (xi3, yi3), respectively.

[0066] 11, the absolute position calculator 34 calculates the position and orientation of the marker 4 in a three-dimensional coordinate system (camera coordinate system) with the image capturing device 11 as the origin (i.e., the position and orientation of the marker 4 as seen from the image capturing device 11) based on the coordinates of the marker 4 detected in step S21. For example, the absolute position calculator 34 calculates the position and orientation of the marker 4 as seen from the image capturing device 11 by solving a perspective n point (PnP) problem based on the coordinates of the four vertices of the marker 4 in the two-dimensional image coordinate system and the coordinates of the four vertices of the marker 4 in the three-dimensional marker coordinate system.

[0067] In step S23, the absolute position calculator 34 calculates the position and orientation of the camera 11 in the marker coordinate system (i.e., the position and orientation of the camera 11 as seen from the marker 4). Here, the position of the marker 4 as seen from the camera 11 is represented by a translation vector t, and the orientation of the marker 4 as seen from the camera 11 is represented by a rotation matrix R. In this case, the orientation of the camera 11 as seen from the marker 4 is expressed as R -1 The position of the camera 11 as viewed from the marker 4 is represented by -R -1 It is represented by t.

[0068] In step S24, the absolute position calculator 34 executes a marker evaluation process (described later with reference to FIGS. 17 to 46) to determine whether the marker 4 has been photographed from an appropriate angle and an appropriate distance, that is, whether the recognized marker 4 can be trusted. If it is determined that the recognized marker 4 can be trusted, the process proceeds from step S25 to step S26; if not, the process proceeds to step S4 in FIG.

[0069] In step S26, the absolute position calculator 34 reads out the position and orientation of the marker 4 in the world coordinate system (ie, the absolute position and absolute orientation of the marker 4) from the storage device 35 based on the identifier of the marker 4 detected in step S21.

[0070] In step S27, the absolute position calculator 34 calculates the position and attitude of the vehicle 1 in the world coordinate system (i.e., the absolute position and attitude of the vehicle 1) based on the position and attitude of the image capturing device 11 in the marker coordinate system calculated in step S23 and the position and attitude of the marker 4 in the world coordinate system read out in step S26. The position and attitude of the vehicle 1 in the world coordinate system are obtained by adding the position and attitude of the marker 4 in the world coordinate system as an offset value to the position and attitude of the image capturing device 11 in the marker coordinate system. The absolute position calculator 34 sends the calculated absolute position and attitude of the vehicle 1 to the corrector 36.

[0071] [Correction process] FIG. 14 is a flowchart showing a subroutine of step S4 (correction process) in FIG.

[0072] When performing the correction process of FIG. 14, the corrector 36 initially holds therein the latest absolute position and absolute attitude previously calculated.

[0073] In step S31, the corrector 36 calculates a corrected movement amount of the vehicle 1 based on the relative position and absolute position of the vehicle 1.

[0074] FIG. 15 is a diagram for explaining the correction movement amount calculated in step S31 of FIG. 14. FIG. 15 shows the position of the vehicle 1 calculated by the positioning device 12. At times n-2 to n+2, the vehicle 1 has relative positions P(n-2) to P(n+2). When the vehicle 1 arrives at the relative position P(n), the positioning device 12 detects the marker 4. At this time, even if the relative position P(n) is far from the marker 4 on the map, the vehicle 1 is considered to be actually located near the marker 4. Therefore, the positioning device 12 calculates the difference between the relative position P(n) and the position of the marker 4 as the correction movement amount τ. The positioning device 12 adds the correction movement amount τ to the relative position P(n) to obtain a corrected position P'(n). Thereafter, the positioning device 12 similarly adds the correction movement amount τ to the relative positions P(n+1), P(n+2), . . . to obtain corrected positions P'(n+1), P'(n+2), .

[0075] In step S32 of FIG. 14, the corrector 36 adds the correction amount of the vehicle 1 to the relative position of the vehicle 1.

[0076] In step S33, the corrector 36 calculates a correction rotation amount of the vehicle 1 based on the relative attitude and absolute attitude of the vehicle 1.

[0077] FIG. 16 is a diagram for explaining the correction rotation amount calculated in step S33 of FIG. 14. FIG. 16 shows the positions P, P' and attitudes A, A' of the vehicle 1 calculated by the positioning device 12. At times n-2 to n+2, the vehicle 1 has relative positions P(n-2) to P(n+2) and relative attitudes A(n-2) to A(n+2). In FIG. 16, the thick arrow indicates the attitude of the vehicle 1. When the vehicle 1 arrives at the relative position P(n), the positioning device 12 detects the marker 4 and calculates the absolute attitude of the vehicle 1. The positioning device 12 calculates the difference between the relative attitude A(n) and the absolute attitude as the correction rotation amount ρ, with the relative position P(n) as the center of rotation. The positioning device 12 adds the correction rotation amount ρ to the relative attitude A(n) to obtain a corrected attitude A'(n). Thereafter, the positioning device 12 similarly adds the correction rotation amount ρ to the relative attitudes A(n+1), A(n+2), . . . to obtain corrected attitudes A'(n+1), A'(n+2), .

[0078] In step S33, the corrector 36 may calculate the difference between the absolute attitude and the average value of multiple relative attitudes calculated over a predetermined time period (or corresponding to a predetermined number of consecutive images) as the corrected rotation amount of the vehicle 1. The actual traveling direction of the vehicle 1 (e.g., whether the vehicle 1 is moving along the passage 101 or not) cannot be determined only from the instantaneous value of the relative attitude of the vehicle 1. In addition, the relative attitude of the vehicle 1 may contain an error. Therefore, by using the average value of multiple relative attitudes, the actual traveling direction of the vehicle 1 can be accurately determined.

[0079] In step S34 of FIG. 14, the corrector 36 adds the correction rotation amount of the vehicle 1 to the relative attitude of the vehicle 1.

[0080] [Marker evaluation process] FIG. 17 is a flowchart showing a subroutine of step S24 (marker evaluation process) in FIG.

[0081] [Step S41] In step S41 of FIG. 17, the absolute position calculator 34 determines whether or not the surface of the marker 4 appears perpendicular to the optical axis of the photographing device 11 in the image captured by the photographing device 11; if the result is YES, the process proceeds to step S48; if the result is NO, the process proceeds to step S42.

[0082] Depending on the position of the marker 4 in the image captured by the image capture device 11, it may be difficult to distinguish between markers 4 having surfaces at different angles to the optical axis of the image capture device 11. For example, at the end of the image captured by the image capture device 11, it is difficult to distinguish between markers 4 having surfaces perpendicular to the optical axis of the image capture device 11 and markers 4 having surfaces parallel to the optical axis of the image capture device 11.

[0083] FIG. 18 is a diagram showing an exemplary arrangement of the markers 4a-1 to 4a-12 photographed by the photographing device 11 of FIG. 1. Each of the markers 4a-1 to 4a-12 has a square-shaped pattern and an actual size of 0.6 m×0.6 m (not including the margins around the pattern). In addition, each of the markers 4a-1 to 4a-12 is arranged with their faces perpendicular to the optical axis of the photographing device 11, that is, parallel to the Xw-Zw plane. FIG. 19 is a diagram showing an example of an image 40B of the markers 4a-1 to 4a-12 of FIG. 18 photographed by the photographing device 11.

[0084] FIG. 20 is a diagram showing an exemplary arrangement of the markers 4b-1 to 4b-12 photographed by the photographing device 11 of FIG. 1. Each of the markers 4b-1 to 4b-12 has a square-shaped pattern and an actual size (not including the margin around the pattern) of 0.6 m×0.6 m. Also, each of the markers 4b-1 to 4b-12 is arranged so that their faces are parallel to the optical axis of the photographing device 11, that is, parallel to the Yw-Zw plane. FIG. 21 is a diagram showing an example of an image 40C of the markers 4b-1 to 4b-12 of FIG. 20 photographed by the photographing device 11. Comparing FIG. 21 with FIG. 19, it can be seen that the marker 4b-6 of FIG. 21 appears to be almost the same size and shape in the image 40C as each of the markers 4a-1 to 4a-12 of FIG. 19, even though the orientation of the face is different by 90 degrees.

[0085] FIG. 22 is a diagram for explaining the fluctuation of the position of the photographing device 11 in the marker coordinate system caused by erroneous detection of the angle of the surface of the marker 4 with respect to the optical axis of the photographing device 11 in FIG. 1. As described above, the markers 4a-1 to 4a-12 in FIG. 19 and the marker 4b-6 in FIG. 21 cannot be distinguished from each other in appearance. Therefore, the photographing device 11 may be erroneously determined to have the position and attitude of "11b" in the marker coordinate system of FIG. 22 when it should have the position and attitude of "11a", and vice versa. As will be described later with reference to FIGS. 35 to 38, when the optical axis of the photographing device 11 passes substantially through the center of the surface of the marker 4 and is substantially perpendicular to the surface of the marker 4, it is difficult to determine the position of the photographing device 11 in the marker coordinate system with the marker 4 as the origin, and the marker 4 is not suitable for calculating the absolute position and absolute attitude of the vehicle 1. 17, the absolute position calculator 34 treats the markers 4 that appear to have a surface perpendicular to the optical axis of the camera 11 as markers not to be used in the calculation of the absolute position and attitude of the vehicle 1. Here, the markers 4 that appear to have a surface perpendicular to the optical axis of the camera 11 include markers 4 that actually have a surface perpendicular to the optical axis of the camera 11 (markers 4-1 to 4a-12 in FIG. 18) and markers 4 that are seemingly indistinguishable from those that have a surface perpendicular to the optical axis of the camera 11 (marker 4b-6 in FIG. 20).

[0086] Fig. 23 is a flowchart showing a subroutine of step S41 in Fig. 17. Step S41 may include, for example, the following steps.

[0087] In step S51, the absolute position calculator 34 judges whether the apparent size of the marker 4 in the image captured by the image capture device 11 is larger than a threshold value. If YES, the process proceeds to step S42 in FIG. 17, and if NO, the process proceeds to step S52. The apparent size of the marker 4 in the image is, for example, the apparent height and width of the marker 4 in the image, and is expressed, for example, in terms of the number of pixels. The threshold value in step S51 is, for example, 30 pixels. A marker 4 that occupies a small area in the image may be erroneously recognized. On the other hand, a marker 4 that occupies a large area in the image is considered less likely to be erroneously recognized.

[0088] In step S52, the absolute position calculator 34 judges whether the difference or ratio between the apparent height and width of the marker 4 in the image captured by the image capture device 11 is greater than a threshold value. If YES, proceed to step S42 in FIG. 17, and if NO, proceed to step S53. The apparent height and width of the marker 4 in the image are expressed, for example, in pixels. In step S52, when the difference between the apparent height and width of the marker 4 in the image is compared with a threshold value, the threshold value is, for example, 3 pixels. Markers 4 having approximately the same apparent height and width in the image may appear to have a surface perpendicular to the optical axis of the image capture device 11. On the other hand, markers 4 having substantially different apparent heights and widths in the image are considered to have a surface inclined with respect to the optical axis of the image capture device 11.

[0089] In step S53, the absolute position calculator 34 judges whether the apparent angle difference between the adjacent corners of the marker 4 in the image captured by the image capture device 11 is greater than a threshold value. If YES, proceed to step S42 in FIG. 17, and if NO, proceed to step S48 in FIG. 17. The threshold value in step S53 is, for example, 30 degrees. A marker 4 whose adjacent corners in the image have approximately equal angles may appear to have a surface perpendicular to the optical axis of the image capture device 11. On the other hand, a marker 4 whose adjacent corners in the image have substantially different angles is considered to have a surface inclined with respect to the optical axis of the image capture device 11.

[0090] FIG. 24 is a diagram for explaining the apparent height of the marker 4 in the image, which relates to steps S51 and S52 in FIG. 23. FIG. 25 is a diagram for explaining the apparent width of the marker 4 in the image, which relates to steps S51 and S52 in FIG. 23. In the image, the pattern of the marker 4 has a rectangular shape. With reference to FIG. 24, the apparent height of the marker 4 in the image is represented by the length of the shortest perpendicular line among the perpendicular lines h1 to h4 from the four vertices to the opposite side. Similarly, with reference to FIG. 25, the apparent width of the marker 4 in the image is represented by the length of the shortest perpendicular line among the perpendicular lines w1 to w4 from the four vertices to the opposite side. The apparent height and width of the marker 4 in the image indicate the apparent height and width of the pattern in the image, and do not include margins around the pattern.

[0091] Fig. 26 is a diagram for explaining the apparent angle difference between adjacent corners of the marker 4 in the image, in accordance with step S53 in Fig. 23. The apparent angle difference between adjacent corners of the marker 4 in the image indicates the apparent angle difference between adjacent corners of the pattern in the image, and does not include the margins around the pattern. In the example of Fig. 26, the apparent angle of the upper left corner of the marker 4 is 80 degrees, and the apparent angle of the upper right corner of the marker 4 is 100 degrees. Therefore, the apparent angle difference between the upper left corner and the upper right corner of the marker 4 in the image is 20 degrees.

[0092] Fig. 27 is a diagram showing an example of an image 40D including a marker 4a having a surface perpendicular to the optical axis of the image capturing device 11 of Fig. 1. Fig. 28 is a diagram showing an example of an image 40E including a marker 4b having a surface parallel to the optical axis of the image capturing device 11 of Fig. 1, in which the difference between the height and width of the marker 4b in the image 40E is one pixel. Comparing Figs. 27 and 28, it can be seen that when the difference between the height and width of the marker 4b in the image is less than the threshold value of three pixels, the marker 4b having a surface perpendicular to the optical axis of the image capturing device 11 (Fig. 28) is difficult to distinguish from the marker 4a having a parallel surface (Fig. 27).

[0093] Fig. 29 is a diagram showing an example of an image 40F including a marker 4a having a surface perpendicular to the optical axis of the image capturing device 11 of Fig. 1. Fig. 30 is a diagram showing an example of an image 40G including a marker 4b having a surface parallel to the optical axis of the image capturing device 11 of Fig. 1, in which the difference between the height and width of the marker 4b in the image 40G is four pixels. Comparing Fig. 29 and Fig. 30, when the difference between the height and width of the marker 4b in the image is greater than the threshold value of three pixels, the marker 4b having a surface perpendicular to the optical axis of the image capturing device 11 (Fig. 30) can be distinguished from the marker 4a having a parallel surface (Fig. 29).

[0094] Fig. 31 is a diagram showing an example of an image 40H including a marker 4a having a surface perpendicular to the optical axis of the image capturing device 11 of Fig. 1. Fig. 32 is a diagram showing an example of an image 40I including a marker 4b having a surface parallel to the optical axis of the image capturing device 11 of Fig. 1, in which the angle difference between adjacent corners of the marker 4b in the image 40I is 35 degrees. Comparing Figs. 31 and 32, when the angle difference between adjacent corners of the marker 4b in the image is greater than the threshold value of 30 degrees, the marker 4b having a surface perpendicular to the optical axis of the image capturing device 11 (Fig. 32) can be distinguished from the marker 4a having a parallel surface (Fig. 31).

[0095] According to step S41 in FIG. 17, markers 4 that are difficult to distinguish in appearance despite having surfaces at different angles relative to the optical axis of the imaging device 11, i.e., markers 4 that are unsuitable for calculating the absolute position and absolute attitude of the vehicle 1, can be detected and removed.

[0096] [Step S42] 17, the absolute position calculator 34 counts the number of images from which markers 4 having the same identifier are extracted, among images continuously captured by the image capture device 11. Furthermore, in step S42, the absolute position calculator 34 determines whether a predetermined number of markers 4 extracted from a predetermined number of images continuously captured by the image capture device 11 have the same identifier, and if YES, the process proceeds to step S43, and if NO, the process proceeds to step S48.

[0097] Fig. 33 is a diagram showing an example of an image 40J captured by the image capturing device 11 of Fig. 1. When a marker 4 is captured in an environment including a complex background such as a warehouse or a factory, there is a possibility that some object 5 may be erroneously recognized as a marker, i.e., as a visually identifiable area having a marker identifier (e.g., ID: 668). In particular, when the pattern of the marker 4 is simple, other objects 5 are likely to be erroneously recognized as a marker.

[0098] FIG. 34 is a table including the markers 4 detected by the image recognizer 33 of FIG. 3 and the objects erroneously detected as the markers 4. Each row of the table in FIG. 34 indicates a candidate for the marker 4 (the marker 4 or the object erroneously detected) extracted from each image captured by the image capture device 11. One or more candidates for the marker 4 may be detected from one image, and no candidate for the marker 4 may be detected from a certain image. The table in FIG. 34 indicates, for each candidate for the marker 4, an identifier (ID) (first column), the time when the image including the candidate for the marker 4 was captured (second column), the position (third to fifth columns), and the angle (sixth column). When the image recognizer 33 detects a certain marker 4, there is a possibility that an object 5 close to the marker 4 may be erroneously recognized as a marker temporarily. However, even if an object 5 is erroneously detected as a marker 4 in a certain image, it is considered that the possibility of erroneously detecting the object 5 as a marker 4 in the same manner across multiple consecutive images is very low. Therefore, even if an object 5 having a certain identifier (e.g., ID: 668) is detected in an image, if the object 5 is not detected in multiple consecutive images, it can be determined that the object 5 is not a marker 4. On the other hand, if a marker 4 with the same identifier (e.g., ID: 010) is detected in multiple images captured consecutively by the image capture device 11, it can be determined that the marker 4 has been detected.

[0099] The threshold value (predetermined number) in step S42 in FIG. 17 is set to, for example, "3," but may be set to any other value.

[0100] [Step S43] 17, the absolute position calculator 34 calculates the position and attitude of the camera 11 (i.e., the vehicle 1) in a predetermined number of marker coordinate systems that have as their origins markers 4 having the same identifiers extracted from a predetermined number of images continuously captured by the camera 11. Furthermore, in step S43, the absolute position calculator 34 judges whether the coordinates of the camera 11 (i.e., the vehicle 1) in the marker coordinate systems have consecutively the same sign across a predetermined number (threshold) of images, and if YES, proceeds to step S44, and if NO, proceeds to step S48.

[0101] FIG. 35 is a diagram showing an example of an image 40K including a marker 4 that is photographed by the photographing device 11 of FIG. 1 and is suitable for calculating the absolute position and absolute attitude of the vehicle 1. FIG. 36 is a diagram showing an example of an image 40L including a marker 4 that is photographed by the photographing device 11 of FIG. 1 and is not suitable for calculating the absolute position and absolute attitude of the vehicle 1. In an image, for example, when the opposite sides of a rectangular marker 4 have different lengths, it is possible to clearly determine from where the photographing device 11 photographs the marker 4, that is, the position of the photographing device 11 in the marker coordinate system. For example, as shown in FIG. 35, when the right side of the marker 4 is longer than the left side, it is possible to determine that the marker 4 is photographed from the right side of the marker 4. On the other hand, in an image, for example, when the opposite sides of a rectangular marker 4 have the same length, it is difficult to determine the position of the photographing device 11 in the marker coordinate system. For example, as shown in FIG. 36, when the lengths of the right side and the left side of the marker 4 are the same, it is impossible to determine whether the marker 4 is photographed from the right side of the marker 4 or from the left side of the marker 4.

[0102] Fig. 37 is a diagram for explaining the inversion of the position of the camera 11 in the marker coordinate system when the optical axis of the camera 11 in Fig. 1 passes substantially through the center of the surface of the marker 4 and is substantially perpendicular to the surface of the marker 4. In the example of Fig. 38, the Xm coordinate of the camera 11c located to the right of the normal to the marker 4 is positive, and the Xm coordinate of the camera 11d located to the left of the normal to the marker 4 is negative.

[0103] FIG. 38 is a table showing the change in the position of the camera 11 in the marker coordinate system calculated by the position calculator 34 in FIG. 3. Each row of the table in FIG. 38 corresponds to each image captured by the camera 11, and shows the identifier of one marker 4 included in each image, and the position and attitude of the camera 11 in the marker coordinate system with the marker 4 as the origin. If the opposite sides of the square marker 4 have the same length and it is difficult to determine the position of the camera 11 in the marker coordinate system, the sign of the coordinates of the camera 11 in the marker coordinate system is likely to fluctuate. Therefore, when the sign of the coordinates of the camera 11 in the marker coordinate system changes frequently, it can be determined that the image including the marker 4 is not suitable for calculating the absolute position and absolute attitude of the vehicle 1. On the other hand, when the coordinates of the camera 11 in the marker coordinate system have the same sign consecutively over a predetermined number of images, it can be determined that the image including the marker 4 is suitable for calculating the absolute position and absolute attitude of the vehicle 1.

[0104] 38, in lines 2 to 4, the sign of the Xm coordinate alternates, so the corresponding images are not suitable for calculating the absolute position and attitude of vehicle 1. On the other hand, in lines 4 to 6, the Xm, Ym, and Zm coordinates have the same sign, so the corresponding images are suitable for calculating the absolute position and attitude of vehicle 1.

[0105] The threshold value (predetermined number) in step S43 in FIG. 17 is set to, for example, "3," but may be set to any other value.

[0106] [Step S44] 17, the absolute position calculator 34 calculates the apparent size of the marker 4 in the image captured by the image capture device 11. Furthermore, in step S44, the absolute position calculator 34 determines whether the apparent size of the marker 4 in the image is greater than a predetermined threshold value, and if YES, the process proceeds to step S45, and if NO, the process proceeds to step S48.

[0107] FIG. 39 is a diagram showing an example of an image of the marker 4 photographed by the photographing device 11 of FIG. 1 and having a width suitable for calculating the absolute position and absolute attitude of the vehicle 1. FIG. 40 is a diagram showing an example of an image of the marker 4 photographed by the photographing device 11 of FIG. 1 and not having a width suitable for calculating the absolute position and absolute attitude of the vehicle 1. FIG. 39 and FIG. 40 show the apparent size (number of pixels) of the marker 4 in the image photographed by the photographing device 11. In the examples of FIG. 39 and FIG. 40, the apparent size of the marker 4 in the image is represented by the number of pixels from each vertex to its opposite side. In order to correctly recognize the pattern of the marker 4, the marker 4 needs to have a sufficiently large apparent size (number of pixels) in the image photographed by the photographing device 11. For example, the marker 4 shown in FIG. 5 has a pattern consisting of 7×7 white or black square cells in the vertical and horizontal directions. In this case, the number of pixels of the marker 4 in FIG. 39 is sufficient to recognize each cell of the pattern. On the other hand, the number of pixels of the marker 4 in FIG. 40 is obviously insufficient to recognize each cell of the pattern.

[0108] Fig. 41 is a graph showing the error in the absolute position of the vehicle 1 when the threshold value in step S44 of Fig. 17 is changed. Fig. 41 shows that the error is smaller when the threshold value is set to 14 or more. Therefore, when the apparent size of the marker 4 in the image captured by the photographing device 11 is smaller than a predetermined threshold value, it can be determined that the image including the marker 4 is not suitable for calculating the absolute position and attitude of the vehicle 1. On the other hand, when the apparent size of the marker 4 in the image is larger than the predetermined threshold value, it can be determined that the image including the marker 4 is suitable for calculating the absolute position and attitude of the vehicle 1.

[0109] The threshold value in step S44 in Fig. 17 may be set to a value different from the threshold value in step S51 in Fig. 23. In the example described, the threshold value in step S51 in Fig. 23 is set to a relatively large value (30 pixels) so as to detect and extract markers 4 having an apparent size suitable for calculating the absolute position and attitude of the vehicle 1 (i.e., if step S51 is YES). On the other hand, the threshold value in step S44 in Fig. 17 is set to a relatively small value (14 pixels) so as to detect and remove markers 4 having an apparent size that is inappropriate for calculating the absolute position and attitude of the vehicle 1 (i.e., if step S44 is NO).

[0110] The apparent size of the marker 4 in the image may be represented by the length of each side, the sum of the lengths of the sides, the length of the diagonal, or the area of ​​the marker 4, instead of the distance (number of pixels) from each vertex to its opposite side.

[0111] [Step S45] 17, the absolute position calculator 34 calculates the angle of the direction of the vehicle 1 with respect to the normal line of the surface of one marker 4 based on the position and attitude of the vehicle 1 in the marker coordinate system with one marker 4 as the origin. Furthermore, in step S45, the absolute position calculator 34 judges whether the angle of the direction of the photographing device 11 (i.e., the vehicle 1) with respect to the normal line of the surface of one marker 4 in the marker coordinate system is larger than a predetermined threshold value αth, and if YES, proceeds to step S46, and if NO, proceeds to step S48. Note that when the photographing device 11 and the vehicle 1 are in different directions, the absolute position calculator 34 judges whether the angle of the normal line of the surface of the photographing device 11 and the marker 4 is larger than the threshold value αth, and if YES, proceeds to step S46, and if NO, proceeds to step S48.

[0112] 42 is a diagram for explaining the angles and distances at which an image of the marker 4 suitable for calculating the absolute position and absolute attitude of the vehicle 1 can be captured in the marker coordinate system. As described above in relation to step S43, when, for example, opposite sides of a rectangular marker 4 have different lengths in an image, the position of the photographing device 11 in the marker coordinate system can be clearly determined. Therefore, the photographing device 11 photographs the marker 4 not from the normal direction (front) of the marker 4, but from a direction sufficiently inclined from the normal direction of the marker 4.

[0113] FIG. 43 is a diagram for explaining the angle conditions when photographing an image of the marker 4 suitable for calculating the absolute position and absolute attitude of the vehicle 1. FIG. 44 is a diagram showing an example of an image 40M including the marker 4 photographed by the photographing device 11 of FIG. 43. In FIG. 43, α indicates an angle in the direction of the photographing device 11 with respect to the normal to the surface of the marker 4. The angle α satisfies α=arccos(a / b). Here, a and b indicate the apparent size of the marker 4 in the image 40M photographed by the photographing device 11 as shown in FIG. 44. As the ratio approaches a:b=1:1, the accuracy decreases. The positive and negative of the angle α can be determined from the orientation of the trapezoid in FIG. 44. However, when the photographing device 11 photographs the marker 4 from the front, or when the distance d from the photographing device 11 to the marker 4 increases, the determination of the positive and negative of the angle α becomes unstable. β is the angle of the direction of the marker 4 with respect to the optical axis of the photographing device 11. In other words, angle β indicates the position of marker 4 in image 40M captured by image capture device 11. Distance d from image capture device 11 to marker 4 satisfies d=1 / b. The accuracy of the obtained results is in the order of β>d>α.

[0114] 17 is set to, for example, "45 degrees," but may be set to any other value. In addition, this threshold value αth may vary depending on the distance from the image capturing device 11 to the marker 4.

[0115] When the angle of the direction of the camera 11 with respect to the normal to the surface of the marker 4 approaches 90 degrees, the apparent size a of the marker 4 in the image captured by the camera 11 becomes small, making it difficult to satisfy the condition of step S44. The angle of the direction of the camera 11 with respect to the normal to the surface of the marker 4 needs to satisfy the conditions of both steps S44 and S45.

[0116] [Step S46] 17, the absolute position calculator 34 calculates the distance from the vehicle 1 to the marker 4 based on the image captured by the image capture device 11. Furthermore, in step S46, the absolute position calculator 34 determines whether the distance from the image capture device 11 (i.e., the vehicle 1) to the marker 4 is smaller than a predetermined threshold value or not, and if YES, the process proceeds to step S47, and if NO, the process proceeds to step S48.

[0117] 42 again, the imaging device 11 captures an image of the marker 4 at a position that is closer to the marker 4 than the distance Dth. If the distance from the imaging device 11 to the marker 4 is large, the accuracy of the position and attitude of the vehicle 1 calculated by the positioning device 12 decreases.

[0118] 17 is set to, for example, "6 m" when the marker 4 has, for example, an A2 size (420 x 594 mm), but may be set to any other value. Also, this threshold Dth may vary depending on the apparent size of the marker 4 in the image.

[0119] [Steps S47 to S48] 17, the absolute position calculator 34 determines that the marker 4 is reliable. In this case, the absolute position calculator 34 sends the absolute position and absolute attitude calculated based on the position and attitude of the marker 4 to the corrector 36. On the other hand, in step S48, the absolute position calculator 34 determines that the marker 4 is unreliable. In this case, the absolute position calculator 34 does not send the absolute position and absolute attitude calculated based on the position and attitude of the marker 4.

[0120] 17, the absolute position calculator 34 determines whether the marker 4 has been photographed from an appropriate angle and an appropriate distance, that is, whether the recognized marker 4 is reliable. As a result, in the correction process of step S4 in FIG. 8, the corrector 36 operates as follows.

[0121] When the surface of the marker 4 appears perpendicular to the optical axis of the camera 11 in the image captured by the camera 11 (YES in step S41), the corrector 36 does not use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1. When the surface of the marker 4 does not appear perpendicular to the optical axis of the camera 11 in the image captured by the camera 11 (NO in step S41), the corrector 36 can use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1.

[0122] When the marker 4 does not have the same identifier across a number of images equal to a predetermined threshold (step S42 is NO), the corrector 36 does not use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1. On the other hand, when the marker 4 has the same identifier across a number of images equal to or greater than the threshold (step S42 is YES), the corrector 36 can use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1.

[0123] When the coordinates of the vehicle 1 in each marker coordinate system across a number of images equal to a predetermined threshold value do not have the same sign (step S43 is NO), the corrector 36 does not use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1. On the other hand, when the coordinates of the vehicle 1 in each marker coordinate system across a number of images equal to or greater than the threshold value (step S43 is YES), the corrector 36 can use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1.

[0124] When the apparent size of the marker 4 in the image is equal to or smaller than a predetermined threshold (step S44: NO), the corrector 36 does not use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1. On the other hand, when the apparent size of the marker 4 in the image is larger than the threshold (step S44: YES), the corrector 36 can use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1.

[0125] When the angle of the direction of the vehicle 1 relative to the normal to the surface of one of the markers 4 is equal to or smaller than a predetermined threshold (NO in step S45), the corrector 36 does not use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1. On the other hand, when the angle of the direction of the vehicle 1 relative to the normal to the surface of one of the markers 4 is greater than the threshold (YES in step S45), the corrector 36 can use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1.

[0126] When the distance from the vehicle 1 to the marker 4 is equal to or greater than a predetermined threshold (step S46: NO), the corrector 36 does not use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1. On the other hand, when the distance from the vehicle 1 to the marker 4 is smaller than the threshold (step S46: YES), the corrector 36 can use the absolute position and attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1.

[0127] In the example of Figure 17, when all of the conditions of steps S41 to S46 are met, the corrector 36 uses the absolute position and absolute attitude calculated based on the position and attitude of the marker 4 to correct the position and attitude of the vehicle 1.

[0128] 23 is also referred to as the "first threshold", the threshold of step S53 is also referred to as the "second threshold", and the threshold of step S51 is also referred to as the "third threshold". Also, in this specification, the threshold of step S45 is also referred to as the "fourth threshold", the threshold of step S42 is also referred to as the "fifth threshold", the threshold of step S43 is also referred to as the "sixth threshold", and the threshold of step S46 is also referred to as the "seventh threshold".

[0129] FIG. 45 is a diagram showing a trajectory 103 of the vehicle 1 calculated by executing a correction process according to a comparative example of the first embodiment. FIG. 46 is a diagram showing a trajectory 104 of the vehicle 1 calculated by executing the marker evaluation process of FIG. 17. In FIG. 45 to FIG. 46, for the sake of explanation, the front of the marker 4 (Zm axis in FIG. 5) is shown by a protrusion provided on one side of each marker 4, but in reality, there is no such protrusion. According to FIG. 45, an error occurs in the calculated position of the vehicle 1 at the position surrounded by a dashed line due to the inability to correctly recognize the marker 4 having a surface at a different angle with respect to the optical axis of the imaging device 11. On the other hand, according to FIG. 46, the positioning device 12 does not use the marker 4 that appears to have a surface perpendicular to the optical axis of the imaging device 11 in the calculation of the absolute position and absolute attitude of the vehicle 1, so that the position and attitude of the vehicle 1 can be measured with high accuracy.

[0130] [Display of vehicle position and attitude] FIG. 47 is a diagram showing a first example of an image displayed on the display device 14 or 25 of FIG. 2. FIG. 48 is a diagram showing a second example of an image displayed on the display device 14 or 25 of FIG. 2. FIG. 49 is a diagram showing a third example of an image displayed on the display device 14 or 25 of FIG. 2. As described above, the display device 14, 25 displays an image showing the position and attitude of the vehicle 1. The image showing the position and attitude of the vehicle 1 may be an image in which the position and attitude of the vehicle 1 are superimposed on a map, or may be an image photographed by the photographing device 11. In the examples of FIG. 47 to FIG. 49, the images 111 to 113 displayed on the display device 14 or 25 have a window 120 including an image photographed by the photographing device 11 superimposed on a map.

[0131] 47 and 48 show the current relative position and relative attitude of the vehicle 1 calculated by the relative position calculator 32 by the position and orientation of a triangle. Also, FIG. 47 and FIG. 48 show trajectories 105, 106 of the vehicle 1 based on the relative position and relative attitude, respectively. Here, trajectory 106 is a continuation of trajectory 105. In the example of FIG. 47 and FIG. 48, the vehicle 1 is actually moving straight up along the aisle 101 in front of the marker 4 having the identifier "003", but due to a calculation error, the vehicle 1 is displayed as if it is running into the shelf 102.

[0132] Fig. 49 shows the current position and attitude of the vehicle 1, corrected based on the absolute position and attitude calculated based on the marker 4 with identifier "003", by the position and orientation of a triangle. Fig. 49 also shows a trajectory 107 of the vehicle 1, including a section 107a based on the corrected position and attitude. The trajectory 107 is also a continuation of the trajectory 105.

[0133] The marker 4 having the identifier "003" is photographed as shown in the window 120 of FIG. 47. The absolute position calculator 34 calculates the absolute position and absolute attitude of the vehicle 1 based on the marker 4 having the identifier "003". The corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude. At this time, the display device 14, 25 may associate the marker 4 used in the calculation of the absolute position and absolute attitude with the corrected position and corrected attitude, and may superimpose the marker 4 and the corrected position and corrected attitude on the map. In the example of FIG. 49, the color of the vehicle 1 and the marker 4 is changed from black to white in order to associate the marker 4 used in the calculation of the absolute position and absolute attitude with the corrected position and corrected attitude. Here, instead of changing the color of the vehicle 1 and the marker 4, the pattern of the vehicle 1 and the marker 4 may be changed, the size of the vehicle 1 and the marker 4 may be increased, or the vehicle 1 and the marker 4 may be surrounded by a frame.

[0134] 49, in order to indicate that the position and attitude of the vehicle 1 have been corrected, the width of the section 107a of the trajectory 107 of the vehicle 1 that is based on the corrected position and attitude is changed. Instead of changing the width of the section 107a, the color of the section 107a may be changed, for example, from black to gray, or the section 107a may be changed from a solid line to a dotted line, in order to indicate that the position and attitude of the vehicle 1 have been corrected. This allows the user to confirm which section of the trajectory of the vehicle 1 has been corrected based on the marker 4.

[0135] As shown in FIG. 49, the current position and attitude of the vehicle 1 can be displayed accurately by correcting the relative position and relative attitude based on the absolute position and absolute attitude.

[0136] 47, the recognized marker 4 or the marker 4 used in the calculation of the absolute position and the absolute attitude may be highlighted by, for example, surrounding it with a frame 121. Furthermore, the identifier of the marker 4 (ID:003) may be displayed near the recognized marker 4 or the marker 4 used in the calculation of the absolute position and the absolute attitude.

[0137] In addition, the display devices 14, 25 may selectively display either a first image (see FIG. 48) in which the relative position and relative attitude of the vehicle 1 are superimposed on a map, or a second image (see FIG. 49) in which the corrected position and corrected attitude are superimposed on a map, without superimposing a window 120 containing an image captured by the photographing device 11 on the map.

[0138] Furthermore, the display devices 14 and 25 may selectively display either an image in which the position and attitude of the vehicle 1 is superimposed on a map, or an image captured by the image capturing device 11.

[0139] Furthermore, the display devices 14, 25 may display an image showing either the position or the attitude of the vehicle 1, but not both.

[0140] [Summary of the first embodiment] According to the first embodiment, the position and orientation of the vehicle 1 can be measured with high accuracy by correcting the relative position and relative orientation calculated based on the feature points 41 using Visual-SLAM or the like based on the absolute position and absolute orientation calculated based on the marker 4. Furthermore, according to the first embodiment, it is determined whether the recognized marker 4 is reliable, and only when the marker 4 is reliable, the position and orientation of the vehicle 1 are corrected based on the absolute position and absolute orientation, thereby making it possible to measure the position and orientation of the vehicle 1 with high accuracy.

[0141] According to the first embodiment, by using the image capturing device 11, the position of the vehicle 1 can be measured inexpensively even indoors where radio waves from GPS satellites cannot be received, such as in a warehouse or a factory. Since there is no need to install a large number of wireless transmitters for transmitting wireless signals, the initial implementation cost can be reduced.

[0142] According to the first embodiment, business operations can be improved based on the movement route of the vehicle 1 obtained from the positioning result.

[0143] According to the first embodiment, the travel distance of the vehicle 1 can be used to determine whether maintenance is required, whether the lease contract needs to be renewed, and the like.

[0144] According to the first embodiment, the layout of aisles, shelves, etc. in a warehouse or factory can be optimized based on a heat map of the movement of the vehicle 1.

[0145] According to the first embodiment, based on the differences in the trajectories of each vehicle 1, it is possible to visualize the locations where each vehicle 1 passes each other while moving, and it is possible to review the route and passage width to improve safety.

[0146] [Modification of the first embodiment] 18 to 24, cases have been described in which it is difficult to distinguish between markers 4 having a surface perpendicular to the optical axis of the camera 11 and markers 4 having a surface parallel to the optical axis of the camera 11, but it may also be difficult to distinguish between markers having a surface at another angle (such as 45 degrees) to the optical axis of the camera 11. Similarly in this case, the absolute position calculator 34 treats the marker 4 that appears to have a surface perpendicular to the optical axis of the camera 11 as one that will not be used in calculating the absolute position and absolute attitude of the vehicle 1.

[0147] Steps S41 to S46 in Fig. 17 may be executed in any other order. The result will not change depending on the order of the processes, but the processing speed will differ. For example, by executing step S43 before step S45, it is possible to efficiently eliminate cases where the photographing device 11 is located in front of the marker 4. Also, only some of steps S42 to S46 may be executed.

[0148] 11 may be executed by the corrector 36 instead of the absolute position calculator 34. In this case, the absolute position calculator 34 always sends the absolute position and absolute attitude calculated based on the position and attitude of the marker 4 to the corrector 36. The corrector 36 corrects the position and attitude of the vehicle 1 based on the absolute position and absolute attitude only when the marker 4 can be trusted.

[0149] In the above example, the corrector 36 does not overwrite the relative position and relative orientation with the absolute position and absolute orientation, and manages the absolute position and absolute orientation separately from the relative position and relative orientation. Alternatively, the corrector 36 may overwrite the relative position and relative orientation with the absolute position and absolute orientation, and then calculate the relative position and relative orientation using the absolute position and absolute orientation as a new reference position and reference orientation.

[0150] The image capture device 11 may be configured to generate an image of the subject and detect the distance from the image capture device 11 to each point on the subject. The image capture device 11 may include, for example, a depth sensor, such as an RGB-D camera, or a Time of Flight (ToF) sensor to detect the distance to the subject. Alternatively, the image capture device 11 may be a stereo camera including two cameras positioned a predetermined distance apart to detect the distance to the subject.

[0151] When the image capturing device 11 detects the distance, the relative position calculator 32 may calculate the relative position and relative attitude of the vehicle 1 using a known ICP (iterative closest point) algorithm or the like.

[0152] 9, an example in which the FAST and KLT trackers are used has been described, but other methods may be used. For example, feature point detection processing and feature point matching processing using SIFT (Scale Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF), which are common in image processing, may be used.

[0153] If the marker 4 is placed in the middle of a straight section of the passage 101, it is expected that the absolute position and absolute attitude can be calculated with high accuracy. On the other hand, if the marker 4 is placed near an intersection or an entrance / exit of the passage 101, the vehicle 1 does not necessarily move straight in the vicinity of the marker 4, and there is a possibility that the error of the calculated absolute position and absolute attitude will be large. Therefore, some of the multiple markers 4 may be auxiliary markers that are not used to calculate the absolute position and absolute attitude (i.e., to correct the position and attitude). The auxiliary marker is placed, for example, near some structure that may be the starting point, destination, or other checkpoint of the vehicle 1 (an entrance / exit of the warehouse 100, an intersection of the passage 101, a specific shelf 102, etc.). The auxiliary marker does not have to be placed along the passage 101 as long as it can be photographed from the vehicle 1. The positioning device 12 can recognize that the vehicle 1 has reached a specific checkpoint by detecting the auxiliary marker. In this case, the table of marker information stored in the storage device 35 further includes an item indicating whether each marker 4 is an auxiliary marker or not. In this case, the marker information table does not need to have information on the position and orientation of the auxiliary markers. The marker information table may include an item indicating the reliability of the absolute position and absolute orientation calculated based on the marker 4, instead of an item indicating whether or not each marker 4 is an auxiliary marker.

[0154] The vehicle 1 and the server device 2 may use a removable storage medium such as an SD card instead of the communication devices 13 and 22. The position and attitude of the vehicle calculated in the vehicle may be written to the storage medium, and the server device 2 may read the position and attitude of the vehicle from the storage medium.

[0155] Instead of a forklift, the vehicle 1 may be a manned vehicle such as a truck or a towing vehicle. The vehicle 1 may also be an unmanned luggage transport device such as an automated guided vehicle (AGV) or a pallet transport robot. In this case, the vehicle 1 moves by controlling the drive mechanism 15 under the control of the server device 2. The vehicle 1 may also be a human-powered vehicle without a drive mechanism, such as a handcart.

[0156] The positioning device 12 may be provided in the server device 2 instead of in the vehicle 1. In this case, the image captured by the imaging device 11 is sent from the vehicle 1 to the server device 2 by the communication device 13 (or a removable storage medium). The positioning device 12 of the server device 2 calculates the position and attitude of the vehicle 1 based on the image acquired from the vehicle 1, similar to the positioning device 12 of FIG. 2.

[0157] [Advantages of the first embodiment] According to the first embodiment, the positioning device 12 includes a relative position calculator 32, a storage device 35, an absolute position calculator 34, and a corrector 36. The relative position calculator 32 calculates the relative position and relative attitude of the vehicle 1, which indicate the relative position and relative attitude of the vehicle 1 with respect to a predetermined reference position and reference attitude, based on a plurality of images taken by a photographing device mounted on the vehicle 1. The storage device 35 stores information on identifiers, positions, and attitudes of a plurality of markers 4 arranged at predetermined positions and visually identifiable, and information on a map including a path for the vehicle 1. The absolute position calculator 34 extracts one of the plurality of markers 4 from the image taken by the photographing device, and calculates the absolute position and absolute attitude of the vehicle 1, which indicate the position and attitude of the vehicle 1 on the map, based on the position and attitude of the one extracted marker 4. The corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, and calculates the corrected position and corrected attitude of the vehicle 1. The corrector 36 does not use the absolute position and absolute orientation calculated based on the position and orientation of the marker 4 when the difference or ratio between the apparent height and width of the marker 4 in the image is less than or equal to a first threshold, and calculates a corrected position and corrected orientation using the absolute position and absolute orientation calculated based on the position and orientation of the marker 4 when the difference or ratio between the apparent height and width of the marker 4 in the image is greater than the first threshold.

[0158] According to the first embodiment, the corrector 36 may calculate a corrected position and attitude without using the absolute position and attitude calculated based on the position and attitude of the marker 4 when the apparent angular difference between adjacent corners of the marker 4 in the image is less than or equal to the second threshold, and may calculate a corrected position and attitude using the absolute position and attitude calculated based on the position and attitude of the marker 4 when the apparent angular difference between adjacent corners of the marker 4 in the image is greater than the second threshold.

[0159] According to the first embodiment, the positioning device 12 includes a relative position calculator 32, a storage device 35, an absolute position calculator 34, and a corrector 36. The relative position calculator 32 calculates the relative position and relative attitude of the vehicle 1, which indicate the relative position and relative attitude of the vehicle 1 with respect to a predetermined reference position and reference attitude, based on a plurality of images taken by a photographing device mounted on the vehicle 1. The storage device 35 stores information on identifiers, positions, and attitudes of a plurality of markers 4 arranged at predetermined positions and visually identifiable, and information on a map including a path for the vehicle 1. The absolute position calculator 34 extracts one of the plurality of markers 4 from the image taken by the photographing device, and calculates the absolute position and absolute attitude of the vehicle 1, which indicate the position and attitude of the vehicle 1 on the map, based on the position and attitude of the one extracted marker 4. The corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, and calculates the corrected position and corrected attitude of the vehicle 1. The corrector 36 does not use the absolute position and absolute orientation calculated based on the position and orientation of the marker 4 when the apparent angular difference between adjacent corners of the marker 4 in the image is less than or equal to the second threshold, and calculates a corrected position and corrected orientation using the absolute position and absolute orientation calculated based on the position and orientation of the marker 4 when the apparent angular difference between adjacent corners of the marker 4 in the image is greater than the second threshold.

[0160] According to the first embodiment, the absolute position calculator 34 may calculate the apparent size of the marker 4 in the image captured by the image capture device. The corrector 36 does not use the absolute position and absolute orientation calculated based on the position and orientation of the marker 4 when the apparent size of the marker 4 in the image is equal to or smaller than a third threshold, and calculates a corrected position and corrected orientation using the absolute position and absolute orientation calculated based on the position and orientation of the marker 4 when the apparent size of the marker 4 in the image is greater than the third threshold.

[0161] As a result, the corrector 36 corrects the position and attitude of the vehicle 1 based on the position and attitude of the marker 4 without using a marker 4 that appears to have a surface perpendicular to the optical axis of the imaging device 11.

[0162] According to the first embodiment, the absolute position calculator 34 may calculate an angle in the direction of the vehicle 1 relative to the normal to the surface of one marker 4 based on the position and attitude of the vehicle 1 in a marker coordinate system with one marker 4 as the origin. The corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, and calculates a corrected position and corrected attitude of the moving body. When the angle is equal to or less than the fourth threshold, the corrector 36 does not use the absolute position and absolute attitude calculated based on the position and attitude of the marker 4, and calculates the corrected position and corrected attitude using the absolute position and absolute attitude calculated based on the position and attitude of the marker 4 when the angle is greater than the fourth threshold.

[0163] As a result, the corrector 36 corrects the position and attitude of the vehicle 1 based on the position and attitude of the marker 4 only when the angle in the direction of the vehicle 1 with respect to the normal to the surface of the marker 4 is greater than the fourth threshold value.

[0164] According to the first embodiment, the absolute position calculator 34 may count the number of images successively captured by the image capture device 11 from which a marker 4 having the same identifier is extracted. In this case, the corrector 36 does not use the absolute position and absolute orientation calculated based on the position and orientation of the marker 4 when the marker 4 does not have the same identifier across a number of images equal to the fifth threshold, and calculates a corrected position and corrected orientation using the absolute position and absolute orientation calculated based on the position and orientation of the marker 4 when the marker 4 has the same identifier across a number of images equal to or greater than the fifth threshold.

[0165] As a result, the corrector 36 corrects the position and attitude of the vehicle 1 based on the position and attitude of the marker 4 only when the marker 4 has the same identifier across a number of images equal to or greater than the fifth threshold.

[0166] According to the first embodiment, the absolute position calculator 34 may calculate the position and attitude of the vehicle 1 in each marker coordinate system having as its origin a marker 4 having the same identifier extracted from a number of images successively captured by the image capture device 11. In this case, the corrector 36 does not use the absolute position and attitude calculated based on the position and attitude of the marker 4 when the coordinates of the vehicle 1 in each marker coordinate system do not have the same sign across a number of images equal to the sixth threshold, but calculates a corrected position and attitude using the absolute position and attitude calculated based on the position and attitude of the marker 4 when the coordinates of the vehicle 1 in each marker coordinate system have the same sign across a number of images equal to or greater than the sixth threshold.

[0167] As a result, the corrector 36 corrects the position and attitude of the vehicle 1 based on the position and attitude of the marker 4 only when the coordinates of the vehicle 1 in each marker coordinate system have the same sign across a number of images equal to or greater than the sixth threshold.

[0168] According to the first embodiment, the absolute position calculator 34 may calculate the distance from the vehicle 1 to the marker 4 based on the image captured by the imaging device 11. In this case, the corrector 36 does not use the absolute position and absolute attitude calculated based on the position and attitude of the marker 4 when the distance from the vehicle 1 to the marker 4 is equal to or greater than the seventh threshold, and calculates a corrected position and corrected attitude using the absolute position and absolute attitude calculated based on the position and attitude of the marker 4 when the distance from the vehicle 1 to the marker 4 is smaller than the seventh threshold.

[0169] Thereby, the corrector 36 corrects the position and attitude of the vehicle 1 based on the position and attitude of the marker 4 only when the marker 4 has a distance from the vehicle 1 that is less than the seventh threshold value.

[0170] According to the first embodiment, a vehicle 1 including an imaging device 11 and a positioning device 12 may be provided.

[0171] According to the first embodiment, the positioning system includes at least one vehicle 1 and a server device 2. The server device 2 includes a communication device 22 communicatively connected to each vehicle 1, a processing device 21 that acquires information indicating at least one of the position and attitude of each vehicle 1 from each vehicle 1 via the communication device 22, and a storage device 24 that stores the information indicating at least one of the position and attitude of each vehicle 1.

[0172] This allows the server device 2 to, for example, acquire and record information indicating the position and attitude of each vehicle 1 from a plurality of vehicles 1, and generate an accurate map based on the movement route of each vehicle 1.

[0173] According to the first embodiment, the server device 2 may further include a display device 25 that displays an image indicating at least one of the position and attitude of each vehicle 1. When the corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, the display device 25 associates the marker 4 used in calculating the absolute position and absolute attitude with at least one of the corrected position and corrected attitude, and displays the marker 4 and at least one of the corrected position and corrected attitude by superimposing them on a map.

[0174] This allows the user to easily recognize that the position or attitude of the vehicle 1 has been corrected based on the marker 4.

[0175] The server device 2 may further include a display device 25 that displays an image indicating at least one of the position and attitude of each vehicle 1. The display device 25 displays an image captured by the imaging device, and when the corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, highlights in the image the markers 4 used in the calculation of the absolute position and absolute attitude.

[0176] This allows the user to easily recognize the marker 4 used to correct the position or attitude of the vehicle 1.

[0177] According to the first embodiment, the position and orientation of the vehicle 1 can be measured with high accuracy by correcting the relative position and relative orientation calculated based on the feature points 41 using Visual-SLAM or the like based on the absolute position and absolute orientation calculated based on the marker 4. Furthermore, according to the first embodiment, it is determined whether the recognized marker 4 is reliable, and only when the marker 4 is reliable, the position and orientation of the vehicle are corrected based on the absolute position and absolute orientation, thereby making it possible to measure the position and orientation of the vehicle 1 with high accuracy.

[0178] [Second embodiment] [Configuration of the second embodiment] Fig. 50 is a block diagram showing a configuration of a positioning device 12A according to the second embodiment. The positioning device 12A further includes a display device 14 that displays an image showing at least one of the position and the attitude of the vehicle 1, in addition to the components of the positioning device 12 in Fig. 3.

[0179] When the corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, the display device 14 may associate the marker 4 used in calculating the absolute position and absolute attitude with at least one of the corrected position and corrected attitude, and display the marker 4 and at least one of the corrected position and corrected attitude by superimposing them on a map.

[0180] The display device 14 may selectively display one of a first image in which at least one of the relative position and relative attitude of the vehicle 1 is superimposed on a map, and a second image in which at least one of the corrected position and corrected attitude is superimposed on a map.

[0181] The display device 14 displays the image captured by the imaging device, and when the corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, the marker 4 used in calculating the absolute position and absolute attitude may be highlighted in the image.

[0182] The components of the positioning device 12A other than the display device 14 are configured and operate in the same manner as in the first embodiment.

[0183] [Advantages of the second embodiment] According to the second embodiment, the positioning device 12A includes a relative position calculator 32, a storage device 35, an absolute position calculator 34, a corrector 36, and a display device 14. The relative position calculator 32 calculates the relative position and relative attitude of the vehicle 1, which indicate the relative position and relative attitude of the vehicle 1 with respect to a predetermined reference position and reference attitude, based on a plurality of images taken by a photographing device mounted on the vehicle 1. The storage device 35 stores information on identifiers, positions, and attitudes of a plurality of markers 4 arranged at predetermined positions and visually identifiable, and information on a map including a path for the vehicle 1. The absolute position calculator 34 extracts one of the plurality of markers 4 from the image taken by the photographing device, and calculates the absolute position and absolute attitude of the vehicle 1, which indicate the position and attitude of the vehicle 1 on the map, based on the position and attitude of the one extracted marker 4. The corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, and calculates the corrected position and corrected attitude of the vehicle 1. The display device 14 displays an image indicating at least one of the position and attitude of the vehicle 1. When the corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, the display device 14 associates the marker 4 used in calculating the absolute position and absolute attitude with at least one of the corrected position and corrected attitude, and displays the marker 4 and at least one of the corrected position and corrected attitude by superimposing them on a map.

[0184] This allows the user to easily recognize that the position or attitude of the vehicle 1 has been corrected based on the marker 4.

[0185] According to the second embodiment, the display device 14 may selectively display one of a first image in which at least one of the relative position and the relative attitude of the vehicle 1 is superimposed on a map, and a second image in which at least one of the corrected position and the corrected attitude is superimposed on a map.

[0186] This allows the user to easily check the effect of correcting the position or attitude of the vehicle 1.

[0187] According to the second embodiment, the positioning device 12A includes a relative position calculator 32, a storage device 35, an absolute position calculator 34, a corrector 36, and a display device 14. The relative position calculator 32 calculates the relative position and relative attitude of the vehicle 1, which indicate the relative position and relative attitude of the vehicle 1 with respect to a predetermined reference position and reference attitude, based on a plurality of images taken by a photographing device mounted on the vehicle 1. The storage device 35 stores information on identifiers, positions, and attitudes of a plurality of markers 4 arranged at predetermined positions and visually identifiable, and information on a map including a path for the vehicle 1. The absolute position calculator 34 extracts one of the plurality of markers 4 from the image taken by the photographing device, and calculates the absolute position and absolute attitude of the vehicle 1, which indicate the position and attitude of the vehicle 1 on the map, based on the position and attitude of the one extracted marker 4. The corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, and calculates the corrected position and corrected attitude of the vehicle 1. The display device 14 displays an image indicating at least one of the position and attitude of the vehicle 1. The display device 14 displays the image captured by the imaging device, and when the corrector 36 corrects the relative position and relative attitude based on the absolute position and absolute attitude, the marker 4 used in calculating the absolute position and absolute attitude is highlighted in the image.

[0188] This allows the user to easily recognize the marker 4 used to correct the position or attitude of the vehicle 1.

[0189] [Third embodiment] A positioning device and a moving object including the same according to the third embodiment will be described with reference to FIG.

[0190] [Configuration of the third embodiment] Fig. 51 is a block diagram showing a configuration of a positioning device 12B according to the third embodiment. The vehicle 1 according to the first embodiment may be equipped with a positioning device 12B instead of the positioning device 12 in Fig. 3. The positioning device 12B includes an image recognizer 37 and a data combiner 38 in addition to the components of the positioning device 12 in Fig. 3.

[0191] The image recognizer 37 recognizes a predetermined object from an image captured by the image capture device 11. The image recognizer 37 may recognize a person (e.g., a driver of the vehicle 1, a person around the vehicle 1). The image recognizer 37 may recognize a specific baggage 3 that has been learned in advance. The image processor 31, the image recognizer 33, and the image recognizer 37 may acquire images from the same image capture device 11. Alternatively, the image recognizer 37 may capture an image including a person and / or baggage 3 using an image capture device different from the image capture device 11 that captures an image (i.e., an image for measuring the position and attitude of the vehicle 1) that is supplied to the image processor 31 and the image recognizer 33. In this case, the image capture device 11 may be provided to capture, for example, an aisle 101 in front of the vehicle 1, and the other image capture device may be provided to capture, for example, the driver's seat or a luggage rack of the vehicle 1. The image capture device 11 and the other image capture device are synchronized with each other in advance.

[0192] The data combiner 38 acquires data on the corrected position and corrected attitude of the vehicle 1 from the corrector 36 together with a time stamp of the time when an image corresponding to the position and attitude was captured by the image capture device 11 (or the time when the position and attitude were calculated). The data combiner 38 combines image recognition data of the object recognized by the image recognizer 37 with data on the position and attitude of the vehicle 1. The data combiner 38 also acquires sensor data generated by the sensor group 16B including one or more sensors mounted on the vehicle 1, and combines the sensor data with data on the position and attitude of the vehicle 1. The data combiner 38 may acquire sensor data including at least one of the acceleration and angular velocity of the vehicle 1. The data combiner 38 may also acquire sensor data including the weight of the baggage 3 carried by the vehicle 1. Each sensor of the sensor group 16B assigns a time stamp of the time when the sensor data was acquired to the sensor data. The data combiner 38 synchronizes and combines the data based on the time stamp of the position and attitude data of the vehicle 1, the time stamp of the image recognition data, and the time stamp of the sensor data.

[0193] When the timestamps of these data do not match, the data combiner 38 may associate the image recognition data or sensor data with the position and attitude data of the vehicle 1 having the closest timestamp. When the timestamps of these data do not match, the data combiner 38 may interpolate the position and attitude data of the vehicle 1 using linear interpolation or internal division, and associate the image recognition data or sensor data with the interpolated position and attitude data of the vehicle 1 having the corresponding timestamp.

[0194] By providing the data synthesizer 38, various data related to the work of the vehicle 1 can be recorded in association with the position and trajectory of the vehicle 1. For example, by recording a person whose image has been recognized, it is possible to track a person related to the work of the vehicle 1. By recording a baggage 3 whose image has been recognized, it is possible to track the baggage 3 transported by the vehicle 1. By recording the acceleration and angular velocity of the vehicle 1, it is possible to detect unevenness of the road surface in the warehouse. By recording the weight of the baggage 3, it is possible to monitor the workload of the vehicle 1.

[0195] [Advantages of the third embodiment] According to the third embodiment, the vehicle 1 may further include a data combiner 38 that acquires sensor data generated by one or more sensors mounted on the vehicle 1 and combines the sensor data into corrected position and corrected attitude data.

[0196] According to a third embodiment, the data combiner 38 may obtain sensor data including at least one of the acceleration and angular velocity of the vehicle 1 .

[0197] According to a third embodiment, the data combiner 38 may obtain sensor data including the weight of the load 3 carried by the vehicle 1 .

[0198] According to the third embodiment, the positioning device 12B may further include an image recognizer 37 that recognizes a predetermined object from an image captured by the imaging device 11. In this case, the data combiner 38 combines information about the object recognized by the image recognizer 37 with data of a corrected position and a corrected attitude.

[0199] According to a third embodiment, the image recognizer 37 may recognize people.

[0200] According to the third embodiment, the image recognizer 37 may recognize a specific piece of luggage 3 that has been previously learned.

[0201] According to the third embodiment, various data related to the work of the vehicle 1 can be recorded in association with the position and trajectory of the vehicle 1.

[0202] According to the third embodiment, the vehicle 1 may include a photographing device 11 that photographs images for measuring the position and attitude of the vehicle 1, and another photographing device that photographs other objects. In this case, the data combiner 38 can mutually associate the position and attitude data of the vehicle 1 generated based on the image photographed by the photographing device 11 with the image recognition data generated based on the image photographed by the other photographing device. If the position and attitude data of the vehicle 1 are associated with the image recognition data of other objects photographed during the movement of the vehicle 1, this is very useful when performing a business analysis based on the position and trajectory of the vehicle 1 on a map. For example, when a suspicious behavior of a person is detected by visual inspection or the like, it is possible to search for and read out images or videos previously photographed near the position and / or related to the person by referring to the position of the person on the map.

[0203] [Other embodiments] In each embodiment, the positioning device may be provided in a four-wheeled vehicle such as a forklift or a truck, or in a vehicle with one to three wheels, or five or more wheels. In each embodiment, the positioning device may be provided in a moving body without wheels, such as an airplane, a helicopter, a drone, or a hovercraft, regardless of the number of wheels and / or the presence or absence of wheels. The positioning device according to this embodiment can estimate the position of the moving body based on an image captured by an imaging device, rather than estimating the position of the moving body from the number of rotations of the wheels.

[0204] As long as the marker has a visually identifiable pattern and is formed so that the posture of the marker itself can be detected from an image of the marker, the marker may be formed not only in the form of a square flat plate, but also in the form of other polygonal or curved contours, or may be formed in a curved shape. [Industrial Applicability]

[0205] According to the positioning device according to each aspect of the present disclosure, it is possible to measure the position of a moving object in a warehouse, a factory, etc. This makes it possible to track the trajectory (traffic line) of the moving object, route the moving object, optimize the placement of luggage, etc. in the warehouse or factory, monitor the operating rate, improve work efficiency, etc. [Explanation of symbols]

[0206] 1 vehicle 1a Cargo platform 1b Lifting mechanism 1c Console 2. Server equipment 3. Luggage 4. Markers 11 Imaging Device 12, 12A, 12B Positioning device 13. Communications Equipment 14 Display device 15 Drive mechanism 16B Sensor Group 21 Processing equipment 22 Communication Equipment 23 Input Devices 24 Storage device 25 Display device 31 Image processor (feature points) 32 Relative Position Calculator 33 Image Recognizer (Marker) 34 Absolute Position Calculator 35 Storage device 36 Corrector 37 Image recognition device (person and luggage) 38 Data Synthesizer 40,40A~40M Images 41 Minutiae 100 warehouse Aisle 101 102 Shelf 103~107 Trajectory 111~113 Display images 120 Windows 121 Frame showing recognized markers

Claims

1. a first calculator that calculates a first position and a first attitude of the moving body, the first position and the first attitude indicating a relative position and a relative attitude of the moving body with respect to a predetermined reference position and a reference attitude, based on a plurality of images captured by an image capturing device mounted on the moving body; A storage device that stores information on identifiers, positions, and attitudes of a plurality of visually identifiable markers arranged at predetermined positions, and information on a map including a path for the moving body; a second calculator that extracts one of the plurality of markers from an image captured by the imaging device, and calculates a second position and a second orientation of the moving object indicating the position and orientation of the moving object on the map based on the position and orientation of the one extracted marker; a corrector that corrects the first position and the first attitude based on the second position and the second attitude and calculates a corrected position and a corrected attitude of the moving body, the corrector does not use the second position and the second orientation calculated based on the position and orientation of the marker when an apparent angular difference between adjacent corners of the marker in the image is equal to or smaller than a first threshold, and calculates the corrected position and the corrected orientation using the second position and the second orientation calculated based on the position and orientation of the marker when an apparent angular difference between adjacent corners of the marker in the image is greater than the first threshold. Positioning device.

2. The second calculator calculates an apparent size of the marker in an image captured by the image capture device; the corrector does not use the second position and the second orientation calculated based on the position and orientation of the marker when the apparent size of the marker in the image is equal to or smaller than a third threshold, and calculates the corrected position and the corrected orientation using the second position and the second orientation calculated based on the position and orientation of the marker when the apparent size of the marker in the image is greater than the third threshold. The positioning device according to claim 1.

3. the second calculator calculates an angle of the direction of the moving body with respect to a normal to a surface of the one marker, based on a position and an orientation of the moving body in a marker coordinate system having the one marker as an origin; the corrector does not use the second position and the second orientation calculated based on the position and orientation of the marker when the angle is equal to or less than a fourth threshold, and calculates the corrected position and the corrected orientation using the second position and the second orientation calculated based on the position and orientation of the marker when the angle is greater than the fourth threshold.

3. The positioning device according to claim 1 or 2.

4. The second calculator counts the number of images from which markers having the same identifier are extracted, the images being successively captured by the image capture device; the corrector does not use the second position and the second orientation calculated based on the position and orientation of the marker when the marker does not have the same identifier across a number of images equal to a fifth threshold, and calculates the corrected position and the corrected orientation using the second position and the second orientation calculated based on the position and orientation of the marker when the marker has the same identifier across a number of images equal to or greater than the fifth threshold. Positioning device according to one of claims 1 to 3.

5. The second calculator calculates the position and orientation of the moving object in each marker coordinate system having an origin in a marker having the same identifier extracted from each of a plurality of images continuously captured by the image capture device, and the corrector does not use the second position and the second orientation calculated based on the position and orientation of the marker when the coordinates of the moving body in each of the marker coordinate systems do not have the same sign across a number of images equal to a sixth threshold, and calculates the corrected position and the corrected orientation using the second position and the second orientation calculated based on the position and orientation of the marker when the coordinates of the moving body in each of the marker coordinate systems have the same sign across a number of images equal to or greater than the sixth threshold. Positioning device according to one of claims 1 to 4.

6. The second calculator calculates a distance from the moving object to the marker based on an image captured by the imaging device; the corrector does not use the second position and the second orientation calculated based on the position and orientation of the marker when the distance from the moving body to the marker is equal to or greater than a seventh threshold, and calculates the corrected position and the corrected orientation by using the second position and the second orientation calculated based on the position and orientation of the marker when the distance from the moving body to the marker is smaller than the seventh threshold. Positioning device according to one of claims 1 to 5.

7. The vehicle further includes a data combiner that acquires sensor data generated by one or more sensors mounted on the vehicle and combines the sensor data into the corrected position and attitude data. Positioning device according to one of the preceding claims.

8. The data combiner acquires sensor data including at least one of an acceleration and an angular velocity of the moving body. The positioning device according to claim 7.

9. the data combiner acquires sensor data including a weight of a load carried by the vehicle; 9. The positioning device according to claim 7 or 8.

10. The positioning device further includes an image recognizer that recognizes a predetermined object from the image captured by the image capture device, The data synthesizer synthesizes information of the object recognized by the image recognizer into data of the corrected position and the corrected attitude. Positioning device according to one of claims 7 to 9.

11. The image recognizer recognizes people. The positioning device according to claim 10.

12. The image recognizer recognizes a specific piece of luggage that has been previously trained. The positioning device according to claim 10.

13. the positioning device further includes a first display device that displays an image indicating at least one of a position and an attitude of the moving object; when the corrector corrects the first position and the first attitude based on the second position and the second attitude, the first display device associates a marker used in calculation of the second position and the second attitude with at least one of the corrected position and the corrected attitude, and displays the marker and at least one of the corrected position and the corrected attitude on the map in a superimposed manner. Positioning device according to one of the claims 1 to 12.

14. the first display device selectively displays one of a first image in which at least one of a first position and a first attitude of the moving object is superimposed on the map, and a second image in which at least one of the corrected position and the corrected attitude is superimposed on the map. The positioning device according to claim 13.

15. the positioning device further includes a first display device that displays an image indicating at least one of a position and an attitude of the moving object; the first display device displays an image captured by the imaging device, and when the corrector corrects the first position and the first orientation based on the second position and the second orientation, highlights in the image a marker used in calculating the second position and the second orientation. Positioning device according to one of the claims 1 to 12.

16. a first calculator that calculates a first position and a first attitude of the moving body, the first position and the first attitude indicating a relative position and a relative attitude of the moving body with respect to a predetermined reference position and a reference attitude, based on a plurality of images captured by an image capturing device mounted on the moving body; A storage device that stores information on identifiers, positions, and attitudes of a plurality of visually identifiable markers arranged at predetermined positions, and information on a map including a path for the moving body; a second calculator that extracts one of the plurality of markers from an image captured by the imaging device, and calculates a second position and a second orientation of the moving object indicating the position and orientation of the moving object on the map based on the position and orientation of the one extracted marker; a corrector that corrects the first position and the first attitude based on the second position and the second attitude and calculates a corrected position and a corrected attitude of the moving body; a first display device that displays an image indicating at least one of a position and an attitude of the moving object; the first display device displays an image captured by the imaging device, and when the corrector corrects the first position and the first orientation based on the second position and the second orientation, highlights in the image a marker used in calculating the second position and the second orientation. Positioning device.

17. A photographing device; and a positioning device according to any one of claims 1 to 16, Mobile body.

18. A positioning system comprising at least one mobile object according to claim 17 and a server device, The server device includes: A communication device communicably connected to each of the mobile objects; a processing device that acquires information indicating at least one of a position and an attitude of each of the moving objects from each of the moving objects via the communication device; a storage device for storing information indicating at least one of the position and the attitude of each of the moving bodies; Positioning system.

19. the server device further includes a second display device that displays an image indicating at least one of a position and an attitude of each of the moving objects; when the corrector corrects the first position and the first attitude based on the second position and the second attitude, the second display device associates a marker used in calculating the second position and the second attitude with at least one of the corrected position and the corrected attitude, and displays the marker and at least one of the corrected position and the corrected attitude on the map in a superimposed manner.

20. The positioning system of claim 18.

20. the server device further includes a second display device that displays an image indicating at least one of a position and an attitude of each of the moving objects; the second display device displays an image captured by the imaging device, and when the corrector corrects the first position and the first orientation based on the second position and the second orientation, highlights in the image a marker used in calculating the second position and the second orientation.

20. The positioning system of claim 18.

Images