Positioning device and mobile object

Abstract

A relative position calculator (32) calculates a relative position and relative attitude of a vehicle (1) on the basis of an image captured by an image-capturing device (11) mounted on the vehicle (1). An absolute position calculator (34) extracts at least one of a plurality of markers (4) from the image captured by the image-capturing device (11) and calculates an absolute position and absolute attitude of the vehicle (1) on the basis of the position and attitude of the extracted marker (4). A reliability calculator (36) calculates the reliability of the relative position and relative attitude calculated by the relative position calculator (32) and the reliability of the absolute position and absolute attitude calculated by the absolute position calculator (34). A position and attitude selector (37) selectors the relative position and relative attitude as the position and attitude of the vehicle (1) when the reliability of the relative position and relative attitude is greater than or equal to the reliability of the absolute position and absolute attitude, and otherwise selects the absolute position and absolute attitude as the position and attitude of the vehicle (1).

Description

[Technical field]

[0001] The present disclosure relates to a positioning device that measures the position of a moving body such as a vehicle, and to a moving body equipped with such a positioning device. [Background technology]

[0002] When a vehicle for transporting luggage is used to move luggage within a predetermined area or between predetermined points, it may be necessary to measure and track the position of the vehicle. For example, Patent Document 1 discloses a luggage location management device that measures the position of the vehicle using positioning techniques such as GPS, wireless LAN positioning, and infrared positioning.

[0003] When positioning a vehicle moving outdoors, a positioning method using GPS is generally used. On the other hand, indoors, such as in a warehouse or a factory, radio waves from GPS satellites cannot be received, so the positioning method using GPS cannot be used. As an indoor positioning method, for example, there is one using a wireless signal such as UWB (ultra wide band), Wi-Fi, or BLE (Bluetooth (registered trademark) Low Energy). However, the positioning method using a wireless signal requires a large number of wireless transmitters for transmitting wireless signals to be installed within the moving range of the vehicle, so the initial introduction cost is high. In addition, as an indoor positioning method, there is one called PDR (Pedestrian Dead Reckoning). However, it is difficult for PDR to measure the position with high accuracy.

[0004] In order to measure and track the position of a moving object such as a vehicle with high accuracy without requiring a large number of wireless transmitters for transmitting wireless signals, there is a technology called Visual-SLAM (Visual Simultaneous Localization and Mapping), as disclosed in, for example, Non-Patent Document 1. According to Visual-SLAM, a moving object equipped with an image capture device moves while capturing images of its surroundings, and the amount of movement of the moving object is calculated based on the amount of movement of feature points in the multiple captured images. This makes it possible to estimate the current position of the moving object and generate a map based on the trajectory of the moving object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2011-219229 A [Non-patent literature]

[0006] [Non-Patent Document 1] R. Mur-Artal, et al., "ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras", IEEE Transactions on Robotics, Volume: 33, Issue: 5, Oct. 2017 Summary of the Invention [Problem to be solved by the invention]

[0007] In scenes where the background changes daily, such as in factories and warehouses, it is difficult to identify the current position using a map created in advance. In such cases, the position of a moving object obtained by Visual-SLAM is calculated as a relative position to a certain reference position (for example, the position where the moving object started moving), so the error increases cumulatively over time. Therefore, there is a demand for a positioning device that can measure the position of a moving object with smaller error than conventional devices using a camera.

[0008] An object of the present disclosure is to provide a positioning device that uses an imaging device to measure the position of a moving object, and that can measure the position of the moving object with a smaller error than conventional positioning devices. [Means for solving the problem]

[0009] According to one aspect of the present disclosure, a first calculator that calculates a first position and a first orientation of the moving body, the first position and the first orientation indicating a relative position and a relative orientation of the moving body with respect to a predetermined reference position and a reference orientation, based on a plurality of images captured by an image capturing device mounted on the moving body; A storage device that stores information on identifiers, positions, and attitudes of a plurality of visually identifiable markers arranged at predetermined positions, and information on a map including a path for the moving body; a second calculator that extracts one of the plurality of markers from an image captured by the image capture device, and calculates a second position and a second orientation of the moving object indicating the position and orientation of the moving object on the map based on the position and orientation of the one extracted marker; a confidence calculator that calculates a first confidence indicating a confidence of the first position and the first orientation calculated by the first calculator, and a second confidence indicating a confidence of the second position and the second orientation calculated by the second calculator; and a position and attitude determiner that determines the first position and the first attitude as the position and attitude of the moving body when the first reliability is equal to or greater than the second reliability, and that determines the second position and the second attitude as the position and attitude of the moving body when the first reliability is smaller than the second reliability.

[0010] These general and specific aspects may be realized by a system, a method, a computer program, or any combination of a system, a method, and a computer program. Effect of the Invention

[0011] According to one aspect of the present disclosure, the reliability of the relative position and relative orientation calculated by a first calculator and the reliability of the absolute position and absolute orientation calculated by a second calculator are calculated, and the one with the higher reliability is determined to be the position and orientation of the moving body, thereby making it possible to measure the position and orientation of the moving body with high accuracy. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram showing a configuration of a vehicle 1 according to a first embodiment. [Diagram 2] 2 is a block diagram showing a configuration of a positioning system including the vehicle 1 of FIG. [Diagram 3] 3 is a block diagram showing a configuration of a positioning device 12 in FIG. 2. [Figure 4] 2 is a map of a warehouse 100 including an aisle 101 along which the vehicle 1 of FIG. 1 travels. [Diagram 5] 5A and 5B are diagrams showing an example of the marker 4 in FIG. 4, in which (a) shows a front view of the marker 4 and (b) shows a top view of the marker 4. [Figure 6] 4 is a table showing an example of marker information stored in the storage device 35 of FIG. 3. [Figure 7] 2 is a diagram showing an example of an image 40 captured by the image capturing device 11 of FIG. 1. FIG. [Figure 8] 3 is a flowchart showing a positioning process executed by the positioning device 12 of FIG. 2. [Figure 9] 9 is a flowchart showing a subroutine of step S2 (relative position calculation process) in FIG. 8. [Figure 10] FIG. 4 shows feature points extracted by image processor 31 in FIG. 3, where (a) shows feature points F1 and F2 extracted from image 40(n) at time n, and (b) shows feature points F1' and F2' extracted from image 40(n') at time n'. [Figure 11] 4 is a diagram for explaining the calculation of the reliability of the relative position and the relative orientation calculated by the relative position calculator 32 in FIG. 3. [Figure 12] 9 is a flowchart showing a subroutine of step S4 (absolute position calculation process) in FIG. 8. [Figure 13] 13 is a diagram showing the coordinates of the vertices of a marker 4 in the marker coordinate system. FIG. [Figure 14] 2 is a diagram showing the coordinates of vertices of a marker 4 in an image 40 captured by the imaging device 11 of FIG. 1. FIG. [Figure 15] 4 is a diagram for explaining calculation of the reliability of the absolute position and the absolute attitude calculated by the absolute position calculator 34 in FIG. 3. [Figure 16] 2 is a graph illustrating that the absolute position error depends on the apparent size of the marker 4 in the image captured by the image capture device 11 of FIG. 1. [Figure 17] 2 is a graph illustrating that an error in absolute attitude depends on the apparent size of a marker 4 in an image captured by the image capturing device 11 of FIG. 1. [Figure 18] 10A to 10C are diagrams illustrating calculation of reliability of a relative position and a relative attitude in a positioning process according to a first modified example of the first embodiment. [Figure 19] 13A to 13C are diagrams illustrating calculation of reliability of a relative position and a relative attitude in a positioning process according to a second modified example of the first embodiment. [Figure 20] FIG. 13 is a diagram for explaining calculation of the reliability of a relative position and a relative attitude in the positioning process according to the third modified example of the first embodiment, and is a diagram showing a first trajectory of the vehicle 1. [Figure 21]21 is a graph showing an outline of the amount of change in the relative position or the relative attitude of the vehicle 1 in FIG. 20. [Figure 22] FIG. 13 is a diagram for explaining calculation of the reliability of a relative position and a relative attitude in the positioning process according to the third modified example of the first embodiment, and is a diagram showing a second trajectory of the vehicle 1. [Figure 23] 23 is a graph showing an outline of the amount of change in the relative position or the relative attitude of the vehicle 1 in FIG. 22. [Figure 24] 13A to 13C are diagrams illustrating calculation of the reliability of an absolute position and an absolute attitude in a positioning process according to a fourth modified example of the first embodiment. [Diagram 25] 2 is a diagram showing an exemplary arrangement of markers 4a-1 to 4a-12 photographed by the photographing device 11 of FIG. [Figure 26] 26 is a diagram showing an example of an image 40 obtained by photographing the markers 4a-1 to 4a-12 in FIG. 25 with the photographing device 11. FIG. [Figure 27] 2 is a diagram showing an exemplary arrangement of markers 4b-1 to 4b-12 photographed by the photographing device 11 of FIG. [Figure 28] 28 is a diagram showing an example of an image 40 obtained by photographing the markers 4b-1 to 4b-12 in FIG. 27 with the photographing device 11. FIG. [Figure 29] 1. FIG. 2 is a diagram for explaining the fluctuation in the position of the photographing device 11 in the marker coordinate system caused by erroneous detection of the angle of the surface of the marker 4 relative to the optical axis of the photographing device 11 in FIG. [Diagram 30] 13A to 13C are diagrams illustrating calculation of the reliability of an absolute position and an absolute attitude in a positioning process according to a fifth modified example of the first embodiment. [Diagram 31] FIG. 11 is a block diagram showing a configuration of a positioning device 12B according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following embodiments, similar components are denoted by the same reference numerals.

[0014] [First embodiment] A positioning device and a moving object including the same according to a first embodiment will be described below.

[0015] [Configuration of the first embodiment] [Overall configuration] FIG. 1 is a schematic diagram showing a configuration of a vehicle 1 according to a first embodiment. The vehicle 1 may be, for example, a forklift. The vehicle 1 includes a loading platform 1a on which luggage 3 is mounted. The vehicle 1 may further include a lifting mechanism 1b for loading and unloading the luggage 3 onto and from the loading platform 1a. The vehicle 1 also includes a console 1c that receives user operations such as forward movement, reverse movement, steering, and stopping. An imaging device 11 is installed on the body of the vehicle 1 so as to capture images of a predetermined direction (forward, backward, sideways, upward, and / or downward) relative to the vehicle 1.

[0016] Fig. 2 is a block diagram showing a configuration of a positioning system including the vehicle 1 of Fig. 1. The positioning system of Fig. 2 includes at least one vehicle 1 and a server device 2. Each vehicle 1 is equipped with a positioning device 12 that measures its position based on an image captured by an imaging device 11. The server device 2 acquires the positions of each vehicle 1 from the vehicles 1 and records the positions of each vehicle 1.

[0017] [Vehicle 1 Configuration] The vehicle 1 further includes an imaging device 11 , a positioning device 12 , a communication device 13 , a display device 14 , and a drive mechanism 15 .

[0018] The image capturing device 11 generates images of a subject in a predetermined orientation with respect to the vehicle 1 at predetermined time intervals while the vehicle 1 is moving. The image capturing device 11 includes, for example, at least one camera. The image capturing device 11 may capture still images at predetermined time intervals, or may extract frames at predetermined time intervals from a series of frames of a moving image. The image capturing device 11 sends the captured images to the positioning device 12. The image capturing device 11 assigns to each image a timestamp indicating the time when the image was captured.

[0019] The positioning device 12 calculates the position and attitude of the vehicle 1 based on the images captured by the image capture device 11. The positioning device 12 extracts feature points from multiple images captured by the image capture device 11, associates the extracted feature points between the images, and calculates the relative position and relative attitude of the vehicle 1 with respect to a predetermined reference position and reference attitude based on the amount of change in the feature points between the images. The positioning device 12 also extracts one of multiple markers that are arranged at a predetermined position and are visually identifiable from the images captured by the image capture device 11, and calculates the absolute position and absolute attitude of the vehicle 1 in a predefined map based on the one extracted marker. The positioning device 12 further calculates the reliability of the relative position and relative attitude and the reliability of the absolute position and absolute attitude, and determines the one with the higher reliability as the position and attitude of the vehicle 1. The reliability is represented by an estimated error calculated based on the images captured by the image capture device 11, for example, with reference to a model of theoretical error (i.e., statistical error from the true value of the position and attitude) determined experimentally in advance. The smaller the estimation error, the greater the reliability.

[0020] In this specification, the "attitude" of the vehicle 1 indicates, for example, the angle of the traveling direction of the vehicle 1 with respect to the coordinate axes of a predetermined coordinate system (the "world coordinate system" or "marker coordinate system" described later).

[0021] The communication device 13 includes a module such as Wi-Fi or Bluetooth and a control program for the module, and wirelessly communicates with the server device 2. The communication device 13 transmits the position and attitude of the vehicle 1 calculated by the positioning device 12 to the server device 2.

[0022] The display device 14 may display on a map the position of the vehicle 1. The display device 14 may also display warnings related to the operation of the vehicle 1, etc.

[0023] The drive mechanism 15 includes an engine or a motor, a steering device, a braking device, and control devices thereof of the vehicle 1. The drive mechanism 15 is controlled by a user via, for example, the console 1c.

[0024] [Server device 2 configuration] The server device 2 in FIG. 2 includes a processing device 21, a communication device 22, an input device 23, a storage device 24, and a display device 25. The processing device 21 is, for example, a general-purpose computer including a processor and a memory. The communication device 22 is communicatively connected to the communication device 13 of the vehicle 1. The input device 23 includes a keyboard, a pointing device, and the like. The storage device 24 records the position and attitude of the vehicle 1 received from the vehicle 1. The display device 25 displays the position and attitude of the vehicle 1 received from the vehicle 1. The processing device 21 acquires the positions of the vehicles 1 from the vehicles 1 via the communication device 22, records the positions of the vehicles 1 in the storage device 24, and displays the positions of the vehicles 1 on the display device 25.

[0025] The display device 25 displays the position and attitude of the vehicle 1 calculated by the positioning device 12 of the vehicle 1. The processing device 21 may acquire a map of the movement range of the vehicle 1 (such as a warehouse or a factory) in advance, and superimpose the position and attitude of the vehicle 1 calculated by the positioning device 12 on this map and display it on the display device 25. Alternatively, the processing device 21 may generate a map by itself based on the movement route of the vehicle 1, and display this map on the display device 25.

[0026] [Configuration of positioning device 12] Fig. 3 is a block diagram showing the configuration of the positioning device 12 of Fig. 2. The positioning device 12 includes an image processor 31, a relative position calculator 32, an image recognizer 33, an absolute position calculator 34, a storage device 35, a reliability calculator 36, and a position and attitude determiner 37.

[0027] The storage device 35 stores information on the identifiers, positions, and attitudes of a plurality of visually identifiable markers 4 arranged at predetermined positions, and information on a map (e.g., a map of a warehouse 100 described with reference to FIG. 4) including paths for the vehicle 1. The positions of the markers 4 may be represented as relative positions to a predetermined reference position and / or may be represented in association with the map.

[0028] 4 is a map of a warehouse 100 including an aisle 101 along which the vehicle 1 of FIG. 1 moves. The warehouse 100 includes structures such as a plurality of aisles 101 and a plurality of shelves 102. A plurality of markers 4 are pre-arranged at a plurality of predetermined positions in the warehouse 100. The vehicle 1 of FIG. 1 moves along the aisle 101 to transport luggage 3 from one shelf 102 to another shelf 102. The positions of the vehicle 1 and each marker 4 are represented by a world coordinate system (Xw, Yw, Zw) determined with respect to the entire warehouse 100.

[0029] FIG. 5 is a diagram showing an example of the marker 4 in FIG. 4, where FIG. 5(a) shows a front view of the marker 4 and FIG. 5(b) shows a top view of the marker 4. In the example of FIG. 5, the marker 4 is formed as a square flat plate. The marker 4 has a visually identifiable pattern that encodes the identifier of the marker 4 itself on one surface. In the example of FIG. 5, the marker 4 has a pattern consisting of 7×7 white or black square cells in a vertical and horizontal direction. The pattern of the marker 4 is further formed so that the attitude of the marker 4 itself can be detected from an image of the marker 4, such as a marker used in the field of augmented reality (also called an "AR marker"). Each marker 4 has a marker coordinate system (Xm, Ym, Zm) with an arbitrary point (for example, a center or one vertex) as the origin. In the lower part of FIG. 5 and other drawings, the front of the marker 4 (positive direction of the Zm axis) is indicated by an arrow at the center of the surface along the Xm-Ym plane.

[0030] FIG. 6 is a table showing an example of marker information stored in the storage device 35 of FIG. 3. In the example of FIG. 6, information of two markers 4 shown in FIG. 4 is shown. Each marker 4 has an identifier 001, 002. This identifier is encoded in the pattern of the marker 4. Each marker 4 has a predetermined coordinate in the world coordinate system (Xw, Yw, Zw). Each marker 4 is arranged in an orientation such that its front (positive direction of the Zm axis) has an angle θ (i.e., azimuth angle) with respect to the Xw axis in the Xw-Yw plane. The orientation of each marker 4 may be represented by an azimuth angle and an elevation angle. Each marker 4 has a square-shaped pattern and an actual size of 30 cm×30 cm. The size of the marker 4 indicates the size of the area of ​​the pattern and does not include margins around the pattern (not shown in FIG. 5).

[0031] The storage device 35 stores marker information including items such as those shown in Fig. 6 for all markers 4. The storage device 35 also stores map information for all passages 101 including their orientations, dimensions, and locations.

[0032] FIG. 7 is a diagram showing an example of an image 40 captured by the image capture device 11 of FIG. 1. The image 40 includes a plurality of feature points 41. The feature points 41 are points whose brightness values ​​or colors can be distinguished from surrounding pixels, and whose positions can be accurately determined. The feature points 41 are detected, for example, from vertices or edges of structures such as the passage 101 or shelves 102 along which the vehicle 1 moves, or patterns on the floor, wall, or ceiling. Furthermore, when the vehicle 1 passes near the marker 4, the image 40 includes the marker 4. The positions of the feature points 41 and the marker 4 in the image 40 are represented, for example, by an image coordinate system (Xi, Yi) with an arbitrary point (for example, the upper left corner) of the image 40 as the origin.

[0033] Referring again to FIG. 3, the image processor 31 extracts (tracks) the coordinates of corresponding feature points from a plurality of images captured by the image capture device 11 at a plurality of times separated by a predetermined time. The relative position calculator 32 calculates the movement amount of the vehicle 1 based on the movement amount of the feature points in two temporally adjacent images. As a result, the relative position calculator 32 calculates the relative position and relative orientation of the vehicle 1 with respect to a predetermined reference position and reference orientation (for example, the position and orientation when the vehicle 1 starts moving) based on the coordinates of the feature points in the plurality of images. The relative position calculator 32 may calculate the relative position and relative orientation of the vehicle 1 using a known image processing and positioning technique such as Visual-SLAM or Visual-Odometry. The reference position and reference orientation are associated with map information stored in the storage device 35. The relative position calculator 32 also assigns the timestamp of the image (the latter of the two temporally adjacent images) associated with the calculation to the relative position and relative orientation.

[0034] The relative position calculator 32 may express the calculated position of the vehicle 1, for example, in Cartesian coordinates (XYZ coordinates). The relative position calculator 32 may calculate the speed and / or acceleration of the vehicle 1 based on the calculated position of the vehicle 1 and the time. The relative position calculator 32 may express the calculated attitude of the vehicle 1 in terms of roll (left / right tilt), pitch (front / back tilt), and yaw (rotation around an axis perpendicular to the floor surface (i.e., the Zw axis in FIG. 4)). This makes it possible to express not only the orientation of the vehicle 1 in a horizontal plane parallel to the ground, but also the tilt of the body of the vehicle 1 and the movement of the vehicle 1 in the height direction.

[0035] In this specification, the image processor 31 and the relative position calculator 32 are collectively referred to as a "first calculator." Furthermore, in this specification, the relative position and the relative orientation are also referred to as a "first position" and a "first orientation."

[0036] The image recognizer 33 extracts one of a plurality of visually identifiable markers 4 arranged at predetermined positions from the image captured by the image capture device 11. The absolute position calculator 34 calculates the absolute position and absolute attitude of the vehicle 1 indicating the position and attitude of the vehicle 1 in the map (i.e., the world coordinate system) based on the position and attitude of the extracted one marker 4, by referring to the information of the marker 4 and the map information stored in the storage device 35. The absolute position calculator 34 also assigns a timestamp of the image associated with these calculations to the absolute position and absolute attitude.

[0037] In this specification, the image recognizer 33 and the absolute position calculator 34 are collectively referred to as a "second calculator." Furthermore, in this specification, the absolute position and the absolute orientation are also referred to as a "second position" and a "second orientation."

[0038] The reliability calculator 36 calculates the reliability of the relative position and relative attitude calculated by the relative position calculator 32 and the reliability of the absolute position and absolute attitude calculated by the absolute position calculator 34. The reliability calculator 36 calculates the reliability of the relative position and relative attitude, for example, so that the reliability increases as the moving distance and rotation angle from the reference position and reference attitude of the vehicle 1 become smaller, and the reliability decreases as the moving distance and rotation angle become larger. For example, as described below, when the position and attitude determiner 37 determines the absolute position and absolute attitude as the position and attitude of the vehicle 1, the reference position and reference attitude of the vehicle 1 are reset. In other words, the absolute position and absolute attitude when the reliability of the absolute position and absolute attitude exceeds the reliability of the relative position and relative attitude are set as the reference position and reference attitude of the vehicle 1. In addition, the reliability calculator 36 calculates the reliability of the absolute position and absolute orientation so that, for example, the reliability increases as the apparent size of the marker 4 in the image captured by the photographing device 11 increases, and the reliability decreases as the apparent size of the marker 4 in the image decreases.

[0039] The position and attitude determiner 37 determines the relative position and relative attitude as the position and attitude of the vehicle 1 when the reliability of the relative position and relative attitude is equal to or greater than the reliability of the absolute position and absolute attitude. Moreover, the position and attitude determiner 37 determines the absolute position and absolute attitude as the position and attitude of the vehicle 1 when the reliability of the relative position and relative attitude is smaller than the reliability of the absolute position and absolute attitude. The position and attitude determiner 37 synchronizes the absolute position and absolute attitude with the relative position and relative attitude based on the timestamps of the relative position and relative attitude and the timestamps of the absolute position and absolute attitude. For example, the position and attitude determiner 37 may consider the relative position and relative attitude and the absolute position and absolute attitude having the closest timestamps, having a time difference smaller than a predetermined threshold, as being calculated from the same image.

[0040] At least some of the components 31 to 37 of the positioning device 12 may be integrated. For example, the image processor 31 and the image recognizer 33 may be integrated. Furthermore, the components 31 to 37 of the positioning device 12 may be implemented as a dedicated circuit, or may be implemented as a program executed by a general-purpose processor.

[0041] [Operation of the first embodiment] According to the positioning device 12, when the error in the relative position and relative attitude calculated using Visual-SLAM increases, the position and attitude of the vehicle 1 can be corrected using the absolute position and absolute attitude calculated based on the marker 4.

[0042] However, when calculating the relative position and relative orientation of the vehicle 1 using Visual-SLAM, the error in the relative position and relative orientation increases as the moving distance and rotation angle of the vehicle 1 from the reference position and reference orientation increase. In addition, when calculating the absolute position and absolute orientation of the vehicle 1 based on the marker 4, the error in the absolute position and absolute orientation increases depending on the conditions under which the marker 4 is photographed by the imaging device 11. For example, the error in the absolute position and absolute orientation increases as the distance from the imaging device 11 to the marker 4 increases. In addition, when the angle of the surface of the marker 4 with respect to the optical axis of the imaging device 11 is close to 0 degrees, the error in the absolute position and absolute orientation increases. In addition, when the marker 4 is located at the edge of the photographed image, the error in the absolute position and absolute orientation may increase.

[0043] When the errors in the absolute position and attitude are large, it is desirable not to use the absolute position and attitude as the position and attitude of the vehicle 1. On the other hand, when the errors in the relative position and attitude have accumulated too much, it is desirable to determine the absolute position and attitude as the position and attitude of the vehicle 1 even if the errors in the absolute position and attitude are large to a certain extent. In this case, it is necessary to evaluate the magnitude of the errors in the relative position and attitude and the magnitude of the errors in the absolute position and attitude.

[0044] Therefore, the positioning device 12 according to the embodiment calculates the reliability of the relative position and relative attitude and the reliability of the absolute position and absolute attitude, and determines the position and attitude of the vehicle 1 based on the one with the higher reliability.

[0045] The operation of the positioning device 12 will now be described in detail.

[0046] [Overall positioning process] FIG. 8 is a flowchart showing the positioning process executed by the positioning device 12 of FIG.

[0047] In step S1, the positioning device 12 acquires an image captured by the image capturing device 11.

[0048] In step S2, the image processor 31 and the relative position calculator 32 execute a relative position calculation process to calculate the relative position and relative orientation of the vehicle 1. In step S3, the reliability calculator 36 calculates the reliability Q1 of the relative position and relative orientation of the vehicle 1.

[0049] In step S4, the image recognizer 33 and the absolute position calculator 34 execute an absolute position calculation process to calculate the absolute position and absolute attitude of the vehicle 1. In step S5, the reliability calculator 36 calculates the reliability Q2 of the absolute position and absolute attitude of the vehicle 1.

[0050] Steps S2 to S5 may be executed in parallel as shown in FIG. 8, or may be executed sequentially.

[0051] In step S6, the confidence calculator 36 judges whether or not at least one of "confidence of absolute position>confidence of relative position" and "confidence of absolute attitude>confidence of relative attitude" is satisfied, and if YES, the process proceeds to step S7, and if NO, the process proceeds to step S9. The position and attitude determiner 37 may execute step S6 instead of the confidence calculator 36.

[0052] In step S7, the position and attitude determiner 37 determines the absolute position and absolute attitude of the vehicle 1 as the position and attitude of the vehicle 1. The position and attitude determiner 37 outputs the determined position and attitude of the vehicle 1 to the communication device 13 and the display device 14. In step S8, the position and attitude determiner 37 resets the reference for the relative position and relative attitude of the vehicle 1.

[0053] In step S9, the position and attitude determiner 37 determines the relative position and device attitude of the vehicle 1 as the position and attitude of the vehicle 1. The position and attitude determiner 37 outputs the determined position and attitude of the vehicle 1 to the communication device 13 and the display device 14.

[0054] After steps S8 and S9, the process returns to step S1 to repeat the process.

[0055] [Relative position calculation process] FIG. 9 is a flowchart showing a subroutine of step S2 (relative position calculation process) in FIG.

[0056] In step S11, the image processor 31 acquires first and second images taken at first and second times, respectively, that are separated by a predetermined time (eg, frames adjacent in time).

[0057] In step S12, the image processor 31 detects feature points from the first image. In order to detect feature points from the image, for example, an image processing technique such as FAST (Features from Accelerated Segment Test) may be used.

[0058] In step S13, the image processor 31 detects feature points from the second image that correspond to the feature points from the first image. In order to detect corresponding feature points between images, a known image processing technique such as a KLT (Kanade-Lucas-Tomasi) tracker may be used.

[0059] FIG. 10 is a diagram showing feature points extracted by the image processor 31 of FIG. 3. FIG. 10(a) shows feature points F1 and F2 extracted from an image 40(n) at time n. FIG. 10(b) shows feature points F1' and F2' extracted from an image 40(n') at time n'. In the image coordinate system of the image 40(n) of FIG. 10(a), the feature point F1 has coordinates (xi1, yi1), and the feature point F2 has coordinates (xi2, yi2). In the image coordinate system of the image 40(n') of FIG. 10(b), the feature point F1' has coordinates (xi1', yi1'), and the feature point F2' has coordinates (xi2', yi2'). The feature points F1' and F2' of FIG. 10(b) correspond to the feature points F1 and F2 of FIG. 10(a), respectively.

[0060] 9, the image processor 31 acquires a set of coordinates of corresponding feature points in the first and second images. For example, the image processor 31 acquires a set of coordinates (xi1, yi1, xi1', yi1') of feature points F1 and F1', and acquires a set of coordinates (xi2, yi2, xi2', yi2') of feature points F2 and F2'.

[0061] In step S15, the relative position calculator 32 calculates a fundamental matrix E consisting of 3×3 elements, for example, using a five-point algorithm, based on the coordinates of the feature points acquired in step S14.

[0062] In step S16, the relative position calculator 32 performs singular value decomposition on the fundamental matrix E to calculate a rotation matrix R and a translation vector t that represent the movement of the vehicle 1 between the times when the first and second images were captured. The rotation matrix R indicates the change in attitude of the vehicle 1 between the times when the first and second images were captured. The translation vector t indicates the change in position of the vehicle 1 between the times when the first and second images were captured.

[0063] The calculation of the rotation matrix R and the translation vector t is formulated, for example, as follows:

[0064] The fundamental matrix E is obtained by singular value decomposition as E=UΣV T Here, Σ is a diagonal matrix Σ consisting of 3 × 3 elements, and U and V are orthogonal matrices consisting of 3 × 3 elements.

[0065] The rotation matrix R is calculated by using the following matrix W, which consists of 3 × 3 elements: R = UW -1 V T It is calculated as follows:

[0066]

number

[0067] In addition, to calculate the translation vector t, we use a 3×3 matrix T=VWΣV TThe matrix T satisfies E=TR and is expressed by the following equation.

[0068]

number

[0069] The translation vector t is expressed as t=(t x ,t y ,t z ) T It is expressed as:

[0070] In step S17, the relative position calculator 32 calculates the relative position and relative attitude of the vehicle 1. When the vehicle 1 has a relative position t(n-1) and a relative attitude R(n-1) at the immediately preceding time n-1, the relative position t(n) of the vehicle 1 at the current time n is expressed by t(n)=t(n-1)+tR(n-1) using the translation vector t calculated in step S16. Also, the relative attitude R(n) of the vehicle 1 at the current time n is expressed by R(n)=RR(n-1) using the rotation matrix R calculated in step S16. In this way, the relative position calculator 32 cumulatively adds a plurality of translation vectors and cumulatively multiplies a plurality of rotation matrices to calculate the relative position and relative attitude of the vehicle 1 with respect to a predetermined reference position and reference attitude. The relative position calculator 32 sends the calculated relative position and relative attitude of the vehicle 1 to the position and attitude determiner 37.

[0071] [Reliability of relative position and orientation] FIG. 11 is a diagram for explaining the calculation of the reliability of the relative position and relative orientation calculated by the relative position calculator 32 in FIG.

[0072] The reliability calculator 36 calculates the reliability of the relative position and relative attitude, for example, so that the reliability increases as the moving distance and rotation angle from the reference position and reference attitude of the vehicle 1 become smaller, and the reliability decreases as the moving distance and rotation angle become larger. The reliability of the relative position and relative attitude is represented by an estimated error calculated based on the image captured by the image capture device 11, with reference to a model of theoretical error (i.e., statistical error from the true values ​​of the position and attitude) determined in advance through experiments. The reliability increases as the estimated error becomes smaller.

[0073] As a preliminary experiment, the vehicle 1 is actually driven under various conditions, and the difference between the position and attitude of the vehicle 1 on the actual route and the position and attitude of the vehicle 1 on the route estimated using Visual-SLAM is calculated. This allows the theoretical errors of Visual-SLAM, i.e., the theoretical error Era of the moving distance of the vehicle 1 and the theoretical error Erb of the rotation angle during the movement of the vehicle 1, to be obtained. Here, the theoretical error Era of the moving distance indicates, for example, the ratio of the error of the moving distance estimated using Visual-SLAM to the actual moving distance calculated and averaged under various conditions. The theoretical error Erb of the rotation angle indicates, for example, the ratio Erb of the error of the rotation angle estimated using Visual-SLAM to the actual rotation angle calculated and averaged under various conditions.

[0074] In the example of FIG. 11, the vehicle 1 actually travels along a path from point P0 to point P1. The actual path P0→P1 has, for example, a length d1=50 m and includes two 90-degree corners. Thus, the vehicle 1 rotates through A1=|90 degrees|×2=180 degrees while traveling along the path P0→P1. Furthermore, when the vehicle 1 arrives at point P1, the position of the vehicle 1 estimated using Visual-SLAM is at point P2. Points P1 and P2 have a position error E1aa=3.85 m along the traveling direction of the vehicle 1 and a position error E1ab=1.6 m along the direction perpendicular to the traveling direction of the vehicle 1. Thus, the path P0→P2 estimated using Visual-SLAM has a length d2=((d1-E1aa) 2 +E1ab 2 ) 1 / 2=((50m-3.85m) 2 +1.6 2 ) 1 / 2 = 46.2 m. In addition, the path P0 → P1 and the path P0 → P2 have an angle error E1b = arcsin(E1ab / d2) = arcsin(1.6 / 46.2) = 2 degrees. In this case, the theoretical errors of Visual-SLAM, that is, the theoretical error Era of the moving distance and the theoretical error Erb of the rotation angle, are calculated as follows:

[0075] Era=E1aa / d1=3.85 / 50=0.077 Erb=E1b / A1=2 / 180=0.011

[0076] Here, the theoretical error Era of the travel distance is calculated based on the component of the travel distance along the traveling direction of the vehicle 1.

[0077] Given such a theoretical error model, which has been experimentally determined in advance, the estimation error of the relative position and relative attitude of the vehicle 1 when the vehicle 1 moves along an arbitrary path is calculated as follows:

[0078] Consider a case where the theoretical errors of Visual-SLAM are given as Era=0.077 and Erb=0.011, and a vehicle 1 travels along a path P0→P1 having a length d1=50m and including two 90-degree corners. The length d1=50m and rotation angle A1=180 degrees of the actual path P0→P1 are unknown. On the other hand, it is known that the path P0→P2 estimated using Visual-SLAM has, for example, a length d2=46.2m and a rotation angle A2=182 degrees. Therefore, the estimation error E1a of the relative position is calculated as follows:

[0079] E1b=A2×Erb=182 degrees×0.011=2 degrees E1ab=d2×sin(E1b)=46.2m×sin(2 degrees)=1.6m E1aa=d2×cos(E1b)×Era / (1±Era)=46.2m×cos(2 degrees)×0.077 / (1±0.077)=3.3m,3.85m E1a=(E1aa2 +E1ab 2 ) 1 / 2 =(3.85 2 +1.6 2 ) 1 / 2 =4.2m

[0080] On the other hand, the estimation error of the relative attitude is equal to the estimation error E1b of the angle between the path P0→P1 and the path P0→P2.

[0081] As the moving distance and rotation angle of vehicle 1 from the reference position and reference attitude increase, the estimation error of the relative position and relative attitude increases (i.e., the reliability of the relative position and relative attitude decreases). Also, as the moving distance and rotation angle decrease, the estimation error of the relative position and relative attitude decreases (i.e., the reliability of the relative position and relative attitude increases). Therefore, the reliability of the relative position and relative attitude can be expressed using the estimation error of the relative position and relative attitude.

[0082] [Absolute position calculation process] FIG. 12 is a flowchart showing a subroutine of step S4 (absolute position calculation process) in FIG.

[0083] In step S21, the image recognizer 33 detects the marker 4 from the image. Here, the image recognizer 33 detects the coordinates of the four vertices (corners) of the quadrangular marker 4 in the image coordinate system, and also decodes the pattern of the marker 4 to obtain the identifier of the marker 4. Note that the image recognizer 33 may detect the coordinates of several predetermined points instead of the four vertices of the marker 4.

[0084] 13 is a diagram showing the coordinates of the vertices of the marker 4 in the marker coordinate system. In the marker coordinate system (Xm, Ym, Zm), the four vertices of the marker 4 have coordinates (xm0, ym0, zm0), (xm1, ym1, zm1), (xm2, ym2, zm2), and (xm3, ym3, zm3), respectively. Since the dimensions of the marker 4 are known, the coordinates of the four vertices of the marker 4 in the marker coordinate system are also known. For example, if the top left vertex of marker 4 in Figure 13 is the origin of the marker coordinate system (Xm, Ym, Zm) and marker 4 has an actual size of 30 cm x 30 cm, the vertices of marker 4 have, for example, coordinates (xm0, ym0, zm0) = (0, 0, 0), (xm1, ym1, zm1) = (0.3, 0, 0), (xm2, ym2, zm2) = (0, 0.3, 0), and (xm3, ym3, zm3) = (0.3, 0.3, 0).

[0085] Fig. 14 is a diagram showing the coordinates of the vertices of the marker 4 in the image 40 captured by the imaging device 11 in Fig. 1. In the image coordinate system (Xi, Yi), the four vertices of the marker 4 have coordinates (xi0, yi0), (xi1, yi1), (xi2, yi2), and (xi3, yi3), respectively.

[0086] In step S22 of FIG. 12, the absolute position calculator 34 obtains the apparent size of the marker 4 in the image captured by the image capture device 11. When the vehicle 1 moves mainly in a horizontal plane (i.e., in the Xw-Yw plane) in the world coordinate system, the apparent size of the marker 4 in the image may be represented, for example, by the length (number of pixels) of the marker 4 in the vertical direction (i.e., the Yi direction). Alternatively, the apparent size of the marker 4 in the image may be represented by a combination of the vertical and horizontal lengths of the marker 4. The apparent size of the marker 4 in the image corresponds to the distance from the image capture device 11 to the marker 4. Note that the horizontal length of the marker 4 (i.e., the Xi direction) varies not only depending on the distance but also depending on the angle at which the marker 4 is captured by the image capture device 11.

[0087] In step S23, the absolute position calculator 34 calculates the position and orientation of the marker 4 in a three-dimensional coordinate system (camera coordinate system) with the image capturing device 11 as the origin (i.e., the position and orientation of the marker 4 as seen from the image capturing device 11) based on the coordinates of the marker 4 detected in step S21. For example, the absolute position calculator 34 calculates the position and orientation of the marker 4 as seen from the image capturing device 11 by solving a perspective n point (PnP) problem based on the coordinates of the four vertices of the marker 4 in the two-dimensional image coordinate system and the coordinates of the four vertices of the marker 4 in the three-dimensional marker coordinate system.

[0088] In step S24, the absolute position calculator 34 calculates the position and orientation of the camera 11 in the marker coordinate system (i.e., the position and orientation of the camera 11 as seen from the marker 4). Here, the position of the marker 4 as seen from the camera 11 is represented by a translation vector t, and the orientation of the marker 4 as seen from the camera 11 is represented by a rotation matrix R. In this case, the orientation of the camera 11 as seen from the marker 4 is expressed as R -1 The position of the camera 11 as viewed from the marker 4 is represented by -R -1 It is represented by t.

[0089] In step S25, the absolute position calculator 34 reads out, from the storage device 35, the position and orientation of the marker 4 in the world coordinate system (ie, the absolute position and absolute orientation of the marker 4) based on the identifier of the marker 4 detected in step S21.

[0090] In step S26, the absolute position calculator 34 calculates the position and attitude of the vehicle 1 in the world coordinate system (i.e., the absolute position and attitude of the vehicle 1) based on the position and attitude of the camera 11 in the marker coordinate system calculated in step S24 and the position and attitude of the marker 4 in the world coordinate system read out in step S25. The position and attitude of the vehicle 1 in the world coordinate system are obtained by adding the position and attitude of the camera 11 in the marker coordinate system as an offset value. The absolute position calculator 34 sends the calculated absolute position and attitude of the vehicle 1 to the position and attitude determiner 37.

[0091] [Confidence of absolute position and attitude] FIG. 15 is a diagram for explaining the calculation of the reliability of the absolute position and the absolute attitude calculated by the absolute position calculator 34 in FIG.

[0092] The reliability calculator 36 calculates the reliability of the absolute position and absolute orientation so that, for example, the reliability increases as the apparent size of the marker 4 in the image captured by the image capture device 11 increases, and the reliability decreases as the apparent size of the marker 4 in the image decreases. The reliability of the absolute position and absolute orientation is represented by an estimated error calculated based on the image captured by the image capture device 11, with reference to a model of theoretical error (i.e., statistical error from the true values ​​of the position and orientation) determined in advance through experiments. The reliability increases as the estimated error decreases.

[0093] As a preliminary experiment, the marker 4 is photographed by the photographing device 11 under various conditions, and the error between the actual position and attitude of the vehicle 1 and the absolute position and attitude of the vehicle 1 calculated based on the marker 4 is calculated. At this time, the marker 4 is photographed by the photographing device 11 from a plurality of predetermined observation points located at various distances and angles relative to the marker 4. The observation points may be, for example, lattice points having a predetermined interval. The marker 4 may be photographed while changing the orientation of the photographing device 11.

[0094] The example of Fig. 15 shows an image capturing device 11-1 relatively close to the marker 4, and an image capturing device 11-2 relatively far from the marker 4. A vehicle 1 equipped with the image capturing device 11-1 has an absolute position error E2a-1 and an absolute attitude error E2b-1. A vehicle 1 equipped with the image capturing device 11-2 has an absolute position error E2a-2 larger than the absolute position error E2a-1, and an absolute attitude error E2b-2 larger than the absolute attitude error E2b-1. Therefore, according to Fig. 15, the errors in the absolute position and absolute attitude are considered to depend on the distance from the image capturing device 11 to the marker 4.

[0095] In the example of FIG. 15, the absolute attitude errors E2b-1, E2b-2 are shown in relation to the marker coordinate system, but as explained in step S26 of FIG. 12, the attitude in the marker coordinate system is converted into the attitude in the world coordinate system.

[0096] FIG. 16 is a graph for explaining that the error in the absolute position depends on the apparent size of the marker 4 in the image captured by the camera device 11 of FIG. 1. FIG. 17 is a graph for explaining that the error in the absolute attitude depends on the apparent size of the marker 4 in the image captured by the camera device 11 of FIG. 1. The points plotted in FIG. 16 and FIG. 17 show the results of the above-mentioned preliminary experiment. Here, the apparent size of the marker 4 in the image captured by the camera device 11 changes depending on the distance from the camera device 11 to the marker 4. According to FIG. 16 and FIG. 17, it is considered that there is a correlation between the vertical length of the marker 4 and the errors in the absolute position and absolute attitude. Therefore, a curve is fitted to the graphs of FIG. 16 and FIG. 17, and the theoretical error E2a of the absolute position and the theoretical error E2b of the absolute attitude are expressed, for example, as follows based on the fitted curve.

[0097] E2a=40 / vertical length (number of pixels) E2b=400 / vertical length (number of pixels)

[0098] Given a model of the theoretical error thus experimentally determined in advance, when the vehicle 1 moves along an arbitrary path, the theoretical error E2a of the absolute position and the theoretical error E2b of the absolute attitude can be calculated as the estimated errors of the absolute position and absolute attitude.

[0099] The smaller the apparent size of the marker 4 in the image captured by the imaging device 11, the greater the estimation error of the absolute position and orientation (i.e., the lower the reliability of the absolute position and orientation). Also, the larger the apparent size of the marker 4 in the image, the lower the estimation error of the absolute position and orientation (i.e., the higher the reliability of the absolute position and orientation). Therefore, the reliability of the absolute position and orientation can be expressed using the estimation error of the absolute position and orientation.

[0100] According to the first embodiment, it is possible to measure with high accuracy the position and attitude of the vehicle 1 by calculating the reliability of the relative position and relative attitude and the reliability of the absolute position and absolute attitude, and determining the one with the higher reliability as the position and attitude of the vehicle 1. According to the first embodiment, for example, it is possible to calculate the estimation error of the relative position and relative attitude as the reliability of the relative position and relative attitude, calculate the estimation error of the absolute position and absolute attitude as the reliability of the absolute position and absolute attitude, and determine the one with the smaller estimation error as the position and attitude of the vehicle 1.

[0101] According to the first embodiment, by using the image capturing device 11, the position of the vehicle 1 can be measured inexpensively even indoors where radio waves from GPS satellites cannot be received, such as in a warehouse or a factory. Since there is no need to install a large number of wireless transmitters for transmitting wireless signals, the initial implementation cost can be reduced.

[0102] According to the first embodiment, business operations can be improved based on the movement route of the vehicle 1 obtained from the positioning result.

[0103] According to the first embodiment, the travel distance of the vehicle 1 can be used to determine whether maintenance is required, whether the lease contract needs to be renewed, and the like.

[0104] According to the first embodiment, the layout of aisles, shelves, etc. in a warehouse or factory can be optimized based on a heat map of the movement of the vehicle 1.

[0105] According to the first embodiment, based on the differences in the trajectories of each vehicle 1, it is possible to visualize the locations where each vehicle 1 passes each other while moving, and it is possible to review the route and passage width to improve safety.

[0106] [First Modification of the First Embodiment] 18 is a diagram for explaining the calculation of the reliability of the relative position and the relative attitude in the positioning process according to the first modified example of the first embodiment. Instead of calculating the reliability of the relative position and the relative attitude based on the moving distance and the rotation angle of the vehicle 1 as described with reference to FIG. 11, the reliability calculator 36 may calculate the reliability of the relative position and the relative attitude as follows.

[0107] The reliability calculator 36 calculates the reliability of the relative position and orientation so that the reliability increases as the difference between the relative position and orientation and the absolute position and orientation decreases, and the reliability decreases as the difference between the relative position and orientation and the absolute position and orientation increases. In this case, the reliability of the relative position and orientation is represented by the difference between the relative position and orientation and the absolute position and orientation (also called the "estimation error of the relative position and orientation").

[0108] In Fig. 18, vehicle 1 has an absolute position P10 and absolute orientation A10 calculated based on marker 4. Vehicle 1 also has a relative position P11 and relative orientation A11 (shown as "vehicle 1'" in Fig. 18) or a relative position P12 and relative orientation A12 (shown as "vehicle 1" in Fig. 18) calculated using Visual-SLAM. Vehicle 1 has an absolute position estimation error E2a and absolute orientation estimation error E2b.

[0109] The reliability calculator 36 calculates the distance between the relative position and the absolute position of the vehicle 1 as the estimation error of the relative position of the vehicle 1. When the distance between the relative position and the absolute position of the vehicle 1 is greater than the estimation error E2a of the absolute position, the position and orientation determiner 37 determines the absolute position as the position of the vehicle 1. Also, when the distance between the relative position and the absolute position of the vehicle 1 is less than or equal to the estimation error E2a of the absolute position, the position and orientation determiner 37 determines the relative position as the position of the vehicle 1. Therefore, in the example of FIG. 18, when the vehicle 1 has the relative position P11, the relative position P11 of the vehicle 1 has the estimation error d11 < E2a, and the position and orientation determiner 37 determines the relative position P11 as the position of the vehicle 1. On the other hand, when the vehicle 1 has the relative position P12, the relative position P12 of the vehicle 1 has the estimation error d12 > E2a, and the position and orientation determiner 37 determines the absolute position P10 as the position of the vehicle 1.

[0110] The reliability calculator 36 calculates the angular difference between the relative orientation and the absolute orientation of the vehicle 1 as the estimation error of the relative orientation of the vehicle 1. When the angular difference between the relative orientation and the absolute orientation of the vehicle 1 is greater than the estimation error E2b of the absolute orientation, the position and orientation determiner 37 determines the absolute orientation as the orientation of the vehicle 1. Also, when the angular difference between the relative orientation and the absolute orientation of the vehicle 1 is less than or equal to the estimation error E2b of the absolute orientation, the position and orientation determiner 37 determines the relative orientation as the orientation of the vehicle 1. Therefore, in the example of FIG. 18, when the vehicle 1 has the relative orientation A11, the relative orientation A11 of the vehicle 1 has the estimation error θ1 < E2b, and the position and orientation determiner 37 determines the relative orientation A11 as the orientation of the vehicle 1. On the other hand, when the vehicle 1 has the relative orientation A12, the relative orientation A12 of the vehicle 1 has the estimation error θ2 > E2b, and the position and orientation determiner 37 determines the absolute orientation A10 as the orientation of the vehicle 1.

[0111] The larger the difference between the absolute position and the relative orientation and the relative position and orientation, the larger the estimation error of the relative position and the relative orientation (i.e., the reliability of the relative position and the relative orientation decreases). Also, the smaller the difference between the absolute position and the relative orientation and the relative position and orientation, the smaller the estimation error of the relative position and the relative orientation (i.e., the reliability of the relative position and the relative orientation increases). Therefore, the reliability of the relative position and the relative orientation can be expressed using the estimation error of the relative position and the relative orientation.

[0112] In the example of FIG. 18, the estimation errors of the absolute position and attitude are calculated in the same manner as in the case described with reference to FIGS.

[0113] According to the first variant of the first embodiment, the estimation error of the relative position and relative attitude and the estimation error of the absolute position and absolute attitude are calculated, and the estimation error with the smaller estimation error (i.e., the estimation error with the higher reliability) is determined as the position and attitude of the vehicle 1, thereby making it possible to measure the position and attitude of the vehicle 1 with high accuracy.

[0114] [Second Modification of the First Embodiment] Fig. 19 is a diagram for explaining calculation of the reliability of the relative position and the relative attitude in the positioning process according to the second modification of the first embodiment. The reliability calculator 36 may take the reliability of tracking into consideration when calculating the estimation error of the relative position and the relative attitude as described with reference to Fig. 11 or Fig. 18.

[0115] The relative position calculator 32 calculates the relative position and the relative orientation based on the feature points tracked in the multiple images adjacent in time. The reliability calculator 36 calculates the reliability of the tracking based on the number of feature points successfully tracked in the multiple images adjacent in time, and calculates the reliability of the relative position and the relative orientation so that the reliability of the relative position and the relative orientation increases as the reliability of the tracking increases. In this case, the reliability of the relative position and the relative orientation is represented by a value obtained by multiplying the estimation error of the relative position and the relative orientation calculated in the same manner as in the case described with reference to Fig. 11 or Fig. 18 by a coefficient indicating the reliability of the tracking.

[0116] With reference to Fig. 19(a), feature points F1 to F4 are detected in image 40(n). With reference to Fig. 19(b), feature point F2' corresponding to feature point F2 in Fig. 19(a) and feature point F4' corresponding to feature point F4 in Fig. 19(a) are detected in image 40(n'), but detection of feature points corresponding to feature points F1 and F3 in Fig. 19(a) fails.

[0117] The estimated error E1a' of the relative position and the estimated error E1b' of the relative attitude taking into account tracking are calculated by the following equations.

[0118] E1a' = λ × (1 / Tf) × E1a E1b' = λ × (1 / Tf) × E1b

[0119] where λ is a predetermined constant. Tf indicates the reliability of tracking and is calculated by the following formula:

[0120] Tf = total number of successfully tracked feature points / distance traveled by vehicle 1 (m)

[0121] For example, if vehicle 1 travels a distance of 10 m, 40,000 feature points are successfully tracked in two temporally adjacent images, and λ=3000, the estimation error of the relative position and relative orientation is calculated as follows:

[0122] Tf=40000 / 10=4000 E1a' = 3000 × (1 / 4000) × E1a E1b' = 3000 × (1 / 4000) × E1b

[0123] The estimated error E1a of the relative position and the estimated error E1b of the relative attitude are calculated in the same manner as described with reference to FIG. 11 or FIG.

[0124] The lower the reliability of tracking, the greater the estimation error of the relative position and relative orientation (i.e., the reliability of the relative position and relative orientation decreases). Also, the higher the reliability of tracking, the smaller the estimation error of the relative position and relative orientation (i.e., the reliability of the relative position and relative orientation increases). Therefore, the reliability of the relative position and relative orientation can be expressed using the estimation error of the relative position and relative orientation that takes into account the reliability of tracking.

[0125] The reliability of the absolute position and the absolute attitude is calculated in the same manner as described with reference to FIGS.

[0126] According to the second modification of the first embodiment, the position and attitude of the vehicle 1 can be measured with higher accuracy by taking into account the reliability of tracking.

[0127] [Third Modification of the First Embodiment] The reliability calculator 36 may calculate the variance of the change in the relative position and relative orientation as the reliability of the relative position and relative orientation, instead of calculating the estimation error of the relative position and relative orientation as in the case described with reference to FIG. 11, FIG. 18 or FIG. 19.

[0128] The reliability calculator 36 calculates the reliability of the relative position and the relative orientation so that the reliability increases as the variance of the change amount of the relative position or the relative orientation decreases, and the reliability decreases as the variance of the change amount of the relative position or the relative orientation increases. In this case, the reliability of the relative position and the relative orientation is represented by the variance of the change amount of the relative position or the relative orientation.

[0129] Fig. 20 is a diagram for explaining calculation of reliability of relative position and relative attitude in positioning processing according to a third modified example of the first embodiment, and is a diagram showing a first trajectory of vehicle 1. Fig. 21 is a graph showing an outline of the amount of change in the relative position or relative attitude of vehicle 1 in Fig. 20. As shown in Fig. 20, at times n-2 to n+2, vehicle 1 has relative positions P(n-2) to P(n+2) and relative attitudes A(n-2) to A(n+2). In this case, the amount of change in the relative position and relative attitude is small, and the variance is also small.

[0130] Fig. 22 is a diagram for explaining calculation of reliability of relative position and relative attitude in positioning processing according to a third modified example of the first embodiment, and is a diagram showing a second trajectory of the vehicle 1. Fig. 23 is a graph showing a schematic representation of the amount of change in the relative position or relative attitude of the vehicle 1 in Fig. 22. As shown in Fig. 22, at times n-2 to n+2, the vehicle 1 has relative positions P(n-2) to P(n+2) and relative attitudes A(n-2) to A(n+2). In this case, the amount of change in the relative position and relative attitude varies more greatly from frame to frame than in Fig. 20, and the variance is also larger than in Fig. 20.

[0131] When the amount of change in the relative position and the relative orientation varies greatly from frame to frame, it is considered difficult to calculate the relative position and the relative orientation with high accuracy. The greater the variance of the amount of change in the relative position or the relative orientation, the lower the reliability of the relative position and the relative orientation. Also, the smaller the variance of the amount of change in the relative position or the relative orientation, the higher the reliability of the relative position and the relative orientation. Therefore, the reliability of the relative position and the relative orientation can be expressed using the variance of the amount of change in the relative position and the relative orientation.

[0132] The reliability of the absolute position and the absolute attitude is calculated in the same manner as described with reference to FIGS.

[0133] According to the third variant of the first embodiment, the change in variance of the relative position and relative attitude is calculated as the reliability of the relative position and relative attitude, and the estimation error of the absolute position and absolute attitude is calculated as the reliability of the absolute position and absolute attitude, and the one with the higher reliability can be determined as the position and attitude of vehicle 1.

[0134] [Fourth Modification of the First Embodiment] 24 is a diagram for explaining calculation of the reliability of the absolute position and the absolute attitude in the positioning process according to the fourth modified example of the first embodiment. The reliability calculator 36 may take into consideration the position of the marker 4 in the image captured by the image capturing device 11 when calculating the estimation error of the absolute position and the absolute attitude as described with reference to FIGS.

[0135] 24, in an image 40 captured by the image capture device 11, the marker 4 is located at a distance d21 from the center P21 of the image 40. The reliability calculator 36 calculates the reliability of the absolute position and orientation in the image 40 captured by the image capture device 11 such that the reliability increases as the distance d21 from the center P21 of the image 40 to the marker 4 decreases, and the reliability decreases as the distance d21 increases. In this case, the reliability of the absolute position and orientation is represented by a value obtained by multiplying the estimation error of the absolute position and orientation calculated in the same manner as in the case described with reference to FIGS. 15 to 17 by a coefficient including the distance d21 from the center P21 of the image 40 to the marker 4.

[0136] The estimation error E2a' of the absolute position and the estimation error E2b' of the absolute attitude taking into account the position of the marker 4 in the image are expressed by the following equations.

[0137] E2a' = λ × d21 × E2a E2b' = λ × d21 × E2b

[0138] Here, λ is a predetermined constant. The estimated error E2a of the absolute position and the estimated error E2b of the absolute attitude are calculated in the same manner as described with reference to FIGS.

[0139] The reliability of the relative position and the relative orientation is calculated in the same manner as described with reference to FIGS.

[0140] Depending on the position of the marker 4 in the image captured by the image capture device 11, it may be difficult to distinguish between markers 4 having surfaces at different angles to the optical axis of the image capture device 11. For example, at the edge of the image captured by the image capture device 11, it becomes difficult to distinguish between markers 4 having surfaces perpendicular to the optical axis of the image capture device 11 and markers 4 having surfaces parallel to the optical axis of the image capture device 11.

[0141] Fig. 25 is a diagram showing an exemplary arrangement of markers 4a-1 to 4a-12 photographed by the photographing device 11 of Fig. 1. Each of the markers 4a-1 to 4a-12 has a square shape of 0.6 m x 0.6 m. Furthermore, each of the markers 4a-1 to 4a-12 is arranged with their faces perpendicular to the optical axis of the photographing device 11, that is, parallel to the Xw-Zw plane. Fig. 26 is a diagram showing an example of an image 40 of the markers 4a-1 to 4a-12 of Fig. 25 photographed by the photographing device 11.

[0142] FIG. 27 is a diagram showing an exemplary arrangement of the markers 4b-1 to 4b-12 photographed by the photographing device 11 of FIG. 1. Each of the markers 4b-1 to 4b-12 has a square shape of 0.6 m×0.6 m. Also, each of the markers 4b-1 to 4b-12 is arranged so that their faces are parallel to the optical axis of the photographing device 11, that is, parallel to the Yw-Zw plane. FIG. 28 is a diagram showing an example of an image 40 obtained by photographing the markers 4b-1 to 4b-12 of FIG. 27 by the photographing device 11. Comparing FIG. 28 with FIG. 26, it can be seen that the marker 4b-6 of FIG. 28 appears to be almost the same size and shape as each of the markers 4a-1 to 4a-12 of FIG. 26, even though the orientation of the face is different by 90 degrees.

[0143] FIG. 29 is a diagram for explaining the fluctuation of the position of the photographing device 11 in the marker coordinate system caused by erroneous detection of the angle of the surface of the marker 4 with respect to the optical axis of the photographing device 11 in FIG. 1. As described above, the markers 4a-1 to 4a-12 in FIG. 26 and the marker 4b-6 in FIG. 28 cannot be distinguished from each other in appearance. Therefore, the photographing device 11 may be erroneously determined to have the position and attitude of "11b" in the marker coordinate system of FIG. 29 when it should have the position and attitude of "11a", and vice versa. When the optical axis of the photographing device 11 passes substantially through the center of the surface of the marker 4 and is substantially perpendicular to the surface of the marker 4, it is difficult to determine the position of the photographing device 11 in the marker coordinate system with the marker 4 as the origin, and the marker 4 is not suitable for calculating the absolute position and absolute attitude of the vehicle 1. Therefore, it is considered that the reliability is low for the markers 4 (markers 4-1 to 4a-12 in FIG. 25) having a surface perpendicular to the optical axis of the image capturing device 11 and the marker 4 (marker 4b-6 in FIG. 27) that appears to have a surface perpendicular to the optical axis of the image capturing device 11. The reliability calculator 36 increases the estimation error of the absolute position and absolute attitude of such markers 4.

[0144] The larger the distance d21 from the center P21 of the image 40 to the marker 4 (i.e., the closer the marker 4 is to the end in the image 40), the larger the estimation error of the absolute position and orientation (i.e., the lower the reliability of the absolute position and orientation). Also, the smaller the distance d21 (i.e., the closer the marker 4 is to the center P21 in the image 40), the lower the estimation error of the absolute position and orientation (i.e., the higher the reliability of the absolute position and orientation). Therefore, the reliability of the absolute position and orientation can be expressed using the estimation error of the absolute position and orientation that takes into account the position of the marker 4 in the image.

[0145] According to the modification of FIG. 24, by taking into consideration the positions of the markers 4 in the image captured by the image capturing device 11, the position and attitude of the vehicle 1 can be measured with higher accuracy.

[0146] [Fifth Modification of the First Embodiment] 30 is a diagram for explaining the calculation of the reliability of the absolute position and the absolute attitude in the positioning process according to the fifth modified example of the first embodiment. The reliability calculator 36 may take into account the apparent size of the marker 4 in the image captured by the image capture device 11.

[0147] The reliability calculator 36 may calculate the reliability of the absolute position and the absolute orientation so that the reliability increases as the apparent size of the marker 4 in the image captured by the image capturing device 11 increases, and the reliability decreases as the apparent size of the marker 4 in the image decreases. In this case, the reliability of the absolute position and the absolute orientation is represented by a value obtained by multiplying the estimation error of the absolute position and the absolute orientation calculated in the same manner as in the cases described with reference to Figures 15 to 17 and 24 to 29 by a coefficient indicating the apparent size of the marker 4 in the image.

[0148] 30, in an image 40 captured by the camera 11, the marker 4a has an apparent size d31, and the marker 4b has an apparent size d32. The apparent sizes of the markers 4a and 4b in the image are represented, for example, by the vertical length (number of pixels). The apparent size of the marker 4 in the image corresponds to the distance from the camera 11 to the marker 4.

[0149] The estimated error E2a of the absolute position and the estimated error E2b of the absolute attitude are calculated in the same manner as in the cases described with reference to FIGS.

[0150] The reliability of the relative position and the relative orientation is calculated in the same manner as described with reference to FIGS.

[0151] The smaller the apparent size of the marker 4 in the image (i.e., the greater the distance from the camera 11 to the marker 4), the greater the estimation error of the absolute position and attitude (i.e., the lower the reliability of the absolute position and attitude). Also, the larger the apparent size of the marker 4 in the image (i.e., the smaller the distance from the camera 11 to the marker 4), the lower the estimation error of the absolute position and attitude (i.e., the higher the reliability of the absolute position and attitude). Therefore, the reliability of the absolute position and attitude can be expressed using the estimation error of the absolute position and attitude that takes into account the apparent size of the marker 4 in the image.

[0152] Additionally or alternatively, the position and orientation determiner 37 may determine the relative position and orientation as the position and orientation of the vehicle 1 whenever the apparent size of the marker 4 in the images captured by the image capture device 11 is smaller than a predefined threshold. In other words, the reliability of the absolute position and orientation is lowest when the apparent size of the marker 4 in the images is smaller than the threshold. The threshold for the apparent size of the marker 4 in the images is set to, for example, 7 pixels in the vertical direction.

[0153] According to the modification of FIG. 30, by taking into consideration the apparent size of the marker 4 in the image captured by the image capturing device 11, the position and attitude of the vehicle 1 can be measured with higher accuracy.

[0154] [Another Modification of the First Embodiment] The image capture device 11 may be configured to generate an image of the subject and detect the distance from the image capture device 11 to each point on the subject. The image capture device 11 may include, for example, a depth sensor, such as an RGB-D camera, or a Time of Flight (ToF) sensor to detect the distance to the subject. Alternatively, the image capture device 11 may be a stereo camera including two cameras positioned a predetermined distance apart to detect the distance to the subject.

[0155] When the image capturing device 11 detects the distance, the relative position calculator 32 may calculate the relative position and relative attitude of the vehicle 1 using a known ICP (iterative closest point) algorithm or the like.

[0156] 9, an example in which the FAST and KLT trackers are used has been described, but other methods may be used. For example, feature point detection processing and feature point matching processing using SIFT (Scale Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF), which are common in image processing, may be used.

[0157] If the marker 4 is placed in the middle of a straight section of the passage 101, it is expected that the absolute position and absolute attitude can be calculated with high accuracy. On the other hand, if the marker 4 is placed near an intersection or an entrance / exit of the passage 101, the vehicle 1 does not necessarily move straight in the vicinity of the marker 4, and there is a possibility that the error of the calculated absolute position and absolute attitude will be large. Therefore, some of the multiple markers 4 may be auxiliary markers that are not used to calculate the absolute position and absolute attitude (i.e., to correct the position and attitude). The auxiliary marker is placed, for example, near some structure that may be the starting point, destination, or other checkpoint of the vehicle 1 (an entrance / exit of the warehouse 100, an intersection of the passage 101, a specific shelf 102, etc.). The auxiliary marker does not have to be placed along the passage 101 as long as it can be photographed from the vehicle 1. The positioning device 12 can recognize that the vehicle 1 has reached a specific checkpoint by detecting the auxiliary marker. In this case, the table of marker information stored in the storage device 35 further includes an item indicating whether each marker 4 is an auxiliary marker or not. In this case, the marker information table does not need to have information on the position and orientation of the auxiliary markers. The marker information table may include an item indicating the reliability of the absolute position and absolute orientation calculated based on the marker 4, instead of an item indicating whether or not each marker 4 is an auxiliary marker.

[0158] The vehicle 1 and the server device 2 may use a removable storage medium such as an SD card instead of the communication devices 13 and 22. The position and attitude of the vehicle calculated in the vehicle may be written to the storage medium, and the server device 2 may read the position and attitude of the vehicle from the storage medium.

[0159] Instead of a forklift, the vehicle 1 may be a manned vehicle such as a truck or a towing vehicle. The vehicle 1 may also be an unmanned luggage transport device such as an automated guided vehicle (AGV) or a pallet transport robot. In this case, the vehicle 1 moves by controlling the drive mechanism 15 under the control of the server device 2. The vehicle 1 may also be a human-powered vehicle without a drive mechanism, such as a handcart.

[0160] The positioning device 12 may be provided in the server device 2 instead of in the vehicle 1. In this case, the image captured by the imaging device 11 is sent from the vehicle 1 to the server device 2 by the communication device 13 (or a removable storage medium). The positioning device 12 of the server device 2 calculates the position and attitude of the vehicle 1 based on the image acquired from the vehicle 1, similar to the positioning device 12 of FIG. 2.

[0161] [Advantages of the first embodiment] According to the first embodiment, the positioning device 12 includes a relative position calculator 32, a storage device 35, an absolute position calculator 34, a reliability calculator 36, and a position and attitude determiner 37. The relative position calculator 32 calculates the relative position and relative attitude of the vehicle 1, which indicate the relative position and relative attitude of the vehicle 1 with respect to a predetermined reference position and reference attitude, based on a plurality of images taken by the photographing device 11 mounted on the vehicle 1. The storage device 35 stores information on identifiers, positions, and attitudes of a plurality of markers 4 arranged at predetermined positions and visually identifiable, and information on a map including a path for the vehicle 1. The absolute position calculator 34 extracts one of the plurality of markers 4 from the image taken by the photographing device 11, and calculates the absolute position and absolute attitude of the vehicle 1, which indicate the position and attitude of the vehicle 1 on the map, based on the position and attitude of the one extracted marker 4. The reliability calculator 36 calculates the reliability of the relative position and relative attitude calculated by the relative position calculator 32 and the reliability of the absolute position and absolute attitude calculated by the absolute position calculator 34. The position and attitude determiner 37 determines the relative position and relative attitude as the position and attitude of the vehicle 1 when the reliability of the relative position and relative attitude is equal to or greater than the reliability of the absolute position and absolute attitude, and determines the absolute position and absolute attitude as the position and attitude of the vehicle 1 when the reliability of the relative position and relative attitude is less than the reliability of the absolute position and absolute attitude.

[0162] This allows the reliability of the relative position and relative attitude and the reliability of the absolute position and absolute attitude to be calculated, and the one with the higher reliability to be determined as the position and attitude of vehicle 1, thereby making it possible to measure the position and attitude of vehicle 1 with high accuracy.

[0163] According to the first embodiment, the reliability calculator 36 may calculate the reliability of the relative position and relative attitude such that the reliability increases as the moving distance and rotation angle from the reference position and reference attitude of the vehicle 1 decrease. When the position and attitude determiner 37 determines the absolute position and absolute attitude as the position and attitude of the vehicle 1, the reliability calculator 36 resets the reference position and reference attitude.

[0164] This makes it possible to calculate errors in the relative position and relative attitude based on the moving distance and rotation angle of the vehicle 1.

[0165] According to the first embodiment, the confidence calculator 36 may calculate the confidence of the relative position and orientation such that the confidence increases as the difference between the relative position and orientation and the absolute position and orientation decreases.

[0166] This makes it possible to calculate errors in the relative position and orientation based on the difference between the absolute position and the relative orientation and the relative position and orientation.

[0167] According to the first embodiment, the relative position calculator 32 may calculate the relative position and orientation based on feature points tracked in multiple temporally adjacent images. The confidence calculator 36 may calculate the confidence of the tracking based on the number of feature points successfully tracked in multiple temporally adjacent images, and may calculate the confidence of the relative position and orientation such that the higher the confidence of the tracking, the higher the confidence of the relative position and orientation.

[0168] This makes it possible to calculate errors in the relative position and relative orientation based on the reliability of the tracking.

[0169] According to the first embodiment, the confidence calculator 36 may calculate the confidence of the relative position and the relative orientation such that the confidence increases as the variance of the change amount of the relative position or the relative orientation decreases.

[0170] This makes it possible to calculate errors in the relative position and relative orientation based on the variance of the amount of change in the relative position or relative orientation.

[0171] According to the first embodiment, the reliability calculator 36 may calculate the reliability of the absolute position and orientation in an image captured by the image capture device 11 such that the reliability increases as the distance from the center of the image to the marker 4 decreases.

[0172] This makes it possible to calculate errors in the absolute position and attitude, taking into account the position of the marker 4 in the image captured by the imaging device 11.

[0173] According to the first embodiment, the confidence calculator 36 may calculate the confidence of the absolute position and attitude such that the confidence increases as the apparent size of the marker 4 in the image captured by the image capture device 11 increases.

[0174] This makes it possible to calculate errors in the absolute position and absolute attitude based on the apparent size of the marker 4 in the image captured by the imaging device 11.

[0175] According to the first embodiment, the position and attitude determiner 37 may determine the relative position and relative attitude as the position and attitude of the vehicle 1 when the apparent size of the marker 4 in the image captured by the photographing device 11 is smaller than a predetermined threshold.

[0176] This makes it possible to calculate errors in the absolute position and attitude, taking into account the apparent size of the marker 4 in the image captured by the imaging device 11.

[0177] [Second embodiment] A positioning device and a moving object including the same according to the second embodiment will be described with reference to FIG.

[0178] [Configuration of the second embodiment] Fig. 31 is a block diagram showing a configuration of a positioning device 12B according to the second embodiment. The vehicle 1 according to the first embodiment may be equipped with a positioning device 12B instead of the positioning device 12 in Fig. 3. The positioning device 12B includes an image recognizer 38 and a data combiner 39 in addition to the components of the positioning device 12 in Fig. 3.

[0179] The image recognizer 38 recognizes a predetermined object from an image captured by the image capture device 11. The image recognizer 38 may recognize a person (e.g., a driver of the vehicle 1, a person around the vehicle 1). The image recognizer 38 may recognize a specific baggage 3 that has been learned in advance. The image processor 31, the image recognizer 33, and the image recognizer 38 may acquire images from the same image capture device 11. Alternatively, the image recognizer 38 may capture an image including a person and / or baggage 3 using a capture device different from the image capture device 11 that captures an image (i.e., an image for measuring the position and attitude of the vehicle 1) that is supplied to the image processor 31 and the image recognizer 33. In this case, the image capture device 11 may be provided to capture, for example, an aisle 101 in front of the vehicle 1, and the other image capture device may be provided to capture, for example, the driver's seat or a luggage rack of the vehicle 1. The image capture device 11 and the other image capture device are synchronized with each other in advance.

[0180] The data combiner 39 acquires data on the position and attitude of the vehicle 1 determined by the position and attitude determiner 37 from the position and attitude determiner 37 together with a time stamp of the time when an image corresponding to the position and attitude was captured by the image capture device 11 (or the time when the position and attitude were calculated). The data combiner 39 combines image recognition data of an object recognized by the image recognizer 38 with the position and attitude data of the vehicle 1. The data combiner 39 also acquires sensor data generated by a sensor group 16B including one or more sensors mounted on the vehicle 1, and combines the sensor data with the position and attitude data of the vehicle 1. The data combiner 39 may acquire sensor data including at least one of the acceleration and angular velocity of the vehicle 1. The data combiner 39 may also acquire sensor data including the weight of the baggage 3 transported by the vehicle 1. Each sensor of the sensor group 16B assigns a time stamp of the time when the sensor data was acquired to the sensor data. The data combiner 39 combines the data based on the timestamps of the position and attitude data of the vehicle 1, the timestamps of the image recognition data, and the timestamps of the sensor data, in synchronization with each other.

[0181] When the timestamps of these data do not match, the data combiner 39 may associate the image recognition data or sensor data with the position and attitude data of the vehicle 1 having the closest timestamp. When the timestamps of these data do not match, the data combiner 39 may interpolate the position and attitude data of the vehicle 1 using linear interpolation or internal division, and associate the image recognition data or sensor data with the interpolated position and attitude data of the vehicle 1 having the corresponding timestamp.

[0182] By providing the data synthesizer 39, various data related to the work of the vehicle 1 can be recorded in association with the position and trajectory of the vehicle 1. For example, by recording a person whose image has been recognized, it is possible to track a person related to the work of the vehicle 1. By recording a baggage 3 whose image has been recognized, it is possible to track the baggage 3 transported by the vehicle 1. By recording the acceleration and angular velocity of the vehicle 1, it is possible to detect unevenness of the road surface in the warehouse. By recording the weight of the baggage 3, it is possible to monitor the workload of the vehicle 1.

[0183] [Advantages of the second embodiment] According to the second embodiment, the vehicle 1 may further include a data combiner 39 that acquires sensor data generated by one or more sensors mounted on the vehicle 1 and combines the sensor data with position and attitude data of the vehicle 1 determined by the position and attitude determiner 37.

[0184] According to a second embodiment, the data combiner 39 may obtain sensor data including at least one of the acceleration and angular velocity of the vehicle 1 .

[0185] According to a second embodiment, the data combiner 39 may obtain sensor data including the weight of the load 3 carried by the vehicle 1 .

[0186] According to the second embodiment, the positioning device 12B may further include an image recognizer 38 that recognizes a predetermined object from an image captured by the imaging device 11. In this case, the data combiner 39 combines information on the object recognized by the image recognizer 38 with the data on the position and attitude of the vehicle 1 determined by the position and attitude determiner 37.

[0187] According to a second embodiment, the image recognizer 38 may recognize people.

[0188] According to the second embodiment, the image recognizer 38 may recognize a specific piece of luggage 3 that has been previously trained.

[0189] According to the second embodiment, various data related to the work of the vehicle 1 can be recorded in association with the position and trajectory of the vehicle 1.

[0190] According to the second embodiment, the vehicle 1 may include a photographing device 11 that photographs images for measuring the position and attitude of the vehicle 1, and another photographing device that photographs other objects. In this case, the data combiner 39 can mutually associate the position and attitude data of the vehicle 1 generated based on the image photographed by the photographing device 11 with the image recognition data generated based on the image photographed by the other photographing device. If the position and attitude data of the vehicle 1 are associated with the image recognition data of other objects photographed during the movement of the vehicle 1, this is very useful when performing a business analysis based on the position and trajectory of the vehicle 1 on a map. For example, when a suspicious behavior of a person is detected by visual inspection or the like, it is possible to search for and read out images or videos previously photographed near the position and / or related to the person by referring to the position of the person on the map.

[0191] [Other embodiments] In each embodiment, the positioning device may be provided in a four-wheeled vehicle such as a forklift or a truck, or in a vehicle with one to three wheels, or five or more wheels. In each embodiment, the positioning device may be provided in a moving body without wheels, such as an airplane, a helicopter, a drone, or a hovercraft, regardless of the number of wheels and / or the presence or absence of wheels. The positioning device according to this embodiment can estimate the position of the moving body based on an image captured by an imaging device, rather than estimating the position of the moving body from the number of rotations of the wheels. [Industrial Applicability]

[0192] According to the positioning device according to each aspect of the present disclosure, it is possible to measure the position of a moving object in a warehouse, a factory, etc. This makes it possible to track the trajectory (traffic line) of the moving object, route the moving object, optimize the placement of luggage, etc. in the warehouse or factory, monitor the operating rate, improve work efficiency, etc. [Explanation of symbols]

[0193] 1 vehicle 1a Cargo platform 1b Lifting mechanism 1c Console 2. Server equipment 3. Luggage 4. Markers 11 Imaging Device 12,12B Positioning device 13. Communications Equipment 14 Display device 15 Drive mechanism 16B Sensor Group 21 Processing equipment 22 Communication Equipment 23 Input Devices 24 Storage device 25 Display device 31 Image processor (feature points) 32 Relative Position Calculator 33 Image Recognizer (Marker) 34 Absolute Position Calculator 35 Storage device 36 Reliability Calculator 37 Position and Attitude Determinator 38 Image recognition device (person and luggage) 39 Data Synthesizer 40 images 41 Minutiae 100 warehouse Aisle 101 102 Shelf

Claims

1. a first calculator that calculates a first position and a first attitude of the moving body, the first position and the first attitude indicating a relative position and a relative attitude of the moving body with respect to a predetermined reference position and a reference attitude, based on a plurality of images captured by an image capturing device mounted on the moving body; A storage device that stores information on identifiers, positions, and attitudes of a plurality of visually identifiable markers arranged at predetermined positions, and information on a map including a path for the moving body; a second calculator that extracts one of the plurality of markers from an image captured by the imaging device, and calculates a second position and a second orientation of the moving object indicating the position and orientation of the moving object on the map based on the position and orientation of the one extracted marker; a confidence calculator that calculates a first confidence indicating a confidence of the first position and the first orientation calculated by the first calculator, and a second confidence indicating a confidence of the second position and the second orientation calculated by the second calculator; a position and attitude determiner that determines the first position and the first attitude as the position and attitude of the moving body when the first reliability is equal to or greater than the second reliability, and determines the second position and the second attitude as the position and attitude of the moving body when the first reliability is smaller than the second reliability, Positioning device.

2. the reliability calculator calculates the first reliability such that the first reliability increases as a moving distance and a rotation angle of the moving object from a reference position and a reference attitude are reduced; when the position and attitude determiner determines the second position and the second attitude as the position and attitude of the moving body, the reliability calculator resets the reference position and the reference attitude. The positioning device according to claim 1.

3. the confidence calculator calculates the first confidence such that the first confidence increases as a difference between the first position and the first orientation and the second position and the second orientation decreases. The positioning device according to claim 1.

4. the first calculator calculates the first position and the first orientation based on feature points tracked in a plurality of temporally adjacent images; the reliability calculator calculates a reliability of tracking based on a number of feature points that have been successfully tracked in a plurality of temporally adjacent images, and calculates the first reliability such that the higher the reliability of tracking, the higher the first reliability.

4. The positioning device according to claim 2 or 3.

5. the reliability calculator calculates the first reliability such that the first reliability increases as the variance of the change amount of the first position or the first orientation decreases. The positioning device according to claim 1.

6. the reliability calculator calculates the second reliability such that the second reliability increases as the distance from the center of the image to the marker decreases in the image captured by the image capture device; Positioning device according to one of claims 1 to 5.

7. the reliability calculator calculates the second reliability such that the second reliability increases as the apparent size of the marker in the image captured by the imaging device increases; Positioning device according to one of the preceding claims.

8. the position and attitude determiner determines the first position and the first attitude as the position and attitude of the moving object when an apparent size of the marker in the image captured by the image capture device is smaller than a predetermined threshold value; Positioning device according to one of the preceding claims.

9. a data combiner for acquiring sensor data generated by one or more sensors mounted on the moving body and combining the sensor data with position and attitude data of the moving body determined by the position and attitude determiner; Positioning device according to one of the claims 1 to 8.

10. The data combiner acquires sensor data including at least one of an acceleration and an angular velocity of the moving body. The positioning device according to claim 9.

11. the data combiner acquires sensor data including a weight of a load carried by the vehicle; The positioning device according to claim 9 or 10.

12. The positioning device further includes an image recognizer that recognizes a predetermined object from the image captured by the image capture device, The data synthesizer synthesizes the information of the object recognized by the image recognizer with the data of the position and attitude of the moving body determined by the position and attitude determiner. Positioning device according to one of claims 9 to 11.

13. The image recognizer recognizes people. The positioning device according to claim 12.

14. The image recognizer recognizes a specific piece of luggage that has been previously trained. The positioning device according to claim 12.

15. A photographing device; and a positioning device according to any one of claims 1 to 14, Mobile body.

Images