Mobile system and orthoimage map generation method

Abstract

To generate scaled image map data reliably and accurately in a mobile body system.SOLUTION: A system has a scale adjustment unit that adds scale information to a camera image by comparing the position attitude of a distance sensor with the relative position attitude of the camera, and a floor face extraction unit that extracts information on the floor face of the camera image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a mobile system and an orthoimage map generation method. [Background technology]

[0002] In recent years, with the increase in logistics within factories, forklifts and other vehicles are being equipped with sensors such as laser scanners and cameras, and SLAM (Simultaneous Localization And Mapping) technology is being applied to make automated guided vehicles (AGVs) trackless.

[0003] AGVs utilizing this technology can improve transportation efficiency by moving autonomously. Trackless AGVs are often operated in environments where people coexist. In such cases, there are many instructions regarding people's movement, such as safety passages and stop signs, and it is necessary to create an AGV operation plan that takes these into consideration.

[0004] A prior art document in this technical field is Patent Document 1. Patent Document 1 discloses a technology for generating map data from an image acquired by a monocular camera by using the results of position estimation by a laser for position estimation by a monocular camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-161141 Summary of the Invention [Problem to be solved by the invention]

[0006] Although scale information cannot be obtained from images acquired by a monocular camera, in Patent Document 1, dimensional information of the image is calculated by comparing it with map data containing a scale generated by a laser.

[0007] However, since the features of structures measured by laser do not necessarily match those captured in the image, it is not possible to generate image map data with a scale reliably and accurately.

[0008] An object of the present invention is to generate scaled image map data reliably and accurately in a mobile system. [Means for solving the problem]

[0009] One aspect of the mobile system of the present invention is a mobile system having a distance sensor and a camera mounted on a mobile body that moves within a site having a floor surface, and an orthoimage map generation device that generates an orthoimage map of the floor surface from camera images taken by the camera, wherein the orthoimage map generation device is characterized by having a distance sensor position and attitude detection unit that detects the position and attitude of the distance sensor by comparing the distance data measured by the distance sensor with map data, a camera relative position and attitude detection unit that detects the relative position and attitude of the camera from the camera images taken by the camera, a scale adjustment unit that assigns scale information to the camera image by comparing the position and attitude of the distance sensor with the relative position and attitude of the camera, and a floor surface extraction unit that extracts information about the floor surface from the camera image. [Effects of the Invention]

[0010] According to one aspect of the present invention, scaled image map data can be generated reliably and accurately in a mobile system. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the configuration of a mobile system according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing an example of a site where a moving object operates and a measurement example using a distance sensor; [Figure 3] FIG. 1 is a diagram illustrating an example in which a mobile object is measuring a site. [Figure 4A] FIG. 10 is a diagram showing an example of an acquired image. [Figure 4B] FIG. 10 is a diagram showing an example of an acquired image. [Figure 5] FIG. 10 is a diagram showing an example in which a projective transformation is performed on an acquired image. [Figure 6] FIG. 10 is a diagram illustrating an example of processing for extracting a floor surface pattern. [Figure 7] FIG. 10 is a diagram illustrating an example of a comparison of the movement trajectories of a laser sensor and a monocular camera. [Figure 8] FIG. 10 is a diagram showing an example of an image in which laser map data and an orthoimage are superimposed. [Figure 9] 1 is a flowchart showing the process of the present invention. [Figure 10] FIG. 10 is a diagram showing another example of a comparison of the movement trajectories of the laser sensor and the monocular camera. DETAILED DESCRIPTION OF THE INVENTION [Example]

[0012] The present invention relates to a technique for defining the movement paths of moving objects such as forklifts and automated guided vehicles (AGVs) that operate in a factory.

[0013] Hereinafter, an embodiment for realizing the present invention will be described with reference to the drawings. In this embodiment, an example will be described in which image map data is generated for a mobile object such as an AGV in a factory.

[0014] FIG. 1 is a diagram showing an example of the configuration of a mobile system according to an embodiment of the present invention.

[0015] The mobile object (AGV) 10 is equipped with a distance sensor 105 for detecting its position, a camera 101, an orthoimage map generation device 104 for generating an orthoimage map of the road surface from images taken by the camera 101, a display 102 for displaying image data generated by the orthoimage map generation device 104, and a vehicle control device 120 for controlling the drive wheels 121.

[0016] The orthoimage map generating device 104 includes a camera relative position and orientation detection unit 106, a distance sensor position and orientation detection unit 107, a scale adjustment unit 112, a distance sensor position conversion unit 108, a coordinate system adjustment unit 113, a floor extraction unit 122, and an image map synthesis unit 110. These are mainly processed by a processor 119. Furthermore, the orthoimage map generating device 104 has a memory 118. The memory 118 stores, as reference data when processing is performed by the processor 119, a map 114 used to detect the position and orientation of the range sensor 105 and a sensor relative position 117 used when converting the range sensor position into a camera position.

[0017] The distance sensor 105 is a sensor capable of measuring the distance to an object in the surrounding environment and the measurement direction. In this embodiment, a two-dimensional laser scanner is used as a typical distance sensor. A two-dimensional laser scanner emits a laser pulse in the horizontal direction and calculates the distance to the object by measuring the time it takes for the laser light to be diffused when it hits the object and then returns.

[0018] The direction of irradiation of the laser pulse is rotated by a motor (not shown) provided inside the distance sensor 105, and the irradiation direction and distance are acquired, thereby making it possible to acquire shape data of the surroundings.

[0019] The measurement range of a laser scanner can be, for example, an irradiation range of 270 degrees, an angular resolution (pulse irradiation interval) of 0.25 degrees, and a scanner that can acquire 1081 points of distance data at 25 msec intervals. However, this is not limited to this, and the present invention can be applied to any sensor that can acquire multiple sets of distance data and irradiation direction, even if the irradiation range and angular resolution are different.

[0020] Furthermore, the present invention can be applied to any device that can obtain multiple distances to surrounding structures in different angular directions, such as a three-dimensional laser scanner or stereo camera that can measure the distance to surrounding structures in three dimensions, not just a two-dimensional laser scanner.

[0021] The distance sensor position and orientation detection unit 107 detects the position and orientation of the distance sensor 105 by comparing the distance data with the map 114 .

[0022] The camera relative position and orientation detection unit 106 calculates the position and orientation of the camera 101 from the image captured by the camera 101 using V-SLAM technology.

[0023] In this embodiment, the processor 119 is an arithmetic device for executing a program, such as a CPU (Central Processing Unit), but it can be anything that can execute a program, and may be realized as an FPGA (Field Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit).

[0024] The vehicle control device 120 is a control device for moving the mobile body 10 by issuing commands to the drive wheels 121. This is for realizing the function of setting a rough travel route based on the map 114 and traveling along that travel route.

[0025] Figure 2 shows the field environment in which the mobile object 10 is operated. Information such as pedestrian safety passages 202 and 203 is marked on the floor, but because a two-dimensional laser is irradiated horizontally, it is unable to measure the information marked on the road surface. A three-dimensional laser scanner can measure the floor surface, but it is difficult to accurately measure floor surface patterns with the resolution of a laser.

[0026] In Figure 2, map 114 represents the shape of the surface of an object that can be measured by a laser. The positions of these structures are called, for example, an occupancy grid map. In an occupancy grid map, space is divided into a grid, and the grid is expressed in an image-like format, with 1 representing a grid that has a structure that can be measured by a laser, and 0 representing a grid that does not have a structure. An origin 204 is defined for map 114. Also, by defining a scale for the length of one side of the grid, it is possible to include dimensional information in map 114.

[0027] Any method of representation of the map 114 can be used as long as it can be compared with the shape data measured by a laser and whether or not it matches, and it is also possible to use an NDT (Normal Distribution Transform) that approximates the point cloud data within a grid using a Gaussian distribution.

[0028] The map (map data) 114 is like a cross-sectional view of the structures located at the site, sliced ​​at a certain height. This map 114 can be created using SLAM technology, but if precise CAD data or the like is available, this data can also be used as the map 114 for position detection.

[0029] The distance sensor position and orientation detection unit 107 compares the map 114 with the measurement data 205 of the distance sensor 105, and calculates the position and orientation (x, y, θ) on the coordinate system 204 defined on the map 114. The distance sensor position and orientation detection unit 107 is realized by a function for quickly comparing the map 114 with the distance data 205, and can use, for example, the ICP (Iterative Closest Point) method or the NDT method. It may also perform a coarse-to-fine search using maps with different resolutions.

[0030] FIG. 3 shows a mobile object 10 using a camera 101 to capture an image of a work environment including a cart 201, a white line 202, and the like.

[0031] The moving object 10 continuously records images 307 and 308 at regular time intervals while moving. The camera position and orientation detection unit 106 uses the images captured by the camera 1-01 to detect the position and orientation of the camera 101 using techniques such as monocular Visual SLAM or SfM (Structure from Motion).

[0032] Examples of Visual SLAM include a method that extracts feature points using HARRIS features, SIFT features, etc., and performs triangulation by finding correspondences between points (pixels) with the same features, and a direct method that directly calculates correlation functions between pixels and finds correspondences for each pixel. Furthermore, a method that uses a DNN (Deep Neural Network) to estimate the position and orientation of the camera 101 may also be used.

[0033] In the present invention, any method may be used as long as it can accurately calculate the orientation R and position t304 of the camera 101 that captured each of the above two or more images. Here, R is a three-dimensional rotation matrix, and t represents a three-dimensional vector. However, the amount of movement t304 estimated monocularly cannot obtain absolute scale information, and can only obtain the direction of movement.

[0034] The scale adjustment unit 112 calculates a scale adjustment parameter by comparing the movement amount t304 with the movement amount 305 measured by the distance sensor 105.

[0035] The distance sensor position conversion unit 108 calculates the position and orientation of the camera 101 based on the position (x, y, θ) of the distance sensor 105 and the sensor relative position 117. It is assumed that the camera 101 is attached at position b with the sensor coordinate system of the distance sensor 105 as the reference.

[0036] If the position and orientation of the distance sensor in the coordinate system of map 114 is (x, y, θ) and the Z axis of the three-dimensional space is taken to be perpendicular to the floor surface, the position of the distance sensor 105 can be written as [x, y, 0]^T (where T represents the transpose of the vector) using a column vector.

[0037] Using a rotation matrix U(θ) that represents the orientation in a three-dimensional coordinate system, and assuming that the relative position of the camera is d=[x, y, 0]^T, the camera position can be described from the coordinates of the distance sensor 105 by the following equation.

[0038] (Number 1) JPEG0007739254000001.jpg34152 By comparing the displacement of the camera position (Equation 2) calculated from the movement amount 305 of the distance sensor 105 with the displacement t (304) calculated from the image of the camera 101, the scale ratio can be calculated and the scale can be applied to the image of the camera 101.

[0039] (Math 2) m(d)=g_f(θ, d)-g_f-1(θ, d) Fig. 4A shows an acquired camera image 307. Fig. 4B shows an acquired camera image 308. In the camera image 307 of Fig. 4A, 307p is a pixel or point, and 307e is an edge. In the camera image 308 of Fig. 4B, 308p is a pixel or point, and 308e is an edge.

[0040] By performing a projective transformation (homography) on this image using matrix G, correspondence is established between the two images, and three-dimensional information such as white lines written on the floor is restored.

[0041] The projective transformation can be expressed by (Equation 3) using a projective transformation matrix G=(g_ij) where p=[u,v,1]^T is a vector representing two-dimensional coordinates on an image.

[0042] (Number 3) JPEG0007739254000002.jpg44119 Moreover, G can be expressed as (Equation 4).

[0043] (Number 4) JPEG0007739254000003.jpg1298 Here, n is the normal vector of the floor surface with respect to the orientation of the camera 101, and K is the internal parameter matrix of the camera 101. The normal vector n is a vector determined from the relative positions of the camera 101 and the distance sensor 105. The internal parameter K can be calculated by performing calibration in advance.

[0044] Both values ​​can be calculated in advance, and therefore, since R and t are known, it is possible to calculate G. Furthermore, since the camera 101 moves on a plane, R is a rotation matrix with n as the rotation axis, and therefore it is also possible to calculate n from R.

[0045] Figure 5 shows an image obtained by performing projective transformation on image 308 at time t. Projective transformation is a process for extracting the pattern on the floor surface. By calculating G in (Equation 2) and performing the transformation in (Equation 1), the floor surface portion is transformed so that it overlaps exactly with the image 307 at time t-1, and a projectively transformed image 308h can be obtained.

[0046] FIG. 6 is a schematic diagram of the processing of the floor surface extraction unit 121, in which an edge-extracted image 307 and an image 308h are superimposed.

[0047] Because the cart part 601 is not on the floor, it does not overlap with the cart in the image 307 at time t-1, but the pattern 602 on the floor is on the same plane, so it can be superimposed exactly. By leaving only the overlapping part due to this property, it is possible to extract only the pattern on the floor.

[0048] This transformation also makes it possible to determine the correspondence between each pixel of image 307 at time t-1 and image 308 at time t, and from this correspondence and the relative relationship of the camera positions, it is possible to reconstruct the three-dimensional distance of the floor surface, i.e., the three-dimensional coordinates of the actual size floor pattern.

[0049] In a typical method using feature points, locations where a one-to-one correspondence can be established between images, such as information about corners in an image, are used. However, by using the present invention, even in locations where a unique correspondence cannot be determined by comparing pixels, such as edges 307e and 308e, it is possible to establish correspondence by assuming a plane and restore three-dimensional information.

[0050] FIG. 7 shows a movement trajectory 702 of the camera 101 calculated from the camera images of all the cameras 101, and a movement trajectory 705 of the positions calculated by the distance sensor 105 converted into camera positions.

[0051] These correspond to the measured times (one-to-one time correspondence 703), and by performing coordinate transformation so that the positions at corresponding times overlap, the coordinate system 701 of the camera 101 and the coordinate system 204 of the distance sensor 105 can be expressed on the same coordinate system.

[0052] Here, 304 is the movement amount of the camera 101, and 305m is the movement amount of the distance sensor 105. In this way, the movement amount of the distance sensor 105, which defines the installation position of the camera 101, is converted into a movement amount converted into the camera position relative to the installation position of the distance sensor 105. The above processing is performed by the coordinate system adjustment unit 113 and the scale adjustment unit 112.

[0053] Figure 8 shows the result of adjusting the coordinate system. In this way, by using the present invention, it is possible to accurately combine the map (map data) 114 captured by the distance sensor 105 with the floor patterns 202 and 203 captured by the camera 101. Here, 308N is a partial orthoimage.

[0054] FIG. 9 is a flowchart showing the process of the present invention.

[0055] First, the camera relative position and orientation detection unit 106 reads two frames of image data and the relative positions and orientations R and t calculated in advance (step 902).

[0056] Next, the distance sensor position conversion unit 107 calculates the scale conversion parameter s (=m(d) / |t|) from the movement distance |t| of the camera position and the movement distance d of the distance sensor (step 903). Next, the floor surface extraction unit 122 obtains a projective transformation matrix G from the position and orientation R, s×t, and normal vector n of the camera 101 (step 904).

[0057] Next, the floor surface extraction unit 122 performs a projective transformation G on the edge-extracted image, and extracts only the overlapping portions as the floor surface (step 905). Next, the floor surface extraction unit 122 restores scaled 3D point cloud data of the floor surface (step 906) using the orientation R, the movement amount st, and the correspondence of edge pixels according to G (step 905).

[0058] Finally, the scale adjustment unit 112 and the coordinate system adjustment unit 113 compare the movement trajectory of the distance sensor 105 with the movement trajectory of the camera 101, and synthesize them by adjusting the coordinate system with the map 114 (step 907). FIG. 10 is a diagram showing another example of a comparison between the movement trajectories of the laser sensor and the monocular camera.

[0059] In Figure 7, the measured times are correlated (one-to-one time correspondence 703), and by performing coordinate transformation so that the positions at corresponding times overlap, the coordinate system 701 of the camera 101 and the coordinate system 204 of the distance sensor 105 are expressed on the same coordinate system.

[0060] 10, the correspondence between the measured times (one-to-one time correspondence 703) is unknown, but by evaluating the overlap between the movement trajectory 1001 of the range sensor and the movement trajectory 1002 of the monocular camera, it is possible to calculate the relationship between the coordinate systems and information on the scale. Specifically, it is possible to calculate the correspondence between the coordinate systems and information on the scale by minimizing the sum of the distance 1003a between the position and orientation constituting the movement trajectory 1001 of the range sensor and the nearest point on the movement trajectory of the camera, and the distance 1003b between the position and orientation constituting the movement trajectory 1002 of the camera and the nearest point on the movement trajectory 1001 of the range sensor. Needless to say, it is not necessary for the sensors to emit laser light and capture images with the cameras at the same time.

[0061] In this way, it is possible to express the coordinate system 701 of the camera 101 and the coordinate system 204 of the distance sensor 105 on the same coordinate system from the information on the movement trajectory without using information on the measurement times of the camera and laser.

[0062] In the above embodiment, the orthoimage map generating device 104 is provided inside the mobile body 10, but the present invention is not limited to this, and the orthoimage map generating device 104 may be provided in an external server (not shown) connected via a communication line.

[0063] In this way, in the above embodiment, the robot estimates its own position by comparing distance data measured by the distance sensor 105 with the map 114 of the environment in which it is moving, estimates its own relative position from multiple images measured by the camera 101, and adds scale information to the image data by comparing the self-position and orientation determined by the distance sensor 105 with the self-position and orientation determined by the camera 101.

[0064] According to the above embodiment, instead of using laser map data, traffic signs for workers that are written on the floor can be written as scaled information on the map data, and operation plans for moving objects can be created efficiently. [Explanation of symbols]

[0065] 10: moving body, 102: display, 105: distance sensor, 104: orthoimage map generating device, 119: processor (CPU), 106: camera relative position and attitude detection unit, 107: distance sensor position and attitude detection unit, 113: coordinate system adjustment unit, 112: scale adjustment unit, 110: orthoimage synthesis unit, 108: distance sensor position conversion unit, 118: memory (storage device), 114: map, 117: sensor relative position, 120: vehicle control device, 121: drive wheel, 201: bogie, 202: safety passage, 204: map origin, 307: image at time t-1, 308: image at time t

Claims

1. A mobile system including a distance sensor and a camera provided on a mobile body that moves within a site having a floor surface, and an orthoimage map generating device that generates an orthoimage map of the floor surface from a camera image taken by the camera, The orthoimage map generating device includes: a distance sensor position and orientation detection unit that detects the position and orientation of the distance sensor by comparing distance data measured by the distance sensor with map data; a camera position and orientation detection unit that detects the position and orientation of the camera from the camera image captured by the camera; a scale adjustment unit that adds scale information to the camera image by comparing the position and orientation of the distance sensor with the position and orientation of the camera; a floor surface extraction unit that extracts information about the floor surface of the camera image; A mobile system comprising:

2. The camera position and orientation detection unit Detecting the position and orientation of the camera from a plurality of camera images captured by the camera; The scale adjustment unit 2. The mobile body system according to claim 1, wherein the scale information is provided by comparing the amount of movement obtained by the camera with the amount of movement obtained by the distance sensor.

3. The scale adjustment unit 2. The mobile system according to claim 1, wherein information about a pattern on the floor surface is added to the map data.

4. The scale adjustment unit 4. The mobile system according to claim 3, wherein information on a safety passage for workers at the work site is added to the map data as information on the pattern drawn on the floor surface.

5. The floor surface extraction unit The mobile system described in claim 1, characterized in that the camera images are subjected to a projective transformation to establish correspondence between the multiple camera images, thereby restoring three-dimensional point cloud data of the floor surface to which the scale information has been assigned, and extracting information about the floor surface.

6. The floor surface extraction unit The mobile system according to claim 3, wherein information about the floor surface is extracted by overlaying a plurality of edge-extracted camera images on each other and leaving the patterns on the overlapping floor surface.

7. The orthoimage map generating device includes: a coordinate system adjustment unit that expresses the coordinate system of the map data and the coordinate system of the camera on the same coordinate system by comparing the movement trajectory of the distance sensor with the movement trajectory of the camera; an image map synthesis unit that synthesizes the map data and information on the floor surface of the camera image to generate the orthoimage map to which the scale information is assigned; 10. The mobile system of claim 1, further comprising:

8. The coordinate system adjustment unit a movement trajectory of the distance sensor is calculated based on a position obtained by converting the sensor position calculated by the distance sensor into a camera position of the camera; The mobile system according to claim 7, characterized in that the coordinate system of the map data and the coordinate system of the camera are expressed on the same coordinate system by correlating the measurement times of the movement trajectory of the distance sensor and the movement trajectory of the camera and performing coordinate transformation so that the positions of the corresponding measurement times overlap.

9. The moving body is 2. The mobile system according to claim 1, wherein the camera images are continuously taken at regular time intervals by the camera while moving around the site.

10. 1. An orthoimage map generation method for generating an orthoimage map of a floor surface from camera images taken by a camera attached to a mobile object moving over a site having a floor surface, comprising: a distance sensor position and orientation detection step of detecting a position and orientation of the distance sensor by comparing distance data measured by the distance sensor provided on the moving body with map data; a camera position and orientation detection step of detecting a position and orientation of the camera from a camera image captured by the camera; a scale adjustment step of adding scale information to the camera image by comparing the position and orientation of the distance sensor detected by the distance sensor with the position and orientation of the camera detected by the camera; a floor surface extraction step of extracting information about the floor surface of the camera image; An orthoimage map generating method comprising:

Images