Information processing system, information processing device, information processing method and program

Abstract

To provide an information processing system.SOLUTION: An information processing system 1 to create map information using an imaging device 20 provided at a moving object 10 is provided. The information processing system 1 includes a map information creating unit 60 that creates the map information on the basis of a plurality of images picked up by the imaging device 20 through a route between a predetermined start position A of the imaging device 20 and a predetermined end position B thereof. The information processing system 1 also includes a map correcting unit 61 that creates the map information having the scale corrected on the basis of a virtual distance between a start point corresponding to the start position A on the map information and an end point corresponding to the end position B thereon, and of the actually measured distance that is given in advance to a set of the start position A and the end position B of the imaging device 20.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to information processing technology, and more particularly to an information processing system, an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, there are known technologies (SLAM, Simultaneous Localization and Mapping) that simultaneously estimate the self-location of a moving object and generate an environmental map. Among them, a technology that estimates the self-location and generates an environmental map using images captured by an imaging device as input is referred to as VLSAM (Visual SLAM).

[0003] VLSAM includes technologies that combine a camera and depth sensor, or use a stereo camera, as well as technologies that use only monocular images as input. When only monocular images are input, the distance units are unknown in the map generated by VLSAM, and separate scaling is required to convert them into positions and distances in real space. For this reason, auxiliary sensors other than imaging devices, such as wheel odometry, are generally required for scaling.

[0004] A technology disclosed in Japanese Patent Laid-Open Publication No. 2021-092465 (Patent Document 1) is known in relation to the self-localization described above. Patent Document 1 discloses a system that corrects the estimated position of an unmanned aircraft, which is equipped with an imaging device capable of capturing images of the upper sky, extracts feature points from images acquired by the imaging device, and flies while estimating its own position based on the feature points, while capturing images of a bridge from below the bridge based on a predetermined flight plan route. In the conventional technology disclosed in Patent Document 1, a connecting route has a first end and a second end in the flight order of the unmanned aircraft, and the first end of the flight plan route is set as a waypoint. When the unmanned aircraft, while flying along the crossing route, detects that an area where the feature points are distributed corresponds to a bias in the image on the opposite side of the unmanned aircraft's direction of travel by at least a first predetermined degree before the estimated position of the unmanned aircraft estimated by self-localization reaches the waypoint, the estimated position of the unmanned aircraft at that time is corrected so that it corresponds to the waypoint.

[0005] However, the conventional technology disclosed in Patent Document 1 determines whether a moving object has reached a waypoint based on the distribution of image features, which is uncertain. Furthermore, even if the algorithm determines that a moving object has reached a waypoint, the current location of the moving object does not necessarily coincide with the waypoint, making it difficult to accurately identify the object's location. In particular, such errors make it difficult to accurately identify the work location when sorting or picking packages or pallets in a situation where there is little space between them. Therefore, the problem of difficulty in accurately identifying a location has not yet been fully resolved. Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure has been made in consideration of the above points, and aims to provide an information processing system that enables simple and accurate scale correction of map information created using images captured by an imaging device as input. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present disclosure provides an information processing system for creating map information using an imaging device provided in a mobile body, the information processing system having the following features: The information processing system includes a map information generation unit that generates map information based on a plurality of images captured by the imaging device along a route between predetermined start and end positions of the imaging device, and a map correction unit that generates map information with scale correction based on a virtual distance between a start point corresponding to the start position and an end point corresponding to the end position on the map information, and a measured distance given in advance for a pair of the start and end positions of the imaging device. [Effects of the Invention]

[0008] With the above configuration, it is possible to easily and accurately correct the scale of map information created using an image captured by an imaging device as an input. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 10 is a diagram illustrating an example of creating map information within a warehouse. [Figure 2] 1 is a diagram illustrating an example of the overall configuration of a map creation system according to an embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a server device that constitutes the map creation system according to the present embodiment. [Figure 4] FIG. 1 is a block diagram illustrating an example of the functional configuration of a map creation system according to an embodiment of the present invention. [Figure 5] FIG. 2 is a block diagram showing a detailed functional configuration around a mobile object position acquisition unit in the map creation system according to the present embodiment. [Figure 6] 10A and 10B are diagrams illustrating a scale correction process for map information in the map creation system according to the present embodiment. [Figure 7]5 is a flowchart showing an example of a map creation process performed by a server device constituting the map creation system according to the present embodiment. [Figure 8] 10 illustrates an example of a map display screen displayed on a display unit provided in a forklift truck in the map creation system according to this embodiment. [Figure 9] 10 illustrates an example of a map display screen displayed on a browser screen on the display of a server device in the map creation system according to this embodiment. [Figure 10] 10 illustrates an example of a map display screen that shows the destination of an object to be transported by a forklift, which is displayed on a browser screen on the display of the server device in the map creation system according to this embodiment. [Figure 11] A method for setting the start and end positions of measurement by the omnidirectional camera in the map creation system according to this embodiment will be described. [Figure 12] Another method for setting the start and end positions of measurement by the omnidirectional camera in the map creation system according to this embodiment will be described. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described, but the embodiment of the present invention is not limited to the embodiment described below. Note that in the embodiment described below, as an example of an information processing system and an information processing device, a map creation system 1 including a forklift 10 as a mobile body, an omnidirectional camera 20 as an imaging device, and a server device 50 as an information processing device will be described with reference to the server device 50 as an information processing device. Note that in each drawing, the same components are denoted by the same reference numerals, and duplicated explanations may be omitted.

[0011] The information processing device according to this embodiment estimates the position of a moving object (imaging device) based on images captured by the imaging device while the moving object is equipped with an imaging device, and generates environmental map information. Although not particularly limited, in a specific embodiment, the moving object may be a forklift, and the imaging device may be a wide-angle camera such as a spherical camera. The information processing device according to this specific embodiment estimates the position of the forklift 10 (spherical camera 20) based on images captured by the spherical camera 20 mounted on the forklift 10 while the forklift 10 is moving along a predetermined route, and generates an environmental map of the location where the forklift 10 is moving.

[0012] Here, Fig. 1 is a diagram for explaining an example of creating map information in a warehouse. Fig. 1 shows the inside of a warehouse 100 and the surrounding area of ​​the warehouse 100 as viewed from above (the ceiling side).

[0013] Warehouse 100 is a terminal warehouse (a warehouse established at a transit point in transportation). This terminal warehouse is a type of warehouse known as a cross-docking type. In a cross-docking type warehouse, multiple pallets for each product are received from a factory or wholesaler, and temporarily stored in the warehouse. Then, at the time of shipment, multiple types of pallets are combined while still packed on the same pallet, and shipped to the respective retail stores.

[0014] In Fig. 1, a truck yard 200 is located around a warehouse 100. Fig. 1 shows that the truck yard 200 has detached containers 300 transported by trailers and truck beds connected to the warehouse. A forklift 10 removes a pallet 31 from at least one of the beds of trucks that have arrived at the truck yard 200 and the containers 300 transported by trailers.

[0015] Thereafter, the forklift 10 carries the pallet 31 to the temporary storage location 40 and stores it there temporarily. Thereafter, at the time of shipping, the forklift 10 carries the pallet 31 to a location close to the truck yard 200 in the warehouse 100, arranges the items, and then loads the pallet 31 onto the bed of a truck or into a container 300.

[0016] In order to ensure flexible space for the daily changes in the types and quantities of goods coming in and out, temporary storage locations 40 often do not have designated sections for each product. However, because multiple workers temporarily store pallets 31 in arbitrary locations, when shipping, it is necessary to search for the desired pallet from among the multiple temporarily stored pallets.

[0017] To efficiently perform this search operation, it is necessary to recognize and track the movement of the pallet 31 within the warehouse 100, while effectively utilizing space by not specifying a temporary storage location for the pallet 31, and to visualize it. Furthermore, to recognize and track the movement of the pallet 31 within the warehouse 100, a technology is required that can accurately estimate the position of the forklift 10 and create highly accurate map information within the warehouse. In particular, VSLAM technology, which simultaneously estimates the forklift's own position and creates environmental map information based on captured images, is often used to create map information. The map creation system 1 according to this embodiment is configured in an environment such as that shown in FIG. 1, by installing an imaging device on a moving object such as the forklift 10 and applying VSLAM technology based on images from the imaging device to simultaneously estimate the forklift's own position and create an environmental map.

[0018] For general information on the technical details of VSLAM, see, for example, "Explanation: Current Status and Future Prospects of SLAM," by Tomono Masahiro et al., Systems / Control / Information, 2020, Vol. 64, No. 2, pp. 45-50 (https: / / www.jstage.jst.go.jp / article / isciesci / 64 / 2 / 64_45 / _article / -char / ja / ), and Sumikura, S, et. al., "OpenVSLAM: A Versatile Visual SLAM Framework," in MM '19: Proceedings of the 27th ACM International Conference on Multimedia, October 2019, Pages 2292-2295 (https: / / dl.acm.org / doi / 10.1145 / 3343031.3350539).

[0019] The map creation system 1 according to this embodiment will be described below.

[0020] (Example of the overall configuration of Map Creation System 1) Fig. 2 is a diagram showing an example of the overall configuration of a map creation system 1 according to this embodiment. As shown in Fig. 2, the map creation system 1 includes a forklift 10 as a mobile object, a spherical camera 20 as an imaging device, a server device 50 as an information processing device, and a touch panel information terminal 23 as a display device provided on the mobile object. These are communicably connected via a network 400 such as a LAN (Local Area Network). Note that devices other than those described above, such as an external server or an image forming device, may also be communicably connected to the network 400.

[0021] The forklift 10 is an example of a mobile body that transports the pallet 31 and the cargo 32 by holding and transporting cargo 32 placed on the pallet 31. Transport by a mobile body is an example of movement by a mobile body. The pallet 31 and the cargo 32 are each an example of an object. In the following, when there is no particular need to distinguish between the pallet 31 and the cargo 32, they will be collectively referred to as the object 30. The forklift 10 is a generic term for multiple forklifts, the pallet 31 is a generic term for multiple pallets, and the cargo 32 is a generic term for multiple cargoes.

[0022] The forklift 10 may transport the object 30 in response to the driving operation of an operator, or may transport the object 30 by automatic driving without the intervention of an operator.

[0023] The omnidirectional camera 20 is an example of an imaging device provided on the forklift 10. The omnidirectional camera 20 is a camera that can capture images in all directions of 360 degrees around the omnidirectional camera 20. The direction 20a indicates the direction in which the omnidirectional camera 20 can capture images.

[0024] The omnidirectional image (all-directional image) captured by omnidirectional camera 20 is one example of a captured image. However, the imaging device is not limited to omnidirectional camera 20, and may be any other wide-angle camera, a camera with a normal angle of view, or anything that can capture an image of the area around forklift 10. Furthermore, the captured image does not necessarily have to be a omnidirectional image.

[0025] The spherical image includes an image capturing a scene in the conveying direction 11 of the object 30 as seen from the forklift 10, and a scene in the vertically upward direction 12 as seen from the forklift 10. In other words, the conveying direction 11 is in front of the forklift 10, and the vertically upward direction 12 is above the forklift 10. Since the spherical camera 20 can capture images in all directions, it can capture an image including both the front and the above of the forklift 10 in a single image. The conveying direction 11 is an example of a moving direction.

[0026] The spherical camera 20 is preferably mounted on the roof of the forklift 10 or on a support member 22 that supports the forks 21. This ensures a good field of view for capturing images in front of and above the forklift 10.

[0027] The omnidirectional camera 20 has a wireless communication function and transmits the captured omnidirectional image to the server device 50 via the network 400.

[0028] The touch panel information terminal 23 is a terminal device equipped with a display device provided on the mobile body side, and can display map information created by the map creation system 1 on a touch panel. The touch panel information terminal 23 has a wireless communication function, receives map information from the server device 50 via the network 400, and outputs the map information as an image on its own display device. Examples of the touch panel information terminal 23 include a tablet terminal and a small liquid crystal display equipped with a touch screen sensor. Note that the touch panel information terminal 23 will be described as having calculation and display functions as a terminal, but is not limited to this, and each function may be configured as a separate device, and the specific implementation is not particularly limited.

[0029] The map information can also be displayed on a liquid crystal display 506 provided in the server device 50 or connected to the server device 50 .

[0030] In the described embodiment, the omnidirectional camera 20 and the touch panel information terminal 23 each have a communication function and communicate with the server device 50 via the network 400. However, the present invention is not limited to this. In another embodiment, the omnidirectional camera 20 may communicate with the touch panel information terminal 23 via a wired or wireless connection, and may communicate with the server device 50 via the touch panel information terminal 23 via the network 400. In another embodiment, the touch panel information terminal 23 may communicate with the server device 50 via the omnidirectional camera 20 via the network 400.

[0031] The cargo 32 is provided with a barcode 33, which is an example of identification information that identifies the cargo 32. Such a barcode may be provided on the pallet 31 and used as identification information that identifies the pallet 31. The barcode 33 is read by a reader such as a barcode reader, and the identification information resulting from the reading is transmitted to the server device 50 via the network 400. Note that the identification information is not limited to a barcode, and may be a QR code (registered trademark), an ID (identifier) ​​number, or the like.

[0032] Server device 50 is an example of an information processing device that estimates position information of forklift 10 and creates map information of the interior of warehouse 100. Server device 50 may also be installed in warehouse 100 and process position information of object 30 transported by forklift 10. Server device 50 can process the position information of object 30 based on the omnidirectional image and identification information indicating object 30 received via network 400. Server device 50 can also acquire position information of object 30 held by multiple forklifts 10 by using the omnidirectional images captured by multiple omnidirectional cameras 20, respectively.

[0033] (Example of hardware configuration of server device 50) 3 is a block diagram showing an example of the hardware configuration of the server device 50. The server device 50 is constructed by a computer.

[0034] 3, the server device 50 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, and a display 506. The server device 50 also includes an external device connection I / F (Interface) 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0035] Of these, the CPU 501 controls the overall operation of the server device 50. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0036] The HD 504 stores various data such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, menus, windows, characters, or images. The display 506 can display map information as described above.

[0037] The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using the network 400. The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501 shown in FIG. 3.

[0038] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0039] Although the above describes the hardware configuration of the server device 50, the hardware configuration of the touch panel information terminal 23 provided on the forklift 10 can also be implemented in a similar configuration to that shown in Figure 3 by adding or deleting components as needed.

[0040] (Example of functional configuration of map creation system 1) Fig. 4 is a block diagram showing an example of the functional configuration of the map creation system 1. As shown in Fig. 4, the server device 50 includes a communication unit 51, a mobile object position acquisition unit 52, a retained information acquisition unit 53, an identification information acquisition unit 54, a time acquisition unit 55, an object position acquisition unit 56, an output unit 57, a storage unit 58, and an input unit 59.

[0041] 3 operates in response to an instruction from CPU 501 in accordance with a program loaded from HD 504 onto RAM 503. Storage unit 58 is configured to include storage areas provided by HD 504 to RAM 503, etc.

[0042] The forklift 10 is also provided with a spherical camera 20, a communication unit 101, and a display unit 102. The function of the communication unit 101 is realized by an electric circuit provided in any one of the forklift 10, the spherical camera 20, or the touch panel information terminal 23, or can also be realized by software (CPU). It may also be realized by a plurality of circuits or a plurality of pieces of software. The function of the display unit 102 is realized by an LCD (Liquid Crystal Display) or organic EL (electroluminescence) panel and electric circuit provided in the touch panel information terminal 23 shown in FIG. 2.

[0043] The server device 50 estimates the position information of the forklift 10 based on the omnidirectional image captured by the omnidirectional camera 20, and generates environmental map information. The server device 50 can also acquire the position information of the object 30 based on the position information of the forklift 10 and holding information indicating whether the forklift 10 is holding or not holding the object 30. The server device 50 can then output the estimated position information of the forklift 10, the generated map information, and the acquired position information of the object 30 to the outside via the output unit 57.

[0044] Communication unit 51 receives, via network 400, the spherical images captured by omnidirectional camera 20 on forklift 10 and transmitted via communication unit 101, and outputs the images to mobile object position acquisition unit 52 and retained information acquisition unit 53. Communication unit 51 also receives, via network 400, identification information read by a reader such as a barcode reader, and outputs the information to identification information acquisition unit 54.

[0045] Mobile object position acquisition unit 52 estimates the position of forklift 10 (omnidirectional camera 20) based on the multiple omnidirectional images that are continuously input, creates map information, and stores the created map information and position information in storage unit 58. Here, input unit 59 receives input of the measured distance between two points (start position and end position of a distance measurement section) that have been measured in advance as a parameter to be used when creating the map, and stores it in storage unit 58.

[0046] The generated map information and estimated position information may be output via output unit 57. The output destination of output unit 57 may be an external device such as a PC (Personal Computer), a display device such as display 506, or a storage device such as HD 504. The generated map information and estimated position information may be transmitted by communication unit 51 to touch panel information terminal 23 on forklift 10 via network 400 and displayed on touch panel information terminal 23. Mobile object position acquisition unit 52 also outputs the position information of forklift 10 acquired by calculation to object position acquisition unit 56.

[0047] The holding information acquisition unit 53 can acquire holding information indicating whether the object 30 is being held or not held by the forklift 10 through calculation based on the input spherical image, and output the information to the object position acquisition unit 56.

[0048] The identification information acquisition unit 54 acquires the identification information by inputting the identification information from the communication unit 51, and outputs the identification information to the object position acquisition unit 56. However, acquisition of the identification information by the identification information acquisition unit 54 is not limited to via the network 400. For example, the identification information acquisition unit 54 may acquire identification information input by a user such as a manager using the keyboard 511 or the pointing device 512 in FIG. 3, or may acquire identification information stored in advance in the storage unit 58, or may acquire identification information via the external device connection I / F 508. The manager is the manager of the map creation system 1 or the warehouse 100.

[0049] The time acquisition unit 55 acquires information indicating the time when the communication unit 51 receives the spherical image and the identification information, and outputs the information to the object position acquisition unit 56 .

[0050] The object position acquisition unit 56 can acquire the position information of the object 30 based on the position information of the forklift 10 and the holding information. The object position acquisition unit 56 may also output the position information of the object 30, identification information indicating the object 30, and time information via the output unit 57 in a mutually associated manner.

[0051] The storage unit 58 can store parameters used in creating the map, as well as estimated position information of the forklift 10, generated map information, and identification information indicating the object 30 such as a pallet 31 or cargo 32.

[0052] (Details of the map creation function of Map Creation System 1) FIG. 5 is a block diagram showing a more detailed functional configuration around the mobile object position acquisition unit 52 in the map creation system 1.

[0053] The mobile object position acquisition unit 52 estimates the position of the forklift 10 (the omnidirectional camera 20) and creates map information based on the plurality of omnidirectional images continuously input from the communication unit 51 and the parameters read from the storage unit 58.

[0054] Here, the estimated position will be described assuming that the omnidirectional camera 20 is fixed to the forklift 10 and that the position of the omnidirectional camera 20 (the position of the forklift 10) is estimated. However, the position of the omnidirectional camera 20 does not necessarily need to be fixed to the forklift 10. In other embodiments, the omnidirectional camera 20 may be displaceable relative to the forklift 10. In an embodiment in which the omnidirectional camera 20 is displaceable relative to the forklift 10, the mobile object position acquisition unit 52 directly estimates the position of the omnidirectional camera 20, but the position of the forklift 10 is found from the position of the omnidirectional camera 20 based on a known displacement.

[0055] More specifically, the mobile object position acquisition unit 52 includes a map information generation unit 60 and a map correction unit 61.

[0056] The map information generator 60 generates map information based on multiple images captured by the spherical camera 20 while the forklift 10 moves along a predetermined route and continuously input during that time. The route of the forklift 10 typically travels from a predetermined start position to a predetermined end position, passing through multiple visit points. The route of the forklift 10 may include revisited points that are visited multiple times, or may include a closed loop portion where the route portion connecting the multiple visit points makes a full circle.

[0057] More specifically, the map information generating unit 60 includes a tracking unit 64 , a local optimization unit 65 , and a global optimization unit 66 .

[0058] The tracking unit 64 detects key points for all frames that are continuously input, performs key point matching, and performs pose optimization to estimate the pose of the omnidirectional camera 20. The tracking unit 64 also determines a key frame to determine whether to insert a new key frame. When the tracking unit 64 registers an appropriate frame as a new key frame, it outputs the key frame to be registered to the local optimization unit 65 and the global optimization unit 66. The self-location estimation result is output to the map correction unit 61.

[0059] The local optimizer 65 performs optimization on frames near the latest frame. The local optimizer 65 triangulates points on 3D coordinates using the inserted keyframe, thereby creating and expanding a map. The local optimizer 65 also performs local bundle adjustment. The partial map around the keyframe is referred to as the local map 67.

[0060] If loop closure is possible in the travel area for which a map is to be created, it is preferable to perform loop closure by setting revisit points in the route of the forklift 10 to eliminate distortions in the map. The global optimization unit 66 performs overall optimization using loop closure. The global optimization unit 66 executes loop detection, pose graph optimization, and global bundle adjustment. This optimization eliminates distortions in the overall map as much as possible. The overall map obtained so far is referred to as a global map 68.

[0061] The omnidirectional camera 20 has multiple optical systems and a field of view in all directions. However, unlike a stereo camera, the generated images are equivalent to images captured with a single eye. Therefore, the map information (local map 67 and global map 68) generated by the map information generation unit 60 has an unclear distance unit, and scaling (scale correction) is required to convert the distance into positions and distances in real space. The map correction unit 61 corrects the scale of the map information 67, 68 based on predetermined parameters, and generates map information with the corrected scale. In this embodiment, this scale correction is performed based on the virtual distances between the start and end points on the map information 67, 68 corresponding to the start and end positions of the omnidirectional camera 20 (the forklift 10) and the measured distances given in advance for the pair of the start and end positions.

[0062] (Map correction processing) Hereinafter, with reference to FIG. 6, a process for generating map information in a predetermined distance unit by correcting the scale of map information in which the distance unit is unknown will be described.

[0063] As described above, the distances on the VSLAM system (map) generated by the monocular camera deviate from the actual distances because the distance units are unknown. Scaling is necessary to resolve this. In this embodiment, at least before scaling, as shown in FIG. 6(A), the distance between the corresponding start position A and end position B of the omnidirectional camera 20 when the forklift 10 is parked in a predetermined posture at each of the measurement start point and measurement end point (for example, when the forklift 10 is parked in a predetermined posture facing a car stop provided at a predetermined position) is measured, known, and stored in storage unit 58.

[0064] Although the distance unit is unknown in the map information 67, 68 generated by the map information generation unit 60, some distance is defined between the start point and end point on the map information 67, 68 of the omnidirectional camera 20 that correspond to the start position and end position, respectively. The map correction unit 61 calculates a correction coefficient for converting a virtual distance in any unit (virtual distance) between the start point and end point of the omnidirectional camera 20 on the map information 67, 68 into the actually measured distance between the start position A and the end position B of the omnidirectional camera 20.

[0065] The map information generated (VLSAM-generated) by the map information generation unit 60 described above includes a set of points on three-dimensional coordinates with no fixed distance unit. As shown in Fig. 6(B), the map correction unit 61 applies the correction coefficient to each of the included sets of points on three-dimensional coordinates (including the coordinates of the key frames) to generate map information corresponding to the actual distance.

[0066] In the above description, it has been explained that the forklift 10 comes to a stop against, for example, a bollard provided at the measurement start point and the measurement end point, and the omnidirectional camera 20 is positioned at the start position A and the end position B, respectively.

[0067] However, the method for ensuring that the omnidirectional camera 20 is located at the start position A and the end position B is not limited to this, and the form thereof is not particularly limited as long as the conditions that the forklift 10 is located at each point and the three-dimensional distance between each point can be accurately measured are satisfied. For example, the forklift 10 may be provided with a means (a start-side detection unit or an end-side detection unit) that detects that the omnidirectional camera 20 is placed at a predetermined position and in a predetermined attitude so as to be located at the start position A or the end position B. For example, in a specific embodiment, markings or flags may be provided on the road surface, and the start-side detection unit and the end-side detection unit may detect and confirm that the forklift 10 is located at each point by performing image recognition on images captured by the omnidirectional camera 20 with the markings or flags included in its field of view. Alternatively, the presence of the forklift 10 or the omnidirectional camera 20 at a predetermined position may be detected by providing other sensors at bollards, for example.

[0068] Furthermore, the method for measuring the distance between two points is not particularly limited. Measurement may be performed using any distance measurement method, such as a laser distance sensor or an ultrasonic distance sensor, and the measurement value input from the sensor may be stored in storage unit 58. Alternatively, the distance value measured using any method may be stored in storage unit 58 manually.

[0069] (Map creation process by server device 50) FIG. 7 is a flowchart showing an example of a map creation process by the server device 50. The process shown in FIG. 7 is executed in response to the server device 50 receiving an instruction to start map creation. The instruction to start map creation may be given by a user such as an administrator using, for example, the pointing device 512 in FIG. 3. For example, the driver of the forklift 10 used for map creation may wait at the start point and notify the administrator that he or she is ready, and then give the instruction. Alternatively, the driver of the forklift 10 used for map creation may notify the touch panel information terminal 23 that he or she is ready, and the notification may be received via the network 400, thereby triggering the start of map creation.

[0070] First, as a preliminary preparation for the process shown in Fig. 7, bollards are installed at the positions that will be the measurement start point and measurement end point in the area to be mapped. The bollards are installed so that when forklift 10 is parked against the bollards installed at these two points, the position (start position A and end position B) of omnidirectional camera 20 installed on forklift 10 will be in an appropriate position and the distance between the two points will be known. It is also desirable that start position A and end position B be locations that can be easily illustrated on a design drawing of the work area.

[0071] The process shown in FIG. 7 starts at step S100, and at step S101, the server device 50 reads out from the storage unit 58 the measured distance predefined for the pair of the start position A and the end position B as a correction parameter.

[0072] In step S102, the server device 50 starts map creation assuming that the omnidirectional camera 20 is located at the start position A. Map creation may be started when the forklift 10 stops at a stop sign at the start point and, for example, in response to a notification from the driver of the forklift 10 that preparations for starting map creation are complete. After map creation starts, for example, the driver starts the forklift 10 traveling along a predetermined route.

[0073] In step S103, in response to the forklift 10 completing traveling along the entire route, the server device 50 generates a map using the map information generator 60. Note that the map obtained at this stage has an indefinite distance unit. Here, the forklift 10 travels along all possible routes within the work area, and as it travels, the omnidirectional camera 20 captures images of the forklift 10, and the resulting omnidirectional images are continuously input to the server device 50. In step S103, the local optimizer 65 optimizes the latest frame with respect to nearby key frames.

[0074] In step S104, in response to the completion of the closed-loop travel by the forklift 10, the server device 50 performs a map update. Note that the map obtained at this stage also has an indefinite distance unit. Here, if a location allows for loop closure, a revisit point is set to eliminate distortion in the map, and the loop is closed. The global optimization unit 66 detects this loop and eliminates distortion in the map as much as possible through global optimization.

[0075] In step S105, the forklift 10 is stopped by hitting the buffer stop at the end point, and, for example, in response to a notification from the driver of the forklift 10 that map creation is complete, the server device 50 completes map creation by assuming that the omnidirectional camera 20 is positioned at end position B. At this time, the driver of the forklift 10 stops the forklift 10 at the buffer stop set up at the end point as expected, and after stopping the forklift 10, notifies the server device 50 using the touch panel information terminal 23 or the like that the forklift 10 has stopped at the end position.

[0076] In step S106, server device 50 performs scale correction using map correction unit 61 to calculate a scale coefficient based on the virtual distance between the start point corresponding to the start position and the end point corresponding to the end position on the map information generated without a fixed distance unit, and the actual measured distance given in advance for the pair of start position and end position of omnidirectional camera 20 prepared in step 101. A coefficient is calculated that can convert the distance from start position A to end position B into a distance unit in real space (meters, for simplicity's sake).

[0077] In step S107, the server device 50 uses the calculated scale coefficient to perform scaling using the map correction unit 61 to generate scale-corrected map information. More specifically, in step S107, the scale coefficient calculated in step S106 is applied to each piece of coordinate information included in the map information. By multiplying the coordinate information of the point cloud and the coordinate information included in the key frame by the scaling coefficient, the coordinate values ​​on the system are converted to meters, and the self-localization results using that map are also converted to meters thereafter.

[0078] (Scale-corrected map visualization) Below we explain how to visualize scale-corrected map information on a blueprint. Scale-corrected map information is a collection of points on a 3D coordinate system. To visualize it on a 2D map, the point cloud is projected onto the XZ plane (horizontal plane) in the camera coordinate system to create a bird's-eye view from the point cloud map, and image data is generated that serves as a point cloud-based orthoimage. Here, an orthoimage refers to an image that has been orthogonally transformed into one in which each point is positioned in the correct position (the distance between two points is correct), as if viewed from directly above, without tilt.

[0079] The ortho-view created above is fitted to the design map information of the work area. At this time, the scale-corrected map information is enlarged or reduced so that the corrected start point corresponding to the start position A on the map information and the corrected end point corresponding to the end position B on the map information coincide with the design start point and design end point corresponding to the start position on the design map information of the predefined travel area, and is also rotated or translated, or both, so that the self-location estimation result is superimposed on the design map and visualized.

[0080] The output destination for showing the position of the forklift 10 on the design map is preferably the display unit 102 (a display device such as a touch panel) provided on the forklift 10. Alternatively, the output destination of the output unit 57 may be an external device such as a PC or a display device such as the display 506.

[0081] Fig. 8 shows an example of a map display screen 90 displayed on the display unit 102 of the forklift 10. The map display screen shown in Fig. 8 shows the position 92 of the forklift 10 and the positions of objects 30 such as luggage and pallets on a design map 91.

[0082] 9 shows an example of a map display screen 93 displayed on a browser screen on the display 506 of the server device 50. The map display screen 93 shown in FIG. 9 also shows the current position 95 of the forklift and the positions of objects 30 such as luggage and pallets on a design map 94.

[0083] The method for visualizing a scale-corrected map on a design drawing has been described above with reference to Figures 8 and 9. However, the method for displaying scale-corrected map information is not particularly limited.

[0084] For example, scale-corrected map information can be displayed as image data that becomes a point cloud-based ortho-view. In this case, a scale bar (displaying distance units) can be displayed with scale marks. FIG. 10 illustrates a map display screen 96 displayed on a browser screen on the display 506 of the server device 50, showing the destination of the object 30 transported by the forklift 10. The display screen 96 displays a location map 97. The location map 97 includes a source location 98A and a destination location 98B. The source location 98A indicates the source location, and the destination location 98B indicates the destination location. Displaying the location of a pre-registered temporary storage location or the like on the screen can provide information that makes work easier. FIG. 10 also illustrates a scale bar 99. The scale bar 99 is an example of a GUI component (graphical display) that indicates the displayed scale and is attached when displaying scale-corrected map information.

[0085] 10 is created by the mobile object position acquisition unit 52 based on the omnidirectional image captured by the omnidirectional camera 20. The position map 97 is created by projecting a point cloud including the acquired three-dimensional coordinate information onto a two-dimensional plane. Although it is assumed that there are multiple forklifts 10, the position map 97 may be created, for example, according to the movement of one forklift 10 and used to acquire the position information of all the forklifts 10 (self-position recognition). This makes it possible to express the positions of the multiple forklifts 10 in the same coordinate system.

[0086] A scale bar 99 may be displayed on the map display screen shown in FIGS.

[0087] (How to set the start and end positions) In the above explanation, it has been simply explained that the start position A and end position B of the omnidirectional camera 20 are set at locations that can be easily illustrated on a blueprint of the work area. Below, a method for setting the start position A and end position B will be described in more detail.

[0088] When using wheel chocks for the forklift 10 at the measurement start point and measurement end point, the chocks may be for either the front or rear wheels. When the forklift 10 is at the specified start point and end point in a specified attitude (direction), it is required that the omnidirectional camera 20 (its center) is reliably at the start position A and the end position B. In other words, the forklift 10 must be parked at the specified position in a specified attitude, and as long as the start and end states are known, the form of the forklift 10 is not important.

[0089] On the other hand, it is desirable that the distance between the start and end positions be as long as possible in a straight line in an open area. This is because it is assumed that a globally optimized map with loop closure is used for self-localization, and the estimation error is theoretically distributed evenly anywhere on the map, so the smaller the proportion of the error relative to the measured distance, the smaller the error in the scaling coefficient will be.

[0090] For example, as shown in FIG. 11, it is possible to use known distances (distance H from one wall of the warehouse 100 to the opposing wall and displacement W between start position A and end position B in a direction parallel to these walls) that are listed on a design drawing of the area within the warehouse 100 where the forklift 10 can move. In that case, as shown in FIG. 12(A), the distance D between the wall W and the omnidirectional camera 20 is measured with the tires of the forklift 10 in contact with the car stopper S of the rear wheels (or the front wheels, as another example) (the horizontal distance between the contact point of the car stopper with the tire and the camera and the horizontal distance between the contact point of the car stopper and the wall are measured and added together), and the distance D (2D) is subtracted from the distance H between the walls to obtain the actual measured distance (L) between the cameras in the start state and the end state. 2 =(H-2D) 2 +W 2 ) can be obtained.

[0091] (Modification of the method for setting the start position and end position) In the above-described embodiment, the omnidirectional camera 20 is fixed to the forklift 10, and the predetermined start and end positions of the omnidirectional camera 20 correspond to states in which the forklift 10 is stopped at different points within a movable area. However, this is not limiting, and in another embodiment, as shown in FIG. 12(B), the omnidirectional camera 20 can be mounted on a single-axis linear motion mechanism 13 that is fixed to the forklift 10. In this case, by changing the amount of movement of the single-axis linear motion mechanism between the start and end of measurement, it is possible to make the start position A and end position B of the omnidirectional camera 20 different while stopping the forklift 10 at the same point. In this case, the physical distance converted from the set amount of movement (the number of pulses of the pulse motor) is stored as a parameter in storage unit 58.

[0092] The single-axis linear motion mechanism 13 may be, for example, a mechanism such as a linear ladder truck, or a stage that slides on rails. Although the above example has been described as a single-axis linear motion mechanism, the mechanism may take any form as long as the omnidirectional camera 20 can move between two points whose distance in real space is known.

[0093] Additionally, although the distance between the start position A and the end position B has been described as being measured using a separate device such as a laser distance sensor or an ultrasonic sensor, it is also possible to measure the coordinates of the positions (start position and end position) of the omnidirectional camera 20 using a beacon and a receiving device, and perform scale measurement with the distance between the two points known.

[0094] Furthermore, in the above-described embodiment, the measured distance is set between two points, the measurement start position A and the measurement end position B, but in cases where the distance is not a straight line, a relay point C may be provided midway for a temporary stop, and the measured distance between the measurement start position A and the measurement end position B may be defined by dividing it into the measured distance between the measurement start position A and the relay point C and the measured distance between the relay point C and the measurement end position B. However, from the viewpoint of reducing sources of error, it is preferable to use the measured distance measured directly between the two points, the measurement start position A and the measurement end position B.

[0095] Furthermore, the start position A and the end position B are points for measuring virtual distances in the sense that they correspond to two actually measured points, but they do not necessarily have to coincide with the start and end points for map creation. For example, map creation may start before departing from a stopped state at the start position A, or map creation may continue as the vehicle continues moving after stopping at the end position.

[0096] (Modification of functional part arrangement) In the embodiment described below, the mobile object position acquisition unit 52 is described as being provided in the server device 50. However, the mobile object position acquisition unit 52 may be provided in a device other than the server device 50. In other embodiments, the mobile object position acquisition unit 52 may be provided in a device on the forklift 10 side, such as the touch panel information terminal 23 or the omnidirectional camera 20, or in another computer device mounted on the forklift 10. In this case, the generated map information may be used for automatic driving or autonomous traveling of the forklift 10.

[0097] In still another embodiment, the mobile object position acquisition unit 52 may be provided on the cloud (its physical or virtual server), and may be provided as a cloud service that provides map information generated in response to continuous input of omnidirectional images from the omnidirectional camera 20 (as video data or as a sequence of still images, and as files or as stream data).

[0098] (advantage) As described above, according to this embodiment, it is possible to provide an information processing system, an information processing device, an information processing method, and a program that can easily and accurately correct the scale of map information created using an image captured by an imaging device as input.

[0099] In particular, in a configuration in which self-location estimation and environmental map information are simultaneously performed based on captured images, it is possible to correct the scale of map information with indefinite units to map information with a length in a predetermined unit. In this case, the start and end positions of a known distance section are specified, and when map creation is completed (or when the end point of the measured distance section is reached), the travel distance in the system is converted to the known measured distance, allowing accurate map scaling processing to be performed automatically.

[0100] Another possible scaling method is to use an object with a known size, but this method can result in object recognition and reprojection errors and is not necessarily accurate. Furthermore, the pixel size and resolution of the camera affect the estimation accuracy, so the error contained in the true measurement value varies from camera to camera. In contrast, the scale correction method described above has the advantage that the error that can be contained in the true measurement value is constant regardless of the camera model as long as the same measurement method is used, so the camera's resolution and pixel size do not affect the measurement error of the true value.

[0101] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each of the above-described functions.

[0102] The devices described in the embodiments are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In some embodiments, the information processing apparatus includes a plurality of computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0103] The above has described the embodiments and examples of the present invention, but the embodiments and examples of the present invention are not limited to the above-described embodiments and examples, and may be modified within the scope of what a person skilled in the art could conceive, such as other embodiments, other examples, additions, changes, deletions, etc., and any aspect is included in the scope of the present invention as long as it exhibits the functions and effects of the present invention. [Explanation of symbols]

[0104] 1...map creation system, 10...forklift (an example of a moving body), 11...conveying direction (an example of a moving direction), 12...vertical upward direction, 20...omnidirectional camera (an example of an imaging device), 20a...orientation, 21...fork, 22...support member, 30...object, 31...pallet, 32...cargo, 33...barcode, 40...temporary storage location, 50...server device (an example of an information processing device), 51...communication unit, 52...moving body position acquisition unit, 53...retained information acquisition unit, 54...identification information acquisition unit, 55...time acquisition unit, 56...object position acquisition unit, 57...output unit, 58...storage unit, 59...input unit, 60...map information generation unit, 61...map correction unit, 62...omnidirectional image (an example of an image), 63...parameter, 64...traffic Stacking unit, 65...local optimization unit, 66...global optimization unit, 67...local map, 68...global map, 100...warehouse, 101...communication unit, 102...display unit, 200...truck yard, 300...container, 400...network, 500...personal computer, 501...CPU, 502...ROM, 503...RAM, 504...HDD, 505...HDD controller, 506...display, 508...external device connection I / F, 509...network I / F, 511...keyboard, 512...pointing device, 514...DVD-RW drive, 513...DVD-RW media, 516...media I / F, 515...recording media [Prior art documents] [Patent documents]

[0105] [Patent Document 1] Patent Publication No. 2021-092465

Claims

1. An information processing system for creating map information using an imaging device provided in a moving object, a map information generating unit that generates map information based on a plurality of images captured by the imaging device along a route between different start positions and end positions of the imaging device in space; a map correction unit that generates map information corrected to a distance unit in the space based on a virtual distance between a start point on the map information corresponding to the start position and an end point on the map information corresponding to the end position, and an actual measured distance that is given in advance for a pair of the start position and the end position of the imaging device; An information processing system comprising:

2. The information processing system according to claim 1 , further comprising a display device that displays the corrected map information.

3. 3. An information processing system according to claim 2, wherein when the corrected map information is displayed on the display device, a graphic display indicating a scale on the display is added.

4. the display device is provided on the moving body, The information processing system of claim 2 or 3, wherein the corrected map information is enlarged or reduced so that the corrected start point corresponding to the start position on the map information and the corrected end point corresponding to the end position match the design start point and design end point corresponding to the start position in the design map information of the predefined movement area of ​​the mobile body, and is configured to show the position of the mobile body on the design map information.

5. The information processing system according to any one of claims 1 to 4, wherein the map information generated by the map information generation unit includes a set of points on three-dimensional coordinates with an indefinite distance unit, and the map correction unit applies a scale coefficient to each of the sets of points on the three-dimensional coordinates to convert the distance between the start point and the end point on the map information into the measured distance.

6. The imaging device is fixedly mounted on the moving body, and the predetermined start and end positions of the imaging device correspond to different points within a range in which the moving body can move, or 6. The information processing system according to claim 1, wherein the imaging device is provided so as to be movable relative to the moving body, and the predetermined start position and end position of the imaging device include a movement amount of the imaging device relative to the moving body.

7. The information processing system includes: A moving object and The imaging device provided on the moving body is a wide-angle camera that captures images of scenery in the moving direction of the moving body and scenery vertically above the moving body; a start-side detection unit that detects that the moving body is placed at a predetermined position and in a predetermined attitude so that the imaging device is at the start position; an end-side detection unit that detects that the moving body is placed at a predetermined position and in a predetermined attitude so that the imaging device is at the end position; 7. The information processing system according to claim 1, further comprising at least one of:

8. An information processing device for creating map information using an imaging device provided in a moving body, a map information generating unit that generates map information based on a plurality of images captured by the imaging device along a route between different start positions and end positions of the imaging device in space; a map correction unit that generates map information corrected to a distance unit in the space based on a virtual distance between a start point on the map information corresponding to the start position and an end point on the map information corresponding to the end position, and an actual measured distance that is given in advance for a pair of the start position and the end position of the imaging device; An information processing device comprising:

9. An information processing method for creating map information using an imaging device provided in a moving object, comprising: acquiring a plurality of images captured by the imaging device along a path between different start and end positions of the imaging device in space; generating map information based on the plurality of images; generating map information corrected to a distance unit in the space based on a virtual distance between a start point on the map information corresponding to the start position and an end point on the map information corresponding to the end position, and an actual measured distance given in advance for a pair of the start position and the end position of the imaging device; An information processing method, including:

10. A program for realizing an information processing device for generating map information using an imaging device provided in a moving object, the program comprising: a map information generating unit that generates map information based on a plurality of images captured by the imaging device along a route between different start positions and end positions of the imaging device in space; and a map correction unit that generates map information corrected to a distance unit in the space based on a virtual distance between a start point on the map information corresponding to the start position and an end point on the map information corresponding to the end position, and an actual measured distance that is given in advance for a pair of the start position and the end position of the imaging device; A program to function as a

Images