Conveyance device, conveyance system, and control method of conveyance device

Abstract

This conveyance device conveys an object to be conveyed in a travel space, and includes: a first sensor that acquires an image around the conveyance device; a control unit that estimates the position of the conveyance device on the basis of the image acquired by the first sensor; and an analysis unit that analyzes disturbances affecting the estimation of the position of the conveyance device based on the image acquired by the first sensor. The control unit includes: a position estimation unit that estimates the position of the conveyance device by using any of a plurality of position estimation modes including a first mode for estimating the position of the conveyance device on the basis of the image acquired by the first sensor; and a disturbance reduction unit that reduces the influence of the disturbances affecting the position estimation of the conveyance device in the first mode.

Description

Conveyance device, conveyance system, and method for controlling the conveyance device Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2023-144360, filed on September 6, 2023, the contents of which are incorporated herein by reference.

[0002] The present invention relates to a transport device, and more particularly to a control technique for eliminating factors that inhibit self-position estimation.

[0003] As a method for estimating the self-position of an automatic guided vehicle (AGV), a method is known in which a plurality of self-position estimation methods are combined to improve accuracy and robustness. For example, a technique is known in which a Visual SLAM (Simultaneous Localization and Mapping) method using image feature points is combined with a Wheel Odometry (hereinafter, referred to as "odometry") method using wheel rotation information (encoder information).

[0004] In SLAM position estimation, if an image taken in a disturbed environment is used, the accuracy of extracting feature points decreases, and the accuracy of position estimation also decreases.

[0005] The following prior art exists as background art in this technical field: Patent Literature 1 (JP 2021-96731 A) discloses a control device for an autonomously traveling automated guided vehicle that performs autonomous driving control based on a current self-position calculated by a self-position estimation process, the control device performing an image processing self-position estimation process that calculates a first self-position based on landmarks photographed by a photographing device mounted on the autonomously traveling automated guided vehicle, and an odometry self-position estimation process that calculates a second self-position based on driving information of the autonomously traveling automated guided vehicle, the control device including a self-position estimation processing unit that estimates the current self-position using the first self-position and the second self-position, and a storage unit that stores a calculation accuracy map of the image processing self-position estimation process within a predetermined area including a driving route, the self-position estimation processing unit changing the contribution of the first self-position and the second self-position in estimating the current self-position based on the calculation accuracy of the calculation accuracy map.

[0006] Patent Document 2 (WO 2018 / 38131) discloses a three-dimensional information processing method including an acquisition step of acquiring map data including first three-dimensional position information via a communication channel; a generation step of generating second three-dimensional position information from information detected by a sensor; a determination step of determining whether the first three-dimensional position information or the second three-dimensional position information is abnormal by performing an abnormality determination process on the first three-dimensional position information or the second three-dimensional position information; a determination step of determining a countermeasure action for the abnormality if the first three-dimensional position information or the second three-dimensional position information is determined to be abnormal; and an operation control step of performing control necessary for implementing the countermeasure action.

[0007] Patent Document 3 (WO 2019 / 111701) discloses a control device including a self-posture detection unit that detects its own posture based on sensor information, a posture change detection unit that detects unpredictable posture changes based on the detection result by the self-posture detection unit, and a self-position estimation unit that estimates its own position using a first estimation model based on the sensor information and, if the posture change detection unit detects the unpredictable posture change, estimates its own position using a second estimation model different from the first estimation model.

[0008] In SLAM position estimation, it is necessary to reduce the influence of disturbances that cause deterioration in the accuracy of feature point extraction and improve the accuracy of position estimation.

[0009] The present invention aims to provide a transport device equipped with a highly accurate position estimation system that reduces the influence of disturbances that cause deterioration in the accuracy of feature point extraction in a disturbance environment.

[0010] A representative example of the invention disclosed in the present application is as follows: That is, a transport device for transporting an object to be transported includes: a first sensor that acquires an image of the surroundings of the transport device; a control unit that estimates a position of the transport device based on the image acquired by the first sensor; and an analysis unit that analyzes disturbances that affect the estimation of the position of the transport device based on the image acquired by the first sensor, wherein the control unit has a position estimation unit that estimates the position of the transport device using one of a plurality of position estimation modes including a first mode that estimates the position of the transport device based on the image acquired by the first sensor; and a disturbance reduction unit that reduces the influence of disturbances that affect the position estimation of the transport device in the first mode.

[0011] According to one aspect of the present invention, it is possible to improve the accuracy of position estimation. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0012] FIG. 1 is a diagram showing an example of the layout of a logistics center of a conveying system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of a conveying system according to an embodiment of the present invention. FIG. 3 is a perspective view of a conveying device according to an embodiment of the present invention. FIG. 4 is a side view of a conveying device according to an embodiment of the present invention. FIG. 5 is a flowchart of processing for a frame image according to an embodiment of the present invention. FIG. 6 is a flowchart of disturbance determination (reduction) processing according to an embodiment of the present invention. FIG. 7 is a flowchart of disturbance determination (reduction) processing according to an embodiment of the present invention. FIG. 8 is a flowchart of divided area determination processing according to an embodiment of the present invention. FIG. 9 is a flowchart of disturbance reduction processing according to an embodiment of the present invention.

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments are illustrative for explaining the present invention, and appropriate omissions and simplifications have been made for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0014] Although various types of information may be described using expressions such as "table," "list," and "queue" as examples, the various types of information may also be expressed using data structures other than these. For example, various types of information such as "XX table," "XX list," and "XX queue" may also be expressed as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

[0015] When there are multiple components having the same or similar functions, they may be described using the same reference numeral with different subscripts. When there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0016] In the embodiments, processing performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU) and performs processing defined by the program using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity performing the processing by executing the program may be the processor. Similarly, the entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity performing the processing by executing the program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0017] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in the embodiments, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0018] FIG. 1 is a diagram showing an example of the layout of a logistics center of a transport system according to an embodiment of the present invention.

[0019] In the conveying system of this embodiment, the conveying device 2 conveys the shelf 51 to the picking station, and a worker or a picking robot takes out the corresponding product from the shelf 51 and stores the product on the shelf 51 .

[0020] The logistics center has a storage space 52. A plurality of shelves 51 are arranged in a grid pattern in the vertical and horizontal directions within the storage space 52. The shelves 51 form an "island" consisting of a predetermined number of shelves 51 (for example, 2 x 6 or 1 x 6).

[0021] A plurality of conveying devices 2 are arranged in the storage space 52. The conveying devices 2 move under the shelves 51 and lift up the shelves 51. A plurality of charging stations for charging the conveying devices 2 are arranged at predetermined locations around the storage space 52.

[0022] A plurality of work stations are arranged on the outer periphery of the storage space 52. At the work stations, workers carry out the work of storing and retrieving items, and at the robot station, robots carry out the work of storing and retrieving items.

[0023] A safety light curtain is installed at a work station adjacent to the storage space 52 to detect an intrusion of a worker into the storage space 52. The space between the safety light curtains forms an opening 80 where shelves 51 are placed and picking work is performed.

[0024] When the transport device 2 places the shelf 51 in the opening 80, the safety light curtain is turned off, allowing the worker to perform the picking work. On the other hand, when the picking work is completed and the transport device 2 moves the shelf 51 from the opening 80, the safety light curtain is turned on, and an alarm is output if a worker or the like enters through the opening 80.

[0025] At the work station where the worker works, a station terminal is placed near the frontage 80. In addition, a work space for sorting and packing is set up at a predetermined position on the periphery of the work station.

[0026] A plurality of cameras 3 are installed in the storage space 52. It is preferable to install a plurality of cameras 3 so as to avoid creating blind spots in the storage space 52 that cannot be photographed by any of the cameras 3. Images of the transport device 2 captured by the cameras 3 are used to estimate the position of the transport device 2 using motion capture technology. Note that estimating the position of the transport device 2 using motion capture is an auxiliary position estimation method, and cameras 3 do not necessarily have to be installed in the storage space 52.

[0027] FIG. 2 is a block diagram showing an example of the configuration of the transport system of this embodiment.

[0028] The conveyance system of this embodiment includes a warehouse control device 1 and multiple conveyance devices 2. The multiple conveyance devices 2 are connected to the warehouse control device 1 via a network 4. The warehouse control device 1 is also connected to the station terminals described above, but these are not shown in FIG.

[0029] The warehouse control device 1 is a computer having an arithmetic unit 11, a memory 12, an input device 13, an output device 14, a storage device 15, and a communication interface 17. The warehouse control device 1 is not limited to the configuration shown in FIG. 2. The warehouse control device 1 may be composed of a single computer or multiple computers. Furthermore, each device included in the warehouse control device 1 may be located on a single computer or may be distributed across multiple devices. The programs and information stored in the storage device 15 may be stored in a single storage device or may be distributed across multiple storage devices.

[0030] The storage device 15 has a non-volatile storage medium and stores programs executed by the calculation device 11 and data used by the programs. Examples of the programs stored in the storage device 15 are a route creation program 151, a transport device control program 156, a disturbance influence collection program 158, and a disturbance influence output program 159. The calculation device 11 loads the necessary programs into the memory 12 and executes them. Examples of the data stored in the storage device 15 are floor information 164, map information 165, device information 166, and route data 167.

[0031] The route creation program 151 refers to the map information 165 and calculates the movement route of the conveyance device 2 based on the position of the item (or product) to be picked and the position of the destination work station, etc. The route creation program 151 may refer to the disturbance information collected by the disturbance influence collection program 158 and create a travel route that avoids positions affected by the disturbance.

[0032] The transport device control program 156 refers to the route calculated by the route creation program 151, floor information 164, and device information 166 (e.g., the status of the transport device 2), etc., to determine the transport device 2 that will transport the shelf 51, and sends a transport instruction to the transport device 2 that includes information on the shelf 51 to be transported and the work station to which it is to be transported.

[0033] The disturbance effect collection program 158 collects information on disturbances that are affecting position estimation from the transport device 2, stores the information in the storage device 15 together with the position information where the disturbance occurred, and creates a disturbance effect potential map in which the disturbances that have occurred are mapped onto map information.

[0034] The disturbance effect output program 159 outputs the disturbance information collected by the disturbance effect collection program 158 .

[0035] The floor information 164 stores information indicating the floor condition for each floor area. The map information 165 stores map information within the warehouse. The device information 166 stores the position and operating status of the transport device 2. The device information 166 stores the identification information, position, operating status, etc. of the transport device 2. The route data 167 stores information on the route for each transport device 2 within the warehouse.

[0036] The input device 13 is composed of a keyboard, a mouse, a touch panel, etc. The output device 14 is composed of a display, etc. The communication interface 17 communicates with the transport device 2 and other computers via the network 4, for example, wirelessly.

[0037] The conveying device 2 is an autonomous mobile object that automatically conveys a shelf 51 loaded with items in response to a conveying instruction from the warehouse control device 1. The conveying device 2 is an autonomous conveying device that includes a side camera 211, a LiDAR 212, a marker sensor 213, a floor sensor 214, a memory unit 22, running wheels 23, a communication interface 24, a control unit 25, an analysis unit 26, lighting 27, and a table 28. The conveying device 2 also includes sensors (e.g., a vibration sensor and an acceleration sensor) not shown. The conveying device 2 can travel by switching between at least two movement modes: linear movement and rotational movement (also referred to as turning). Here, linear movement refers to traveling directional from one point to another. The trajectory of the linear movement may be a straight trajectory or a curved trajectory. Here, rotational movement refers to movement in which the conveying device 2 rotates (turns) to change its direction of travel on the spot.

[0038] The side camera 211 is provided on the side of the conveyance device 2 and captures images of the surroundings of the conveyance device 2. The LiDAR 212 is provided on the side of the conveyance device 2 and irradiates the surroundings of the conveyance device 2 with laser light to acquire three-dimensional point cloud data based on reflected light. The marker sensor 213 is a camera provided on the underside of the conveyance device 2 and captures images of markers provided on the floor of the storage space 52. The floor sensor 214 is a sensor provided on the underside of the conveyance device 2 and is used to detect guide lines (e.g., magnetic tape, light-reflecting tape, electromagnetic induction cable, etc.) provided on the floor of the storage space 52 and control the conveyance device 2 to travel along the detected guide lines. The floor sensor 214 can be a magnetic sensor that detects magnetic tape provided on the floor, a camera that detects light-reflecting tape, or a proximity sensor that detects electromagnetic induction cable. The observation results of the side camera 211, LiDAR 212, marker sensor 213, and floor sensor 214 are used by the conveyance device 2 to estimate its own position.

[0039] The control unit 25 includes a calculation device and a memory. The memory stores a travel control program, a position estimation program, an estimation mode switching program, and a disturbance reduction program. The programs stored in the memory are executed by the calculation device. The calculation device is composed of a microcomputer or a processor.

[0040] The travel control unit 251 controls the rotation of the running wheels 23 in accordance with the transport instruction sent from the warehouse control device 1 to cause the transport device 2 to travel.

[0041] The position estimation unit 252 estimates its own position using the observation results of various sensors (such as the side camera 211, the LiDAR 212, the marker sensor 213, and the floor sensor 214). For example, the position estimation unit 252 estimates its own position in a first mode using a position estimation model that analyzes an image of the surroundings of the transport device 2 captured by the side camera 211 and derives the position by comparing it with an image previously captured at the same position and stored together with position information. Note that the position may also be derived by comparing the feature amounts of the image captured by the side camera 211 with the feature amounts of an image previously captured. Each feature point having three-dimensional information is mapped to the estimated own position, thereby constructing an environmental map. Furthermore, the position estimation unit 252 estimates its own position in a LiDAR mode using a position estimation model that analyzes three-dimensional point cloud data of the surroundings of the transport device 2 acquired by the LiDAR 212 and compares it with previously captured three-dimensional point cloud data. Similarly, in the LiDAR mode, the 3D point cloud data acquired at each time point is mapped to the estimated self-position to construct an environmental map based on the 3D point cloud. The position estimation unit 252 estimates the position in a marker mode, in which the marker on the floor imaged by the marker sensor 213 is read and the position of the storage space 52 where the marker is installed is estimated as the self-position. The position estimation unit 252 also estimates the position in a guide line mode, in which the self-position is estimated based on the detection result of the guide line by the floor sensor 214. The position estimation unit 252 also acquires images of the transport device 2 captured by the camera 3 from the warehouse control device 1 and estimates the position of the transport device 2 using motion capture technology. For example, the motion capture technology involves installing several infrared cameras and estimating the position and orientation of an AGV equipped with reflective markers from images captured by the infrared cameras. Reflective markers can be installed, for example, at the four corners of the AGV, and the position and orientation of the installation surface can be determined in real time. Therefore, it is advisable to control the reflective markers to minimize the difference from the target route based on this position and orientation information. Environmental maps typically contain absolute errors, so it is necessary to calibrate the positional relationship with the motion capture device during the environmental map creation stage. For example, if it is determined in advance that a certain point is off by 2 cm in the north direction, it is necessary to add a north offset.In this way, it is advisable to calibrate in advance the positional relationship between the environmental map, the real environment, and the measurement results obtained by motion capture.

[0042] The estimation mode switching unit 253 switches the method of estimating the self-position used by the position estimation unit 252 .

[0043] When an image captured by the side camera 211 is difficult to analyze due to disturbances, the disturbance reduction unit 254 removes the influence of the disturbances from the image and generates an image that can be analyzed.

[0044] The storage unit 22 is configured as a non-volatile storage medium and stores programs and data used by the programs. Examples of the data include map information 221 and travel distance information 222. The map information 221 is map information received from the warehouse control device 1. The travel distance information 222 is data calculated using odometry to determine the distance traveled by the transport device 2 from the reference point of the grid to which it belongs.

[0045] As shown in Figure 4, the running wheels 23 are composed of two drive wheels and four auxiliary wheels (casters) and are attached to the underside of the transport device 2. The two drive wheels are arranged parallel to the direction of straight travel. Each drive wheel is equipped with a separate motor and encoder, which serve as a drive device, and the motor is supplied with power from a battery. If the two drive wheels rotate forward at the same rotation speed, the transport device 2 moves straight, and if the two drive wheels rotate in opposite directions, the transport device 2 turns.

[0046] The analysis unit 26 includes a calculation device and a memory. The memory stores a feature point extraction program, a disturbance area estimation program, a disturbance type recognition program, and a disturbance influence calculation program. The programs stored in the memory are executed by the calculation device. The calculation device is composed of a microcomputer or a processor. The feature point extraction unit 261 extracts feature points to be used for position estimation from the observation results of various sensors. The disturbance area estimation unit 262 estimates areas in the image that are affected by disturbances. The disturbance type recognition unit 263 recognizes the type of disturbance that is affecting the image. The disturbance influence calculation unit 264 recognizes the type of disturbance that is affecting the image.

[0047] The arithmetic unit and memory of the analysis unit 26 may be the same hardware as the arithmetic unit and memory of the control unit 25, or may be separate hardware.

[0048] FIG. 3 is a perspective view of the conveying device 2 of this embodiment, and FIG. 4 is a side view of the conveying device 2. As shown in FIG.

[0049] The transport device 2 has a table 28 on its upper surface on which a shelf 51 can be mounted. The table 28 can be rotated horizontally by a drive device, and the shelf 51 rotates when the table 28 rotates.

[0050] The transport device 2 has lights 27 at the corners of its sides. The lights 27 brightly illuminate the area around the transport device 2, reducing disturbances caused by brightness and making it easier to check the surrounding conditions.

[0051] FIG. 5A is a flowchart showing the overall processing for a frame image in this embodiment.

[0052] The transport device 2 refers to the created environmental map and generates a movement route for the transport device 2 based on the positional relationship between the destination and the current location.

[0053] First, the transport device 2 acquires an image captured by a sensor at a certain point (S121), and executes a disturbance determination (reduction) process (S122). Details of the disturbance determination (reduction) process will be described with reference to FIGS. 5B and 5C.

[0054] Thereafter, the transport device 2 executes a self-position estimation process in a predetermined self-position estimation mode (S123), and determines whether the self-position estimation is successful (S124). If the self-position estimation is successful, the transport device 2 calculates the deviation between the position on the target route and the estimated self-position (S125), and performs movement control while changing the current movement speed and turning speed so as to reduce the calculated deviation (S126).

[0055] On the other hand, if the self-position estimation fails and the current self-position is unknown, the robot requests the warehouse control device 1 to re-estimate its self-position. In this case, it is recommended that the robot move at a low speed and / or turn at a low turning speed so as to reduce the area determined to be a disturbance area.

[0056] When controlling operation in map creation mode or self-location estimation mode using a created environmental map, a disturbance influence potential map with continuous values ​​of influence is created at each location in the warehouse by reflecting the influence level of each disturbance (obstruction factor) according to the location on the environmental map in the environmental map. This disturbance influence potential map allows for the creation of a travel route that avoids areas with high disturbance influence when generating a route connecting the current location and the destination. Since stopping the movement of the transport device 2 due to an inability to estimate its own location reduces throughput, using such a disturbance influence potential map allows the manager to visualize a route that will arrive quickly while taking into account the disturbance influence level, and then actually controls movement using that route. A method for avoiding locations with high disturbance influence levels involves installing virtual obstacles in areas with disturbance influence levels higher than a predetermined threshold, determining a predetermined safety distance from the installed virtual obstacle, and generating a route that avoids areas that are the obstacle area plus the safety distance.

[0057] 5B and 5C are flowcharts of the disturbance determination (reduction) process of this embodiment.

[0058] The disturbance determination (reduction) process is executed by the analysis unit 26 to determine a position estimation mode when the position estimation unit 252 estimates the position of the transport device 2. Then, the position estimation unit 252 estimates the position of the transport device 2 using the position estimation mode determined by the analysis unit 26.

[0059] In the disturbance determination (reduction) process, first, the feature point extraction unit 261 extracts feature points from the image to be used for position estimation (S101). The feature points extracted may be corners of figures in the image or points with large brightness gradients.

[0060] Next, the disturbance region estimation unit 262 divides the image into predetermined regions (S102). For example, the image is divided into a predetermined number (or a predetermined size) of grids, such as 4 to 10 grids in both the vertical and horizontal directions.

[0061] Next, the disturbance type recognition unit 263 executes a divided region determination process for determining a disturbance in each divided region (S103). Details of the divided region determination process will be described with reference to FIG.

[0062] Next, the disturbance reduction unit 254 executes a disturbance reduction process to reduce the influence of disturbances in the image (S104). Details of the disturbance reduction process will be described with reference to FIG.

[0063] Thereafter, the disturbance type recognition unit 263 executes a divided region determination process to determine whether the influence of the disturbance has been reduced to the extent that the image can be used by the process of step S104 (S105).

[0064] Thereafter, the disturbance reduction unit 254 determines whether all the divided regions have been processed (S106). If some divided regions have not been processed, the process returns to step S103 and repeats the process for the unprocessed divided regions. On the other hand, if all the divided regions have been processed, the process proceeds to step S107.

[0065] Next, the disturbance influence calculation unit 264 calculates the percentage of disturbance determination areas by dividing the number of disturbance determination areas determined to have a disturbance by the total number of divided areas, and determines whether the percentage of disturbance determination areas is greater than a predetermined threshold value Th1 (S107).

[0066] If the proportion of areas determined to have disturbances is greater than a predetermined threshold Th1, many areas in the image cannot be used, so the disturbance reduction unit 254 determines that all areas of the frame image are not to be used (S108) and terminates the disturbance determination (reduction) process.

[0067] On the other hand, if the proportion of areas determined to have disturbances is equal to or less than a predetermined threshold Th1, there is usable area in the image, so the disturbance reduction unit 254 rejects the divided areas determined to have disturbances in step S105 and adopts the divided areas determined to have no disturbances (S109).

[0068] Next, the disturbance reduction unit 254 divides the number of blur determination areas that have been determined to have a disturbance due to shaking and are labeled "blur" by the total number of areas to calculate the percentage of areas that have been determined to have a disturbance due to shaking, and determines whether this percentage is greater than a predetermined threshold value Th2 (S110).

[0069] If the proportion of the blur determination area is greater than a predetermined threshold Th2, the disturbance reduction unit 254 changes the vehicle control to a rotation speed reduction mode (S111) and turns on the lights 27 (S112). Furthermore, the disturbance reduction unit 254 may switch the position estimation mode to a motion capture mode that estimates the position using motion capture technology provided by the warehouse control device 1, a marker mode that estimates the position using floor markers, or a guide line mode that travels along a guide line (S113). The motion capture mode, marker mode, and guide line mode should be switched in order of decreasing movement constraints. Generally, the priority of the motion capture mode, marker mode, and guide line mode should be selected. For example, the motion capture mode should be used in an environment within the motion capture detection area and where there is no particular occlusion. When using floor markers, it is recommended to install markers other than the floor markers in the surrounding environment and switch in an environment where marker-based detailed position control can be performed using a side camera. In areas where occlusion occurs or when guidance lines are drawn on the floor, it is advisable to switch to the guidance line mode as the final candidate position estimation method. The position estimation mode may be switched by estimating the position in all modes, or may be switched sequentially depending on the availability of each mode.

[0070] On the other hand, if the proportion of areas including feature points determined to have disturbance due to shaking is equal to or less than a predetermined threshold Th2, the disturbance influence calculation unit 264 determines that there is disturbance due to lighting, calculates the proportion of lighting determination areas by dividing the number of lighting determination areas labeled with "lighting" by the total number of divided areas, and determines whether the proportion of lighting determination areas is greater than a predetermined threshold Th3 (S114).

[0071] If the proportion of the illumination determination area is greater than a predetermined threshold Th3, the disturbance reduction unit 254 determines that the influence of disturbances caused by lighting is large, and switches to one of the following modes (S117): LiDAR mode, which estimates the position using three-dimensional point cloud data observed by the LiDAR 212; marker mode, which estimates the position using floor markers; and guidance line mode, which travels along a guidance line. The switching of each mode may be selected in order of priority from the viewpoint of movement constraints: LiDAR mode, marker mode, and guidance line mode. The switching of the position estimation mode may be performed by estimating the position in all modes, or may be performed sequentially depending on the availability of each mode.

[0072] On the other hand, if the proportion of the illumination judgment area is equal to or less than the predetermined threshold value Th3, the disturbance influence calculation unit 264 determines that there is disturbance due to a dark place, calculates the proportion of the dark place judgment area by dividing the number of dark place judgment areas labeled "dark place" by the total number of divided areas, and determines whether the proportion of the dark place judgment area is greater than the predetermined threshold value Th4 (S115).

[0073] If the proportion of the dark place determination area is greater than a predetermined threshold Th4, the disturbance reduction unit 254 switches to one of the following modes: a LiDAR mode that estimates the position using the observation results of the LiDAR 212, a marker mode that estimates the position using floor markers, or a guidance line mode that travels along a guidance line (S117). The position estimation mode may be switched by estimating the position in all modes, or may be switched sequentially depending on the availability of each mode.

[0074] On the other hand, if the proportion of dark judgment areas is less than or equal to a predetermined threshold Th4, the disturbance influence calculation unit 264 determines that there is little change in the image of the area and that there is a disturbance that interferes with recognition, and calculates the proportion of textureless judgment areas by dividing the number of textureless judgment areas labeled "textureless" by the total number of divided areas, and determines whether the proportion of textureless judgment areas is greater than a predetermined threshold Th5 (S116).

[0075] If the proportion of the textureless determination region is greater than a predetermined threshold Th5, the disturbance reduction unit 254 switches to one of the following modes: a LiDAR mode that estimates the position using the observation results of the LiDAR 212, a marker mode that estimates the position using floor markers, or a guide line mode that travels along a guide line (S117). The position estimation mode may be switched by estimating the position in all modes, or may be switched sequentially depending on the availability of each mode.

[0076] On the other hand, if the proportion of textureless determination areas is equal to or less than the predetermined threshold Th5, the disturbance reduction unit 254 ends the disturbance determination (reduction) process without switching the position estimation mode.

[0077] As described above, the conveying device 2 of this embodiment identifies the type of disturbance and the degree of influence of the disturbance that affects the self-location estimation of the conveying device 2, and identifies and executes a method of self-location estimation based on the information on the type of disturbance and the degree of influence of the disturbance. Identifying the method of self-location estimation based on the information on the type of disturbance and the degree of influence of the disturbance includes, for example, a disturbance reduction process according to the type of disturbance, and a process of determining whether or not to switch the self-location estimation mode based on information on the degree of influence of a specific type of disturbance. This makes it possible to adjust the method of self-location estimation according to the type and degree of influence of the disturbance, and effectively reduce deterioration in the accuracy of self-location estimation.

[0078] An example of a conveying device 2 described in this embodiment is a conveying device 2 that conveys items in a traveling area including multiple facilities, and is equipped with a control unit 25 that estimates the position of the conveying device 2 based on image information acquired by a sensor unit provided in the conveying device 2, and an analysis unit 26 that identifies the type of disturbance that affects the estimation of the position of the conveying device 2 and the degree of influence of the disturbance from the image information, wherein the control unit 25 is capable of estimating the position of the conveying device 2 by switchably executing multiple different position estimation methods, including a first estimation method that estimates the position of the conveying device 2 based on the image information, and when estimating the position of the conveying device 2 using the first estimation method, it is capable of identifying a second estimation method from the multiple different position estimation methods to reduce the influence of the disturbance based on information on the type of disturbance and the degree of influence of the disturbance, and switching from the first estimation method to the second estimation method to estimate the position of the conveying device 2.

[0079] FIG. 6 is a flowchart of the divided region determination process (S103) of this embodiment.

[0080] In the divided region determination process, the disturbance type recognition unit 263 first determines the number of edges extracted from the image (S1030). The edge extraction method may use, for example, a Sobel filter or a Laplacian filter. By applying a Laplacian filter, a gradient magnitude value is calculated for each pixel. If the number of pixels determined to be edges based on a predetermined gradient magnitude threshold is less than a predetermined threshold Th6, the image is determined to be blurred due to camera shake, and the region is determined to contain a disturbance due to shake, and the region is labeled as "blur" (S1031).

[0081] On the other hand, if the number of pixels determined to be edges is equal to or greater than a predetermined threshold Th6, the disturbance type recognition unit 263 determines for each pixel in the region whether the luminance difference within a predetermined range is smaller than a predetermined threshold Th7 (S1032). If the luminance difference within a predetermined range (e.g., several pixels) is smaller than the predetermined threshold Th7, the disturbance type recognition unit 263 determines whether the number of pixels in the region whose luminance values ​​are equal to or greater than a predetermined value is greater than a predetermined threshold Th8 (S1033). If the number of pixels whose luminance values ​​are equal to or greater than a predetermined value is greater than the predetermined threshold Th8, the disturbance type recognition unit 263 determines that lighting is reflected in the region and that the disturbance with a large luminance difference in the region is caused by lighting, and labels the region as "lighting" (S1034).

[0082] On the other hand, if the brightness difference within a predetermined range (e.g., several pixels) is equal to or greater than a predetermined threshold Th7, or the number of pixels whose brightness values ​​are equal to or greater than a predetermined threshold Th8 is equal to or less than a predetermined threshold Th8, the disturbance type recognition unit 263 determines whether the number of pixels whose brightness values ​​are equal to or less than a predetermined threshold Th9 in the region is greater than a predetermined threshold Th9 (S1035). If the number of pixels whose brightness values ​​are equal to or less than a predetermined threshold Th9 is greater than the predetermined threshold Th9, the disturbance type recognition unit 263 determines that the region is disturbed by a dark place where there are many low-brightness pixels, and labels the region as a "dark place" (S1036).

[0083] The disturbance type recognition unit 263 then determines the amount of edges extracted from the image (S1037) and determines the number of feature points extracted from the image (S1038). If the number of pixels determined to be edges in the region is less than a predetermined threshold Th10, or if the number of feature points extracted from the image in the region is less than a predetermined threshold Th11, the disturbance type recognition unit 263 determines that there is little change in the image in the region and that there is a disturbance that interferes with recognition, and labels the region as "textureless" (S1039).

[0084] On the other hand, if the number of pixels determined to be edges in the region is equal to or greater than a predetermined threshold Th10 and the number of feature points extracted from the image in the region is equal to or greater than a predetermined threshold Th11, there is a lot of change in the image in the region, so the disturbance type recognition unit 263 terminates the divided region determination process without assigning any label to the region.

[0085] As described above, in the divided area determination process of this embodiment, the area where the disturbance occurs and the type of disturbance are identified, which makes it possible to evaluate the influence of the disturbance and perform disturbance reduction processing according to the type of disturbance.

[0086] FIG. 7 is a flowchart of the disturbance reduction process of this embodiment.

[0087] In the disturbance reduction process, the disturbance reduction unit 254 first determines whether the region is labeled as "blurred" (S1040). If the region is labeled as "blurred," the disturbance reduction unit 254 executes edge clarification processing (S1041). For example, a blur removal filter may be applied as the edge clarification processing method. The blur removal filter may be applied as an anisotropic Gaussian filter with blur intensity, direction, and aspect ratio as parameters. However, the method is not limited to such a rule-based method; machine learning may also be used to remove blur, and various other methods may be applied.

[0088] On the other hand, if the region is not labeled as "blur," it is determined whether the region is labeled as "lighting" (S1042). If the region is labeled as "lighting," contrast smoothing processing is performed (S1043). As a method for contrast smoothing processing, for example, gamma correction, contrast stretching, or histogram equalization processing may be applied. However, the method is not limited to such a rule-based method, and contrast smoothing may be performed using machine learning, and various other methods can be applied.

[0089] On the other hand, if the region is not labeled as "illumination," it is determined whether the region is labeled as "dark" (S1044). If the region is labeled as "dark," the brightness of the image is adjusted by applying gamma correction processing, histogram smoothing processing, or the like (S1045). However, the method is not limited to such a rule-based method; machine learning may also be used to adjust the brightness, and various other methods can be applied.

[0090] As described above, the control unit 25 of this embodiment has a position estimation unit 252 that estimates the position of the conveying device 2 using one of a plurality of position estimation modes including a first mode that estimates the position of the conveying device 2 based on an image acquired by the first sensor (side camera 211), and a disturbance reduction unit 254 that reduces the influence of disturbances that affect the position estimation of the conveying device 2 in the first mode.Therefore, by performing disturbance reduction processing according to the type of disturbance, it is possible to effectively reduce deterioration in the accuracy of self-position estimation.

[0091] In addition, the conveying device 2 is equipped with a second sensor (LiDAR 212) that acquires three-dimensional information about the surroundings thereof, and when the analysis unit 26 analyzes that the influence of disturbances is large, the position estimation unit 252 estimates the position of the conveying device 2 using a second LiDAR mode that estimates the position of the conveying device 2 based on the three-dimensional information. Therefore, when the surrounding environment has three-dimensional structural characteristics, the influence of disturbances is large in image-based position estimation, making it impossible to perform self-position estimation. In such a situation, the sensor can be replaced with one that acquires three-dimensional information that is less influenced by disturbances.

[0092] In addition, the device is equipped with a third sensor (marker sensor 213) that acquires information about a position marker that indicates the position within the traveling space, and when the analysis unit 26 analyzes that the influence of external disturbances is large, the position estimation unit 252 estimates the position of the conveying device 2 using a third marker mode that estimates the position of the conveying device 2 based on the information about the position marker.Therefore, when markers are affixed to the floor or the environment, the influence of external disturbances on image-based position estimation is large and self-position estimation is not possible, the position can be estimated with high accuracy.

[0093] In addition, the device is equipped with a fourth sensor (floor sensor 214) that acquires information about a guide line that guides the conveying device 2 to the running path, and when the analysis unit 26 analyzes that the influence of external disturbances is large, the position estimation unit 252 controls the position of the conveying device 2 using a fourth guide line mode that controls the running of the conveying device 2 based on information about the guide line.Therefore, when a guide line is installed, the influence of external disturbances on image-based position estimation is large and self-position estimation is not possible, so that the position can be estimated using the guide line.

[0094] The disturbance reduction unit 254 also performs image processing on the image acquired by the first sensor to reduce the effects of disturbances contained in the image by edge definition, contrast smoothing, or gamma correction, thereby enabling image-based position estimation.

[0095] Furthermore, if the disturbance reduction unit 254 determines that the influence of disturbances is large, it controls the first sensor to acquire an image of the surroundings of the conveying device while the light 27 is on, thereby reducing blurring in the image.

[0096] Furthermore, if the disturbance reduction unit 254 determines that the influence of disturbances is large, it controls the first sensor to acquire an image of the surroundings of the conveying device while reducing the operating speed of the drive device (running wheel 23), thereby reducing blurring in the image.

[0097] The analysis unit 26 also has a disturbance influence calculation unit 264 that determines that the influence of the disturbance is large when the proportion of the area affected by the disturbance in the image acquired by the first sensor is larger than a predetermined threshold. By calculating the magnitude of the influence of the disturbance in the entire image, it is possible to determine whether the self-position can be estimated using an image including the disturbance.

[0098] The analysis unit 26 also has a feature point extraction unit 261 that extracts feature points having geometric characteristics from the image acquired by the first sensor, and a disturbance influence calculation unit 264 that determines that the influence of the disturbance is large when the ratio of the number of feature points affected by the disturbance to the total number of feature points in the image acquired by the first sensor is greater than a predetermined threshold. By calculating the magnitude of the influence of the disturbance in the entire image, it is possible to determine whether the self-position can be estimated using an image including the disturbance.

[0099] Furthermore, in the fifth motion capture mode, if the position estimation unit 252 analyzes that the influence of disturbances due to image blurring is large, it estimates the position of the conveying device 2 using motion capture technology using images acquired by the fifth sensor (camera 3).Therefore, in areas with few features over a wide area, image mode and LiDAR mode cannot detect the pattern and structural features in the environment, and self-position estimation is likely to fail.However, by using a motion capture system that can measure a wide area, it is possible to estimate the self-position with high accuracy, without occlusion of markers.

[0100] In addition, the warehouse control device 1 has a route creation unit 151 that references the disturbance information collected by the disturbance influence collection unit 158 ​​and creates a travel route for the conveying device 2 that avoids locations affected by the disturbance.This allows the conveying device 2 to travel on a route that passes through an area with little disturbance influence, without stopping due to the influence of the disturbance, thereby improving the throughput of the entire conveying system.

[0101] In the above embodiment, the conveying device 2 performs the self-position estimation process and the disturbance determination (reduction) process, but the warehouse control device 1 may also perform the position estimation process and the disturbance determination (reduction) process of the conveying device 2.

[0102] As described above, according to the embodiment of the present invention, in a disturbed environment, the influence of disturbances can be reduced and the accuracy of position estimation can be improved by disregarding disturbance areas, applying disturbance reduction processing, or switching position estimation modes depending on the type of disturbance (blur, lighting, dark areas, textureless) that affects image feature extraction and its degree of influence.

[0103] In the above-described embodiment, an example has been described in which the present invention is applied to a conveying device that travels in a warehouse, but the present invention can also be applied to other moving devices such as automobiles, robots, and mobility devices.

[0104] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0105] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0106] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, hard disk, or SSD (Solid State Drive), or in a recording medium such as an IC card, SD card, or DVD.

[0107] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A conveying device that conveys an object to be conveyed within a traveling space, comprising: a first sensor that acquires an image of the surroundings of the conveying device; a control unit that estimates a position of the conveying device based on the image acquired by the first sensor; and an analysis unit that analyzes disturbances that affect the estimation of the position of the conveying device based on the image acquired by the first sensor, wherein the control unit has: a position estimation unit that estimates the position of the conveying device using one of a plurality of position estimation modes including a first mode that estimates the position of the conveying device based on the image acquired by the first sensor; and a disturbance reduction unit that reduces the influence of disturbances that affect the estimation of the position of the conveying device using the first mode.

2. A conveying device as described in claim 1, wherein the analysis unit identifies information on the type and area of ​​disturbance that affects the estimation of the position of the conveying device based on the image acquired by the first sensor, and the disturbance reduction unit reduces the influence of disturbance that affects the estimation of the position of the conveying device in the first mode based on the information identified by the analysis unit.

3. A conveying device as described in claim 1, further comprising a second sensor for acquiring three-dimensional information about the surroundings of the conveying device, and wherein the position estimation unit, when the analysis unit analyzes that the influence of the disturbance is large, estimates the position of the conveying device using a second mode for estimating the position of the conveying device based on the three-dimensional information.

4. A conveying device as described in claim 1, further comprising a third sensor that acquires information of a position marker indicating a position within the traveling space, and wherein the position estimation unit estimates the position of the conveying device using a third mode that estimates the position of the conveying device based on the information of the position marker when the analysis unit analyzes that the influence of the disturbance is large.

5. A conveying device as described in claim 1, further comprising a fourth sensor for acquiring information on a guide line that guides the conveying device along a travel path, and wherein the position estimation unit controls the position of the conveying device using a fourth mode that controls the travel of the conveying device based on the information on the guide line when the analysis unit analyzes that the influence of the disturbance is large.

6. A conveying device as described in claim 1, characterized in that the disturbance reduction unit performs image processing on the image acquired by the first sensor to reduce the effects of disturbances contained in the image by edge clarification, contrast smoothing, or gamma correction.

7. A conveying device as described in claim 1, further comprising a lighting device for illuminating the surroundings of the conveying device, and wherein the disturbance reduction unit, when it is analyzed that the influence of the disturbance is large, controls the first sensor to acquire an image of the surroundings of the conveying device with the lighting device turned on.

8. A conveying device as described in claim 1, further comprising a drive device for driving a table on which the object to be conveyed is loaded, and wherein the disturbance reduction unit, when it is analyzed that the influence of the disturbance is large, controls the first sensor to acquire an image of the surroundings of the conveying device while reducing the operating speed of the drive device.

9. A conveying device as described in claim 1, characterized in that the analysis unit has a disturbance effect calculation unit that determines that the effect of the disturbance is large when the proportion of the area of ​​the image acquired by the first sensor that is affected by the disturbance is greater than a predetermined threshold value.

10. A conveying device as described in claim 1, wherein the analysis unit has: a feature point extraction unit that extracts feature points having geometric characteristics from the image acquired by the first sensor; and a disturbance influence calculation unit that determines that the influence of the disturbance is large when the ratio of the number of feature points affected by the disturbance to the total number of feature points in the image acquired by the first sensor is greater than a predetermined threshold value.

11. A conveying system comprising: a conveying device as described in any one of claims 1 to 10 for conveying an object to be conveyed; a management device for controlling the movement of the conveying device; and a fifth sensor for acquiring images of the conveying device, wherein the position estimation unit estimates the position of the conveying device using a fifth mode in which the position of the conveying device is estimated using the image acquired by the fifth sensor when the analysis unit analyzes that the influence of the disturbance is large.

12. A conveying system as described in claim 11, characterized in that in the fifth mode, when it is analyzed that the influence of disturbances due to image blurring is large, the position estimation unit estimates the position of the conveying device by motion capture technology using images acquired by the fifth sensor.

13. A conveying system as described in claim 11, characterized in that the management device has a disturbance effect collection unit that collects information on disturbances that are affecting the position estimation from the conveying device and stores the information together with the position information where the disturbance occurred.

14. A transport system according to claim 13, wherein said management device has a disturbance effect output section which outputs information on the disturbances collected by said disturbance effect collection section.

15. A conveying system as described in claim 13, characterized in that the management device has a route creation unit that references the information on disturbances collected by the disturbance influence collection unit and creates a travel route for the conveying device while avoiding locations affected by the disturbances.

16. A control method executed by a conveying device, the conveying device having a calculation device that executes a program, a storage device that stores the program, and a first sensor that acquires an image of the surroundings of the conveying device, the control method comprising: a position estimation procedure that estimates a position of the conveying device using one of a plurality of position estimation modes including a first mode that estimates the position of the conveying device based on the image acquired by the first sensor; an analysis procedure that analyzes disturbances that affect the estimation of the position of the conveying device based on the image acquired by the first sensor; and a disturbance reduction procedure that reduces the influence of disturbances that affect the estimation of the position of the conveying device by the first mode.

Images