Transport device, transport system, and method for controlling transport device

Abstract

To provide a transport device mounted with a highly accurate position estimation system.SOLUTION: A transport device for transporting objects to be transported within travel space includes a first sensor for acquiring an image around the transport device, a control part for estimating the position of the transport device on the basis of the image acquired by the first sensor, and an analysis part for analyzing disturbance that exerts influence on the estimation of position of the transport device based on the image acquired by the first sensor, and the control part includes a position estimation part for estimating the position of the transport device by using any of a plurality of position estimation modes including a first mode for estimating the position of the transport device on the basis of the image acquired by the first sensor, and a disturbance reduction part for reducing the influence of the disturbance that exerts influence on the estimation of position of the transport device by the first mode.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a transport device, and more particularly to a control technique for removing factors that impede self-position estimation. [Background technology]

[0002] As a method for estimating the self-location of an automatic guided vehicle (AGV), a method is known that combines multiple self-location estimation methods to improve accuracy and robustness. For example, a technology is known that combines the Visual SLAM (Simultaneous Localization and Mapping) method, which uses image feature points, with the Wheel Odometry (hereinafter, odometry) method, which uses wheel rotation information (encoder information).

[0003] In SLAM position estimation, when images taken in a disturbed environment are used, the accuracy of feature point extraction decreases, and the accuracy of position estimation also decreases.

[0004] The following prior art is included as background art in this technical field. Patent Document 1 (JP 2021-96731 A) discloses a control device for an autonomously traveling automated guided vehicle that performs autonomous driving control based on a current self-position calculated by a self-position estimation process, in which the control device performs an image processing self-position estimation process that calculates a first self-position based on a landmark photographed by a photographing device mounted on the autonomously traveling automated guided vehicle, and an odometry self-position estimation process that calculates a second self-position based on driving information of the autonomously traveling automated guided vehicle, and estimates the current self-position using the first self-position and the second self-position. The control device has a self-position estimation processing unit that estimates the current self-position using the first self-position and the second self-position, and a storage unit that stores a calculation accuracy map of the image processing self-position estimation process within a predetermined area including a driving route, and the self-position estimation processing unit changes the contribution degree of the first self-position and the second self-position in estimating the current self-position based on the calculation accuracy of the calculation accuracy map.

[0005] Patent Document 2 (International Publication No. 2018 / 38131) discloses a three-dimensional information processing method including an acquisition step of acquiring map data including first three-dimensional position information via a communication path, a generation step of generating second three-dimensional position information from information detected by a sensor, a determination step of determining whether the first three-dimensional position information or the second three-dimensional position information is abnormal by performing an abnormality determination process on the first three-dimensional position information or the second three-dimensional position information, a determination step of determining a countermeasure action for the abnormality when the first three-dimensional position information or the second three-dimensional position information is determined to be abnormal, and an operation control step of performing control necessary to implement the countermeasure action.

[0006] Patent Document 3 (WO 2019 / 111701) discloses a control device including a self-posture detection unit that detects its own posture based on sensor information, a posture change detection unit that detects an unpredictable change in posture based on a detection result by the self-posture detection unit, and a self-position estimation unit that estimates a self-position using a first estimation model based on the sensor information and, when the posture change detection unit detects an unpredictable change in posture, estimates a self-position using a second estimation model different from the first estimation model. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2021-96731 [Patent Document 2] International Publication No. 2018 / 38131 [Patent Document 3] International Publication No. 2019 / 111701 Summary of the Invention [Problem to be solved by the invention]

[0008] In SLAM-based position estimation, it is necessary to reduce the influence of disturbances that cause deterioration in the accuracy of feature point extraction and improve position estimation accuracy.

[0009] The present invention aims to provide a transport device equipped with a highly accurate position estimation system that reduces the influence of disturbances that cause deterioration in the accuracy of extracting feature points in a disturbed environment. [Means for solving the problem]

[0010] A representative example of the invention disclosed in the present application is as follows: That is, a transport device for transporting an object to be transported includes a first sensor for acquiring an image of the surroundings of the transport device, a control unit for estimating a position of the transport device based on the image acquired by the first sensor, and an analysis unit for analyzing disturbances that affect the estimation of the position of the transport device based on the image acquired by the first sensor, the control unit having a position estimation unit for estimating the position of the transport device using one of a plurality of position estimation modes including a first mode for estimating the position of the transport device based on the image acquired by the first sensor, and a disturbance reduction unit for reducing the effect of disturbances that affect the estimation of the position of the transport device in the first mode. Effect of the Invention

[0011] According to one aspect of the present invention, it is possible to improve the accuracy of position estimation. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an example of a layout of a logistics center of a transport system according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an example of a configuration of a transport system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a perspective view of a conveying device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a side view of the transport device according to the embodiment of the present invention. [Figure 5A] 11 is a flowchart of processing for a frame image according to an embodiment of the present invention. [Figure 5B]4 is a flowchart of a disturbance determination (reduction) process according to an embodiment of the present invention. [Figure 5C] 4 is a flowchart of a disturbance determination (reduction) process according to an embodiment of the present invention. [Figure 6] 11 is a flowchart of a divided region determination process according to an embodiment of the present invention. [Figure 7] 4 is a flowchart of a disturbance reduction process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment is an example for explaining the present invention, and appropriate omissions and simplifications are made to clarify the explanation. The present invention can be implemented in various other forms. Unless otherwise limited, each component may be singular or plural. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0014] As examples of various information, the various information may be described using expressions such as "table," "list," and "queue," but the various information may be expressed using data structures other than these. For example, various information such as "XX table," "XX list," and "XX queue" may be expressed as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are mutually interchangeable.

[0015] When there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts. When there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0016] In the embodiments, a process performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., CPU, GPU), and performs the process defined by the program while using a storage resource (e.g., memory) and an interface device (e.g., communication port), etc. Therefore, the subject of the process performed by executing the program may be the processor. Similarly, the subject of the process performed by executing the program may be a controller, device, system, computer, or node having a processor. The subject of the process performed by executing the program may be a calculation unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device), etc.

[0017] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource for storing a program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in the embodiments, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0018] FIG. 1 is a diagram showing an example of a layout of a logistics center of a transportation system according to an embodiment of the present invention.

[0019] In the conveyance system of this embodiment, the conveyance device 2 conveys the shelf 51 to a picking station, and a worker or a picking robot takes out the corresponding product from the shelf 51 and stores the product on the shelf 51.

[0020] The logistics center has a storage space 52. Within the storage space 52, a plurality of shelves 51 are arranged in a grid pattern in the vertical and horizontal directions. The shelves 51 form an "island" consisting of a predetermined number of shelves 51 (for example, 2 x 6 or 1 x 6).

[0021] A plurality of conveying devices 2 are arranged in the storage space 52. The conveying devices 2 enter beneath the shelves 51 and lift up the shelves 51 to move. A plurality of charging stations for charging the conveying devices 2 are arranged at predetermined locations around the storage space 52.

[0022] A plurality of work stations are arranged on the outer periphery of the storage space 52. At the work stations, workers carry out storage and retrieval work of articles, and at the robot station, a robot carries out storage and retrieval work of articles.

[0023] A safety light curtain that detects an intrusion of a worker into the storage space 52 is installed at a work station adjacent to the storage space 52. The space between the safety light curtains forms an opening 80 where the shelves 51 are placed and where picking work is performed.

[0024] When the transport device 2 places the shelf 51 in the opening 80, the safety light curtain is turned off, allowing the worker to carry out the picking work. On the other hand, when the picking work is completed and the transport device 2 moves the shelf 51 away from the opening 80, the safety light curtain is turned on, and an alarm is output if a worker or the like enters through the opening 80.

[0025] In the work station where the worker works, a station terminal is disposed near the frontage 80. In addition, a work space for sorting and packing is provided at a predetermined position on the periphery of the work station.

[0026] A plurality of cameras 3 are installed in the storage space 52. It is preferable to install a plurality of cameras 3 so as to avoid creating blind spots in the storage space 52 that cannot be photographed by any of the cameras 3. Images of the transportation device 2 captured by the cameras 3 are used to estimate the position of the transportation device 2 using motion capture technology. Note that the estimation of the position of the transportation device 2 by motion capture is an auxiliary position estimation means, and the cameras 3 do not necessarily have to be arranged in the storage space 52.

[0027] FIG. 2 is a block diagram showing an example of the configuration of the transport system of this embodiment.

[0028] The transportation system of this embodiment includes a warehouse control device 1 and multiple transportation devices 2. The multiple transportation devices 2 are connected to the warehouse control device 1 via a network 4. The warehouse control device 1 is also connected to the station terminals described above, but these are not shown in FIG.

[0029] The warehouse control device 1 is a computer having an arithmetic unit 11, a memory 12, an input device 13, an output device 14, a storage device 15, and a communication interface 17. The warehouse control device 1 is not limited to the configuration shown in FIG. 2. The warehouse control device 1 may be a single computer or may be composed of multiple computers. Furthermore, each device of the warehouse control device 1 may be located on a single computer or may be distributed and located in multiple devices. The programs and information stored in the storage device 15 may be stored in a single storage device or may be distributed and stored in multiple storage devices.

[0030] The storage device 15 has a non-volatile storage medium and stores the programs executed by the arithmetic device 11 and data used by the programs. Examples of the programs stored in the storage device 15 are a route creation program 151, a transport device control program 156, a disturbance effect collection program 158, and a disturbance effect output program 159. The arithmetic device 11 loads the necessary programs into the memory 12 and executes them. Examples of the data stored in the storage device 15 are floor information 164, map information 165, device information 166, and route data 167.

[0031] The route creation program 151 refers to the map information 165 and calculates the movement route of the conveying device 2 from the position of the item (or product) to be picked and the position of the destination work station, etc. The route creation program 151 may refer to the disturbance information collected by the disturbance influence collection program 158 and create a travel route that avoids positions affected by the disturbance.

[0032] The transport device control program 156 refers to the route calculated by the route creation program 151, floor information 164, and equipment information 166 (e.g., the status of the transport device 2), etc., to determine the transport device 2 that will transport the shelf 51, and transmits a transport instruction to the transport device 2, including information on the shelf 51 to be transported and the work station to which it is to be transported.

[0033] The disturbance effect collection program 158 collects information on disturbances that are affecting position estimation from the transport device 2, stores the information in the storage device 15 together with the position information where the disturbance occurred, and creates a disturbance effect potential map in which the disturbances that have occurred are mapped onto map information.

[0034] The disturbance effect output program 159 outputs the disturbance information collected by the disturbance effect collection program 158 .

[0035] The floor information 164 stores information indicating the floor condition for each floor area. The map information 165 stores map information within the warehouse. The equipment information 166 stores the position and operation status of the transport equipment 2. The equipment information 166 stores the identification information, position, operation status, etc. of the transport equipment 2. The route data 167 stores information on the route for each transport equipment 2 within the warehouse.

[0036] The input device 13 is composed of a keyboard, a mouse, a touch panel, etc. The output device 14 is composed of a display, etc. The communication interface 17 communicates with the transport device 2 and other computers via the network 4, for example, wirelessly.

[0037] The conveying device 2 is an autonomous moving body that automatically conveys the shelf 51 loaded with the article in response to a conveying instruction from the warehouse control device 1. The conveying device 2 is an autonomous conveying device having a side camera 211, a LiDAR 212, a marker sensor 213, a floor sensor 214, a memory unit 22, a running wheel 23, a communication interface 24, a control unit 25, an analysis unit 26, a lighting 27, and a table 28. The conveying device 2 also has sensors (e.g., a vibration sensor and an acceleration sensor) that are not shown. The conveying device 2 can travel by switching between at least two movement modes, namely, linear movement and rotational movement (also called turning). Here, linear movement refers to traveling from one point to another point with a directionality. The trajectory of the linear movement may be a straight trajectory or a curved trajectory. Here, rotational movement refers to a movement in which the conveying device 2 rotates (turns) to change the traveling direction on the spot.

[0038] The side camera 211 is provided on the side of the conveying device 2 and captures an image of the surroundings of the conveying device 2. The LiDAR 212 is provided on the side of the conveying device 2 and irradiates the surroundings of the conveying device 2 with laser light to acquire three-dimensional point cloud data by reflected light. The marker sensor 213 is a camera provided on the lower surface of the conveying device 2 and captures an image of a marker provided on the floor surface of the storage space 52. The floor sensor 214 is a sensor provided on the lower surface of the conveying device 2 and is used to detect a guide line (e.g., a magnetic tape, a light reflecting tape, an electromagnetic induction cable, etc.) installed on the floor surface of the storage space 52 and control the conveying device 2 to run along the detected guide line. The floor sensor 214 can be a magnetic sensor that detects a magnetic tape installed on the floor surface, a camera that detects a light reflecting tape, or a proximity sensor that detects an electromagnetic induction cable. The observation results by the side camera 211, the LiDAR 212, the marker sensor 213, and the floor sensor 214 are used by the conveying device 2 to estimate its own position.

[0039] The control unit 25 includes a calculation device and a memory. The memory stores a travel control program, a position estimation program, an estimation mode switching program, and a disturbance reduction program. The programs stored in the memory are executed by the calculation device. The calculation device is composed of a microcomputer or a processor.

[0040] The travel control unit 251 controls the rotation of the running wheels 23 in accordance with the transport instruction transmitted from the warehouse control device 1 to cause the transport device 2 to travel.

[0041] The position estimation unit 252 estimates the self-position using the observation results of various sensors (such as the side camera 211, the LiDAR 212, the marker sensor 213, and the floor sensor 214). For example, the position estimation unit 252 estimates the self-position in a first mode using a position estimation model that analyzes an image of the surroundings of the transport device 2 captured by the side camera 211 and derives the position by comparing it with an image previously captured at the same position and stored. The position may be derived by comparing the feature amount of the image captured by the side camera 211 with the feature amount of an image previously captured. Each feature point is mapped to the estimated self-position by having three-dimensional information, and an environmental map is constructed. In addition, the position estimation unit 252 estimates the self-position in a LiDAR mode using a position estimation model that analyzes three-dimensional point cloud data of the surroundings of the transport device 2 acquired by the LiDAR 212 and compares it with three-dimensional point cloud data previously captured. Similarly, in the LiDAR mode, the three-dimensional point cloud data acquired at each time is mapped to the estimated self-position to construct an environmental map based on the three-dimensional point cloud. The position estimation unit 252 estimates the position in a marker mode in which the marker on the floor surface captured by the marker sensor 213 is read and the position of the storage space 52 in which the marker is provided is estimated as the self-position. The position estimation unit 252 estimates the position in a guide line mode in which the self-position is estimated to be off the guide line based on the detection result of the guide line by the floor sensor 214. The position estimation unit 252 also acquires an image of the transport device 2 captured by the camera 3 from the warehouse control device 1 and estimates the position of the transport device 2 by a motion capture technique using the image. For example, the motion capture technique estimates the position and orientation of the AGV on which the reflective marker is installed from the images captured by the infrared cameras by installing several infrared cameras. The reflective markers are installed, for example, at the four corners of the AGV so that the position and orientation of the installation surface can be known in real time, and it is preferable to control the reflective markers so that the difference from the target route is reduced according to the position and orientation information. Normally, environmental maps often have absolute errors, so it is necessary to calibrate the positional relationship with the motion capture device when creating the environmental map. For example, if it is known in advance that a certain point is off by 2 cm in the north direction, it is necessary to add an offset in the north direction.In this way, it is advisable to calibrate in advance the positional relationship between the environmental map, the actual environment, and the measurement results obtained by motion capture.

[0042] The estimation mode switching unit 253 switches the method of estimating the self-position used by the position estimation unit 252 .

[0043] When an image captured by the side camera 211 is difficult to analyze due to disturbance, the disturbance reduction unit 254 removes the influence of the disturbance from the image and generates an image that can be analyzed.

[0044] The storage unit 22 is configured with a non-volatile storage medium, and stores programs and data used by the programs. Examples of the data include map information 221 and travel distance information 222. The map information 221 is map information received from the warehouse control device 1. The travel distance information 222 is data calculated using odometry on the distance traveled by the transport device 2 from the reference point of the grid to which the transport device 2 belongs.

[0045] As shown in Fig. 4, the running wheels 23 are composed of two drive wheels and four auxiliary wheels (casters) and are attached to the underside of the transport device 2. The two drive wheels are arranged parallel to the straight-line direction. A separate motor and encoder, which serve as a drive device, are attached to each drive wheel, and the motor is supplied with power from a battery. If the two drive wheels rotate forward at the same rotation speed, the transport device 2 moves straight, and if the two drive wheels rotate in opposite directions, the transport device 2 turns.

[0046] The analysis unit 26 includes a calculation device and a memory. The memory stores a feature point extraction program, a disturbance area estimation program, a disturbance type recognition program, and a disturbance influence degree calculation program. The programs stored in the memory are executed by the calculation device. The calculation device is composed of a microcomputer or a processor. The feature point extraction unit 261 extracts feature points to be used for position estimation from the observation results of various sensors. The disturbance area estimation unit 262 estimates an area in the image that is affected by a disturbance. The disturbance type recognition unit 263 recognizes the type of disturbance that is affecting the image. The disturbance influence degree calculation unit 264 recognizes the type of disturbance that is affecting the image.

[0047] The arithmetic unit and memory of the analysis unit 26 may be the same hardware as the arithmetic unit and memory of the control unit 25, or may be separate hardware.

[0048] FIG. 3 is a perspective view of the conveying device 2 of this embodiment, and FIG.

[0049] The transport device 2 has on its upper surface a table 28 on which a shelf 51 can be mounted. The table 28 can be rotated horizontally by a drive device, and the shelf 51 rotates when the table 28 rotates.

[0050] The transport device 2 has lights 27 at the corners of its sides. The lights 27 brightly illuminate the area around the transport device 2, reducing disturbances caused by brightness and making it easier to check the surrounding conditions.

[0051] FIG. 5A is a flow chart showing the overall processing for a frame image in this embodiment.

[0052] The transportation device 2 refers to the created environmental map and generates a movement route for the transportation device 2 based on the positional relationship between the destination and the current location.

[0053] First, the transport device 2 acquires an image captured by a sensor at a certain point (S121), and executes a disturbance determination (reduction) process (S122). The details of the disturbance determination (reduction) process will be described with reference to Figs. 5B and 5C.

[0054] Thereafter, the transport device 2 executes a self-location estimation process in a predetermined self-location estimation mode (S123), and judges whether the self-location estimation is successful (S124). If the self-location estimation is successful, the transport device 2 calculates the deviation between the position on the target route and the estimated self-location (S125), and performs movement control while changing the current movement speed and turning speed so as to reduce the calculated deviation (S126).

[0055] On the other hand, if the self-location estimation fails and the current self-location is unknown, a request is made to re-estimate the self-location to the warehouse control device 1. In that case, it is advisable to move at a low speed and / or turn at a low turning speed so as to reduce the area determined as the disturbance area.

[0056] When controlling the operation in the map creation mode or the self-position estimation mode using the created environmental map, the influence degree of each disturbance (obstruction factor) according to the location of the environmental map is reflected in the environmental map, and a disturbance influence potential map with a continuous value of influence degree is created at each location in the warehouse. This disturbance influence potential map makes it possible to create a travel route that avoids areas with a large disturbance influence degree when generating a route connecting the current location and the destination. Since the stop of the movement of the conveying device 2 due to the inability to estimate the self-position is a factor that reduces throughput, by using such a disturbance influence potential map, a route that takes into account the disturbance influence degree and arrives quickly is visualized to the manager, and movement control is actually performed using that route. A method of avoiding a position with a high disturbance influence degree is to set a virtual obstacle in an area with a disturbance influence degree higher than a predetermined threshold, determine a predetermined safety distance from the set virtual obstacle, and generate a route that avoids an area where the safety distance is added to the obstacle area.

[0057] 5B and 5C are flowcharts of the disturbance determination (reduction) process of this embodiment.

[0058] The disturbance determination (reduction) process is executed by the analysis unit 26 in order to determine a position estimation mode when the position estimation unit 252 estimates the position of the transporting device 2. Then, the position estimation unit 252 estimates the position of the transporting device 2 using the position estimation mode determined by the analysis unit 26.

[0059] In the disturbance determination (reduction) process, first, the feature point extraction unit 261 extracts feature points from an image used for position estimation (S101). It is preferable to extract feature points such as corners of figures in the image or points with large luminance gradients.

[0060] Next, the disturbance region estimation unit 262 divides the image into predetermined regions (S102). For example, the image is divided into a predetermined number (or a predetermined size) of grids, about 4 to 10 vertically and horizontally.

[0061] Next, the disturbance type recognition unit 263 executes a divided region determination process for determining a disturbance in each divided region (S103). The details of the divided region determination process will be described with reference to FIG.

[0062] Next, the disturbance reduction unit 254 executes disturbance reduction processing to reduce the influence of disturbances in the image (S104). Details of the disturbance reduction processing will be described with reference to FIG.

[0063] Thereafter, the disturbance type recognition unit 263 executes a divided region determination process to determine whether the effect of the disturbance has been reduced by the process of step S104 to an extent that the image can be used (S105).

[0064] Thereafter, the disturbance reduction unit 254 determines whether all the divided regions have been processed (S106). If some of the divided regions have not been processed, the process returns to step S103 and repeats the process for the unprocessed divided regions. On the other hand, if all of the divided regions have been processed, the process proceeds to step S107.

[0065] Next, the disturbance influence calculation unit 264 calculates the proportion of the disturbance determination area by dividing the number of disturbance determination areas in which it is determined that a disturbance exists by the total number of divided areas, and determines whether the proportion of the disturbance determination area is greater than a predetermined threshold value Th1 (S107).

[0066] If the proportion of areas determined to have disturbance is greater than a predetermined threshold Th1, then most of the area in the image cannot be used, so the disturbance reduction unit 254 determines that the entire area of ​​the frame image is not to be used (S108) and terminates the disturbance determination (reduction) process.

[0067] On the other hand, if the proportion of areas determined to have disturbances is equal to or less than the predetermined threshold value Th1, there is a usable area in the image, so the disturbance reduction unit 254 rejects the divided areas determined to have disturbances in step S105 and adopts the divided areas determined to have no disturbances (S109).

[0068] Next, the disturbance reduction unit 254 calculates the percentage of areas that are determined to have disturbance due to blur by dividing the number of blur determination areas that are labeled with "blur" by the total number of areas, and determines whether the percentage is greater than a predetermined threshold value Th2 (S110).

[0069] If the proportion of the blur judgment area is greater than a predetermined threshold Th2, the disturbance reduction unit 254 changes the vehicle control to a rotation speed reduction mode (S111) and turns on the lighting 27 (S112). Furthermore, the disturbance reduction unit 254 may switch the position estimation mode to a motion capture mode that estimates the position using a motion capture technique provided by the warehouse control device 1, a marker mode that estimates the position using a floor marker, and a guide line mode that travels along a guide line (S113). The motion capture mode, marker mode, and guide line mode should be switched in order of decreasing movement constraints. Generally, the priority order should be the motion capture mode, the marker mode, and the guide line mode. For example, the motion capture mode should be used in an environment within the motion capture detectable area and where no occlusion occurs. When using floor markers, it is recommended to install markers other than the floor markers in the surrounding environment and switch in an environment where marker-based detailed position control can be performed using a side camera. In areas where occlusion occurs or when guiding lines are drawn on the floor, it is advisable to switch to the guiding line mode as the final position estimation method candidate. The position estimation mode may be switched by estimating the position in all modes, or may be switched sequentially depending on the availability of each mode.

[0070] On the other hand, if the proportion of areas including feature points determined to have disturbance due to shaking is equal to or smaller than the predetermined threshold value Th2, the disturbance influence calculation unit 264 determines that disturbance due to lighting exists, calculates the proportion of the lighting determination areas by dividing the number of lighting determination areas labeled with "lighting" by the total number of divided areas, and determines whether the proportion of the lighting determination areas is greater than a predetermined threshold value Th3 (S114).

[0071] If the ratio of the illumination determination area is larger than a predetermined threshold value Th3, the disturbance reduction unit 254 determines that the influence of disturbance caused by the illumination is large, and switches to one of the LiDAR mode, which estimates the position using three-dimensional point cloud data observed by the LiDAR 212, the marker mode, which estimates the position using floor markers, and the guide line mode, which travels along the guide line (S117). The switching of each mode may be selected in the order of priority of the LiDAR mode, the marker mode, and the guide line mode from the viewpoint of movement constraints. The switching of the position estimation mode may be performed by estimating the position in all modes, or may be performed sequentially depending on the availability of each mode.

[0072] On the other hand, if the proportion of illumination judgment areas is equal to or smaller than a predetermined threshold value Th3, the disturbance influence calculation unit 264 determines that there is disturbance due to a dark place, calculates the proportion of dark place judgment areas by dividing the number of dark place judgment areas labeled "dark place" by the total number of divided areas, and determines whether the proportion of dark place judgment areas is greater than a predetermined threshold value Th4 (S115).

[0073] If the ratio of the dark place determination area is greater than a predetermined threshold value Th4, the disturbance reduction unit 254 switches to any one of the following modes: a LiDAR mode that estimates the position using the observation results of the LiDAR 212, a marker mode that estimates the position using floor markers, and a guide line mode that travels along a guide line (S117). The position estimation mode may be switched by estimating the position in all modes, or may be switched sequentially depending on the availability of each mode.

[0074] On the other hand, if the proportion of dark judgment areas is equal to or less than a predetermined threshold Th4, the disturbance influence calculation unit 264 determines that there is little change in the image of the area and that there is a disturbance that impedes recognition, calculates the proportion of textureless judgment areas by dividing the number of textureless judgment areas labeled "textureless" by the total number of divided areas, and determines whether the proportion of textureless judgment areas is greater than a predetermined threshold Th5 (S116).

[0075] If the ratio of the textureless determination region is greater than a predetermined threshold Th5, the disturbance reduction unit 254 switches to any one of the following modes: a LiDAR mode that estimates the position using the observation results of the LiDAR 212, a marker mode that estimates the position using floor markers, and a guide line mode that travels along a guide line (S117). The position estimation mode may be switched by estimating the position in all modes, or may be switched sequentially depending on the availability of each mode.

[0076] On the other hand, if the proportion of the textureless determination area is equal to or less than the predetermined threshold value Th5, the disturbance reduction unit 254 ends the disturbance determination (reduction) process without switching the position estimation mode.

[0077] As described above, the conveying device 2 of this embodiment identifies the type of disturbance and the influence of the disturbance that affect the self-location estimation of the conveying device 2, and identifies and executes a method of self-location estimation based on the information on the type of disturbance and the influence of the disturbance. Identifying the method of self-location estimation based on the information on the type of disturbance and the influence of the disturbance includes, for example, a disturbance reduction process according to the type of disturbance, and a process of determining whether or not to switch the self-location estimation mode based on the information on the influence of a specific type of disturbance. This makes it possible to adjust the method of self-location estimation according to the type and influence of the disturbance, and effectively reduces the deterioration of the accuracy of self-location estimation.

[0078] An example of a conveying device 2 described in this embodiment is a conveying device 2 that conveys items in a traveling area including multiple facilities, and is equipped with a control unit 25 that estimates the position of the conveying device 2 based on image information acquired by a sensor unit provided in the conveying device 2, and an analysis unit 26 that identifies the type of disturbance that affects the estimation of the position of the conveying device 2 and the degree of influence of the disturbance from the image information, wherein the control unit 25 is capable of estimating the position of the conveying device 2 by switchably executing multiple different position estimation methods, including a first estimation method that estimates the position of the conveying device 2 based on image information, and when estimating the position of the conveying device 2 using the first estimation method, the control unit 25 is capable of identifying a second estimation method for reducing the influence of the disturbance from among the multiple different position estimation methods based on information on the type of disturbance and the degree of influence of the disturbance, and switching from the first estimation method to the second estimation method to estimate the position of the conveying device 2.

[0079] FIG. 6 is a flowchart of the divided region determination process (S103) of this embodiment.

[0080] In the divided region determination process, first, the disturbance type recognition unit 263 determines the amount of edges extracted from the image (S1030). The edge extraction method may use, for example, a Sobel filter or a Laplacian filter. By applying the Laplancian filter, the gradient strength value is calculated for each pixel. If the number of pixels determined to be edges based on a predetermined gradient strength threshold is less than a predetermined threshold Th6, it is determined that the image is blurred due to camera shake, that the region in question has a disturbance due to shake, and the region is labeled as "blur" (S1031).

[0081] On the other hand, if the number of pixels determined to be edges is equal to or greater than the predetermined threshold Th6, the disturbance type recognition unit 263 determines for each pixel in the region whether the luminance difference within the predetermined range is smaller than the predetermined threshold Th7 (S1032). If the luminance difference within the predetermined range (e.g., several pixels) is smaller than the predetermined threshold Th7, it determines whether the number of pixels in the region whose luminance value is equal to or greater than the predetermined value is greater than the predetermined threshold Th8 (S1033). If the number of pixels whose luminance value is equal to or greater than the predetermined value is greater than the predetermined threshold Th8, the disturbance type recognition unit 263 determines that lighting is reflected in the region and that a disturbance with a large luminance difference in the region is caused by lighting, and labels the region as "lighting" (S1034).

[0082] On the other hand, if the brightness difference within a predetermined range (e.g., several pixels) is equal to or greater than a predetermined threshold Th7, or the number of pixels whose brightness values ​​are equal to or greater than a predetermined value is equal to or less than a predetermined threshold Th8, the disturbance type recognition unit 263 determines whether the number of pixels whose brightness values ​​are equal to or less than a predetermined value in the region is greater than a predetermined threshold Th9 (S1035). If the number of pixels whose brightness values ​​are equal to or less than a predetermined value is greater than the predetermined threshold Th9, the disturbance type recognition unit 263 determines that the region is disturbed by a dark place with many low-luminance pixels, and labels the region as a "dark place" (S1036).

[0083] Then, the disturbance type recognition unit 263 determines the amount of edges extracted from the image (S1037), and determines the number of feature points extracted from the image (S1038). If the number of pixels determined to be edges in the region is less than a predetermined threshold Th10, or if the number of feature points extracted from the image in the region is less than a predetermined threshold Th11, the disturbance type recognition unit 263 determines that there is little change in the image in the region and that there is a disturbance that interferes with recognition, and labels the region as "textureless" (S1039).

[0084] On the other hand, if the number of pixels determined to be edges in the region is equal to or greater than a predetermined threshold Th10 and the number of feature points extracted from the image in the region is equal to or greater than a predetermined threshold Th11, there is a lot of change in the image in the region, so the disturbance type recognition unit 263 terminates the divided region determination process without assigning any label to the region.

[0085] As described above, in the divided area determination process of this embodiment, by identifying the area in which a disturbance is occurring and the type of disturbance, it is possible to evaluate the influence of the disturbance and perform disturbance reduction processing according to the type of disturbance.

[0086] FIG. 7 is a flowchart of the disturbance reduction process of this embodiment.

[0087] In the disturbance reduction process, first, the disturbance reduction unit 254 determines whether the region is labeled as "blurred" (S1040). If the region is labeled as "blurred", an edge clarification process is executed (S1041). As a method of edge clarification process, for example, a blur removal filter may be applied. The blur removal filter may be applied as an anisotropic Gaussian filter with the blur strength, direction, and aspect ratio as parameters. However, the method is not limited to such a rule-based method, and blur may be removed using machine learning, and various methods can be applied.

[0088] On the other hand, if the region is not labeled as "blur", it is determined whether the region is labeled as "lighting" (S1042). If the region is labeled as "lighting", contrast smoothing processing is executed (S1043). As a method of contrast smoothing processing, for example, gamma correction, contrast stretching, histogram equalization processing, or the like may be applied. However, the method is not limited to such a rule-based method, and contrast smoothing may be performed using machine learning, and various methods can be applied.

[0089] On the other hand, if the region is not labeled as "illumination", it is determined whether the region is labeled as "dark" (S1044). If the region is labeled as "dark", gamma correction processing, histogram smoothing processing, etc. are applied to adjust the brightness of the image (S1045). However, the method is not limited to such a rule-based method, and machine learning may be used to adjust the brightness, and various other methods can be applied.

[0090] As described above, the control unit 25 of this embodiment has a position estimation unit 252 that estimates the position of the conveying device 2 using one of a plurality of position estimation modes including a first mode that estimates the position of the conveying device 2 based on an image acquired by the first sensor (side camera 211), and a disturbance reduction unit 254 that reduces the influence of disturbances that affect the position estimation of the conveying device 2 in the first mode. Therefore, by performing disturbance reduction processing according to the type of disturbance, deterioration in the accuracy of self-position estimation can be effectively reduced.

[0091] In addition, the conveying device 2 is equipped with a second sensor (LiDAR 212) that acquires three-dimensional information about the surroundings thereof. When the analysis unit 26 analyzes that the influence of disturbances is large, the position estimation unit 252 estimates the position of the conveying device 2 using a second LiDAR mode that estimates the position of the conveying device 2 based on the three-dimensional information. Therefore, when the surrounding environment has three-dimensional structural characteristics, and the influence of disturbances is large in image-based position estimation and self-position estimation is not possible, the conveying device 2 can be substituted with a sensor that acquires three-dimensional information that is less influenced by disturbances.

[0092] In addition, the vehicle is equipped with a third sensor (marker sensor 213) that acquires information on a position marker that indicates a position within the driving space. When the analysis unit 26 analyzes that the influence of external disturbances is large, the position estimation unit 252 estimates the position of the vehicle 2 using a third marker mode that estimates the position of the vehicle 2 based on the information on the position marker. Therefore, when markers are affixed to the floor surface or the environment, the influence of disturbances on image-based position estimation is large, and self-position estimation is not possible, so that the position can be estimated with high accuracy.

[0093] In addition, the device is equipped with a fourth sensor (floor surface sensor 214) that acquires information about a guide line that guides the conveying device 2 along the running path, and when the analysis unit 26 analyzes that the influence of external disturbances is large, the position estimation unit 252 controls the position of the conveying device 2 using a fourth guide line mode that controls the running of the conveying device 2 based on the information about the guide line.Therefore, when a guide line is installed, the influence of external disturbances on image-based position estimation is large and self-position estimation is not possible, so that the position can be estimated using the guide line.

[0094] In addition, the disturbance reduction unit 254 performs image processing on the image captured by the first sensor to reduce the effects of disturbances contained in the image by edge definition, contrast smoothing, or gamma correction, so that image-based position estimation can be performed.

[0095] Furthermore, when it is analyzed that the influence of disturbance is large, the disturbance reducing unit 254 controls the first sensor to capture an image of the surroundings of the transport device with the illumination 27 turned on, thereby reducing blurring in the image.

[0096] In addition, when it is analyzed that the influence of the disturbance is large, the disturbance reduction unit 254 controls the first sensor to acquire an image of the surroundings of the conveying device while reducing the operating speed of the drive device (running wheels 23), thereby reducing blurring in the image.

[0097] The analysis unit 26 also has a disturbance effect calculation unit 264 that determines that the effect of the disturbance is large when the proportion of the area affected by the disturbance in the image acquired by the first sensor is greater than a predetermined threshold value. By calculating the magnitude of the effect of the disturbance in the entire image, it is possible to determine whether the self-position can be estimated using the image including the disturbance.

[0098] The analysis unit 26 also has a feature point extraction unit 261 that extracts feature points having geometric characteristics from the image acquired by the first sensor, and a disturbance influence calculation unit 264 that determines that the influence of the disturbance is large when the ratio of the number of feature points affected by the disturbance to the total number of feature points in the image acquired by the first sensor is greater than a predetermined threshold. By calculating the magnitude of the influence of the disturbance in the entire image, it is possible to determine whether the self-position can be estimated using the image including the disturbance.

[0099] In addition, in the fifth motion capture mode, if the position estimation unit 252 analyzes that the influence of disturbances due to image blurring is large, it estimates the position of the conveying device 2 by motion capture technology using images acquired by the fifth sensor (camera 3). Therefore, in areas with few features over a wide area, the image mode and LiDAR mode are unable to detect the pattern and structural features in the environment, and self-position estimation is likely to fail. However, by using a motion capture system capable of measuring a wide area, the self-position can be estimated with high accuracy without occlusion of markers.

[0100] In addition, the warehouse control device 1 has a route creation unit 151 that refers to the disturbance information collected by the disturbance effect collection unit 158 ​​and creates a driving route for the conveying device 2 while avoiding locations affected by the disturbance. Therefore, the conveying device 2 can travel along a route that passes through areas with little disturbance effect without stopping due to the influence of the disturbance, thereby improving the throughput of the entire conveying system.

[0101] In the above embodiment, the conveying device 2 executes the self-position estimation process and the disturbance determination (reduction) process, but the warehouse control device 1 may execute the position estimation process and the disturbance determination (reduction) process of the conveying device 2.

[0102] As described above, according to an embodiment of the present invention, in a disturbed environment, the influence of disturbances can be reduced and the accuracy of position estimation can be improved by disregarding disturbance areas, applying disturbance reduction processing, or switching position estimation modes depending on the type of disturbance (blur, lighting, dark areas, textureless) that affects image feature extraction and its degree of influence.

[0103] In the above-described embodiment, an example has been described in which the present invention is applied to a conveying device that travels in a warehouse, but the present invention can also be applied to other moving devices such as automobiles, robots, and mobility devices.

[0104] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. Furthermore, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0105] In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0106] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0107] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0108] 1 Warehouse control device 2. Conveyor 3. Camera 4. Network 11 Arithmetic unit 12. Memory 13 Input Devices 14 Output Devices 15 Storage device 17 Communication Interface 22 Memory section 23 Running wheel 24 Communication Interface 25 Control Unit 26 Analysis Department 27 Lighting 28 Table 51 Shelf 52 Storage Space 80 Frontage 151 Route Creation Program 156 Transport device control program 158 Disturbance Effects Collection Program 159 Disturbance effect output program 164 Floor information 165 Map Information 166 Device information 167 Route Data 211 Side Camera 213 Marker Sensor 214 Floor Sensor 221 Map Information 222 Travel distance information 251 Driving control unit 252 Position estimation part 253 Estimation mode switching unit 254 Disturbance reduction section 261 Feature Point Extraction Unit 262 Disturbance area estimation part 263 Disturbance type recognition unit 264 Disturbance Influence Calculation Unit

Claims

1. A conveying device that conveys an object to be conveyed in a traveling space, A first sensor for acquiring an image of the surroundings of the transport device; a control unit that estimates a position of the transport device based on an image acquired by the first sensor; an analysis unit that analyzes a disturbance that affects an estimation of a position of the transport device based on an image acquired by the first sensor; The control unit is a position estimation unit that estimates a position of the transport device using one of a plurality of position estimation modes including a first mode that estimates a position of the transport device based on an image acquired by the first sensor; a disturbance reduction unit that reduces an effect of a disturbance that affects the position estimation of the transport device in the first mode.

2. 2. The conveying device according to claim 1, The analysis unit identifies information on a type and area of ​​disturbance that affects an estimation of a position of the transport device based on the image acquired by the first sensor; The transport device, wherein the disturbance reduction unit reduces an effect of a disturbance that affects a position estimation of the transport device in the first mode based on the information identified by the analysis unit.

3. 2. The conveying device according to claim 1, a second sensor for acquiring three-dimensional information about a periphery of the transport device; The conveying device, characterized in that when the analysis unit analyzes that the influence of the disturbance is large, the position estimation unit estimates the position of the conveying device using a second mode that estimates the position of the conveying device based on the three-dimensional information.

4. 2. The conveying device according to claim 1, a third sensor for acquiring information of a position marker indicating a position within the travel space; The position estimation unit is characterized in that, when the analysis unit analyzes that the influence of the disturbance is large, the position estimation unit estimates the position of the conveying device using a third mode that estimates the position of the conveying device based on information of the position marker.

5. 2. The conveying device according to claim 1, a fourth sensor for acquiring information on a guide line for guiding the conveying device along a travel path; The conveying device is characterized in that, when the analysis unit analyzes that the influence of the disturbance is large, the position estimation unit controls the position of the conveying device using a fourth mode that controls the travel of the conveying device based on information about the guide line.

6. 2. The conveying device according to claim 1, A conveying device characterized in that the disturbance reduction unit performs image processing on the image acquired by the first sensor to reduce the effects of disturbances contained in the image by edge clarification, contrast smoothing, or gamma correction.

7. 2. The conveying device according to claim 1, A lighting device is provided to illuminate the surroundings of the conveying device, The transport device, characterized in that, when it is analyzed that the influence of the disturbance is large, the disturbance reduction unit controls the first sensor to acquire an image of the surroundings of the transport device while the lighting device is turned on.

8. 2. The conveying device according to claim 1, a driving device for driving a table on which the transport object is loaded, A conveying device characterized in that, when it is analyzed that the influence of the disturbance is large, the disturbance reduction unit controls the first sensor to acquire images of the surroundings of the conveying device while reducing the operating speed of the drive device.

9. 2. The conveying device according to claim 1, The analysis unit is characterized in that it has a disturbance influence calculation unit that determines that the influence of the disturbance is large when the proportion of the area affected by the disturbance in the image acquired by the first sensor is greater than a predetermined threshold value.

10. 2. The conveying device according to claim 1, The analysis unit includes: a feature point extraction unit that extracts feature points having geometric characteristics from the image acquired by the first sensor; a disturbance influence calculation unit that determines that the influence of the disturbance is large when a ratio of the number of feature points affected by the disturbance to the total number of feature points in the image acquired by the first sensor is greater than a predetermined threshold value.

11. 1. A conveying system comprising: A conveying device according to any one of claims 1 to 10, which conveys an object to be conveyed; A management device that controls the movement of the transport device; a fifth sensor for acquiring an image of the transport device; A conveying system characterized in that, when the analysis unit analyzes that the influence of the disturbance is large, the position estimation unit estimates the position of the conveying device using a fifth mode in which the position of the conveying device is estimated using an image acquired by the fifth sensor.

12. 12. The transport system of claim 11, A conveying system characterized in that in the fifth mode, when it is analyzed that the influence of disturbances due to image blurring is large, the position estimation unit estimates the position of the conveying device by motion capture technology using images acquired by the fifth sensor.

13. 12. The transport system of claim 11, The transportation system according to claim 1, wherein the management device has a disturbance effect collection unit that collects information on disturbances that are affecting the position estimation from the transportation device and stores the information together with position information where the disturbance occurred.

14. 14. The transport system of claim 13, The transportation system according to the present invention, wherein the management device further comprises a disturbance effect output unit that outputs information about the disturbances collected by the disturbance effect collection unit.

15. 14. The transport system of claim 13, The transportation system is characterized in that the management device has a route creation unit that references the information on the disturbance collected by the disturbance influence collection unit and creates a travel route for the transportation device while avoiding locations affected by the disturbance.

16. A control method executed by a conveying device, comprising: the transport device includes a computing device that executes a program, a storage device that stores the program, and a first sensor that acquires an image of the surroundings of the transport device; The control method includes: a position estimation step of estimating a position of the transport device using any one of a plurality of position estimation modes including a first mode of estimating a position of the transport device based on an image acquired by the first sensor; an analysis step for analyzing disturbances affecting an estimate of the position of the transport device based on the images acquired by the first sensor; and a disturbance reduction step for reducing the effect of a disturbance affecting the position estimation of the transport device in the first mode.

Images