Flight System

Abstract

To provide a flight system that enables effective use of object recognition of luggage in facilities where luggage is placed on storage racks.SOLUTION: A flight system used in facilities where luggage is placed on storage shelves, equipped with a flight vehicle having photographic means is provided, where the flight vehicle performs a flight operation to photograph the luggage while flying along a predetermined flight route, and where object recognition of the luggage is performed for each of the captured image obtained by photography during the flight operation and the luggage image, which is an image of the luggage acquired in advance.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a flight system including an air vehicle. [Background technology]

[0002] Conventionally, monitoring systems have been proposed for monitoring luggage stored in warehouses etc. An example of such a monitoring system is disclosed in Patent Document 1. The luggage monitoring system disclosed in Patent Document 1 is a system that uses a drone flying within a warehouse where pallets carrying luggage are stacked in multiple layers in a flat warehouse.

[0003] The drone flies within the warehouse while avoiding collisions with obstacles in the warehouse using sensors, and acquires image information using a camera. The acquired image information can be transmitted to an external device to monitor the condition of the luggage in the warehouse. In particular, when the drone approaches tilted luggage or pallets, the camera mounted on the drone acquires image information, making it possible to detect the tilted luggage or pallets. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-43815 A Summary of the Invention [Problem to be solved by the invention]

[0005] If object recognition of luggage could be effectively utilized in facilities where luggage is stored on storage shelves, it would be useful for monitoring luggage, etc. An object of the present invention is to provide a flight system that makes it possible to effectively utilize object recognition of luggage in facilities where luggage is stored on storage shelves. [Means for solving the problem]

[0006] In order to achieve the above-mentioned objective, the flight system of the present invention is a flight system that is equipped with an aircraft having an imaging means and is used in a facility where luggage is placed on storage shelves, wherein the aircraft performs a flight operation to photograph the luggage while flying along a predetermined flight route, and is configured to perform object recognition of the luggage for each of the photographed images obtained by the flight operation and luggage images, which are images of the luggage that have been acquired in advance.

[0007] According to this configuration, it becomes possible to effectively utilize object recognition of luggage in a facility where luggage is placed on a storage shelf. More specifically, the luggage image may be acquired as an image showing the luggage in a normal placement state. Note that the "placement state" refers to the state (position or orientation) of how the luggage is placed on the storage shelf.

[0008] More specifically, the luggage image may be an image captured when the luggage is placed on the storage shelf. More specifically, the luggage image may be a configuration in which a difference between a placement state of the luggage in the captured image and a placement state of the luggage in the luggage image is recognized based on a result of the object recognition.

[0009] More specifically, the difference may be a difference between an orientation of the luggage in the captured image and an orientation of the luggage in the luggage image.

[0010] More specifically, the above configuration may be configured such that, when the difference exceeds a predetermined value, the detection result is notified to a predetermined notification destination.

[0011] More specifically, the above configuration may be configured to: photograph the area below while the aircraft flies along the flight route; and based on the information obtained from the photographs, detect any abnormal conditions related to the position or orientation of the luggage, including the luggage having fallen from the storage shelf; and create a distribution map of the locations where the abnormal conditions occurred based on the results of the abnormal condition detection process. Effect of the Invention

[0012] The monitoring system according to the present invention is useful for monitoring luggage in a facility where luggage is stored on storage shelves. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a logistics facility. [Diagram 2] FIG. 2 is a top view of the storage shelf and the article transport device. [Diagram 3] FIG. 2 is a front view of the article transport device. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a support; [Diagram 5] FIG. 1 is a block diagram showing an example of a configuration of a monitoring system. [Figure 6] FIG. 1 is an external view showing an example configuration of a drone. [Figure 7] 4 is a flowchart showing the overall operation of the monitoring system. [Figure 8] 13 is a flowchart showing specific operations in the second flight operation. [Figure 9] 13 is a flowchart showing specific operations in the third flight operation. [Figure 10] 1 is a schematic diagram illustrating an example of a first flight operation. [Figure 11] FIG. 2 is a schematic diagram illustrating an example of a second flight operation. [Figure 12] FIG. 11 is a schematic diagram illustrating an example of a third flight operation. [Figure 13] FIG. 13 is an explanatory diagram regarding deviation of the orientation of luggage. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a first embodiment and a second embodiment of the present invention will be described with reference to the drawings.

[0015] (1) First embodiment 1. Overall configuration of logistics facilities First, a first embodiment of the present invention will be described. Fig. 1 shows an example of the configuration of a logistics facility installed in a warehouse, etc. Note that the monitoring system according to this embodiment will be described by taking an example in which the monitoring system is installed in the logistics facility 100 shown in Fig. 1.

[0016] The logistics facility 100 includes a storage shelf 10, an inbound / outbound conveyor 20, a traveling rail 30, a transport cart 35, a picking conveyor 40, an automatic sorting device 50, a shipping waiting area 55, and a truck berth 60. In Fig. 1, a roof and the like are omitted to make it easier to see the layout of the logistics facility 100. The logistics facility 100 also includes an article transport device 15 (see Fig. 2) that moves articles into and out of the storage shelf 10 and the inbound / outbound conveyor 20. The storage shelf 10 and the article transport device 15 will be described in detail later.

[0017] Furthermore, the logistics facility 100 is equipped with a management device that controls the operation of the article transport device 15, the inbound / outbound conveyor 20, the transport cart 35, etc. The management device controls the operation of each part so that the article transport device 15 takes out the luggage from the storage shelf 10 and transports it to the shipping waiting area 55 via the inbound / outbound conveyor 20, the transport cart 35, the picking conveyor 40, and the automatic sorting device 50. The management device also controls the operation of each part so that the received luggage is stored in the storage shelf 10 by the article transport device 15 via the transport cart 35 and the inbound / outbound conveyor 20.

[0018] 2.Storage shelves and item transport devices Next, the storage shelf 10 and the item transport device 15 will be described in more detail. Fig. 2 is a view of the vicinity of the storage shelf 10 as seen from above. As shown in this figure, in the logistics facility 100, item transport devices 15A-15C (hereinafter sometimes collectively referred to as "item transport devices 15") are arranged close to the storage shelf 10. Fig. 3 is a view of the storage shelf 10 and the item transport device 15 as seen from the front side.

[0019] In the following description, the direction along the path of movement of the item transport device 15 is referred to as the "left-right direction X." Furthermore, of the directions perpendicular to the path of movement of the item transport device 15, the direction along the vertical direction is referred to as the "up-down direction Z," and the direction along the horizontal direction is referred to as the "front-rear direction Y." The up-down direction Z and the left-right direction X correspond to the up-down, left-right, and right directions on the paper surface of FIG. 3. The front-rear direction Y corresponds to the direction perpendicular to the paper surface of FIG. 3. Furthermore, the side of the item transport device 15 with respect to the storage shelf 10, i.e., the side in the direction in which the luggage is removed, is referred to as the "front side," and the opposite side is referred to as the "rear side."

[0020] The storage shelf 10 stores luggage M. As shown in FIG. 3, the storage shelf 10 has a plurality of support posts 102 and a plurality of shelf boards 104. The plurality of support posts 102 are erected in different positions in the left-right direction X along the up-down direction Z. The plurality of shelf boards 104 are arranged in different positions in the up-down direction Z along the left-right direction X. Each shelf board 104 is fixed across a pair of support posts 102 arranged separately in the left-right direction X. A storage section S for storing luggage M is formed as a space surrounded by a pair of support posts 102 adjacent to each other in the left-right direction X and a pair of shelf boards 104 adjacent to each other in the up-down direction Z. The plurality of storage sections S are arranged in a state of being aligned vertically and horizontally.

[0021] The luggage M stored on the storage shelf 10 is stored while placed on a support D. An example of the configuration of the support D is shown in FIG. 4. The support D has a plurality of tabletops Db and a plurality of legs Da. The tabletops Db extend in a direction perpendicular to the direction in which the legs Da extend, and are arranged on the legs Da in a direction perpendicular to the direction in which the legs Da are arranged. The support D is what is called a skid, which consists of a tabletop Db and legs Da and does not have a bottom panel. A typical example of a skid used as the support D is one made of wood.

[0022] 2, a pair of storage shelves 10 are provided facing each other in the front-rear direction Y with one item transport device 15 moving on one traveling rail 16 in between. A plurality of units, each of which is made up of one item transport device 15 and a pair of storage shelves 10 facing each other in the front-rear direction Y with the item transport device 15 in between, are arranged along the front-rear direction Y.

[0023] More specifically, as shown in FIG. 2, a unit is formed from item transport apparatus 15A and storage shelves 10A and 10B sandwiching it in the fore-and-aft direction Y, item transport apparatus 15B and storage shelves 10C and 10D sandwiching it in the fore-and-aft direction Y, and item transport apparatus 15C and storage shelves 10E and 10F sandwiching it in the fore-and-aft direction Y.

[0024] The article transport apparatus 15 is configured as a stacker crane. As shown in FIG. 3, the article transport apparatus 15 has a traveling cart 152, a mast 154, a lifting platform 156, and a transfer device 158.

[0025] The traveling cart 152 travels on traveling rails 16 installed on the floor surface along the left-right direction X. This allows item transport device 15A to travel in the space sandwiched between the front sides of storage shelves 10A and 10B, item transport device 15B to travel in the space sandwiched between the front sides of storage shelves 10C and 10D, and item transport device 15C to travel in the space sandwiched between the front sides of storage shelves 10E and 10F.

[0026] The masts 154 are provided in a pair. The pair of masts 154 are erected in the up-down direction Z at both ends of the traveling carriage 152 in the left-right direction X. The lifting platform 156 rises and falls along the masts 154. The transfer device 158 is placed on the lifting platform 156. The transfer device 158 is configured to be able to move forward and backward relative to the lifting platform 156. The transfer device 158 is configured with a slide fork having a pair of slidable fork parts.

[0027] The article transport device 15 having the transfer device 158 performs unloading and scooping processes for the storage section S and the loading / unloading conveyor 20, and performs transfer work for each. Here, the unloading process is a process of unloading the luggage M placed on the slide fork, and the scooping process is a process of scooping up the luggage M and placing it on the slide fork. In these unloading and scooping processes, the transfer device 158 takes the luggage M in and out of the storage shelf 10 while supporting the luggage M from below. More specifically, the transfer device 158 transfers the luggage M while supporting it from below, with the slide fork constituting the transfer device 158 inserted into the gap space Dc (see FIG. 4) between adjacent legs Da of the support body D.

[0028] For all luggage M placed on the storage shelf 10, an image showing the placement state on the storage shelf 10 is obtained by photographing (hereinafter, for convenience, referred to as "luggage image α"), and the luggage image α is managed in association with the luggage M and the position of the storage shelf 10 on which it is placed. The luggage image α is obtained, for example, by photographing the storage shelf 10 from the side (front side) so that the luggage M is visible when the luggage is placed on the storage shelf 10, and is an image that shows in what direction the product M was placed.

[0029] As shown in Fig. 2, the inlet / outlet conveyors 20 are provided adjacent to the storage shelves 10 in the left-right direction X. The inlet / outlet conveyors 20 are provided in pairs so as to be adjacent to each of a pair of storage shelves 10. One end of each inlet / outlet conveyor 20 faces, in the front-rear direction Y, the traveling rail 16 that constitutes the movement path of the article transport device 15. The other end of each inlet / outlet conveyor 20 faces the traveling rail 30 formed in a loop shape. A transport cart 35 travels on the traveling rail 30.

[0030] 3. Monitoring system configuration In the logistics facility 100, as the monitoring system according to this embodiment, a monitoring system for monitoring the state of the luggage M stored in the storage rack 10 is provided. The configuration of the monitoring system will be described below.

[0031] Fig. 5 is a block diagram showing the configuration of a monitoring system 70 according to this embodiment. The monitoring system 70 includes a drone (an example of an aircraft) 71, a management device 72, and a vibration sensor 73. Fig. 6 is an external view showing an example of the configuration of the drone 71 according to this embodiment. The drone 71 shown in Fig. 6 is a quadcopter equipped with four rotors (rotor wings). The drone 71 includes a frame 71A, a central portion 71B, and rotors 71C to 71F.

[0032] Frame 71A extends radially from a central position in all four directions, forming the diagonals of a square. Rotors 71C to 71F are disposed at each tip of diagonal frame 71A. The rotation shafts of rotors 71C to 71F are disposed at each vertex of the square.

[0033] The central portion 71B is disposed at the center lower side of the frame 71A. A first distance sensor 714, a second distance sensor 715, and a camera 716 (an example of an imaging means) are provided in the central portion 71B. The first distance sensor 714 is a sensor that measures the distance to an object located in front of the central portion 71B. The second distance sensor 715 is a sensor that measures the distance to an object located below the central portion 71B. The first distance sensor 714 and the second distance sensor 715 are configured, for example, by ultrasonic sensors. The camera 716 is capable of capturing images below and to the sides of the central portion 71B. The central portion 71B is also provided with a flight controller 711, a battery 718, and the like.

[0034] As shown in FIG. 5, the drone 71 has a flight controller 711, an ESC (Electronic Speed ​​Controller) 712, a rotor motor 713, a first distance sensor 714, a second distance sensor 715, a camera 716, a communication unit 717, and a battery 718.

[0035] The flight controller 711 sends a control signal to the ESC 712 based on an instruction from the management device 72 received by a communication unit 717, and causes the ESC 712 to drive the rotor motor 713. In this way, the flight controller 711 controls the flight of the drone 71.

[0036] The flight controller 711 has a control unit 711A, a sensor group 711B, and a GPS receiver unit 711C. The control unit 711A is configured by, for example, a microcomputer. Information acquired by the sensor group 711B, the GPS receiver unit 711C, the first distance sensor 714, and the second distance sensor 715 is input to the control unit 711A. The control unit 711A outputs an appropriate control signal to the ESC 712 based on the information acquired by the sensor group 711B and the GPS receiver unit 711C.

[0037] The control unit 711A has a storage unit 711A1. The storage unit 711A1 can store, for example, a program executable by the control unit 711A, flight route information described later, and images captured by a camera 716 described later. The control unit 711A also performs machine learning, and is capable of real-time object recognition of surrounding objects (such as the storage shelf 10, the article transport device 15, and the baggage M) based on the images captured by the camera 716.

[0038] To realize this object recognition, for example, an algorithm such as YOLO (You Only Look Once) can be used. "Machine learning" is a method of autonomously finding laws and rules by repeatedly learning based on given information. In the above machine learning, deep learning may be adopted to enable smooth processing of multidimensional data structures. This "deep learning" is machine learning that uses a multi-layered neural network (an information processing model that mimics the mechanism of the human nervous system).

[0039] The sensor group 711B includes, for example, a three-axis gyro sensor, a three-axis acceleration sensor, and a magnetic sensor. The three-axis gyro sensor detects the forward / backward tilt, the left / right tilt, and the rotational angular velocity to grasp the attitude and movement of the drone 71. The three-axis acceleration sensor detects the acceleration in the forward / backward, left / right, and up / down directions. The magnetic sensor detects the orientation.

[0040] The GPS receiver 711C continuously receives GPS signals in order to utilize a Global Positioning System (GPS) that can be used within the logistics facility 100. This allows the drone 71 to detect its own position information in real time using the GPS, and to fly correctly according to the flight route information based on the detection results. However, other means may be used as a means for flying the drone 71 according to the flight route information.

[0041] The ESC 712 passes a drive current to the rotor motor 713 based on a control signal input from the control unit 711A. The rotor motor 713 is provided at each tip of the frame 71A shown in FIG. 6. That is, four rotor motors 713 are provided. Each rotation shaft of the rotor motor 713 is directly connected to each of the rotors 71C to 71F. One ESC 712 drives one rotor motor 713. That is, four ESCs 712 are provided in accordance with the rotor motors 713.

[0042] Furthermore, for example, a lithium polymer secondary battery is used for the battery 718. Of the four ESCs 712, one ESC 712 has a function of extracting power from the battery 718 and supplying the power to the flight controller 711. The flight controller 711 then supplies power to the communication unit 717. In this way, the power supply unit can be omitted, and the weight of the airframe can be reduced.

[0043] By controlling the rotation speed of each of the rotors 71C to 71F driven by the rotor motor 713, the drone 71 is capable of flying actions such as ascending and descending, moving laterally in the forward / backward and left / right directions, and rotating.

[0044] 5, the management device 72 includes a control unit 721, a communication unit 722, a storage unit 723, and a network interface 724. As described above, the management device 72 is not only used for the monitoring system 70, but also controls each device in the logistics facility 100, such as the article transport device 15.

[0045] The control unit 721 comprehensively controls the management device 72. The communication unit 722 communicates with the communication unit 717 in the drone 71 by wireless signals. The storage unit 723 is configured, for example, with a large-capacity HDD device or a semiconductor memory device. The storage unit 723 can store, for example, a program executable by the control unit 721, various information sent from the drone 71, a database of the luggage M stored in the storage shelf 10, and the like. The network interface 724 is connected to a network NW such as the Internet.

[0046] The vibration sensor 73 is disposed on the storage shelf 10 or in the vicinity of the storage shelf 10 to detect vibrations. Detection information from the vibration sensor 73 is transmitted to the control unit 721. In this way, when vibrations exceeding a specified value occur due to the occurrence of an earthquake or the like, the control unit 721 can detect this situation.

[0047] 4. Abnormalities in luggage stored on storage shelves When the storage shelf 10 is shaken by an earthquake or the like, there is a possibility that an abnormality will occur in the position of the luggage stored on the storage shelf 10. Specifically, when the storage shelf 10 is shaken, there is a possibility that the luggage M stored in the storage section S of the storage shelf 10 will fall to the floor as shown in Fig. 3. When the luggage M falls, it becomes necessary to check the damage status of the luggage M and return the undamaged luggage M to its original position.

[0048] On the other hand, even if the luggage M has not yet fallen, the shaking of the storage shelf 10 may cause the luggage M to shift beyond an allowable range from the appropriate position in the storage section S (the position where the luggage M is normally stored on the storage shelf 10), making it more likely for the luggage M to fall from the storage shelf 10. When the luggage M is in such a state of about to fall (getting caught), it becomes necessary to return the luggage to its original position, etc., in order to prevent the luggage M from falling.

[0049] Therefore, the monitoring system 70 according to the present embodiment performs the operation described below, and is capable of detecting the above-mentioned state in which the luggage M has fallen from the storage shelf 10 to the floor (hereinafter referred to as the "first state") and the state in which the luggage M has been displaced beyond the allowable range in the storage shelf 10 (hereinafter referred to as the "second state") as abnormal states related to the position of the luggage M. Since the luggage M in the first state is on the floor and the luggage M in the second state protrudes outward from the side edge of the storage shelf 10, both can be confirmed by looking from above. The allowable range can be set arbitrarily depending on the importance of preventing the luggage M from falling, etc. In FIG. 2, the luggage M in the first state is designated as luggage M1, and the luggage M in the second state is designated as luggage M2, and an example of an abnormal state related to the position of the luggage M in the storage shelf 10 is shown.

[0050] 5. Operation of the monitoring system When an earthquake or the like occurs, the monitoring system 70 according to this embodiment is capable of detecting abnormal states (first state and second state) regarding the position of the luggage M. Hereinafter, the operation of such monitoring system 70 will be described with reference to the flowcharts shown in Figs. 7 to 9.

[0051] When the vibration sensor 73 detects vibrations exceeding a specified value due to the occurrence of an earthquake or the like, the operation of the flowchart in Fig. 7 is started. The timing at which the operation is started is not limited to this, and may be, for example, when a predetermined operation is performed by an administrator. When the operation is started, the first flight operation of step S10 is first performed. When the operation of the flowchart in Fig. 7 is started, the drone 71 is stopped at a predetermined takeoff and landing location.

[0052] More specifically, in step S10, the control unit 721 detects the position of the item transport device 15 and transmits a predetermined flight route that runs along the front and back sides of the storage shelf 10 and avoids the position of the item transport device 15 to the drone 71 via the communication unit 722. Then, in the drone 71, the control unit 711A stores the predetermined flight route received via the communication unit 717 in the memory unit 711A1. Note that, in cases where the flight route is fixed, information about the route may be stored in advance in the memory unit 711A1, and the transmission may be omitted.

[0053] Then, the control unit 711A drives and controls the rotor motor 713 to start the ascent of the drone 71 stopped at the takeoff and landing point. The control unit 711A then controls the drone 71 to fly along the above flight route at a predetermined altitude. The predetermined altitude is a position higher than the baggage M stored in the uppermost storage unit S of the storage shelf 10.

[0054] As an example of a flight route, a flight route FR is shown in Fig. 2. Note that the drone 71 flies at a height between the storage shelf 10 and the ceiling CE as shown by the dotted arrow in Fig. 3, but since the item transport device 15 is higher than this flight route, the flight of the drone 71 is obstructed. Therefore, the flight route FR is a route that avoids the item transport devices 15A-15C while following the front and rear sides of the storage shelves 10A-10F.

[0055] More specifically, it first enters gap R1 between storage shelves 10A and 10B, advances along the front sides of storage shelves 10A and 10B, and when it reaches a position near item transport device 15A, it retreats and exits through gap R1. It then enters gap R2 between storage shelves 10C and 10D, advances along the front sides of storage shelves 10C and 10D, and when it reaches a position near item transport device 15B, it retreats and exits through gap R2. It then enters gap R3 between storage shelves 10E and 10F, advances along the front sides of storage shelves 10E and 10F, and when it reaches a position near item transport device 15C, it retreats and exits through gap R3.

[0056] Then, it advances along the rear side of the storage shelf 10F, enters the gap R4 between the storage shelves 10F and 10E, advances along the front sides of the storage shelves 10F and 10E, advances to a position near the item transport device 15C, retreats, and exits from the gap R4. Then, it enters the gap R5 between the storage shelves 10D and 10C, advances along the front sides of the storage shelves 10D and 10C, advances to a position near the item transport device 15B, retreats, and exits from the gap R5. Then, it enters the gap R6 between the storage shelves 10B and 10A, advances along the front sides of the storage shelves 10B and 10A, advances to a position near the item transport device 15A, retreats, and exits from the gap R6. Then, it advances along the rear side of the storage shelf 10A, and the flight route ends.

[0057] In this way, when flying through each gap, the drone 71 advances to a position close to the item transport device and then retreats. Therefore, the flying object can fly through the space without colliding with the item transport device. The above flight route covers the space on the front and rear sides of multiple storage shelves 10. Note that the space does not include the area where the rear sides of the storage shelves 10 face each other or the area where the item transport device 15 is located.

[0058] While flying along such a flight route, the drone 71 continuously captures images of the floor surface side (below) with the camera 716. The images captured by the camera 716 are stored in the memory unit 711A1. The captured images thus obtained show the front and back sides of the storage shelf 10 along the flight route and the floor surface in the vicinity thereof, and if there is any baggage M in an abnormal state, the baggage M is also captured.

[0059] The control unit 711A is capable of performing object recognition of the luggage M in the first state and the second state from the captured images stored in the storage unit 711A1, thereby detecting an abnormal area where the first state and the second state have occurred (an abnormal area where an abnormality has occurred regarding the position of the luggage M). Note that, although the abnormal state may be detected by distinguishing between the first state and the second state here, in this embodiment, the first state and the second state are distinguished in the second flight operation described later.

[0060] As described above, in the first flight operation, the monitoring system 70 performs the first flight operation in which the drone 71 (aircraft) flies a predetermined route along the storage shelf 10 while photographing the downward direction, and detects an abnormal state regarding the position of the luggage M, including a state in which the luggage M has fallen from the storage shelf 10, based on information obtained by the photographing. The drone 71 detects the abnormal area as described above, and creates a distribution map of the positions where the abnormal state has occurred by displaying the positions of each of the detected abnormal areas on a map around the storage shelf 10. This distribution map is provided to the manager of the logistics facility 100 (hereinafter, simply referred to as the "manager") and can be used for future responses, etc. For example, if the distribution map identifies a place that frequently becomes an abnormal area (i.e., a place where the position of luggage is likely to be displaced), it is possible to take measures such as inspecting the storage shelf 10 near that place for abnormalities.

[0061] As shown in Fig. 7, after the above-mentioned first flight operation (step S10) is completed, the operation of step S20 is started. However, if no baggage M that has become abnormal is detected in the first flight operation, the administrator is notified of this and the operations after step S20 are omitted. Also, if the purpose of the monitoring system 70 is achieved by obtaining the above-mentioned distribution map, the operations after step S20 may be omitted.

[0062] In step S20, a second flight operation is performed. The second flight operation is performed along a route that overlaps with the flight route of the first flight operation. More specifically, the operation of the flowchart shown in FIG. 8 is performed.

[0063] The operation shown in Fig. 8 is performed for each of the routes in the gaps between the pair of front sides of the storage shelves 10 in the above-mentioned predetermined flight route, and for each route on the rear side of the storage shelves 10. In the example of Fig. 2, the operation is performed for each route in the gaps R1 to R6, and for each route on the rear side of the storage shelves 10A and 10F.

[0064] When the operation of the flowchart in FIG. 8 is started, first, in step S201, when the control unit 711A detects that the drone 71 has moved to the first abnormal area, the control unit 711A descends the drone 71 until the first distance sensor 714 detects the baggage M. Then, in step S202, the control unit 711A photographs the baggage M below by flying the drone 71 at a predetermined low speed above the detected baggage M and photographing the downward direction with the camera 716. Here, the predetermined low speed is a flight speed lower than the flight speed in the first flight operation. By flying the drone 71 at a low speed, a clearer photographed image of the baggage M and its surroundings can be obtained. The photographed image is stored in the storage unit 711A1.

[0065] In the second flight operation, the drone 71 also moves in the height direction, so that the control unit 711A can grasp the height direction position of each piece of luggage M that was found to be in an abnormal state in the first flight operation, and can easily distinguish and recognize whether the luggage M is in the first state or the second state. Instead of distinguishing between the first state and the second state in this manner, the control unit 711A may perform the distinction by object recognition of the luggage M from the captured image stored in the storage unit 711A1. Since the image of the luggage M captured in the second flight operation is clearer than that captured in the first flight operation, the distinction can be made more accurately.

[0066] After step S202, in step S203, the control unit 711A determines whether there is a next abnormal area. If there is a next abnormal area (Y in step S203), the process proceeds to step S204, where the control unit 711A determines whether the drone 71 has moved to the next abnormal area. If the drone 71 has not moved to the next abnormal area, that is, if it is located in a normal area (N in step S204), the process proceeds to step S205, where the control unit 711A flies the drone 71 at a predetermined high speed. Here, the predetermined high speed is a speed faster than the flight speed at the low speed in step S202.

[0067] Until the drone 71 moves to the next abnormal area, it performs a flying operation at a high speed in step S205. Then, when the drone 71 moves to the next abnormal area (Y in step S204), the process proceeds to step S206, where the control unit 711A determines whether the baggage M has been detected based on the distance measurement result by the second distance sensor 715. When the baggage M has been detected (Y in step S206), the process proceeds to step S202, where an image is captured by flying at a low speed.

[0068] On the other hand, if the luggage M is not detected (N in step S206), the process proceeds to step S207, where the control unit 711A determines whether the luggage M photographed immediately before is in the first state. If it is in the first state (Y in step S207), the control unit 711A determines that the luggage M in the second state is present above the current position, and raises the drone 71 until the luggage M is detected by the distance measurement result by the first distance sensor 714. After raising the drone 71, the process proceeds to step S202, where the control unit 711A photographs the luggage M (photographs below) while flying the drone 71 over the luggage M at a low speed.

[0069] That is, after descending to a position near the baggage M in the first state, the drone 71 ascends and detects the baggage M in the second state when it advances to an abnormal area where the baggage M in the first state is not detected. In other words, the drone 71 does not ascend until it reaches the abnormal area where the baggage M in the first state is not detected, and continues flying at the descended height. Note that the abnormal area (the area where either the first state or the second state has occurred) has already been detected by the first flight operation, and this detection result shows that the abnormal area where the baggage M in the first state is not detected is the area where the second state has occurred. Therefore, if the drone 71 ascends in the abnormal area where the baggage M in the first state is not detected, it can detect the baggage M in the second state. This makes it possible to reduce the movement distance of the drone 71 compared to, for example, ascending once to the altitude in the first flight operation and then descending to detect the baggage M in the second state.

[0070] Furthermore, if the luggage M photographed immediately before in step S207 is not in the first state (N in step S207), the control unit 711A, while descending the drone 71, determines whether the luggage M is detected by the second distance sensor 715 before the drone 71 descends to a predetermined minimum position corresponding to the luggage M in the first state. If the luggage M is detected (Y in step S208), the process proceeds to step S202, where the control unit 711A photographs the luggage M (photographs the lower side) while flying the drone 71 at a low speed above the luggage M. In this case, the detected luggage M may be in either the first state or the second state.

[0071] On the other hand, if the second distance sensor 715 does not detect the luggage M before the drone 71 descends to the predetermined lowest position (N in step S208), the process proceeds to step S209, where the control unit 711A determines that the luggage M in the second state is located above, and raises the drone 71 until the luggage M is detected by the distance measurement result by the first distance sensor 714. After raising the drone 71, the process proceeds to step S202, where the control unit 711A photographs the luggage M (photographs the lower side) while flying the drone 71 over the luggage M at a low speed.

[0072] Also, if there is no next abnormal area in step S203 (N in step S203), the process proceeds to step S211, where the control unit 711A causes the drone 71 to rise to a predetermined altitude in the first flight operation. Then, the operation of the flowchart shown in Fig. 8 is completed. For example, since there is no next abnormal area after the abnormal area in which the luggage M1 is located in the gap R1 in Fig. 2, the drone 71 rises to a predetermined altitude after photographing the luggage M1, and then retreats to go outside through the gap R1.

[0073] In addition, for gaps in which no abnormal area was detected during the first flight operation (for example, gaps R3, R5, and R6 in FIG. 2), it has already been confirmed that there is no abnormality, so the drone 71 may not enter the gap during the second flight operation. As described above, in the second flight operation, the drone 71 descends to a position near the baggage M so that the baggage M in the abnormal area can be photographed more clearly. However, instead of the drone 71 descending to a position near the baggage M, the camera 716 provided on the drone 71 may zoom in and photograph the lower part of the abnormal area with a higher resolution than the first flight operation. In this way, it is possible to photograph the baggage M in the abnormal area more clearly without descending the drone 71.

[0074] As shown in Fig. 7, after the second flight operation in step S20, the process proceeds to step S30, where the third flight operation is performed. The third flight operation is performed along a route that overlaps with the flight route of the first flight operation. More specifically, the operation of the flowchart shown in Fig. 9 is performed here.

[0075] The operation shown in Fig. 9 is performed for each of the routes in the gaps between the pair of front sides of the storage shelves 10 in the above-mentioned predetermined flight route, and for each route on the rear side of the storage shelves 10. In the example of Fig. 2, the operation is performed for each route in the gaps R1 to R6, and for each route on the rear side of the storage shelves 10A and 10F.

[0076] Here, based on the image of the luggage M captured in step S202 of the second flight operation, the control unit 711A recognizes in advance the abnormal area of ​​the first state (the area where the first state has occurred) among the abnormal areas detected in the first flight operation. When the operation of the flowchart in FIG. 9 is started, first, in step S301, the control unit 711A causes the drone 71 to descend in the initial abnormal area of ​​the first state while capturing an image of the front side of the storage shelf 10 (capturing an image of the side) with the camera 716. This captures an image of the storage state of the luggage M on the storage shelf 10 in the abnormal area of ​​the first state. The captured image is stored in the memory unit 711A1.

[0077] Thereafter, in step S302, the control unit 711A determines whether there is a next abnormal area of ​​the first state, and if there is (N in step S302), proceeds to step S303. In step S303, the control unit 711A photographs the front side of the storage shelf 10 (photographs the side) with the camera 716 while raising the drone 71 to a predetermined altitude for the first flight operation in the next abnormal area of ​​the first state. This allows an image of the storage state of the luggage M on the storage shelf 10 in the abnormal area of ​​the first state to be captured. The captured image is stored in the memory unit 711A1.

[0078] Thereafter, in step S304, control unit 711A determines whether there is a next abnormal area of ​​the first state, and if there is (N in step S304), proceeds to step S301.

[0079] Also, in step S302, if there is no next abnormal area in the first state (Y in step S302), the process proceeds to step 305, where the control unit 711A raises the drone 71 to a predetermined altitude in the first flight operation. Then, the operation of the flowchart shown in Fig. 9 is completed. Also, in step S304, if there is no next abnormal area in the first state (Y in step S304), the operation shown in Fig. 9 is completed as it is.

[0080] In this way, in the third flight operation, when the drone 71 reaches the abnormal area in the first state, it moves up and down while photographing the storage shelf 10 from the side. By photographing the front side of the storage shelf 10 in the abnormal area in the first state, it becomes possible to identify from which storage section S of the storage shelf 10 the baggage M in the first state has fallen.

[0081] In addition, with regard to gaps in which no abnormal area of ​​the first state was detected during the second flight operation (e.g., gaps R1 to R3, R5, and R6 in Figure 2), it has already been confirmed that no abnormality of the first state has occurred, so it is possible to prevent drone 71 from entering those gaps during the third flight operation.

[0082] 6. Example of flight motion Next, an example of the above-mentioned first to third flight operations will be described with reference to Fig. 10 to Fig. 12. Fig. 10 to Fig. 12 show an example when the drone 71 flies a route on the front side or rear side of the storage shelf 10. In the example of Fig. 10 to Fig. 12, luggage M11 and luggage M12 in a first state that have fallen to the floor surface are shown, and luggage M21 and luggage M22 in a second state that have shifted position on the storage shelf 10 are shown.

[0083] First, as shown in Fig. 10, the drone 71 performs a first flight operation at a predetermined altitude along a flight route FR1 (step S10 in Fig. 7). When the first flight operation is completed, an abnormal area where either the first state or the second state occurs is detected. After that, the drone 71 performs a second flight operation along a flight route FR2 shown in Fig. 11 (step S20 in Fig. 7).

[0084] Specifically, the drone 71 descends to the vicinity of the baggage M11 (step S201 in FIG. 8), and then photographs the three bags M21 (steps S202 to S206 in FIG. 8). After that, the drone 71 moves to the next abnormal area (Y in step S204 in FIG. 8), but the baggage M is not detected here (N in step S206 in FIG. 8), and the baggage M photographed immediately before was in the first state (Y in step S207 in FIG. 8), so the drone 71 rises and detects the baggage M21 in the second state (step S210 in FIG. 8). This causes the drone 71 to photograph the baggage M21 (step S202 in FIG. 8).

[0085] After that, the drone 71 moves to the abnormal area where the luggage M22 is located, but the luggage is not detected at the current height position (N in step S206 in FIG. 8), and the luggage M21 photographed immediately before is not in the first state (N in step S207 in FIG. 8). Therefore, the drone 71 descends to detect the luggage M22 (Y in step S208 in FIG. 8), and photographs the two luggage M22 (steps S202 to S206 in FIG. 8).

[0086] After that, the drone 71 moves to the abnormal area where the luggage M12 is located, but the luggage is not detected at the current height position (N in step S206 in FIG. 8), and the luggage M22 photographed immediately before is not in the first state (N in step S207 in FIG. 8). Therefore, the drone 71 descends to detect the luggage M12 (Y in step S208 in FIG. 8), and photographs four pieces of luggage M12 (steps S202 to S206 in FIG. 8). As described above, the second flight operation proceeds, and if there is no next abnormal area (N in step S203 in FIG. 8), the drone 71 ascends to a predetermined altitude (step S211 in FIG. 8).

[0087] After the second flight operation, the drone 71 performs a third flight operation on the flight route FR3 shown in FIG. 12 (step S30 in FIG. 7). Specifically, the drone 71 photographs the storage shelf 10 with vertical movement at the positions of three pieces of first-state luggage M11 (steps S301 to S304 in FIG. 9), and then photographs the storage shelf 10 with vertical movement at the positions of four pieces of first-state luggage M12 (steps S301 to S304 in FIG. 9). As described above, if the second flight operation proceeds and there is no next abnormal area in the first state (Y in step S302 in FIG. 9), the drone 71 ascends to a predetermined altitude (step S305 in FIG. 9).

[0088] 7. Notification of various information and changes to notification destinations The monitoring system 70 is adapted to appropriately notify the manager of various detection results, captured images, and other useful information obtained during each of the above-mentioned flight operations. This makes it easy for the manager to grasp the current situation and take appropriate measures even if an abnormality occurs in the luggage M due to an earthquake or the like.

[0089] The monitoring system 70 may identify the type of luggage M in an abnormal state and change the notification destination according to the result of the identification. Changing the notification destination is a concept that includes cases where notification destinations are added and cases where some notification destinations are excluded. An example of changing the notification destination will be specifically described below.

[0090] In this example, in the second flight operation described above, when photographing the luggage M in step S202, identification information (e.g., description of contents or barcode, etc.) attached to the luggage M is photographed. Then, the photographed image of the identification information linked to information on the abnormal state of the luggage M (a first state and a second state are distinguished) is transmitted from the control unit 711A of the drone 71 to the control unit 721 of the management device 72.

[0091] The storage unit 723 stores in advance a database in which identification information and luggage type information (information indicating the type of luggage M) for each luggage M are stored. The control unit 721 recognizes the identification information from a captured image of the above-mentioned identification information, and searches the above-mentioned database for luggage type information corresponding to the identification information. If the condition that "the searched luggage type information indicates a specific type (e.g., glass products that are relatively easily broken by impact) and the luggage M is in the first state (a state in which the luggage M has fallen to the floor)" is satisfied, the control unit 721 not only notifies the manager of the occurrence of an abnormal state, but also notifies a predetermined cleaning company. Note that if the condition is not satisfied, the control unit 721 notifies the manager of the occurrence of an abnormal state, but does not notify the cleaning company.

[0092] In this way, when there is a high possibility that the luggage M has been damaged by falling, it is possible to quickly request a cleaning company to clean up the damaged luggage M. Note that the identification of the type of luggage M (identification of whether or not it is relatively easy to be damaged by impact) is not limited to the above-mentioned method, and may be performed by object recognition from a photographed image showing the luggage M. Also, whether or not the luggage M is damaged may be directly identified by object recognition from a photographed image showing the luggage M.

[0093] 8.Other As described above, the monitoring system 70 in this embodiment includes a drone 71 having an imaging means, and is used in a logistics facility 100 in which luggage M is placed on a storage shelf 10. In the monitoring system 70, the drone 71 flies a predetermined flight route along the storage shelf 10 while performing a first flight operation of imaging downward, and detects abnormal conditions related to the position of the luggage M, including a second state in which the luggage M has fallen from the storage shelf 10, based on information obtained by the imaging. In this way, the monitoring system 70 is able to improve the detection function for abnormal conditions of luggage by detecting the first state in which the luggage has fallen from the storage shelf 10.

[0094] (2) Second embodiment Next, a second embodiment of the present invention will be described. The second embodiment is basically the same as the first embodiment, except for detecting a third state as an abnormal state related to the orientation of the luggage M, and points related thereto. In the following description, the emphasis will be placed on points that differ from the first embodiment, and a description of points in common with the first embodiment may be omitted.

[0095] When an earthquake or the like occurs and the storage shelf 10 is shaken, the orientation of the luggage M placed on the storage shelf 10 may be shifted. Here, FIG. 13(a) shows a state in which the orientation of the luggage M is normal (initial state immediately after being placed on the storage shelf 10), and FIG. 13(b) shows a state in which the orientation of the luggage M is shifted by an angle θ from the state of FIG. 13(a), as viewed from above. As shown in FIG. 13(a), if the orientation of the luggage M is normal, it is possible to insert the slide fork of the transfer device 158 into the gap space Dc of the support body D and lift the luggage M appropriately. However, if the angle θ shown in FIG. 13 becomes large (i.e., if the deviation in the orientation of the luggage M becomes large), it may become difficult to insert the slide fork into the gap space Dc or it may become impossible to lift the luggage M appropriately, which may cause problems in taking the luggage M in and out of the storage shelf 10. Therefore, it is important to detect such an abnormality and deal with it appropriately.

[0096] Therefore, the monitoring system 70 according to the second embodiment is adapted to detect not only the first state (a state in which the luggage has fallen from the storage shelf 10 to the floor) and the second state (a state in which a positional deviation that exceeds the allowable range of the storage shelf 10 has occurred), but also a direction deviation that exceeds a predetermined allowable range of the storage shelf 10 (hereinafter referred to as the "third state") as an abnormal state related to the position or direction of the luggage M. Note that the allowable range (corresponding to the threshold value of the angle θ shown in FIG. 13) can be set arbitrarily, but in this embodiment, it is set to 15° as an example.

[0097] In the second embodiment, the third state is detected by using the side photographing (photographing the front side of the storage shelf 10) by the camera 716 in the third flight operation (step S30) described above. However, in the first embodiment, the photographing is performed only in the abnormal area of ​​the first state, but in the second embodiment, the photographing is performed in all areas of the flight route. That is, while flying along the flight route described above, the drone 17 photographs the front side of the storage shelf 10 while descending at the first vertical row position of the storage shelf 10, photographs the front side of the storage shelf 10 while ascending at the next row position, photographs the front side of the storage shelf 10 while descending at the next row position, and so on, photographing the front side of the storage shelf 10 for all rows while moving in a serpentine manner in a side view. This makes it possible to photograph the state of the luggage M as seen from the front side of the storage shelf 10 for all positions of the storage shelf 10 (all positions where the luggage M can be placed).

[0098] The monitoring system 70 executes a process for detecting the third state for all positions on the storage shelf 10 using the captured image obtained in this manner (hereinafter, for convenience, referred to as "captured image β"). More specifically, for all positions on the storage shelf 10, the monitoring system 70 recognizes the difference (angle θ1) between the orientation of the luggage M in the captured image β and the orientation of the luggage M in the luggage image α already described. This detection is performed, for example, by performing object recognition of the luggage M for each of the captured image β and the luggage image α, and using the results of these object recognitions. Note that the luggage M in the normal orientation is shown in the luggage image α, and the luggage M in its current state is shown in the captured image β, so the recognized angle θ1 represents the magnitude of deviation in the orientation of the luggage M.

[0099] Therefore, if there is any luggage M whose angle θ1 exceeds 15° (the above-mentioned allowable range), the monitoring system 70 recognizes it as being in the third state, and is able to detect the occurrence of the third abnormal state and its location. The detection results and information on the third state, such as the captured images, are appropriately notified to the manager along with information on the first and second states. This allows the manager to easily take appropriate measures, such as correcting the orientation of the luggage M that has fallen into the third state.

[0100] (3) Summary In addition to the above-described embodiment, the configuration of the present invention can be modified in various ways without departing from the spirit of the invention. In other words, the above-described embodiment should be considered to be illustrative in all respects and not restrictive, and the technical scope of the present invention is indicated by the claims, not the description of the above-described embodiment, and should be understood to include all modifications that fall within the meaning and scope of the claims. [Explanation of symbols]

[0101] 100 Logistics equipment 10, 10A~10F Storage shelves 15, 15A~15C Item transport device 16 Running rail 20 Incoming and outgoing conveyor 30 Running rail 35 Transport cart 70 Surveillance System 71 Drone 71A Frame 71B Central part 71C~71F rotor 711 Flight Controller 711A Control Unit 711A1 Storage section 711B Sensor Group 711C GPS receiver 712 ESC 713 Rotor Motor 714 1st Distance Sensor 715 Second Distance Sensor 716 Cameras 717 Communications Department 718 Battery 72 Management device 721 Control Unit 722 Communications Department 723 Storage section 724 Network Interface 73 Vibration Sensor M Luggage D Support S Storage section

Claims

1. A flight system for use in a facility where baggage is placed on a storage shelf, the flight system including an aircraft having a photographing means, The flying object performs a flight operation to photograph the baggage while flying along a predetermined flight route, Object recognition of the luggage is performed for each of the captured image obtained by photographing during the flight operation and a luggage image which is an image of the luggage acquired in advance, The flying object photographs a downward direction while flying along the flight route, Based on the information obtained by the photographing, an abnormal state regarding the position or orientation of the luggage, including a state in which the luggage has fallen from the storage shelf, is detected; A flight system characterized by creating a distribution map of the locations where the abnormal conditions occurred based on the results of the execution of the abnormal condition detection process.

2. The flight system described in Claim 1, characterized in that the luggage image is acquired as an image showing the luggage in a normal loaded state.

3. A flight system as described in claim 1 or claim 2, characterized in that the luggage image is an image taken when the luggage is placed on the storage shelf.

4. A flight system described in any of claims 1 to 3, characterized in that, based on the results of the object recognition, the difference between the loading state of the luggage in the captured image and the loading state of the luggage in the luggage image is recognized.

5. A flight system as described in Claim 4, characterized in that the difference is the difference between the orientation of the luggage in the captured image and the orientation of the luggage in the luggage image.

6. A flight system as described in claim 4 or claim 5, characterized in that when the difference exceeds a predetermined value, the detection result is reported to a predetermined reporting destination.

Images