Monitoring system

Abstract

To provide a monitoring system capable of properly monitoring a cargo handling vehicle while suppressing an increase in cost.SOLUTION: A monitoring system includes: a forklift 2 that is a cargo handling vehicle which a cargo handling worker can board and the cargo handling worker steers to perform cargo handling work; a drone 3 that is an unmanned flying object which can autonomously fly and has a camera 4 attached; and an action determination unit 13 that is an imaging place determination unit which determines an imaging place by performing reinforcement learning on the basis of predetermined input data. The action determination unit 13 receives a reward when the forklift 2 is imaged by the camera 4, and determines an imaging place where the reward is the greatest. The drone 3 moves to the imaging place determined by the action determination unit 13, and uses the camera 4 to image the forklift 2 in the imaging place.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a monitoring system capable of monitoring a cargo handling vehicle.

Background Art

[0002] Generally, in a working environment such as a warehouse, it is known to provide cameras at a number of locations for the purpose of, for example, checking events that occur during cargo handling work (see, for example, Patent Document 1).

[0003] Patent Document 1 describes arranging a camera that photographs the vicinity of a forklift, which is a cargo handling vehicle, at a predetermined location such as near the ceiling or at a corner of a travel path, and arranging it on the forklift, for the purpose of determining the possibility of a collision between the forklift and a person.

[0004] However, when cameras are arranged at predetermined locations, the larger the working environment, the more cameras are required to appropriately monitor the cargo handling vehicle, resulting in a problem of increased costs. Also, when cameras are arranged on the cargo handling vehicle, the number of cameras required is proportional to the number of cargo handling vehicles, resulting in a problem of increased costs.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The present invention has been made in view of the above circumstances, and an object thereof is to provide a monitoring system capable of appropriately monitoring a cargo handling vehicle while suppressing an increase in cost.

Means for Solving the Problems

[0007] In order to solve the above problems, the monitoring system of the present invention includes a cargo handling vehicle that can be boarded by a cargo handling worker and performs cargo handling operations by being operated by the cargo handling worker, an unmanned aircraft that can fly autonomously and has a camera attached thereto, and a shooting location determination unit that determines a shooting location by performing reinforcement learning based on predetermined input data. The shooting location determination unit determines the shooting location where the reward is maximized on the assumption that a reward is obtained when the cargo handling vehicle is photographed by the camera. The unmanned aircraft moves to the shooting location determined by the shooting location determination unit and photographs the cargo handling vehicle using the camera at the shooting location.

[0008] Alternatively, the monitoring system of the present invention includes a cargo handling vehicle that can be boarded by a cargo handling worker and performs cargo handling operations by being operated by the cargo handling worker, an unmanned aircraft that can fly autonomously and has a camera attached thereto, a traffic volume prediction unit that predicts the traffic volume of the cargo handling vehicle based on predetermined input data and a traffic volume prediction model generated by machine learning, and a shooting location determination unit that determines a shooting location based on the prediction result of the traffic volume predicted by the traffic volume prediction unit. The unmanned aircraft moves to the shooting location determined by the shooting location determination unit and photographs the cargo handling vehicle using the camera at the shooting location.

[0009] Further, it is preferable that the input data includes one or more types of information among environmental information related to the environment in which the cargo handling operation is performed, cargo information related to the cargo handled by the cargo handling vehicle, vehicle information related to the cargo handling vehicle, and worker information related to the cargo handling worker.

[0010] Further, it is preferable that the environmental information includes one or more types of information among information indicating the width of the passage on which the cargo handling vehicle travels and information indicating the number of steps of the rack where the cargo handling vehicle loads and unloads cargo.

[0011] Further, it is preferable that the cargo information includes one or more types of information among information indicating the shape of the cargo handled by the cargo handling vehicle and information indicating the size of the cargo handled by the cargo handling vehicle.

[0012] In addition, the vehicle information preferably includes one or more types of information among information indicating the type of the loading vehicle, information indicating the size of the loading vehicle, and information indicating the position where the loading worker boards the loading vehicle.

[0013] In addition, the worker information preferably includes one or more types of information among information indicating the height of the loading worker and information indicating the facial features of the loading worker.

Advantages of the Invention

[0014] According to the present invention, it is possible to provide a monitoring system capable of appropriately monitoring a loading vehicle while suppressing an increase in cost.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0016] With reference to the drawings, a monitoring system according to an embodiment of the present invention will be described. As shown in FIG. 1, the monitoring system according to the present embodiment includes a management device 1, a plurality of forklifts 2, and a plurality of drones 3, and monitors the forklifts 2 with the drones 3.

[0017] The management device 1 is a management server configured to be communicable with the forklift 2 and the drone 3, and manages various types of information. The management device 1 instructs the forklift 2 to perform a handling operation by transmitting handling operation instruction information, which will be described later. Further, the management device 1 instructs the drone 3 to photograph the forklift 2 by transmitting photographing instruction information, which will be described later.

[0018] The forklift 2 is a manned vehicle on which a handling worker M can board, and is a handling vehicle that performs a handling operation when the handling worker M operates it. The handling operation includes the warehousing operation and the shipping operation of the cargo N. The warehousing operation includes the operation of picking up the cargo N from the warehousing station S, the operation of transporting the cargo N to the vicinity of the rack R, and the operation of placing the cargo N on the rack R. Further, the shipping operation includes the operation of picking up the cargo N from the rack R, the operation of transporting the cargo N to the vicinity of the shipping station T, and the operation of placing the cargo N on the shipping station T.

[0019] The drone 3 is an unmanned flying body capable of autonomous flight with a camera 4 attached thereto, and is a multicopter with excellent attitude stability. The drone 3 photographs the forklift 2 using the camera 4 and transmits the data of the photographed video to the management device 1.

[0020] The camera 4 includes an image sensor (not shown) that images a scene within the range of the photographing angle of view as a video, and photographs the forklift 2 when the forklift 2 passes through the range of the photographing angle of view.

[0021] Figure 2 shows the schematic configuration of the management device 1, the forklift 2, and the drone 3. As shown in Figure 2, the management device 1 includes a storage unit 11, a handling operation instruction unit 12, an action determination unit 13, and a photographing instruction unit 14.

[0022] The storage unit 11 stores various kinds of information and data. Specifically, the storage unit 11 stores cargo handling work information related to the cargo handling work, environment information related to the environment where the cargo handling work is performed, cargo information related to the cargo N handled by the forklift 2, vehicle information related to the forklift 2, and worker information related to the cargo handler M, etc.

[0023] The cargo handling work information includes information indicating an identifier of the cargo handling work, information indicating the cargo N that is the object of the cargo handling work, information indicating the source of the cargo N (i.e., the incoming station S or the rack R), and information indicating the destination of the cargo N (i.e., the rack R or the outgoing station T), etc.

[0024] The environment information includes information indicating the area where the travel of the forklift 2 is permitted, information indicating the width of the passageway on which the forklift 2 travels, information indicating the position and number of levels of the rack R where the forklift 2 loads and unloads the cargo N, information indicating the area where the flight of the drone 3 is permitted, and information indicating the height at which the flight of the drone 3 is permitted, etc.

[0025] The cargo information includes information indicating an identifier of the cargo N, information indicating the contents of the cargo N, information indicating the source and destination of the cargo N, information indicating the shape of the cargo N, and information indicating the size of the cargo N, etc.

[0026] The vehicle information includes information indicating an identifier of the forklift 2, information indicating the type of the forklift 2, information indicating the size of the forklift 2, and information indicating the position where the cargo handler M boards the forklift 2, etc.

[0027] The worker information includes information indicating an identifier of the cargo handler M, information indicating the working hours of the cargo handler M, information indicating the height of the cargo handler M, and information indicating the facial features of the cargo handler M, etc.

[0028] The handling work instruction unit 12 instructs the forklift 2 and the handling worker M to perform handling work by transmitting handling work instruction information to the forklift 2. The handling work instruction information includes information related to the handling work that the forklift 2 which receives the handling work instruction information should perform, among the handling work information stored in advance in the storage unit 11.

[0029] The action determination unit 13 is a shooting location determination unit that determines the shooting location by performing reinforcement learning based on predetermined input data. The action determination unit 13 determines the action of the drone 3 by determining not only the shooting location but also the shooting direction and the shooting time. The action determination unit 13 acquires various information and data stored in advance in the storage unit 11 as input data.

[0030] The action determination unit 13 determines the shooting location, the shooting direction, and the shooting time at which the reward is maximized, assuming that a reward is obtained when the forklift 2 is photographed by the camera 4, and that the reward is reduced (i.e., a negative reward (penalty) is obtained) when the distance between the drones 3 is short. The shooting location is the position of the camera 4 in three-dimensional space, indicating the position of the drone 3 in the horizontal plane and the position of the drone 3 in the vertical direction. The shooting direction is the orientation of the camera 4 in the horizontal plane, indicating the angle (so-called yaw angle) around the yaw axis (axis parallel to the vertical direction) of the drone 3. The shooting time indicates the start time and the end time of the operation of the camera 4 at the shooting location.

[0031] Note that the conditions for obtaining a reward and the conditions for reducing the reward are not limited to the above conditions only, and may be added as appropriate. For example, it may be assumed that an additional reward is obtained when the face of the handling worker M riding on the forklift 2 is photographed by the camera 4. That is, it may be assumed that the reward is reduced when the face of the handling worker M is not photographed by the camera 4.

[0032] By determining the shooting location, the shooting direction, and the shooting time so that the reward is maximized (i.e., to avoid a decrease in the reward), for example, it is expected that the drone 3 takes the following actions (A) to (C). (A) Move the forklift 2 to a shooting location with a high traffic volume of forklifts 2 so that many forklifts 2 are photographed by one camera 4. (B) Move to a shooting location that does not interfere with the loading and unloading operations of the forklift 2 so that the forklift 2 and the drone 3 do not collide. (C) Move to a shooting location where the face of the loading and unloading worker M on the forklift 2 is not blocked by the load N so that the face of the loading and unloading worker M is photographed.

[0033] The shooting instruction unit 14 instructs the drone 3 to shoot by transmitting shooting instruction information to the drone 3. The shooting instruction information includes information related to the shooting location, shooting direction, and shooting time determined by the action determination unit 13.

[0034] The forklift 2 is provided with a display unit 21. The display unit 21 is composed of a display that presents information to the loading and unloading worker M, and displays the loading and unloading work instruction information received from the management device 1. The loading and unloading worker M operates the forklift 2 with reference to the loading and unloading work instruction information displayed on the display unit 21.

[0035] The drone 3 includes, in addition to the camera 4, a rotary wing 31, a position information acquisition unit 32, a flight control unit 33, a shooting control unit 34, and a video transmission unit 35. A plurality of rotary wings 31 are provided at intervals on concentric circles, and generate lift for the drone 3 to fly. The flight speed, flight direction, flight height, and attitude of the drone 3 are configured to be changeable by controlling the plurality of rotary wings 31.

[0036] The position information acquisition unit 32 acquires the position information of the drone 3 for autonomous flight. The position information acquisition unit 32 is composed of, for example, a laser sensor that acquires data for SLAM (Simultaneous Localization and Mapping) by LiDAR (Light Detection and Ranging), and an ultrasonic sensor for altitude detection.

[0037] The flight control unit 33 controls the flight of the drone 3 based on the shooting instruction information and the like received from the management device 1. Specifically, the flight control unit 33 controls the rotary wings 31 so that the drone 3 flies to the shooting location based on the information related to the shooting location included in the shooting instruction information and the position information acquired by the position information acquisition unit 32.

[0038] In addition, the flight control unit 33 controls the rotary wings 31 so that the camera 4 faces the shooting direction at the shooting location based on the information related to the shooting direction included in the shooting instruction information and the information on the yaw angle of the drone 3 acquired by a azimuth sensor (not shown). Further, the flight control unit 33 controls the rotary wings 31 so that the drone 3 flies at the shooting location for at least the shooting time based on the information related to the shooting time included in the shooting instruction information and the time information output by a real-time clock (not shown).

[0039] The shooting control unit 34 controls the camera 4 based on the shooting instruction information and the like received from the management device 1. Specifically, the shooting control unit 34 controls the operation of the camera 4 based on the information related to the shooting time included in the shooting instruction information and the time information output by a real-time clock (not shown).

[0040] The video transmission unit 35 transmits the video captured by the camera 4. Specifically, the video transmission unit 35 wirelessly transmits the data of the video captured by the camera 4 to the management device 1. Thus, the data of the video transmitted by the drone 3 and received by the management device 1 is stored in the storage unit 11.

[0041] Referring to Fig. 3, the process when the drone 3 monitors the forklift 2 will be described. It is assumed that the storage unit 11 stores various types of information described above in advance.

[0042] First, the action determination unit 13 acquires information from the storage unit 11 (step S1), and determines the shooting location, shooting direction, and shooting time that each of the plurality of cameras 4 is responsible for based on the input data, which is the acquired information (step S2).

[0043] Next, the shooting instruction unit 14 transmits shooting instruction information including the shooting location, shooting direction, and shooting time determined in step S2 to each drone 3, thereby instructing the shooting of the forklift 2 (step S3). In this way, the shooting instruction information transmitted from the management device 1 is received by each drone 3.

[0044] Next, the flight control unit 33 controls the flight of the drone 3 based on the received shooting instruction information, so that the drone 3 moves to the shooting location determined in step S2 (step S4). At the shooting location, the flight control unit 33 controls the flight of the drone 3 based on the received shooting instruction information, so that the camera 4 faces the shooting direction determined in step S2.

[0045] Then, the shooting control unit 34 operates the camera 4 based on the received shooting instruction information, so that the forklift 2 existing in the shooting direction at the shooting time determined in step S2 is shot by the camera 4 (step S5). In this way, the plurality of drones 3 shoot the forklift 2 with the camera 4 according to the shooting location, shooting direction, and shooting time assigned by the management device 1.

[0046] The following effects can be obtained in this embodiment. (1) The action determination unit 13 (shooting location determination unit) determines the shooting location where the reward is maximized on the assumption that a reward is obtained when the forklift 2 (cargo handling vehicle) is shot by the camera 4. The drone 3 moves to the shooting location determined by the action determination unit 13 and shoots the forklift 2 using the camera 4 at the shooting location. According to this configuration, it is possible to appropriately monitor the forklift 2 while suppressing an increase in cost as compared with a configuration in which a large number of cameras are arranged at predetermined locations (that is, a configuration in which the shooting location cannot be changed). In addition, as compared with a configuration in which a camera is arranged on the forklift 2, an increase in cost proportional to the number of forklifts 2 can be suppressed.

[0047] (2) Since the input data includes environmental information, luggage information, vehicle information, and operator information, it is expected that the action determination unit 13 can determine the optimal shooting location for monitoring the forklift 2 compared to the case where any one of these types of information is missing.

[0048] (3) The environmental information includes information indicating the width of the passage where the forklift 2 travels and information indicating the number of levels of the rack R where the forklift 2 loads and unloads the luggage N. Therefore, based on this information, it is possible for the action determination unit 13 to determine, as the shooting location, a position where the drone 3 does not interfere with the cargo handling operation of the forklift 2.

[0049] (4) The luggage information includes information indicating the shape of the luggage N handled by the forklift 2 and information indicating the size of the luggage N handled by the forklift 2. Therefore, based on this information, it is possible for the action determination unit 13 to determine, as the shooting location, a position where the shooting of the forklift 2 is not obstructed by the luggage N.

[0050] (5) The vehicle information includes information indicating the type of the forklift 2, information indicating the size of the forklift 2, and information indicating the position where the cargo handling operator M boards the forklift 2. Therefore, based on this information, it is possible for the action determination unit 13 to determine, as the shooting location, a position where the entire forklift 2 or the cargo handling operator M is photographed.

[0051] (6) The operator information includes information indicating the height of the cargo handling operator M. Therefore, based on this information, it is possible for the action determination unit 13 to determine, as the shooting location, a position where the face of the cargo handling operator M is photographed.

[0052] The present invention is not limited to the above-described embodiments, and the above configuration can also be changed. For example, it can be implemented by making the following changes, or the following changes can be combined and implemented.

[0053] ·(First Modification Example) The system may be configured to identify the cargo handling worker M by face recognition using the camera 4. Hereinafter, the configuration of this modification example will be described with reference to FIG. 4 showing the schematic configurations of the management device 1, the forklift 2, and the drone 3. Note that the description of the same configurations as those in the above embodiment will be omitted.

[0054] In this modification example, the shooting instruction unit 14 is configured to transmit, to the drone 3 together with the shooting instruction information, information indicating the characteristics of the face of the cargo handling worker M included in the worker information stored in the storage unit 11. The drone 3 receives the information indicating the characteristics of the face of the cargo handling worker M transmitted from the management device 1.

[0055] As shown in FIG. 4, the drone 3 further includes a worker identification unit 36. Based on the information indicating the characteristics of the face of the cargo handling worker M, the worker identification unit 36 extracts the face of the cargo handling worker M from the video captured by the camera 4, and identifies and specifies the cargo handling worker M riding on the forklift 2. In addition, the worker identification unit 36 transmits the identification result (specification result) of the cargo handling worker M to the management device 1.

[0056] The following effects can be obtained in this modification example. (7) Based on the information indicating the characteristics of the face of the cargo handling worker M and the video of the cargo handling worker M included in the video captured by the camera 4, the cargo handling worker M can be specified by face recognition.

[0057] ·(Second Modification Example) The shooting location may be determined by machine learning other than reinforcement learning. The configuration of this modification example will be described with reference to FIG. 5 showing the schematic configurations of the management device 1, the forklift 2, and the drone 3. Note that the description of the same configurations as those in the above embodiment will be omitted.

[0058] In this modification example, the storage unit 11 stores the cargo handling work information related to the cargo handling work performed in the past and the traffic volume information related to the past traffic volume of the forklift 2 measured for each predetermined area. In addition, the storage unit 11 stores a traffic volume prediction model, which will be described later.

[0059] As shown in FIG. 5, the management device 1 includes a learning model generation unit 15 and a traffic volume prediction unit 16. The learning model generation unit 15 performs machine learning on the past cargo handling work information and traffic volume information stored in the storage unit 11 to generate a traffic volume prediction model. The traffic volume prediction model is a learning model for predicting the traffic volume from the cargo handling work information. In this way, the traffic volume prediction model generated by machine learning is stored in the storage unit 11.

[0060] The traffic volume prediction unit 16 predicts the traffic volume of the forklift 2 based on the predetermined input data and the traffic volume prediction model generated by machine learning. The traffic volume prediction unit 16 acquires the cargo handling work information related to the cargo handling work stored in advance in the storage unit 11 as input data.

[0061] The action determination unit 13 is a shooting location determination unit that determines the shooting location based on the predicted result of the traffic volume predicted by the traffic volume prediction unit 16. The action determination unit 13 determines a location where the traffic volume is predicted to be large or the vicinity of a location where the traffic volume is predicted to be large as the shooting location. In addition, the action determination unit 13 determines the shooting direction and shooting time at which the forklift 2 can be photographed by the camera 4 at the shooting location.

[0062] The drone 3 of this modification also moves to the shooting location determined by the action determination unit 13 and photographs the forklift 2 using the camera 4 at the shooting location, so that the effect described in (1) above can be obtained.

[0063] · When the direction of the camera 4 can be changed without changing the attitude of the drone 3 (for example, when the camera 4 is configured to be rotatable with respect to the drone 3), the shooting control unit 34 may control the direction of the camera 4 by rotating the camera 4 with respect to the drone 3 based on information related to the shooting direction and the like.

[0064] · The drone 3 may be provided with a storage unit (not shown) that stores the data of the video captured by the camera 4. According to this configuration, it becomes unnecessary to transmit the video data from the drone 3 to the management device 1, and the communication volume between the management device 1 and the drone 3 can be reduced.

[0065] · The input data to the action determination unit 13 is not limited to environmental information, luggage information, vehicle information, and operator information. Also, the environmental information, luggage information, vehicle information, and operator information are not limited to the information described in the above embodiment. That is, the information constituting each of the input data, environmental information, luggage information, vehicle information, and operator information may be changed as appropriate.

[0066] · The handling vehicle on which the handling worker M rides may be a handling vehicle other than the forklift 2. That is, the drone 3 may photograph, for example, a transport cart or a tractor with the camera 4.

Explanation of reference numerals

[0067] 1 Management device 2 Forklift (handling vehicle) 3 Drone (unmanned aerial vehicle) 4 Camera 13 Action determination unit (shooting location determination unit) 16 Traffic volume prediction unit M Handling worker N Luggage R Rack

Claims

1. A cargo handling vehicle that can be boarded by a cargo handling operator and performs cargo handling operations by being operated by the cargo handling operator, An unmanned aerial vehicle that can fly autonomously and has a camera attached thereto, A shooting location determination unit that determines a shooting location by performing reinforcement learning based on predetermined input data, The shooting location determination unit determines the shooting location where the reward is maximized, assuming that a reward is obtained when the cargo handling vehicle is photographed by the camera, The unmanned aerial vehicle moves to the shooting location determined by the shooting location determination unit and photographs the cargo handling vehicle using the camera at the shooting location A monitoring system characterized by the above.

2. A cargo handling vehicle that can be boarded by a cargo handling operator and performs cargo handling operations by being operated by the cargo handling operator, An unmanned aerial vehicle that can fly autonomously and has a camera attached thereto, A traffic volume prediction unit that predicts the traffic volume of the cargo handling vehicle based on predetermined input data and a traffic volume prediction model generated by machine learning, A shooting location determination unit that determines a shooting location based on the prediction result of the traffic volume predicted by the traffic volume prediction unit, The unmanned aerial vehicle moves to the shooting location determined by the shooting location determination unit and photographs the cargo handling vehicle using the camera at the shooting location A monitoring system characterized by the above.

3. The input data includes one or more types of information among environmental information related to the environment in which the cargo handling operation is performed, cargo information related to the cargo handled by the cargo handling vehicle, vehicle information related to the cargo handling vehicle, and operator information related to the cargo handling operator. The monitoring system according to claim 1 or 2, characterized by the above.

4. The environmental information includes one or more types of information among information indicating the width of the passage on which the cargo handling vehicle travels and information indicating the number of levels of the rack where the cargo handling vehicle loads and unloads cargo. The monitoring system according to claim 3, characterized by the above.

5. The cargo information includes one or more types of information among information indicating the shape of the cargo handled by the cargo handling vehicle and information indicating the size of the cargo handled by the cargo handling vehicle. The monitoring system according to claim 3, characterized by the above.

6. The vehicle information includes one or more types of information among information indicating the type of the cargo handling vehicle, information indicating the size of the cargo handling vehicle, and information indicating the position where the cargo handling operator boards the cargo handling vehicle. The monitoring system according to claim 3, characterized by the above.

7. The operator information includes one or more types of information among information indicating the height of the handling worker and information indicating the facial features of the handling worker. The monitoring system according to claim 3, characterized in that.

Images