Mobile object control system and mobile object control method

Abstract

To appropriately prevent a mobile object moving in an unmanned region from entering a manned region.SOLUTION: A transmission unit transmits a permission signal for permitting an operation of a mobile object, to the mobile object that moves in an unmanned region separated from a manned region. A detection unit detects a specific event in which the mobile object moves out of a movement permission area in the unmanned region. A control unit inhibits transmission of the permission signal when the specific event is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a mobile object control technology. [Background technology]

[0002] In recent years, the Ministry of Land, Infrastructure, Transport and Tourism has been formulating safety guidelines for automated and remote construction. According to these safety guidelines, automated construction of earthworks such as land development must be carried out within unmanned areas set up within the construction site. By separating manned areas where workers work from unmanned areas, accidents caused by contact between unmanned construction machinery and people can be prevented.

[0003] A manned area is sometimes called a manned construction area, and an unmanned area is sometimes called an unmanned construction area. Work machines used at construction sites are sometimes called heavy machinery.

[0004] Regarding earthworks, a construction support system for earthworks is known that can improve the efficiency of construction work by instructing changes in the number of transport vehicles depending on the progress of cutting and filling work (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-243039 Summary of the Invention [Problem to be solved by the invention]

[0006] When performing automated construction in an unmanned area, there is a possibility that a work machine traveling in the unmanned area may run away from the unmanned area due to a problem or other reason and run out of control into a manned area.

[0007] In order to prevent such work machines from running away, it is effective to install physical barriers such as embankments or trenches on the boundaries between manned and unmanned areas. However, since the unmanned areas are large and there is a possibility that they will be relocated as the construction progresses, it is not easy to install physical barriers to enclose the unmanned areas.

[0008] One possible method is to have a supervisor who monitors unmanned construction work by radio to make an emergency stop of the work machine if he or she notices that the machine has run away.However, when relying on human judgment, it is difficult to completely prevent human error, and the reliability of safety assurance decreases.

[0009] This problem is not limited to work machines traveling in unmanned areas of construction sites, but occurs in various mobile objects that travel in unmanned areas separated from manned areas where people are active.

[0010] In one aspect, the present invention aims to appropriately prevent a mobile object moving in an unmanned area from invading a manned area. [Means for solving the problem]

[0011] In one embodiment, the mobile object control system includes a transmitter, a detector, and a controller. The transmitter transmits a permission signal to a mobile object moving in an unmanned area separated from a manned area, permitting the operation of the mobile object. The detector detects a specific event in which the mobile object moves outside of a permitted movement area in the unmanned area. The controller inhibits transmission of the permission signal when the specific event is detected.

[0012] In another embodiment, the mobile object control system includes a detection unit and a control unit. The detection unit detects a specific event in which the mobile object moves outside of an allowed movement area within the unmanned area based on an image captured by an imaging device mounted on the mobile object moving within an unmanned area separated from a manned area. The control unit stops the operation of the mobile object when the specific event is detected. [Effects of the Invention]

[0013] According to one aspect, it is possible to appropriately prevent a moving object moving in an unmanned area from entering a manned area. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a configuration diagram of a work machine control system according to an embodiment. [Figure 2] FIG. 1 is a configuration diagram of an operating system. [Figure 3] FIG. 10 is a diagram showing a marker. [Figure 4] 10 is a flowchart of image processing. [Figure 5] FIG. 2 is a functional configuration diagram of a management device. [Figure 6] 10 is a flowchart of a control process. [Figure 7] FIG. 10 is a diagram showing a control process using position information of a work machine. [Figure 8] 4 is a flowchart of a control process that uses position information of a work machine. [Figure 9] FIG. 2 is a hardware configuration diagram of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0016] Fig. 1 shows an example of the configuration of a construction machine control system according to an embodiment. The construction machine control system in Fig. 1 includes a construction machine 101, a management device 102, light-projecting devices 103-1 to 103-4, and light-receiving devices 104-1 to 104-4. The construction machine control system is an example of a mobile object control system.

[0017] The work machine 101 performs unmanned construction work while traveling within an unmanned area 112 set up at a construction site for automated construction. The work machine 101 is, for example, a backhoe. The work machine 101 may also be a bulldozer, a wheel loader, a dump truck, or the like. The work machine 101 is an example of a mobile object.

[0018] A manned area 111 is set outside the unmanned area 112, surrounding the unmanned area 112, and the unmanned area 112 is separated from the manned area 111. The manned area 111 is an area where workers perform work. The management device 102 is installed, for example, in a control room within the manned area 111.

[0019] The light-projecting devices 103-i and the light-receiving devices 104-i (i = 1 to 4) are installed at the boundaries of the movement-permitted areas within the unmanned area 112, and operate as barrier sensors. The boundaries of the movement-permitted areas may be set a predetermined distance inward from the boundary of the unmanned area 112, or may coincide with the boundary of the unmanned area 112.

[0020] The light-projecting device 103-i emits multiple light beams to the light-receiving device 104-i, and the light-receiving device 104-i receives the multiple light beams from the light-projecting device 103-i. When any of the light beams is blocked by the work machine 101, the light-receiving device 104-i detects the passage of the work machine 101 and transmits a detection signal to the management device 102 indicating that the work machine 101 has passed through the boundary of the permitted movement area.

[0021] The number of light-projecting devices 103-i and light-receiving devices 104-i is not limited to four, and five or more light-projecting devices 103-i and light-receiving devices 104-i may be installed at the boundary of the movement-permitted area.

[0022] Fig. 2 shows an example of the configuration of an actuation system mounted on the work machine 101 of Fig. 1. The actuation system of Fig. 2 includes a hydraulic lock lever 211, a switch 212, a hydraulic pump 213, a hydraulic lock solenoid valve 214, a hydraulic actuator 215, an emergency stop device 216, an image processing device 217, and an imaging device 218. The emergency stop device 216 includes a wireless communication device 221 and a controller 222.

[0023] The hydraulic actuator 215 is, for example, a cylinder, a hydraulic motor, etc., and the hydraulic pump 213 delivers oil to the hydraulic actuator 215 via a hydraulic lock solenoid valve 214. The hydraulic actuator 215 operates by the hydraulic pressure of the oil delivered from the hydraulic pump 213. This enables the work machine 101 to travel and drive movable parts such as a boom, arm, and bucket.

[0024] The hydraulic lock solenoid valve 214 releases or activates the hydraulic lock in accordance with a control signal from the switch 212 or the controller 222. Releasing the hydraulic lock causes oil to be sent to the hydraulic actuator 215, and activating the hydraulic lock blocks the oil from being sent. When the oil supply is blocked, the work machine 101 stops traveling and driving of the movable parts.

[0025] When the hydraulic lock lever 211 is operated, the switch 212 detects the operation of the hydraulic lock lever 211 and outputs a control signal to the hydraulic lock electromagnetic valve 214 to release the hydraulic lock.

[0026] The wireless communication device 221 communicates with the management device 102 via the wireless communication network 231 and receives an operation permission signal from the management device 102. The operation permission signal is an example of an permission signal that permits the operation of a mobile object. The wireless communication device 221 can also receive a control signal from a manual switch (not shown) via the wireless communication network 231.

[0027] The imaging device 218 is, for example, an RGB camera, and is attached to the ceiling of the driver's seat of the work machine 101. The imaging device 218 captures images of objects and the like that exist in front of the work machine 101, and outputs the captured images to the image processing device 217.

[0028] The image processing device 217 uses the captured image to estimate the distance between the work machine 101 and the boundary of the allowed movement area, and based on the estimated distance, detects a runaway state in which the work machine 101 runs away from the unmanned area. A runaway state is an example of a specific event in which the work machine 101 moves away from the allowed movement area.

[0029] Instead of the image processing device 217 and the imaging device 218, an AI (Artificial Intelligence) camera in which the image processing device 217 and the imaging device 218 are integrated may be used.

[0030] For example, a plurality of markers having specific shapes are placed on the boundary of the movement-permitted area, and the image processing device 217 recognizes the shape of each marker from the captured image.

[0031] 3 shows an example of markers. In this example, triangular stop signs 301-1 to 301-3 are installed as markers. The image processing device 217 recognizes that the object shown in the captured image is a triangular stop sign 301-j (j=1 to 3) based on the shape of the object.

[0032] Next, the image processing device 217 estimates the distance to each triangular stop sign 301-j (j=1 to 3) from the size of each triangular stop sign 301-j in the image using, for example, a trained machine learning model. Then, the image processing device 217 identifies the position of each triangular stop sign 301-j based on the estimated distance. As the machine learning model, a neural network, a random forest, or the like is used.

[0033] Next, the image processing device 217 generates a line segment 302-j (j=1, 2) connecting the position of the triangular stop plate 301-j and the position of the triangular stop plate 301-(j+1), and estimates the line segments 302-1 and 302-2 as the boundaries of the permitted movement area.

[0034] Next, the image processing device 217 determines the shortest distance from the work machine 101 to the boundary of the allowed movement area and compares the determined shortest distance with a threshold value. If the shortest distance is less than the threshold value, the image processing device 217 determines that a runaway state has occurred. If a runaway state is detected, the image processing device 217 outputs a control signal to the controller 222 to stop the operation of the work machine 101.

[0035] The number of triangular stop plates 301-j is not limited to three, and four or more triangular stop plates 301-j may be installed at a predetermined interval on the boundary of the movement-permitted area. In this case, a polygon generated by sequentially connecting two adjacent triangular stop plates 301-j is estimated as the boundary of the movement-permitted area. Objects of other shapes, such as triangular cones, may also be used as markers.

[0036] 4 is a flowchart showing an example of image processing performed by the image processing device 217. First, the image processing device 217 recognizes markers from a captured image (step 401).

[0037] Next, the image processing device 217 uses the trained machine learning model to estimate the distance to each marker from the size of each marker in the image (step 402).Then, the image processing device 217 identifies the position of each marker based on the estimated distance (step 403).

[0038] Next, the image processing device 217 estimates a polygon connecting the positions of the multiple markers as the boundary of the allowed movement area (step 404), and finds the shortest distance from the work machine 101 to the boundary of the allowed movement area (step 405).The image processing device 217 then compares the shortest distance with a threshold value (step 406).

[0039] If the shortest distance is smaller than the threshold value (step 406, YES), the image processing device 217 outputs a control signal to the controller 222 to stop the operation of the work machine 101 (step 407). If the shortest distance is equal to or greater than the threshold value (step 406, NO), the image processing device 217 ends the processing.

[0040] While the controller 222 continues to receive the operation permission signal from the management device 102, it outputs a control signal to release the hydraulic lock to the hydraulic lock solenoid valve 214. If the operation permission signal from the management device 102 is discontinued, the controller 222 outputs a control signal to activate the hydraulic lock to the hydraulic lock solenoid valve 214, thereby bringing the work machine 101 to an emergency stop.

[0041] Furthermore, when the controller 222 receives a control signal to stop the operation of the work machine 101 from the image processing device 217, it also outputs a control signal to activate the hydraulic lock to the hydraulic lock solenoid valve 214. In this case, the controller 222 transmits a warning signal indicating the occurrence of a runaway state to the management device 102 via the wireless communication device 221.

[0042] Furthermore, the controller 222 outputs a control signal to activate the hydraulic lock to the hydraulic lock solenoid valve 214 when it receives a control signal requesting an emergency stop from a manual switch (not shown) or when a system failure such as a wireless interruption occurs.

[0043] The work machine 101 may be powered by other power sources such as an engine, an electrical source, etc. In this case, the actuation system brings the work machine 101 to an emergency stop by cutting off the power supply in a different manner instead of activating the hydraulic lock.

[0044] Fig. 5 shows an example of the functional configuration of the management device 102 in Fig. 1. The management device 102 in Fig. 5 includes a communication unit 511, a detection unit 512, and a control unit 513. The communication unit 511 is an example of a transmission unit. The warning light 521 is installed, for example, in a control room in the manned area 111.

[0045] The communication unit 511 communicates with the work machine 101 and the light-receiving device 104-i via the wireless communication network 231. The communication unit 511 continues to transmit an operation permission signal to the work machine 101 in accordance with instructions from the control unit 513.

[0046] When a detection signal indicating that the work machine 101 has passed the boundary of the permitted movement area is transmitted from any of the light-receiving devices 104-i, the detection unit 512 receives the detection signal via the communication unit 511. By receiving the detection signal, the detection unit 512 detects that the work machine 101 has run away, and notifies the control unit 513 of the occurrence of the runaway state.

[0047] When the detection unit 512 notifies the control unit 513 that a runaway state has occurred, the control unit 513 cancels the operation permission signal, thereby preventing the communication unit 511 from transmitting the operation permission signal. This causes the work machine 101 to come to an emergency stop. The control unit 513 then turns on the warning light 521. Furthermore, when a warning signal is transmitted from the work machine 101, the control unit 513 receives the warning signal via the communication unit 511 and turns on the warning light 521.

[0048] When the warning light 521 in the control room lights up, the manager becomes aware that the work machine 101 has run away and checks the safety of the construction site. If safety is confirmed, the manager remotely restarts the work machine 101. At this time, the manager performs emergency stop recovery remote operation via the control unit 513 and communication unit 511, and after returning the runaway work machine 101 to the permitted movement area, causes the management device 102 to resume sending an operation permission signal.

[0049] 6 is a flowchart showing an example of control processing performed by the management device 102. First, the control unit 513 transmits an operation permission signal to the work machine 101 via the communication unit 511 (step 601).

[0050] Next, the detection unit 512 checks whether or not a detection signal has been received from any of the light receiving devices 104-i (step 602). If a detection signal has been received (step 602, YES), the detection unit 512 notifies the control unit 513 of the occurrence of a runaway state, and the control unit 513 cancels the operation permission signal (step 603). Then, the control unit 513 turns on the warning light 521 (step 604).

[0051] If a detection signal has not been received (step 602, NO), the control unit 513 checks whether or not a warning signal has been received from the work machine 101 (step 605). If a warning signal has been received (step 605, YES), the control unit 513 performs the processing of step 604. If a warning signal has not been received (step 605, NO), the management device 102 repeats the processing from step 602 onwards.

[0052] 1, if the light receiving device 104-i or the image processing device 217 detects that the work machine 101 traveling in the unmanned area 112 has run away, the work machine 101 will automatically make an emergency stop. This makes it possible to appropriately prevent the work machine 101 from entering the manned area 111, ensuring the safety of automated construction at the construction site.

[0053] The work machine 101 is often brought to the construction site by a partner company or is a rental machine from a rental company. For this reason, it is desirable to ensure the safety of automated construction work without modifying the work machine 101 as much as possible.

[0054] 2, the emergency stop device 216 can be realized with a simple circuit, and can therefore be easily retrofitted without modifying the work machine 101. Furthermore, an AI camera that integrates the image processing device 217 and imaging device 218 can also be easily retrofitted. Therefore, no modification of the work machine 101 is required.

[0055] Instead of the light-projecting device 103-i and the light-receiving device 104-i, other sensors such as a radar sensor or an ultrasonic sensor may be used to detect runaway of the work machine 101.

[0056] If it is difficult to install sensors at the boundaries of the permitted movement area, escape of the work machine 101 may be detected using only the image processing device 217. In this case, there is no need for the management device 102 to cancel the operation permission signal, and the processing is completed within the work machine 101, so the work machine 101 can be brought to a swift emergency stop.

[0057] Incidentally, instead of installing sensors on the boundaries of the permitted movement area, it is also possible to detect runaway of the work machine 101 by using the position information of the work machine 101.

[0058] 7 shows an example of control processing that uses the position information of the work machine 101. The work machine 101 is equipped with a Global Navigation Satellite System (GNSS) receiver.

[0059] The management device 102 stores coordinate information of a pre-set movement-permitted area. The coordinate information of the movement-permitted area includes coordinates of the boundary of the movement-permitted area. The coordinates of the boundary of the movement-permitted area may be coordinates measured using a GPS (Global Positioning System) rover or the like, or may be coordinates set on a construction drawing.

[0060] The GNSS receiver mounted on the work machine 101 measures the position of the work machine 101 performing unmanned construction as needed, and transmits position information indicating the measured position to the management device 102. The detection unit 512 of the management device 102 receives the position information via the communication unit 511.

[0061] Next, the detection unit 512 references the coordinate information of the movement-permitted area, finds the shortest distance from the position of the work machine 101 indicated by the received position information to the boundary of the movement-permitted area, and compares the found shortest distance with a threshold value. If the shortest distance is smaller than the threshold value, the detection unit 512 determines that a runaway state has occurred, and notifies the control unit 513 of the occurrence of the runaway state.

[0062] If position information from the work machine 101 is interrupted for a predetermined period of time, the detection unit 512 presumes that a runaway state has occurred, and notifies the control unit 513 of the occurrence of the runaway state.

[0063] 8 is a flowchart showing an example of control processing using position information of the work machine 101. First, the control unit 513 transmits an operation permission signal to the work machine 101 via the communication unit 511 (step 801).

[0064] Next, the detection unit 512 checks whether position information has been received from the work machine 101 (step 802). If position information has been received (step 802, YES), the detection unit 512 references the coordinate information of the movement-permitted area and determines the shortest distance from the position indicated by the received position information to the boundary of the movement-permitted area (step 803). The detection unit 512 then compares the shortest distance with a threshold value (step 804).

[0065] If the shortest distance is smaller than the threshold value (step 804, YES), the detection unit 512 notifies the control unit 513 of the occurrence of the runaway state, and the control unit 513 cancels the operation permission signal (step 805). Then, the control unit 513 turns on the warning light 521 (step 806).

[0066] If the shortest distance is equal to or greater than the threshold (step 804, NO), the management device 102 repeats the processing from step 802 onwards.

[0067] If the location information has not been received (step 802, NO), the detection unit 512 checks whether the location information has been received for a predetermined period of time (step 807). If the location information has not been received for a predetermined period of time (step 807, YES), the management device 102 performs the processes from step 805 onwards.

[0068] If the position information has not been interrupted for a predetermined period of time (step 807, NO), the control unit 513 checks whether or not a warning signal has been received from the work machine 101 (step 808). If a warning signal has been received (step 808, YES), the control unit 513 performs the processing of step 806. If a warning signal has not been received (step 808, NO), the management device 102 repeats the processing from step 802 onwards.

[0069] According to the control process in Figure 7, even if it is difficult to install sensors because the unmanned area 112 is vast, it is possible to bring the work machine 101 to an emergency stop without installing sensors. The control process in Figure 7 is suitable for an unmanned area 112 that includes a haul path for dump trucks with long traffic lines, etc.

[0070] The escape of the work machine 101 may be detected by using sensors such as the light-projecting device 103-i and the light-receiving device 104-i in combination with the position information of the work machine 101.

[0071] Fig. 9 shows an example of the hardware configuration of an information processing device (computer) used as the management device 102 in Fig. 5. The information processing device in Fig. 9 includes a CPU (Central Processing Unit) 901, a memory 902, and a wireless communication device 903. These components are hardware and are connected to each other by a bus 904. The warning light 521 in Fig. 5 may be connected to the bus 904.

[0072] The memory 902 is, for example, a semiconductor memory such as a read only memory (ROM) or a random access memory (RAM), and stores programs and data used in processing.

[0073] 5 by executing a program using the memory 902. The wireless communication device 903 operates as the communication unit 511 in FIG.

[0074] 1 to 9 can be used to bring various mobile objects moving in unmanned areas to an emergency stop, not just work machines moving in unmanned areas of a construction site. The mobile objects moving in unmanned areas may be unmanned vehicles or robots that move autonomously.

[0075] 1 and 7 are merely examples, and some of the components may be omitted or changed depending on the application or conditions of the work machine control system. The shapes of the manned area 111 and the unmanned area 112 vary depending on the construction site. Multiple work machines 101 may perform unmanned construction work within the unmanned area 112.

[0076] 2 is merely an example, and some of the components may be omitted or changed depending on the application or conditions of the work machine control system. For example, if there is no need to use the image processing device 217 to detect runaway of the work machine 101, the image processing device 217 and the imaging device 218 can be omitted.

[0077] The configuration of the management device 102 in FIG. 5 is merely an example, and some of the components may be omitted or changed depending on the application or conditions of the work machine control system.

[0078] The configuration of the information processing device in FIG. 9 is merely an example, and some of the components may be omitted or changed depending on the use or conditions of the information processing device.

[0079] 4, 6, and 8 are merely examples, and some of the processes may be omitted or changed depending on the configuration or conditions of the work machine control system. The markers shown in Fig. 3 are merely examples, and other markers such as triangular cones may also be used.

[0080] Although the disclosed embodiments and their advantages have been described in detail, those skilled in the art may make various modifications, additions, and omissions without departing from the scope of the invention as clearly set forth in the claims. [Explanation of symbols]

[0081] 101 Work Machinery 102 Management device 103-1~103-4 Floodlights 104-1~104-4 Light receiving device 111 Manned Area 112 Uninhabited area 211 Hydraulic lock lever 212 Switch 213 Hydraulic Pump 214 Hydraulic lock solenoid valve 215 Hydraulic Actuator 216 Emergency stop device 217 Image Processing Device 218 Imaging Device 221, 903 Wireless communication equipment 222 Controller 231 Wireless Communication Network 301-1~301-3 Triangular stop plate 302-1, 302-2 line segments 511 Communications Department 512 Detection unit 513 Control Unit 521 Warning light 901 CPU 902 memory 904 Bus

Claims

1. a transmitter that transmits a permission signal to a moving object moving in an unmanned area separated from a manned area, the permission signal permitting the moving object to operate; a detection unit that detects a specific event in which the moving object deviates from a movement-allowed area within the unmanned area; a control unit that, when the specific event is detected, inhibits transmission of the permission signal; A mobile object control system comprising:

2. the detection unit detects the specific event based on a detection signal transmitted from a sensor installed on a boundary of the movement-permitted area; 2. The mobile object control system according to claim 1, wherein the detection signal indicates that the mobile object has passed through the boundary of the permitted movement area.

3. 2. The mobile object control system according to claim 1, wherein the detection unit detects the specific event based on coordinate information of the movement-permitted area and position information indicating the position of the mobile object.

4. a detection unit that detects a specific event in which a moving object moves outside a movement-permitted area within an unmanned area based on an image captured by an imaging device mounted on the moving object moving in the unmanned area; a control unit that stops the operation of the moving object when the specific event is detected; A mobile object control system comprising:

5. 5. The mobile object control system according to claim 4, wherein the detection unit estimates the distance between the mobile object and the boundary of the movement-permitted area based on the image, and detects the specific event based on the distance.

6. transmitting a permission signal to a moving object moving in an unmanned area separated from a manned area, the permission signal permitting the moving object to operate; detecting a specific event in which the moving object deviates from a movement-allowed area within the unmanned area; If the specific event is detected, the transmission of the permission signal is suppressed. A mobile object control method comprising:

7. detecting a specific event in which a moving object moves outside a permitted movement area within an unmanned area based on an image captured by an imaging device mounted on the moving object moving within the unmanned area; When the specific event is detected, the operation of the moving object is stopped. A mobile object control method comprising:

Images