Control system, alarm system, control method and program

Abstract

To safely transport goods loaded onto the vehicle's cargo bed. [Solution] The system comprises an acquisition unit that acquires an image of the cargo bed including the cargo loaded on the cargo bed of a vehicle, and a launch control unit that controls a launch prevention mechanism that prevents the vehicle from starting. The launch control unit activates the launch prevention mechanism when it determines from the cargo bed image that there is a risk of the cargo falling from the cargo bed.

Description

Technical Field

[0001] The present invention relates to the technology of a control system, an alarm system, a control method, and a program.

Background Art

[0002] In the present invention, excavated material (so-called muck) discharged by tunnel excavation or the like is loaded onto the cargo bed (vessel) of a dump truck by a backhoe and carried out of the work site. However, on transport paths with many curves and steep slopes, there is a risk that part of the loaded excavated material may fall from the vessel of the dump truck and cause an accident. Therefore, in some cases, a supervisor was assigned to monitor whether there was any excavated material likely to fall from the cargo bed when loading the dump truck.

[0003] However, assigning supervisors not only increases labor costs, but also causes problems such as variations in judgment among individual supervisors and safety issues related to supervisors entering the work site.

[0004] Here, Patent Document 1 discloses a technology of displaying a vessel map showing the distribution of the amount of transported material in the vessel on a display device in the cab of a hydraulic excavator, in order to load the transported material into the vessel in a well-balanced manner. In the technology of Patent Document 1, the operator of the hydraulic excavator can also load the transported material while checking the vessel map so as not to cause the material to fall.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the technology described in Patent Document 1, if the hydraulic excavator operator overlooked something or made an incorrect judgment, there was a possibility that the dump truck would start transporting the cargo while there was a risk of it falling. Taking these circumstances into consideration, the present invention aims to safely transport cargo loaded on the bed of a vehicle. [Means for solving the problem]

[0007] [1] A control system comprising an acquisition unit that acquires an image of the cargo bed of a vehicle, including the cargo loaded on the cargo bed, and a launch control unit that controls a launch prevention mechanism that prevents the vehicle from starting, wherein the launch control unit determines from the cargo bed image that there is a risk of the cargo falling from the cargo bed and activates the launch prevention mechanism.

[0008] [2] The launch control unit determines that the working part of the backhoe for loading the cargo onto the cargo bed is included in the cargo bed image, and the launch obstruction mechanism is activated. [1]

[0009] [3] The launch control unit processes the load bed image acquired by the acquisition unit using a generative model that has learned the relationship between a learning load bed image and areas of the load included in the learning load bed image that are likely to fall, thereby identifying areas of fall in the load bed image, and if the areas of fall are identified in the load bed image, it determines that there is a risk of the load falling from the load bed [1] or [2] control system.

[0010] [4] A control system [3] comprising a display panel having an installation surface with a reference diagram of the loading platform attached to it, and an illumination device for irradiating the installation surface with light, further comprising a first illumination control unit for controlling the illumination device, wherein the first illumination control unit controls the illumination device to illuminate a position in the reference diagram corresponding to the identified drop area, and the display panel is positioned so that the installation surface is visible to the driver of a backhoe for loading the load onto the loading platform.

[0011] [5] A control system according to [3] or [4] further comprising a second irradiation control unit for controlling an irradiation unit that irradiates light onto the cargo bed, wherein the second irradiation control unit controls the irradiation unit to irradiate light onto a position in the cargo bed corresponding to the identified fall area.

[0012] [6] The control system according to [1] to [5] further comprises a notification control unit that notifies a terminal device held by the driver of the vehicle of the risk of the load falling if it is determined that the load is in danger of falling from the cargo bed.

[0013] [7] The launch obstruction mechanism is a control system [1] to [6] including at least one of a lock plate or a gate.

[0014] [8] A control system comprising an imaging device for capturing an image of the cargo bed including the cargo loaded on the cargo bed of a vehicle, a starting obstruction mechanism for obstructing the starting of the vehicle, an acquisition unit for acquiring the cargo bed image, and a starting control unit for controlling the starting obstruction mechanism, wherein the starting control unit determines from the cargo bed image that there is a risk of the cargo falling from the cargo bed and activates the starting obstruction mechanism.

[0015] [9] A control method implemented by a computer that acquires an image of the cargo bed of a vehicle including the cargo loaded on the cargo bed, controls a starting obstruction mechanism that obstructs the starting of the vehicle, and in the control, if it is determined from the cargo bed image that there is a risk of the cargo falling from the cargo bed, activates the starting obstruction mechanism.

[0016]

[10] A computer functions as an acquisition unit that acquires an image of the cargo bed of a vehicle, including the cargo loaded on the cargo bed, and a launch control unit that controls a launch prevention mechanism that prevents the vehicle from starting, wherein the launch control unit determines from the cargo bed image that there is a risk of the cargo falling from the cargo bed, and the program activates the launch prevention mechanism. [Effects of the Invention]

[0017] According to the present invention, it is possible to safely transport the load loaded on the cargo bed of a vehicle. [Brief Description of the Drawings]

[0018] [Figure 1] This is an explanatory diagram for explaining a work site according to the first embodiment. [Figure 2] This is a block diagram illustrating the configuration of the control system according to the first embodiment. [Figure 3] This is a block diagram illustrating the functional configuration of the control system according to the first embodiment. [Figure 4] This is an example of the specific image output by the generation model in the first embodiment. [Figure 5] This is a flowchart of the control processing according to the first embodiment. [Figure 6] This is a front view of the start inhibition mechanism according to another configuration of the first embodiment. [Figure 7] This is a front view of the display panel according to the second embodiment. [Figure 8] This is a block diagram illustrating the configuration of the control system according to the second embodiment. [Figure 9] This is a flowchart of the control processing according to the second embodiment. [Figure 10] This is an explanatory diagram for explaining a work site according to the third embodiment. [Figure 11] This is a block diagram illustrating the configuration of the control system according to the third embodiment. [Figure 12] This is a flowchart of the control processing according to the third embodiment. [Figure 13] This is an example of the specific image output by the generation model in the fourth embodiment. [Figure 14] This is a flowchart of the control processing according to the fourth embodiment. [Mode for Carrying Out the Invention]

[0019] <First Embodiment> Figure 1 is an explanatory diagram illustrating a work site where cargo P is loaded onto the cargo bed 11 of vehicle 1 using a backhoe 2. Cargo P is excavated material (so-called spoil) discharged from tunnel excavation, etc.

[0020] Vehicle 1 is a dump truck that includes a cargo bed 11 and multiple wheels 13, and can unload its cargo P by, for example, tilting the cargo bed 11. The cargo bed 11 is a box-shaped container (vessel).

[0021] Backhoe 2 is a heavy machine for loading cargo P onto the cargo bed 11 of vehicle 1, and includes, for example, a machine body 21 and a work section 23. The machine body 21 is the part that includes the driver's cab 211 and the running body 213 (for example, crawlers and wheels 13). The work section 23 can be moved around the connection point with the machine body 21 by operation by the driver D2, and includes, for example, a boom 231, an arm 233 and a bucket 235.

[0022] In this case, if the vehicle 1 starts moving while there is a risk of the cargo P on the cargo bed 11 falling (for example, if there is a part that has been piled up high), there is a risk that some of the cargo P may fall from the cargo bed 11 and cause an accident if there are curves or steep slopes on the transport route. Therefore, in the first embodiment, the vehicle 1 is prevented from starting when there is a risk of the cargo P falling from the cargo bed 11.

[0023] In the first embodiment, a locking plate 3a (flap plate), an imaging device 4, and a control system 5 are used to prevent the vehicle 1 from starting. The control system 5 is connected to the locking plate 3a and the imaging device 4 in a communicative manner. The control system 5 can communicate with the locking plate 3a and the imaging device 4 via a communication network such as short-range wireless communication such as Bluetooth® or Wi-Fi®, or cellular communication.

[0024] The locking plate 3a is provided in front of at least one of the multiple wheels 13. When the locking plate 3a is raised (i.e., the locking plate 3a is activated), it is impossible for the wheels 13 to go over the locking plate 3a, and the vehicle 1 is prevented from starting. On the other hand, when the locking plate 3a is lowered (i.e., the locking plate 3a is not activated), it is possible for the wheels 13 to go over the locking plate 3a, and the vehicle 1 can start. The locking plate 3a is an example of a "starting obstruction mechanism" that prevents the vehicle 1 from starting. The starting obstruction mechanism is installed on the outside of the vehicle 1.

[0025] The imaging device 4 is a camera that captures images of the surrounding subjects. Specifically, the imaging device 4 captures an image (hereinafter referred to as the "cargo bed image") Ga that includes the cargo P loaded on the cargo bed 11 of the vehicle 1. For example, the imaging device 4 is composed of an optical system and an image sensor. The imaging device 4 is installed in a position where the entire surface of the cargo P on the cargo bed 11 can be captured.

[0026] In the first embodiment, an example is given where the imaging device 4 is installed on the backhoe 2. For example, it is attached by magnet to the front of the machine body 21 of the backhoe 2 (for example, the windshield area). Alternatively, the imaging device 4 may be installed inside the driver's cab 211 of the backhoe 2. However, the position and method of installing the imaging device 4 are not particularly limited as long as it is possible to image the load P on the loading platform 11. For example, the imaging device 4 may be installed using a holding device (for example, a tripod) located at a distance from the backhoe 2.

[0027] The generation of the cargo bed image Ga by the imaging device 4 is performed repeatedly, for example, at a very short predetermined cycle, in parallel with the loading of the cargo P by the backhoe 2. The cargo bed image Ga captured by the imaging device 4 is transmitted to the control system 5 via a communication device (not shown).

[0028] Figure 2 is a block diagram illustrating the configuration of the control system 5. As illustrated in Figure 2, the control system 5 comprises a control device 51, a storage device 52, a communication device 53, a display device 54, and an operating device 55. The control system 5 can be implemented using, for example, an information terminal such as a personal computer, a tablet terminal, or a smartphone. The control system 5 can be implemented as a single device or as a group of devices configured separately from each other.

[0029] The control device 51 is one or more processors that control each element of the control system 5. Specifically, the control device 51 is composed of one or more types of processors such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).

[0030] The storage device 52 is one or more recording media that store programs executed by the control device 51 and various data used by the control device 51. The storage device 52 is composed of known recording media such as magnetic recording media or semiconductor recording media. Alternatively, a portable recording media that can be attached to and detached from the control system 5, or a recording media accessible by the control device 51 via a communication network (e.g., cloud storage), may be used as the storage device 52.

[0031] The display device 54 displays various images under the control of the control device 51. The display device 54 is composed of a display panel 6, such as a liquid crystal panel or an organic EL (ElectroLuminescence) panel. The operating device 55 is an input device that receives instructions from the user (e.g., driver D2). For example, the operating device 55 is composed of an operator operated by the user or a touch panel that detects contact by the user. Note that the display device 54 or the operating device 55, which are separate from the control system 5, may be connected to the control system 5 by wire or wireless.

[0032] Figure 3 is a block diagram illustrating the functional configuration of the control system 5. The control device 51 of the control system 5 realizes multiple functions (acquisition unit 511, launch control unit 512, display control unit 513) by executing a program stored in the storage device 52.

[0033] The acquisition unit 511 acquires the cargo bed image Ga. Specifically, the acquisition unit 511 acquires the cargo bed image Ga from the imaging device 4 via the communication device 53. The cargo bed image Ga is acquired, for example, each time the imaging device 4 generates the cargo bed image Ga.

[0034] The launch control unit 512 controls the lock plate 3a. Specifically, the launch control unit 512 activates the lock plate 3a when it determines from the cargo bed image Ga that there is a risk of the load P falling from the cargo bed 11. In the first embodiment, an example is provided in which loading of the load P is started from a state where the lock plate 3a is not activated, and the lock plate 3a is activated (raised) when it is determined that there is a risk of the load P falling from the cargo bed 11.

[0035] In the first embodiment, the launch control unit 512 processes the cargo bed image Ga using the generation model M to determine whether or not there is a risk of the cargo P falling from the cargo bed 11.

[0036] The generative model M of the first embodiment is a trained model that has learned the relationship between a training image of the cargo bed and the areas of the cargo P contained in the training image that are at risk of falling (hereinafter referred to as "falling areas") through machine learning. In other words, the generative model M outputs statistically reasonable falling areas for the cargo bed image Ga.

[0037] The generative model M is composed of, for example, a deep neural network (DNN). Any form of deep neural network, such as a convolutional neural network (CNN), can be used as the generative model M. The generative model M may also be composed of a combination of multiple types of deep neural networks.

[0038] The generation model M is implemented by a combination of a program that causes the control device 51 to perform a calculation to identify the drop area from the cargo bed image Ga, and several variables (e.g., weight values ​​and biases) applied to the calculation. The program and several variables that implement the generation model M are stored in the storage device 52. The numerical values ​​of each of the several variables that define the generation model M are optimized in advance by machine learning.

[0039] The machine learning of the generative model M utilizes multiple training datasets. Each of these training datasets consists of a training image of a cargo bed with a specified (labeled) fall area. An error function is calculated that represents the difference between the fall area output by a provisional model (hereinafter referred to as the "provisional model") when a cargo bed image from each training dataset is input, and the fall area of ​​that training dataset. Then, several variables of the provisional model are updated to reduce the error function. The provisional model at the point when the above process has been repeated for each of the multiple training datasets is determined as the generative model M.

[0040] In the first embodiment, the generation model M, upon receiving a cargo bed image Ga as input, outputs an image (hereinafter referred to as the "specific image") Gb in which an object O1 indicating the fall area is attached to the cargo bed image Ga. An object can be a shape or a character. Figure 4 shows an example of a specific image Gb output by the generation model M. In Figure 4, a specific image Gb is shown as an example in which the fall area is enclosed by a solid line (the circular object O1).

[0041] Then, the launch control unit 512 determines that there is a risk of the load P falling from the loading platform 11 when the generation model M outputs (detects) a fall area from the loading platform image Ga, and activates the lock plate 3a. The lock plate 3a is activated when an instruction signal to activate (raise) is transmitted to the lock plate 3a via the communication device 53. Even after the lock plate 3a is activated, the launch control unit 512 continues to process the loading platform image Ga acquired by the acquisition unit 511 using the generation model M. In the first embodiment, the lock plate 3a continues to operate until no fall area is detected from the loading platform image Ga.

[0042] The display control unit 513 displays a specific image Gb on the display device 54 when it determines that there is a risk of the load P falling from the loading platform 11. Therefore, the backhoe operator D2 can visually check the specific image Gb displayed on the display device 54 to identify the part of the load P on the loading platform 11 that is likely to fall (the part corresponding to the fall area). The operator D2 then operates the work unit 23 of the backhoe 2 to adjust the load P on the loading platform 11 (for example, by removing a part of the load P from the part corresponding to the fall area) until the fall area is no longer detected.

[0043] The elements comprising the locking plate 3a (launching obstruction mechanism), the control system 5, and the imaging device 4 are examples of an "alarm system."

[0044] Figure 5 is a flowchart showing an example of a process (hereinafter referred to as "control process") performed by the control device 51 of the first embodiment. For example, the control process is started in response to an instruction from the driver D2 of the backhoe 2. The control process may also be started when the loading of the cargo P onto the cargo bed 11 of the vehicle 1 is partially completed, or it may be started from the start of loading. In the first embodiment, the control process is started when the lock plate 3a is not activated.

[0045] When control processing begins, the acquisition unit 511 acquires a cargo bed image Ga from the imaging device 4 via the communication device 53 (Sa1). The launch control unit 512 processes the cargo bed image Ga to determine whether or not there is a risk of the cargo P falling from the cargo bed 11 (Sa2). As described above, a generative model M that has learned the relationship between the cargo bed image and the fall area contained in the cargo bed image is used to determine whether or not there is a risk of the cargo P falling. If a fall area is detected in the cargo bed image Ga, it is determined that there is a risk of the cargo P falling. If no fall area is detected in the cargo bed image Ga, it is determined that there is no risk of the cargo P falling. If there is no risk of the cargo P falling from the cargo bed 11 (Sa2; NO), steps Sa1 and Sa2 are repeatedly executed until the end of control processing is instructed.

[0046] On the other hand, if there is a risk of the load P falling from the loading platform 11 (Sa2; YES), the launch control unit 512 activates the lock plate 3a (Sa3). In other words, the vehicle 1 becomes unable to start. The display control unit 513 then displays a specific image Gb on the display device 54, which is a loading platform image Ga with an object O1 indicating the area where the load may fall (Sa4). Therefore, the driver D2 of the backhoe 2 can identify the part of the load P on the loading platform 11 that is likely to fall. The driver D2 then operates the work unit 23 of the backhoe 2 to adjust the load P on the loading platform 11 so that there is no possibility of it falling.

[0047] With the lock plate 3a activated, the process of acquiring the loading platform image Ga (Sa5) and determining whether or not there is a risk of the load P falling (Sa6) is performed. The process in step Sa6 is the same as the process in step Sa2. Steps Sa5 and Sa6 may be performed in parallel with the operation of the work unit 23 of the backhoe 2. If it is determined that there is a risk of the load P falling (Sa6; YES), the process returns to step Sa4 with the lock plate 3a activated, and a specific image Gb is displayed, which is the loading platform image Ga used in step Sa6 with an object O1 indicating the fall area attached. In other words, the specific image Gb is updated.

[0048] On the other hand, if it is determined that there is no risk of the load P falling (Sa6; NO), the launch control unit 512 releases the lock plate 3a (Sa7). That is, the vehicle 1 becomes ready to launch. If the loading of the load P continues, the system may return to step Sa1 after step Sa7.

[0049] As can be understood from the above explanation, in the first embodiment, the lock plate 3a is activated when there is a risk of the cargo P falling from the cargo bed 11. In other words, the vehicle 1 cannot start unless it is determined that there is no risk of the cargo P falling. Therefore, it is possible to safely transport the cargo P loaded on the cargo bed 11 of the vehicle 1.

[0050] In particular, in the first embodiment, a generative model M is used that has learned the relationship between a training image of the loading platform and the areas where the loaded items P contained in the training image are likely to fall. Therefore, the areas where items are likely to fall can be identified with high accuracy by reflecting the trends present in the multiple training data used to train the generative model M. However, methods other than using the generative model M can also be employed to determine whether or not the loaded items P are likely to fall from the loading platform 11. For example, it may be determined whether or not the loaded items P are likely to fall from the loading platform 11 by performing known image analysis processing (e.g., feature extraction) on the loading platform image Ga.

[0051] In the configuration of the first embodiment, the starting obstruction mechanism may be something other than the lock plate 3a. Figure 6 illustrates another example of a starting obstruction mechanism. Figure 6 illustrates the case where the gate 3b is used as the starting obstruction mechanism. When the gate 3b is activated (the state in Figure 6), it is positioned on the vehicle 1's path (in front of the vehicle 1) and obstructs the vehicle 1's departure. On the other hand, when the gate 3b is deactivated, it moves outside the vehicle 1's path and is positioned not to obstruct the vehicle 1's departure. As can be understood from the above explanation, the specific configuration of the starting obstruction mechanism is not particularly limited, as long as it is possible to obstruct the vehicle 1's departure when there is a risk of the load P falling.

[0052] <Second Embodiment> A second embodiment of the present invention will now be described. In each of the embodiments described below, for elements whose operation or function is the same as in the first embodiment, the reference numerals used in the description of the first embodiment will be reused, and detailed descriptions of each will be omitted as appropriate.

[0053] In the second embodiment, a configuration utilizing a display panel 6 is illustrated, from the viewpoint of allowing the driver D2 of the backhoe 2 to easily identify any part of the load P on the loading platform 11 that is in danger of falling.

[0054] Figure 7 shows a front view of a display panel 6 according to an example of the second embodiment. The display panel 6 is installed outside the backhoe 2 in a position visible to the driver D2 inside the driver's cab 211. Specifically, the display panel 6 has an installation surface 61 with a diagram W representing the cargo bed 11 (hereinafter referred to as the "cargo bed diagram") and an illumination device 63 that illuminates the installation surface 61 with light. The display panel 6 is a so-called illuminated sign.

[0055] The cargo bed diagram W includes a diagram (hereinafter referred to as "reference diagram") W1 representing the loading surface (bottom surface of the cargo bed 11) of the cargo bed 11. From the viewpoint of improving visibility, a schematic diagram that schematically represents the cargo bed 11 including the loading surface is preferable as the cargo bed diagram W. However, the cargo bed diagram W may also be a photograph of the actual cargo bed 11 including the loading surface. The cargo bed diagram W only needs to include at least the reference diagram W1.

[0056] The mounting surface 61 to which the loading platform diagram W is attached is formed of a material that transmits light from the illumination device 63, for example, and may be a diffuser plate. The display panel 6 is installed in a position where the mounting surface 61 can be seen by the driver D2 of the backhoe 2.

[0057] The illumination device 63 includes a light source such as an LED (Light Emitting Diode) or a xenon lamp. For example, the illumination device 63 is positioned on the back side of the mounting surface 61. Specifically, the illumination device 63 is configured to individually illuminate each of the multiple sections Ra (9 sections Ra in Figure 7) in the reference diagram W1 (a diagram representing the loading surface) in the cargo bed diagram W. For example, an illumination device 63 equipped with a light source for each section Ra is used. However, the specific configuration of the illumination device 63 is not particularly limited as long as it is possible to illuminate each section Ra with light.

[0058] Figure 8 is a block diagram illustrating the configuration of the control system 5 according to the second embodiment. As illustrated in Figure 8, the control system 5 of the second embodiment includes, in addition to the acquisition unit 511, launch control unit 512, and display control unit 513 of the first embodiment, an illumination control unit 514 that controls the illumination device 63 of the display panel 6. The control system 5 is connected to the display panel 6 in a communication manner.

[0059] Specifically, the irradiation control unit 514 (an example of the "first irradiation control unit") controls the irradiation device 63 to irradiate the position corresponding to the fall area identified by the launch control unit 512 in the reference diagram W1 on the display panel 6. For example, among the multiple sections Ra in reference diagram W1, the section Ra corresponding to the fall area (for example, the section Ra including the center of the fall area) is irradiated by the irradiation device 63.

[0060] The driver D2 of backhoe 2 can determine the position of the load P loaded on the actual loading platform 11 that corresponds to the area where the load will fall by visually checking the illuminated section Ra in the reference diagram W1 on the display panel 6. Then, the driver D2 adjusts the load P on the loading platform 11 while checking the reference diagram W1. In practice, an instruction signal indicating the section to which the illumination device 63 should illuminate is transmitted to the display panel 6 via the communication device 53, enabling the illumination device 63 to illuminate a specific section.

[0061] Figure 9 is a flowchart of an example of the control process according to the second embodiment. The control process in Figure 9 is configured by adding steps Sb1 and Sb2 to the control process in Figure 5. The explanation of the process similar to that in Figure 5 will be omitted as appropriate.

[0062] When the control process starts, the acquisition unit 511 acquires a cargo bed image Ga (Sa1), similar to the first embodiment, and the launch control unit 512 determines whether or not there is a risk of the cargo P falling from the cargo bed 11 (Sa2). If there is a risk of the cargo P falling from the cargo bed 11, that is, if a fall area is identified from the cargo bed image Ga (Sa2; YES), the lock plate 3a is activated (Sa3). Then, the display control unit 513 displays the identified image Gb on the display device 54 (Sa4), and the illumination control unit 514 illuminates the reference diagram W1 on the display panel 6 (Sb1). Specifically, the illumination control unit 514 illuminates the illumination device 63 with the section Ra corresponding to the fall area identified in the cargo bed image Ga from among the multiple sections Ra in the reference diagram W1. Note that the order in which steps Sa4 and Sb1 are executed is not particularly limited and they may be executed in parallel.

[0063] Similar to the first embodiment, with the lock plate 3a activated, the acquisition of the cargo bed image Ga (Sa5) and the determination of whether or not there is a risk of the cargo P falling (Sa6) are performed. If it is determined that there is a risk of the cargo P falling (Sa6; YES), steps Sa4 and Sb1 are repeatedly performed with the lock plate 3a still activated. That is, in step Sa4, similar to the first embodiment, the specific image Gb newly output in step Sa6 is displayed, and in step Sb1, if the section Ra corresponding to the fall area of ​​the specific image Gb newly output in step Sa6 has changed, the changed section Ra is illuminated, and if there has been no change, the same section Ra is continued to be illuminated.

[0064] On the other hand, if it is determined that there is no risk of the load P falling (Sa6; NO), the launch control unit 512 releases the lock plate 3a (Sa7), and the irradiation control unit 514 terminates irradiation to the reference figure W1 (Sb2). Note that the order in which steps Sa7 and Sb2 are executed is not particularly limited and they may be executed in parallel.

[0065] In the second embodiment, light is shone on the display panel 6 at the position corresponding to the fall area in the reference diagram W1, making it easier for the backhoe 2 operator D2 to grasp the parts of the load P on the loading platform 11 that are at risk of falling. In particular, in the second embodiment, the reference diagram W1 is divided into multiple sections Ra, and the section Ra corresponding to the fall area is shone, which has the advantage of making it easier for the backhoe 2 operator D2 to visually grasp the approximate location where there is a risk of falling.

[0066] It is not mandatory for the irradiation device 63 to be located on the back side of the installation surface 61; an irradiation device 63 that irradiates the reference diagram W1 from the front side may also be placed. As long as the driver D2 of the backhoe 2 can identify which of the multiple sections Ra in the reference diagram W1 is being irradiated, the configuration and position of the irradiation device 63 are not particularly limited.

[0067] <Third Embodiment> In the third embodiment, from the viewpoint of allowing the driver D2 of the backhoe 2 to easily grasp the parts of the load P on the loading bed 11 that are in danger of falling, an example is provided in which an illumination unit is used to illuminate the parts of the load P on the loading bed 11 that are in danger of falling.

[0068] Figure 10 is an explanatory diagram illustrating a work site according to the third embodiment. The irradiation unit 7 is installed in a position that can irradiate light onto the cargo P on the cargo bed 11 of the vehicle 1. Specifically, the irradiation unit 7 can change the direction in which light is irradiated. The irradiation unit 7 of the third embodiment comprises, for example, an irradiation section 71 and a support device 73.

[0069] The irradiation unit 71 is an irradiation device that irradiates, for example, laser light L. However, the light emitted by the irradiation unit 71 is not limited to laser light L. The support device 73 supports the irradiation unit 71 and is rotatable in a predetermined direction. For example, a pan-rotating and tilt-rotating pan head can be used as the support device 73. By changing the angle of rotation of the support device 73 (the angle of pan rotation and the angle of tilt rotation), the direction in which the irradiation unit 7 irradiates light can be changed.

[0070] Figure 11 is a block diagram illustrating the configuration of the control system 5 according to the third embodiment. As illustrated in Figure 11, the control system 5 of the third embodiment includes, in addition to the acquisition unit 511, launch control unit 512, and display control unit 513 of the first embodiment, an irradiation control unit 515 that controls the irradiation unit 7. The control system 5 is connected to the irradiation unit 7 in a communication manner.

[0071] Specifically, the irradiation control unit 515 (an example of the "second irradiation control unit") controls the irradiation unit 7 to irradiate light onto the load P on the cargo bed 11 at a position corresponding to the fall area identified by the launch control unit 512. In the third embodiment, the irradiation control unit 515 controls the irradiation unit 7 to irradiate the load P in the section Rb corresponding to the fall area among the multiple sections Rb (9 sections Rb in Figure 10) on the loading surface of the cargo bed 11.

[0072] The irradiation control unit 515 controls the angle at which the support device 73 rotates, for example, so that the irradiation unit 7 irradiates the load P in the section Rb corresponding to the fall area. Specifically, among the multiple sections Rb on the loading surface of the loading bed 11, the load P in the section Rb corresponding to the fall area (for example, the section Rb containing the center of the fall area) is irradiated by the irradiation unit 7. Therefore, the driver D2 of the backhoe 2 can directly grasp the position of the load P loaded on the actual loading bed 11 that is at risk of falling.

[0073] Furthermore, it is sufficient that at least a portion of the load P in the section Rb corresponding to the fall area (for example, near the center of section Rb) be illuminated by the illumination unit 7. In practice, an instruction signal indicating which section Rb to illuminate (for example, a signal instructing the rotation of the support device 73 and illumination by the illumination unit 71) is transmitted to the illumination unit 7 via the communication device 53, enabling the illumination unit 7 to illuminate a specific section Rb.

[0074] Figure 12 is a flowchart of an example of the control process according to the third embodiment. The control process in Figure 12 is configured by adding steps Sc1 and Sc2 to the control process in Figure 5. The explanation of the process, which is the same as in Figure 5, will be omitted as appropriate.

[0075] When the control process starts, the acquisition unit 511 acquires a loading platform image Ga (Sa1), similar to the first embodiment, and the launch control unit 512 determines whether or not there is a risk of the load P falling from the loading platform 11 (Sa2). If there is a risk of the load P falling from the loading platform 11, that is, if a fall area is identified from the loading platform image Ga (Sa2; YES), the lock plate 3a is activated (Sa3). Then, the display control unit 513 displays the identified image Gb on the display device 54 (Sa4), and the illumination control unit 515 illuminates the load P (Sc1). Specifically, the illumination control unit 515 illuminates the load P in the section Rb of the loading surface of the loading platform 11 that corresponds to the fall area identified in the loading platform image Ga, with the illumination unit 7. The order in which steps Sa4 and Sc1 are executed is not particularly limited and they may be executed in parallel.

[0076] Similar to the first embodiment, with the lock plate 3a activated, the acquisition of the cargo bed image Ga (Sa5) and the determination of whether or not there is a risk of the cargo P falling (Sa6) are performed. If it is determined that there is a risk of the cargo P falling (Sa6; YES), steps Sa4 and Sc1 are repeatedly performed with the lock plate 3a still activated. That is, in step Sa4, similar to the first embodiment, the specific image Gb newly output in step Sa6 is displayed, and in step Sc1, if the section Rb corresponding to the falling area of ​​the specific image Gb newly output in step Sa6 has changed, the cargo P in the changed section Rb is illuminated, and if there has been no change, the cargo P in the same section Rb continues to be illuminated.

[0077] On the other hand, if it is determined that there is no risk of the load P falling (Sa6; NO), the launch control unit 512 releases the lock plate 3a (Sa7), and the irradiation control unit 515 terminates the irradiation of the load P (Sc2). The order in which steps Sa7 and Sc2 are executed is not particularly limited and they may be executed in parallel.

[0078] In the third embodiment, since light is shone on the load P on the loading platform 11 at a location corresponding to the area where the load may fall, the driver D2 of the backhoe 2 has the advantage of being able to directly grasp the part of the actual load P that is in danger of falling. The configuration of the third embodiment can also be applied to the second embodiment. In addition, the display of the specific image Gb may be omitted in the second and third embodiments.

[0079] <Fourth Embodiment> In this case, if vehicle 1 starts moving while backhoe 2 is loading cargo P, there is a risk of collision between backhoe 2 and vehicle 1, which is dangerous. Therefore, in the fourth embodiment, the lock plate 3a is kept activated even when the working part 23 of backhoe 2 is near the loading platform 11.

[0080] In the fourth embodiment, the launch control unit 512 activates the lock plate 3a not only when there is a risk of the load P falling from the loading platform 11, but also when the working part 23 of the backhoe 2 is near the loading platform 11. Specifically, the launch control unit 512 activates the lock plate 3a when it determines that the working part 23 (at least a portion) of the backhoe 2 is included in the loading platform image Ga. The generation model M is used to determine whether or not the working part 23 is included in the loading platform image Ga.

[0081] The generation model M of the second embodiment is a trained model that has been further trained by machine learning to learn the relationship between the training platform image and the work unit 23 included in the training platform image, in addition to the relationship between the training platform image and the falling area. Each of the multiple training data used in the machine learning of the generation model M is, for example, data that specifies (labels) the falling area in the training platform image and the area in the training platform image that includes the work unit 23.

[0082] Furthermore, the generative model used to determine whether or not there is a risk of the loaded goods P falling from the loading platform 11, and the generative model used to determine whether or not the loading platform image Ga contains the work area 23, may be prepared separately. In addition, methods other than using the generative model M may be employed to determine whether or not the loading platform image Ga contains the work area 23. For example, the presence or absence of the work area 23 in the loading platform image Ga may be determined by performing known image analysis processing (e.g., feature extraction) on the loading platform image Ga.

[0083] In the fourth embodiment, when a loading platform image Ga is input, the generation model M outputs a specific image Gb, for example, by adding an object O1 indicating the drop area and an object O2 indicating the area representing the work unit 23 (hereinafter referred to as the "work area") to the loading platform image Ga. Figure 13 is an example of a specific image Gb output by the generation model M of the fourth embodiment. In Figure 13, a specific image Gb is shown in which the drop area is enclosed by a solid line (O1) and the work area is enclosed by a dashed line (O2). A part of the bucket 235 of the work unit 23 is detected as the work area.

[0084] Then, when the launch control unit 512 outputs (detects) a work area using the generated model M, it determines that the work unit 23 is near the loading platform 11 and activates the lock plate 3a. The lock plate 3a continues to operate until the work area is no longer detected from the loading platform image Ga.

[0085] Figure 14 is a flowchart of an example of the control process in the fourth embodiment. The control process in Figure 14 is configured by adding steps Sd1 and Sd2 to the control process in Figure 5. The explanation of the process, which is the same as in Figure 5, will be omitted as appropriate.

[0086] When the control process starts, the acquisition unit 511 acquires a cargo bed image Ga (Sa1), similar to the first embodiment, and the launch control unit 512 determines whether or not there is a risk of the cargo P falling from the cargo bed 11 (Sa2). If there is a risk of the cargo P falling from the cargo bed 11, that is, if a fall area is identified from the cargo bed image Ga (Sa2; YES), the lock plate 3a is activated (Sa3).

[0087] On the other hand, if there is no risk of the load P falling from the loading platform 11, that is, if no area of ​​potential fall can be identified from the loading platform image Ga (Sa2; NO), the launch control unit 512 determines whether the working unit 23 of the backhoe 2 is near the loading platform 11 (Sd1). Whether the working unit 23 is included in the loading platform image Ga (whether the working area can be identified) determines whether the working unit 23 of the backhoe 2 is near the loading platform 11.

[0088] If the working section 23 of the backhoe 2 is not close to the loading platform 11 (Sd1; NO), steps Sa1 and Sa2 are repeatedly executed until the control process is instructed to end. On the other hand, if the working section 23 of the backhoe 2 is close to the loading platform 11 (Sd1; YES), the launch control unit 512 activates the locking plate 3a (Sa3). In other words, the locking plate 3a is activated if there is a risk of the load P falling, or if the working section 23 of the backhoe 2 is close to the loading platform 11.

[0089] After step Sa3, steps Sa4 and Sa5 are executed, and with the lock plate 3a activated, step Sa6 (a process to determine whether or not there is a risk of the load P falling) is executed. If it is determined that there is a risk of the load P falling (Sa6; YES), the process returns to step Sa4 with the lock plate 3a activated, similar to the first embodiment. On the other hand, if it is determined that there is no risk of the load P falling (Sa6; NO), the launch control unit 512 determines whether or not the working part 23 of the backhoe 2 is close to the loading bed 11 (Sd2).

[0090] If the working section 23 of the backhoe 2 is not close to the loading platform 11 (Sd2; NO), the launch control unit 512 releases the lock plate 3a (Sa7). On the other hand, if the working section 23 of the backhoe 2 is close to the loading platform 11 (Sd2; YES), the lock plate 3a remains engaged and the process returns to step Sa4. In other words, unless there is no risk of the load P falling and the working section 23 of the backhoe 2 is not close to the loading platform 11, the lock plate 3a will not be released and the vehicle 1 will not be able to start.

[0091] As can be understood from the above explanation, in the fourth embodiment, the lock plate 3a is activated not only when there is a risk of the load P falling, but also when the working part 23 of the backhoe 2 is close to the loading bed 11. Therefore, when the working part 23 of the backhoe 2 is close to the loading bed 11, it is possible to prevent the vehicle 1 from starting. Note that the configuration of the fourth embodiment can also be applied to the second and third embodiments.

[0092] <Variation> Each of the forms exemplified above can be modified in various ways. Specific examples of modifications that can be applied to each of the aforementioned forms are given below. Two or more forms arbitrarily selected from the following examples can be combined as appropriate, provided they do not contradict each other.

[0093] (1) In Figure 5, a configuration is shown in which the lock plate 3a is not activated at the start of the control process, and the lock plate 3a is activated when a risk of the load P falling is detected. However, a configuration may also be adopted in which the control process starts from a state where the lock plate 3a is activated, and the operation of the lock plate 3a is continued when a risk of the load P falling is detected. The launch control unit 512 functions as an element that activates the lock plate 3a (launching obstruction mechanism) when it determines from the load bed image Ga that there is a risk of the load P falling from the load bed 11. In the present invention, "activating the lock plate 3a (launching obstruction mechanism)" includes both a configuration in which the lock plate 3a is activated when it is determined that there is a risk of the load P falling from the load bed 11, and a configuration in which the operation of the lock plate 3a is continued (i.e., the operation of the lock plate 3a is not released) when it is determined that there is a risk of the load P falling from the load bed 11.

[0094] In the fourth embodiment, the configuration for activating the lock plate 3a when it is determined that the working part 23 of the backhoe 2 is included in the loading platform image Ga also includes both a configuration in which the lock plate 3a is activated when it is determined that the working part 23 of the backhoe 2 is included in the loading platform image Ga, and a configuration in which the lock plate 3a continues to operate (i.e., the lock plate 3a is not released) when it is determined that the working part 23 of the backhoe 2 is included in the loading platform image Ga.

[0095] (2) If it is determined that the cargo P is in danger of falling from the cargo bed 11, the vehicle may be further provided with a notification control unit that notifies the driver D1 of the vehicle 1 of the danger of falling to a terminal device. The terminal device is connected to the control system 5 in a communicable manner, for example, by short-range wireless communication. Specifically, the terminal device notifies the driver of the danger of falling, for example, by emitting sound. Information terminals such as tablet terminals and smartphones, or sound-emitting devices such as buzzers may be used as terminal devices. However, the form of notification is not limited to emitting sound, and notification may also be given by display.

[0096] Specifically, when the launch control unit 512 determines that there is a risk of the load P falling from the loading platform 11, the notification control unit transmits an instruction signal to the terminal device of the driver D1 of vehicle 1 to notify the driver of vehicle 1 of this fact. Upon receiving the instruction signal, the terminal device plays a predetermined sound. Therefore, the driver D1 of vehicle 1 can easily understand that there is a risk of the load P falling from the loading platform 11. The driver D1 of vehicle 1 may receive the terminal device when entering the work site and return it when leaving. Similarly, the notification control unit may notify the terminal device held by the driver D1 of vehicle 1 when the working section 23 of the backhoe 2 is near the loading platform 11.

[0097] (3) The functions of the control system 5 according to each of the above embodiments are realized through the cooperation of a computer (e.g., control device 51) and a program. A program according to a preferred embodiment of the present invention is provided in a form stored on a computer-readable recording medium and installed on the computer. The recording medium is, for example, a non-transitory recording medium, such as an optical recording medium (optical disc) like a CD-ROM, but includes any known form of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium except for transient propagation signals, and does not exclude volatile recording media. The program may also be provided to the computer in the form of distribution via a communication network. [Explanation of Symbols]

[0098] 1: Vehicle 2: Backhoe 3a: Locking plate 3b: Gate 4: Imaging device 5: Control System 6: Display Panel 11: Cargo bed 13: Wheels 21: Airframe 23: Work Unit 51: Control device 52: Storage device 53: Communication device 54:Display device 55: Operating device 211: Driver's cab 213: Running body 231: Boom 233: Arm 235: Bucket 511: Acquisition Department 512: Launch Control Unit 513: Display Control Unit D1, D2: Driver Ga: Image of the cargo bed Gb: Specific image P: Cargo

Claims

1. An acquisition unit that acquires an image of the cargo bed including the cargo loaded on the vehicle, The system comprises a starting control unit that controls a starting obstruction mechanism that hinders the starting of the vehicle, If the launch control unit determines from the image of the cargo bed that there is a risk of the loaded material falling from the cargo bed, it activates the launch obstruction mechanism. Control system.

2. If the launch control unit determines that the working part of the backhoe used for loading the cargo onto the cargo bed is included in the image of the cargo bed, it activates the launch obstruction mechanism. The control system according to claim 1.

3. The aforementioned launch control unit, By processing the acquired cargo image using a generative model that has learned the relationship between a training cargo bed image and the areas of the cargo contained in the training cargo bed image that are at risk of falling, the acquisition unit identifies the areas of falling in the cargo bed image. If the area where the goods are to fall is identified in the image of the cargo bed, it is determined that there is a risk that the goods will fall from the cargo bed. The control system according to claim 1.

4. A display panel having an installation surface with a reference diagram representing the aforementioned cargo bed attached, and an illumination device that irradiates light onto the installation surface, further comprising a first illumination control unit that controls the illumination device, The first irradiation control unit controls the irradiation device to irradiate the position corresponding to the identified fall area in the reference diagram, The display panel is positioned so that its mounting surface is visible to the driver of the backhoe used to load the cargo onto the loading platform. The control system according to claim 3.

5. The system further comprises a second irradiation control unit that controls an irradiation unit that irradiates light onto the cargo bed, The second irradiation control unit controls the irradiation unit to irradiate light onto the load on the cargo bed at the location corresponding to the identified fall area. The control system according to claim 3.

6. If it is determined that the cargo may fall from the loading platform, the vehicle further comprises a notification control unit that notifies the driver of the vehicle of the risk of the cargo falling via a terminal device. The control system according to claim 1.

7. The launch obstruction mechanism includes at least one of a locking plate or a gate. The control system according to claim 1.

8. An imaging device that captures images of the cargo bed of a vehicle, including the cargo loaded on it, A starting obstruction mechanism that prevents the vehicle from starting, The control system comprises an acquisition unit for acquiring the cargo bed image and a launch control unit for controlling the launch obstruction mechanism, If the launch control unit determines from the image of the cargo bed that there is a risk of the loaded material falling from the cargo bed, it activates the launch obstruction mechanism. Alarm system.

9. We obtain an image of the cargo bed of the vehicle, including the cargo loaded on it. Control the starting obstruction mechanism that hinders the starting of the aforementioned vehicle, In the aforementioned control, if it is determined from the image of the cargo bed that there is a risk of the loaded material falling from the cargo bed, the launch prevention mechanism is activated. A control method implemented by a computer.

10. An acquisition unit that acquires an image of the cargo bed of a vehicle, including the cargo loaded on the cargo bed, and A computer is used as a starting control unit to control a starting obstruction mechanism that hinders the starting of the vehicle. If the launch control unit determines from the image of the cargo bed that there is a risk of the loaded material falling from the cargo bed, it activates the launch obstruction mechanism. program.

Images