Periphery monitoring device

Abstract

To suppress excessive detection and erroneous detection to an obstacle as much as possible when monitoring the periphery of a facility at a work site where the facility such as a work machine is used.SOLUTION: The surroundings monitoring apparatus 100 of the present disclosure is a surroundings monitoring apparatus that monitors the surroundings of a predetermined facility based on an output from a detection means installed in the facility, and includes the control unit 303 of the image processing apparatus 300 that executes a predetermined process based on an output from the imaging apparatus 200, and the alarm apparatus 400 that outputs a predetermined alarm signal and / or a predetermined alarm. When the photographing device 200 detects an obstacle in a monitoring object area including an area which can be set by selection of a user from detectable areas by the photographing device 200, the control part 303 makes an alarm device 400 output an alarm signal and / or an alarm.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention provides Movable work machine Based on the output from the detection means installed Work machinery The present invention relates to a perimeter monitoring device that monitors the perimeter of a vehicle. [Background technology]

[0002] At work sites where work machines such as excavators and forklifts are used, efforts are being made to prevent accidents involving contact between the work machines and people in the vicinity. For example, it is known that the work machine projects light onto the floor, thereby displaying danger areas on the floor so that people in the vicinity can recognize danger areas where contact with the work machine is likely to occur. It is also known that for machines installed at fixed locations, danger areas where people are prohibited from entering are marked with markers or the like.

[0003] Meanwhile, in recent years, technologies for avoiding collisions by recognizing the external environment around the vehicle using a camera or the like have become widely used in vehicles such as automobiles.

[0004] For example, Patent Document 1 discloses a technology for a surroundings monitoring device that, when a pedestrian is detected around the vehicle, performs avoidance support control to alert the driver of the vehicle and / or control the operation of the vehicle to help the vehicle avoid contact with the pedestrian. Even if a pedestrian is detected, for example, due to factors such as stopping at a traffic light, stopping at a crosswalk, sidewalk conditions, road type, or driving lane, if the possibility of a collision with the pedestrian is low, the technology prohibits or suppresses alerting the driver and controlling the operation of the vehicle. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-143387 Summary of the Invention [Problem to be solved by the invention]

[0006]

[0003] A conventional safety measure at work sites where equipment such as work machines are used is to indicate danger areas using lights, markers, etc., but this safety measure relies on the vision of people around the equipment and is difficult to say that it adequately ensures the safety of those people. Therefore, it is conceivable to install a detection means, such as a sensor, that detects obstacles on the equipment and monitor the area around the equipment based on the output of that detection means. However, the detection means may react to obstacles that do not belong to the danger area. This could result in, for example, an alarm being issued or the equipment being shut down, which could reduce work efficiency at the work site.

[0007] On the other hand, the technology described in Patent Document 1 can suppress or prohibit unnecessary information provision and operation control in consideration of external conditions. Therefore, it appears possible to provide appropriate support according to the situation. However, because various external environments can exist at work sites where equipment such as work machines are used, simply recognizing specific external environments where information provision and operation control are suppressed or prohibited by detecting lanes, etc., may result in excessive detection or false detection of obstacles by the detection means. Furthermore, there is still room for improvement in technology for suppressing excessive detection or false detection of obstacles as much as possible when monitoring the periphery of equipment such as work machines at work sites where the equipment is used.

[0008] The purpose of this disclosure is to Movable work machine In the workplace where Work machinery The present invention provides a technology capable of minimizing excessive detection and erroneous detection of obstacles as much as possible when monitoring the periphery of a vehicle. [Means for solving the problem]

[0009] The periphery monitoring device of the present disclosure comprises: Movable work machine Based on the output from the detection means installed Work machinery The surroundings monitoring device monitors the surroundings of the vehicle, and includes a processor that executes predetermined processing based on the output from the detection means, and an alarm issuing device that outputs a predetermined alarm signal and / or a predetermined alarm. The warning signal is a signal that commands an operation to decelerate or stop the work machine, and the issuance of the alert is a warning using a warning light or a warning sound,The processor: predetermined When the detecting means detects an obstacle in the monitoring area, the alarm signal and / or the alarm is output by the alarm issuing device. The monitoring target area is set in advance by a user's selection from among areas detectable by the detection means, with the travel area and turning area of ​​the work machine being set in advance as the monitoring target area. .

[0010] Such a perimeter monitoring device includes a detection means, a processor, and an alarm issuing device, so that when an obstacle is detected by the detection means, the processor can cause the alarm issuing device to output an alarm signal and / or issue an alarm based on the information. In this case, when an obstacle is detected in the monitored area, the processor causes the above-mentioned alarm signal and / or issue an alarm. And, this monitored area is set in advance by the selection of the user of the perimeter monitoring device, Movable work machine In the workplace where Work machinery When monitoring the surroundings of a vehicle, excessive detection and false detection of obstacles can be suppressed as much as possible. The monitoring target area is set in advance by a user's selection from among areas detectable by the detection means, with the travel area and turning area of ​​the work machine being set in advance as the monitoring target area.

[0011] In the present disclosure, the detecting means is Work machinery The processor may be configured to capture an image of the surroundings of the vehicle. Work machinery Based on the captured image data representing the surrounding image of the Work machinery and receiving a selection of the area to be monitored by the user based on the distance information. In this case, the user can input the selection of the area to be monitored based on the distance information displayed on the display device, for example. ,before The processor may preset a rectangular area as the monitoring area, with the longitudinal direction being the traveling direction of the work machine. This allows the user to add necessary areas to the monitoring area while checking the monitoring area preset by the processor.

[0012] Furthermore, in such a perimeter monitoring device, the processor may update and set the monitored area while the work machine is traveling based on captured image data representing an image of the surroundings of the work machine captured by an imaging device that captures the area around the work machine. In particular, the processor may further execute the following actions while the work machine is traveling: extracting an identification image from the captured image data, the identification image including a predetermined identification that defines a predetermined non-target area; and updating and setting the monitored area while the work machine is traveling based on the extracted identification image, while excluding the area defined by the identification. Note that the above-mentioned identification may be, for example, a gate or a marker displayed on the floor. In this way, dynamically setting the monitored area while the work machine is traveling makes it possible to more effectively suppress excessive detection and erroneous detection of obstacles.

[0013] Furthermore, in the perimeter monitoring device of the present disclosure, even if the detection means detects an obstacle in the monitoring target area, the processor can prohibit the alarm issuing device from outputting the alarm signal and / or issuing the alarm when a predetermined condition is met. ,before The detecting means Work machinery When the monitoring system is configured to include an imaging device that captures images of the surrounding area and a ranging sensor, if the ranging sensor detects an obstacle in the monitored area, the processor may obtain a monitoring judgment of the obstacle by the user, and if the monitoring judgment determines that the obstacle is not a target for monitoring, may prohibit the alarm device from outputting the alarm signal and / or the alarm.

[0014] or ,before The detecting means Work machinery When the monitoring system is configured to include an imaging device that captures images of the surrounding area and a ranging sensor, the processor, when the ranging sensor detects an obstacle in the monitored area, may calculate the arrival time required for the work machine to reach the obstacle, and if the arrival time is longer than a predetermined threshold, may prohibit the alarm device from outputting the warning signal and / or the alarm.

[0015] or ,before The detecting means Work machinery When the monitoring system is configured to include an imaging device that captures images of the surrounding area and a ranging sensor, if the ranging sensor detects an obstacle in the monitored area, the processor may determine the type of obstacle, and if the obstacle is an object installed at the work site, may prohibit the alarm device from outputting the alarm signal and / or the alarm.

[0016] In addition, the processor may determine the overlap state between an obstacle detected by the detection means in the monitored area and the monitored area, and when a predetermined position of the obstacle does not overlap with the monitored area, may prohibit the alarm device from outputting the alarm signal and / or the alarm. [Effects of the Invention]

[0017] According to the present disclosure, Movable work machine In the workplace where Work machinery When monitoring the surroundings of an area, excessive detection and false detection of obstacles can be minimized. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a diagram showing a schematic configuration of a surroundings monitoring device according to a first embodiment. [Figure 2] 5 is a flowchart showing the processing flow performed by a control unit to monitor the periphery of a work machine based on output from an imaging device installed on the work machine, in the periphery monitoring device according to the first embodiment. [Figure 3] FIG. 4 is a diagram illustrating an example of distance information around a work machine. [Figure 4] 10A and 10B are diagrams for explaining the reception of a selection of a monitoring target area based on distance information. [Figure 5] FIG. 10 is a diagram showing an example in which an area in which a work machine is traveling is set in advance as a monitoring target area. [Figure 6]1 is a diagram illustrating a crane apparatus as an example of equipment whose periphery is monitored by the periphery monitoring device of the present disclosure. [Figure 7] FIG. 10 is a diagram for explaining the classification result obtained from an input to a pre-trained model in a modified example of the first embodiment, and the neural network that constitutes the pre-trained model. [Figure 8] 10 is a flowchart showing the processing flow performed by a control unit to monitor the periphery of a work machine based on output from an imaging device installed on the work machine, in a periphery monitoring device according to a modified example of the first embodiment. [Figure 9] 10 is a first flowchart showing a processing flow performed by a control unit in a surroundings monitoring device according to a second embodiment. [Figure 10] 10 is a second flowchart showing the processing flow performed by the control unit in the surroundings monitoring device according to the second embodiment. [Figure 11] 10 is a third flowchart showing the processing flow performed by the control unit in the surroundings monitoring device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The configurations of the following embodiments are examples, and the present disclosure is not limited to the configurations of the embodiments.

[0020] First Embodiment An overview of the periphery monitoring device in the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the schematic configuration of the periphery monitoring device in this embodiment. The periphery monitoring device 100 according to this embodiment is a device for monitoring the periphery of a work machine 10 (corresponding to a predetermined facility in this disclosure) based on output from an imaging device 200 (corresponding to a detection means in this disclosure) installed on the work machine 10. Here, the work machine 10 in this embodiment is a forklift. The periphery monitoring device 100 comprises the imaging device 200, an image processing device 300, and an alarm issuing device 400.

[0021] The image capturing device 200 is a device that is installed on the work machine 10 and captures images of the surroundings of the work machine 10, has a function of accepting input of images such as still images and videos, and is specifically realized by a camera that uses an image sensor such as a Charged-Coupled Device (CCD), a Metal-Oxide-Semiconductor (MOS), or a Complementary Metal-Oxide-Semiconductor (CMOS). The image capturing device 200 in this embodiment is configured as a hemispherical camera, and in this embodiment, as shown in FIG. 1(a), is installed on the top surface of the driver's cab of a forklift, which is the work machine 10, with the optical axis of the hemispherical camera facing downward. In this way, by installing a hemispherical camera on the forklift, it is possible to capture images of the surroundings of the forklift over a wide range without losing any field of view, using as few cameras as possible.

[0022] The image processing device 300 acquires captured image data representing an image of the surroundings of the work machine 10 captured by the image capturing device 200, and executes predetermined processing based on the captured image data.

[0023] Here, the image processing device 300 may be any electronic device, such as a computer, that has the processing capabilities for arithmetic processing and processing such as data acquisition, generation, and updating. That is, the image processing device 300 can be configured as a computer having a processor such as a CPU or GPU, a main storage device such as a RAM or ROM, and an auxiliary storage device such as an EPROM, a hard disk drive, or removable media. The removable media may be, for example, a USB memory or a disc recording medium such as a CD or DVD. The auxiliary storage device stores an operating system (OS), various programs, various tables, etc.

[0024] The alarm issuing device 400 is a device that outputs a predetermined warning signal and / or a predetermined alarm. Here, the predetermined alarm is a warning light, a warning sound, or the like. In this embodiment, as shown in FIG. 1(a), the alarm issuing device 400 may be, for example, a Patlite (registered trademark) disposed on the top surface of the driver's cab of the forklift. In this case, the periphery monitoring device 100 can issue a warning to the operator of the work machine 10 and surrounding people, for example, by the warning light emitted by the Patlite (registered trademark). Note that the alarm issuing device 400 may include, for example, a Patlite (registered trademark) and a speaker. In this case, the periphery monitoring device 100 can issue a warning to the operator of the work machine 10 and surrounding people by the warning light and warning sound.

[0025] In this embodiment, the work machine 10 may be slowed down or stopped in conjunction with the warning from the alarm issuing device 400. As described above, the alarm issuing device 400 is configured to be able to output a predetermined alarm signal. Here, the predetermined alarm signal is, for example, a signal that commands the above-mentioned deceleration or stopping operation, and this signal is transmitted to the controller of the work machine 10 via a Controller Area Network (CAN) or the like. In this way, as will be described later, when an obstacle is detected in the monitored area, the image processing device 300 can cause the alarm issuing device 400 to output the above-mentioned alarm signal, thereby allowing the operation of the work machine 10 to be slowed down or stopped without any driving action by the operator.

[0026] The periphery monitoring device 100 may further include a display device 500. Here, the display device 500 is a device configured to be able to display image data captured by the imaging device 200 and / or predetermined data acquired by processing based on the captured image data. Such a display device 500 is provided, for example, in the driver's cab of the work machine 10. This allows the operator of the work machine 10 to visually recognize obstacles present in the periphery of the work machine 10 via the display device 500. Also,

[0027] The periphery monitoring device 100 of the present disclosure is equipped with the photographing device 200, the image processing device 300, and the alarm issuing device 400, so that when an obstacle is detected by the photographing device 200, the image processing device 300 can output an alarm signal and / or issue an alarm from the alarm issuing device 400 based on that information. This makes it possible to avoid contact between the work machine 10 and people in the vicinity, for example.

[0028] On the other hand, at work sites where equipment such as work machines are used, if an alarm is issued or equipment operation is stopped for obstacles that do not belong to the danger area, there is a risk that work efficiency at the work site will decrease.

[0029] Therefore, in the perimeter monitoring device 100 of the present disclosure, when an obstacle is detected in the monitored area, the image processing device 300 causes the alarm issuing device 400 to output an alarm signal and / or issue an alarm. The monitored area is set in advance by the selection of the user of the perimeter monitoring device 100, so that excessive detection and erroneous detection of obstacles can be minimized when monitoring the periphery of equipment such as a work machine at a work site where the equipment is used.

[0030] Here, FIG. 1(b) is a diagram showing in more detail the components of the image processing device 300 included in the periphery monitoring device 100 in the first embodiment.

[0031] The image processing device 300 has, as functional units, a storage unit 302 and a control unit 303. A program stored in an auxiliary storage device is loaded into a working area of ​​a main storage device and executed, and each functional unit is controlled through the execution of the program, thereby realizing each function that matches the predetermined purpose of each functional unit. However, some or all of the functions may be realized by hardware circuits such as ASICs and FPGAs.

[0032] The storage unit 302 includes a main storage device and an auxiliary storage device. The main storage device is a memory in which the programs executed by the control unit 303 and the data used by the control programs are expanded. The auxiliary storage device is a device in which the programs executed by the control unit 303 and the data used by the control programs are stored.

[0033] The storage unit 302 may also store a pre-trained model, which will be described later. This pre-trained model can be used in the extraction process, which will be described later. Here, if the functions of the image processing device 300 are realized by a hardware circuit such as an FPGA, the pre-trained model may be stored in a memory built into the FPGA. The storage unit 302 also stores a monitoring target area that can be determined in advance.

[0034] The control unit 303 is a functional unit that controls the processing performed by the image processing device 300. The control unit 303 can be realized by an arithmetic processing device such as a CPU. The control unit 303 is further configured to have four functional units: an acquisition unit 3031, a setting unit 3032, a determination processing unit 3033, and an alarm control processing unit 3034. Each functional unit may be realized by the CPU executing a stored program. Note that the control unit 303 functions as a processor according to the present disclosure by executing the processing of the acquisition unit 3031, the setting unit 3032, the determination processing unit 3033, and the alarm control processing unit 3034. Then, by implementing this processor in an integrated circuit that constitutes the image processing device 300, a processor that executes the processing of the above functional units is installed in the work machine 10.

[0035] The processing flow performed by the control unit 303 will now be described with reference to Fig. 2. Fig. 2 is a flowchart showing the processing flow performed by the control unit 303 in the periphery monitoring device 100 according to this embodiment to monitor the periphery of the work machine 10 based on output from the imaging device 200 installed on the work machine 10. In this embodiment, execution of this flow begins when the periphery monitoring device 100 is powered on.

[0036] In this flow, when the acquisition unit 3031 acquires the area to be monitored, distance information is first acquired in S101. At this time, the acquisition unit 3031 acquires distance information around the work machine 10 based on captured image data that represents an image of the surroundings of the work machine 10 captured by the image capture device 200. Note that, for example, if the image capture device 200 is a stereo camera, such distance information can be acquired based on the parallax of the stereo image data. Also, for example, if the image capture device 200 is a hemispherical camera (monocular camera), distance information can be acquired based on the installation specifications of the camera, such as the installation height and installation angle.

[0037] Here, Fig. 3 is a diagram illustrating distance information around the work machine 10. As shown in Fig. 3, the distance information can be generated, for example, as a mesh map with intervals of 1 m. Then, the acquisition unit 3031 displays such distance information on the display device 500, allowing the user of the periphery monitoring device 100 to check a map such as the example shown in Fig. 3 via the display device 500.

[0038] 2, the acquisition unit 3031 acquires an input of a monitoring target area from the user in S102. Here, in this embodiment, the acquisition unit 3031 acquires an input of a monitoring target area from the user by accepting a selection of a monitoring target area by the user based on the distance information.

[0039] 4 is a diagram for explaining the reception of the selection of the area to be monitored based on the distance information. The area to be monitored includes an area that can be set by the user's selection from among the areas detectable by the image capture device 200, and the user can input the selection of the area to be monitored based on the distance information displayed on the display device 500.

[0040] 4, the user can select the area to be monitored by excluding installations at the work site, such as shelves, included in the area detectable by the image capture device 200, more specifically, by excluding the area on the map that includes the installations at the work site. The grayed-out portion in FIG. 4 is the area selected by the user as the area to be monitored.

[0041] 2, next, in S103, the setting unit 3032 sets the monitoring target area based on the above-mentioned selection by the user. At this time, the setting unit 3032 may set the travel area and turning area of ​​the work machine 10 as the monitoring target area in advance. The monitoring target area that can be set in advance in this manner is stored in the memory unit 302.

[0042] Fig. 5 is a diagram showing an example in which the travel area of ​​the work machine 10 is set in advance as the monitoring target area. In this case, the setting unit 3032 can set in advance as the monitoring target area a rectangular area whose longitudinal direction is the travel direction of the work machine 10, as shown in Fig. 5. The part displayed in gray in Fig. 5 is the area set in advance by the setting unit 3032 as the monitoring target area, and in the processing of S102, when distance information is displayed to the user on the display device 500, this rectangular area is displayed in a pre-selected state. Then, in the processing of S102 described above, the user can add required areas to the monitoring target area while checking the monitoring target area set in advance in this way.

[0043] 2, next, in S104, the determination processing unit 3033 determines whether or not the equipment, such as the work machine 10, is in operation. If a positive determination is made in S104, that is, if the equipment is in operation, the determination processing unit 3033 proceeds to the processing of S105, and if a negative determination is made in S104, that is, if the equipment is not yet in operation, the determination processing unit 3033 repeatedly executes the processing of S104.

[0044] If a positive determination is made in S104, then in S105 it is determined whether or not an obstacle has been detected by the image capture device 200. Here, the determination processing unit 3033 can detect obstacles around the work machine 10, for example, by comparing the travel road surface with the height coordinate value of the detected object. Alternatively, the determination processing unit 3033 may detect obstacles around the work machine 10 based on well-known technology using captured image data from the image capture device 200. Then, if a positive determination is made in S105, the determination processing unit 3033 proceeds to processing of S106, and if a negative determination is made in S105, the determination processing unit 3033 proceeds to processing of S108.

[0045] If the determination in S105 is affirmative, then in S106 it is determined whether or not the obstacle detected by the processing in S105 belongs to the area to be monitored. If the determination in S106 is affirmative, the process proceeds to S107, and if the determination in S106 is negative, the process proceeds to S108.

[0046] If a positive determination is made in S106, then in S107 the alarm control processing unit 3034 executes alarm control processing. In this embodiment, the alarm control processing unit 3034 causes the alarm issuing device 400 (for example, PATLITE (registered trademark)) to emit a warning light. Note that if the alarm issuing device 400 is configured, for example, by PATLITE (registered trademark) and a speaker, the alarm control processing unit 3034 may cause the PATLITE (registered trademark) to emit a warning light and the speaker to output an alarm sound. Furthermore, in this embodiment, in addition to the above, processing may be executed to operate the operation control device of the work machine 10 so that the operation of the work machine 10 is stopped.

[0047] If a negative determination is made in S105 or S106, or after the processing of S107, it is then determined in S108 whether or not the operation of the equipment has ended. If a positive determination is made in S108, the execution of this flow is ended, and if a negative determination is made in S108, the processing returns to S105.

[0048] According to the above-described process, even if an obstacle is detected by the process of S105, if the obstacle is not within the monitored area, that is, if a negative determination is made in the process of S106, the alarm control process of S107 is not executed. This makes it possible to minimize situations where an alarm is issued or the operation of equipment is stopped in response to an obstacle that does not belong to the monitored area.

[0049] In the above embodiment, an example has been described in which the perimeter monitoring device 100 monitors the perimeter of a forklift. However, the equipment whose perimeter is monitored by the perimeter monitoring device of the present disclosure is not limited to a work machine such as a forklift, and the equipment may be, for example, a crane. Fig. 6 is a diagram illustrating an example of a crane as an equipment whose perimeter is monitored by the perimeter monitoring device of the present disclosure. The crane is, for example, an overhead crane as shown in Fig. 6.

[0050] Furthermore, there is no intention to limit the detection means of the present disclosure to the image capturing device 200 described in the above embodiment, and the detection means of the present disclosure may be, for example, a distance measuring sensor capable of detecting an obstacle.

[0051] As described above, according to this embodiment, when monitoring the periphery of equipment such as a work machine at a work site where the equipment is used, excessive detection and erroneous detection of obstacles can be minimized.

[0052] <Modification of the first embodiment> A modification of the first embodiment will now be described. In this modification, the monitoring target area is dynamically updated and set while the work machine 10 is traveling.

[0053] In more detail, the setting unit 3032 described in the first embodiment above updates and sets the monitoring area while the work machine 10 is traveling. At this time, the setting unit 3032 updates and sets the monitoring area while the work machine 10 is traveling, based on the extracted identification image, while excluding the area defined by identification.

[0054] Here, the above-mentioned identification image is an image including a predetermined identification that defines a predetermined non-target area, and is extracted from the captured image data by an extraction processing unit, which is a functional unit of the control unit 303 of the image processing device 300. The above-mentioned non-target area is, for example, a pedestrian-only area. The above-mentioned identification is, for example, a gate or a marker displayed on the floor.

[0055] The extraction processing unit can then extract the non-target area using a pre-trained model, which is constructed by performing training using data including images representing a work site.

[0056] FIG. 7 illustrates the classification results obtained from input to a pre-training model in this modification and the neural network that constitutes the pre-training model. In this modification, a neural network model generated by deep learning is used as the pre-training model. The pre-training model 30 in this modification includes an input layer 31 that receives input of predetermined image data, an intermediate layer (hidden layer) 32 that extracts features representing non-target areas from the image data input to the input layer 31, and an output layer 33 that outputs a classification result based on the features. In the example of FIG. 7, the pre-training model 30 includes one intermediate layer 32, with the output of the input layer 31 input to the intermediate layer 32 and the output of the intermediate layer 32 input to the output layer 33. However, the number of intermediate layers 32 does not need to be limited to one; the pre-training model 30 may include two or more intermediate layers 32.

[0057] 7, each of the layers 31 to 33 includes one or more neurons. For example, the number of neurons in the input layer 31 can be set according to the input image data. The number of neurons in the output layer 33 can be set according to the classification result.

[0058] Neurons in adjacent layers are then connected as appropriate, and weights (connection loads) are set for each connection based on the results of machine learning. In the example of Figure 7, each neuron is connected to all neurons in the adjacent layer, but the neuron connections are not limited to this example and can be set as appropriate.

[0059] Such a pre-training model 30 is constructed by performing supervised learning using training data, which is a combination of image data including images representing the gates and markers described above and labels of images representing non-target areas. Specifically, the combination of features and labels is provided to a neural network, and the weights of the connections between neurons are tuned so that the output of the neural network is the same as the label. In this way, the features of the training data are learned and a pre-training model for estimating results from inputs is inductively acquired.

[0060] The pre-trained model 30 may also be constructed by unsupervised learning. For example, domain adaptation, which is a type of transfer learning, can be used for unsupervised learning. This allows a pre-trained model to be acquired without preparing a large amount of labeled training data.

[0061] FIG. 8 is a flowchart showing the processing flow carried out by the control unit 303 in the surroundings monitoring device 100 according to this modified example to monitor the periphery of the work machine 10 based on the output from the imaging device 200 installed on the work machine 10.

[0062] In this flow, an identification image extraction process is executed in S1041 after the process of S104 shown in Fig. 2. This identification image is extracted from the captured image data by the extraction processing section, as described above.

[0063] Next, in S1042, the monitoring target area is updated. In the processing of S1042, the setting unit 3032 updates and sets the monitoring target area while the work machine 10 is traveling, based on the extracted identification image, while excluding the area defined by the identification.

[0064] After the process of S1042, the process of S105 shown in FIG. 2 is executed.

[0065] This allows the monitoring area to be dynamically updated and set while the work machine 10 is traveling, and when monitoring the periphery of equipment such as work machines at a work site where the equipment is used, excessive detection and erroneous detection of obstacles can be minimized.

[0066] Second Embodiment A second embodiment will be described. In the periphery monitoring device according to this embodiment, even if an obstacle is detected in the monitoring area, output of an alarm signal and / or notification by the alarm issuing device is prohibited when a predetermined condition is met.

[0067] Here, in this embodiment, as in the first embodiment, the work machine 10 is a forklift, and the periphery monitoring device 100 comprises an imaging device 200, an image processing device 300, and an alarm issuing device 400.

[0068] The control unit 303 of the image processing device 300 prohibits the alarm device 400 from outputting an alarm signal and / or issuing an alarm when a predetermined condition is met, even if an obstacle is detected in the area to be monitored.

[0069] FIG. 9 is a first flowchart showing the processing flow performed by the control unit 303 in the surroundings monitoring device 100 according to this embodiment.

[0070] In the flow shown in Fig. 9, if a positive determination is made in the process of S106 shown in Fig. 2 above, it is determined in S1061 whether or not an obstacle has been detected by the distance measurement sensor. Here, in the periphery monitoring device 100 in which this flow is executed, a distance measurement sensor is used together with the above-mentioned image capture device 200 as a detection means. Note that the distance measurement sensor is, for example, a so-called LiDAR (Light Detection and Ranging) that measures the distance to an object in the target space by irradiating the object with laser light and receiving light reflected from the object, and is capable of detecting obstacles that are relatively far away. Then, if a positive determination is made in S1061, the process proceeds to S1062, and if a negative determination is made in S1061, the process proceeds to S107.

[0071] If a positive determination is made in S1061, then the arrival time is calculated in S1062. This arrival time is the time it takes for the work machine 10 to reach an obstacle detected by a distance measurement sensor, and can be calculated based on the distance to the obstacle measured by the distance measurement sensor and the relative speed of the obstacle with respect to the work machine 10.

[0072] Next, in S1063, it is determined whether the arrival time calculated in the processing of S1062 is longer than a predetermined threshold. Here, the threshold can be set as the time required for the operator of the work machine 10 to recognize an obstacle and take action, and is, for example, 5 seconds. Then, if a positive determination is made in S1063, the process proceeds to the processing of S108 shown in Figure 2 above, and if a negative determination is made in S1063, the process proceeds to the processing of S107.

[0073] If the determinations at S1061 and S1063 are negative, the alarm control process is executed at S107. Details of the process at S107 are as described above in the description of Fig. 2. After the process at S107, the process proceeds to the process at S108 shown in Fig. 2.

[0074] 9, when the time it takes for the work machine 10 to reach an obstacle detected by the distance measurement sensor is longer than a predetermined threshold, the output of an alarm signal and / or alarm by the alarm issuing device 400 is prohibited. This prevents excessive detection of obstacles, thereby minimizing situations where work efficiency is reduced due to the output of an alarm signal or alarm.

[0075] FIG. 10 is a second flowchart showing the processing flow performed by the control unit 303 in the surroundings monitoring device 100 according to this embodiment.

[0076] In the flow shown in Fig. 10, if a positive determination is made in the process of S106 shown in Fig. 2 above, it is determined in S1061 whether or not an obstacle has been detected by the distance measurement sensor. The process of S1061 is the same as that in Fig. 9 above. If a positive determination is made in S1061, the process proceeds to S1064, and if a negative determination is made in S1061, the process proceeds to S107.

[0077] If a positive determination is made in S1061, then in S1064, the type of obstacle is determined. In the processing of S1064, for example, a pre-learning model constructed by performing learning using data including images of multiple obstacles at the work site is used to determine whether the obstacle is a person or a non-human object. Note that a non-human object may be determined based on a map of the object locations at the work site and the current position of the work machine 10.

[0078] Next, in S1065, it is determined whether the obstacle determined in the processing of S1064 is an object installed at the work site. In the processing of S1065, for example, if the type of obstacle determined in the processing of S1064 is an object other than a person, and the relative speed of the obstacle with respect to the work machine 10 matches the absolute speed of the work machine 10, the obstacle is determined to be an object installed at the work site. Then, if a positive determination is made in S1065, the processing proceeds to S108 shown in Figure 2 above, and if a negative determination is made in S1065, the processing proceeds to S107.

[0079] If a negative determination is made in S1061 and S1065, the alarm control process is executed in S107. Details of the process of S107 are as described above in the description of Fig. 2. After the process of S107, the process proceeds to the process of S108 shown in Fig. 2 above.

[0080] 10, when the obstacle detected by the distance measurement sensor is an object installed at the work site, the output of an alarm signal and / or an alarm by the alarm issuing device 400 is prohibited. This prevents erroneous detection of an obstacle, thereby minimizing situations where work efficiency is reduced due to the output of an alarm signal or an alarm.

[0081] 10 has been described as an example in which the type of obstacle is determined, but the user may determine whether the obstacle is a monitoring target. In this case, when the distance measuring sensor detects an obstacle in the monitoring target area, the control unit 303 acquires a monitoring determination of the obstacle made by the user. Then, when the monitoring determination determines that the obstacle is not a monitoring target, the control unit 303 prohibits the alarm device 400 from outputting an alarm signal and / or issuing an alarm.

[0082] FIG. 11 is a third flowchart showing the processing flow performed by the control unit 303 in the surroundings monitoring device 100 according to this embodiment.

[0083] In the flow shown in Fig. 11, if a positive determination is made in the processing of S106 shown in Fig. 2 above, a determination is made in S1066 as to the manner in which the obstacle detected by the image capture device 200 is superimposed on the area to be monitored. In the processing of S1066, for example, the positions of the skeletal features of a nearby person are identified from captured image data that represents an image of the surroundings of the work machine 10 captured by the image capture device 200, and the manner in which the positions of the skeletal features of the nearby person are superimposed on the area to be monitored is determined. Note that the positions of the skeletal features of the nearby person can be identified using a pre-learning model that has been trained using training data that is a combination of image data including an image representing a person and image labels that represent the positions of the skeletal features of the person.

[0084] Next, in S1067, it is determined whether a predetermined position on the obstacle overlaps with the area to be monitored. Here, the predetermined position on the obstacle is, for example, the torso, which is a skeletal part of a nearby person. Also, the predetermined position on the obstacle is, for example, the center of gravity or central position of the object. Then, if a positive determination is made in S1067, the process proceeds to S108 shown in FIG. 2 above, and if a negative determination is made in S1067, the process proceeds to S107.

[0085] If a negative determination is made in S1067, the alarm control process is executed in S107. Details of the process of S107 are as described above in the description of Fig. 2. After the process of S107, the process proceeds to the process of S108 shown in Fig. 2 above.

[0086] 11, when the predetermined position of an obstacle is not included in the monitored area, the output of an alarm signal and / or an alarm by the alarm issuing device 400 is prohibited. This prevents excessive detection of obstacles, thereby minimizing situations where work efficiency is reduced due to the output of an alarm signal or an alarm.

[0087] <Other variations> The above-described embodiment is merely an example, and the present disclosure may be modified as appropriate within the scope of the present disclosure. For example, the processes and means described in the present disclosure may be freely combined as long as no technical contradiction occurs.

[0088] Furthermore, the processing described as being performed by one device may be shared and executed by multiple devices. For example, the acquisition unit 3031 may be formed in an integrated circuit separate from the image processing device 300. In this case, the separate integrated circuit is configured to be able to cooperate favorably with the image processing device 300. Furthermore, the processing described as being performed by different devices may be executed by a single device. In an integrated circuit, the hardware configuration by which each function is realized can be flexibly changed.

[0089] The present disclosure can also be realized by providing a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer on a non-transitory computer-readable storage medium connectable to the computer's system bus. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (e.g., a floppy disk, a hard disk drive (HDD), etc.), an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, or any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0090] 10. Work machinery 100 Periphery monitoring device 200....imaging device 300 Image processing device 303 Control section 400 Alarm device

Claims

1. A periphery monitoring device that monitors the periphery of a travelable work machine based on an output from a detection means installed on the work machine, a processor that executes a predetermined process based on the output from the detection means; an alarm device that outputs a predetermined alarm signal and / or a predetermined alarm; Equipped with The warning signal is a signal that commands an operation to decelerate or stop the work machine, The issuance of the alert is a warning using a warning light or a warning sound, The processor: When the detection means detects an obstacle in a predetermined monitoring area, the alarm signal and / or the alarm is output by the alarm issuing device; the monitoring target area is set in advance by a user's selection from among areas detectable by the detection means, with the travel area and turning area of ​​the work machine being set in advance as the monitoring target area; Perimeter monitoring device.

2. The detection means is an imaging device that captures images of the surroundings of the work machine, The processor: acquiring distance information around the work machine based on captured image data representing an image of the surroundings of the work machine captured by the imaging device; and accepting a selection of the monitoring target area by the user based on the distance information. The periphery monitoring device according to claim 1 .

3. The detection means is a photographing device that photographs the surroundings of the work machine, The processor: extracting an identification image including a predetermined identification that defines a predetermined non-target area from captured image data representing an image of the surroundings of the work machine captured by the imaging device while the work machine is traveling; and updating and setting the monitoring target area while excluding the area defined by the identification based on the extracted identification image while the work machine is traveling. The periphery monitoring device according to claim 1 .

4. The processor: Even if the detection means detects an obstacle in the monitored area, when a predetermined condition is met, output of the alarm signal and / or the notification by the notification device is prohibited. The periphery monitoring device according to claim 1 .

5. The detection means is configured to include an imaging device that captures images of the surroundings of the work machine and a distance measuring sensor, The processor: When the distance measuring sensor detects an obstacle in the monitored area, a monitoring determination of the obstacle by the user is obtained; When it is determined in the monitoring determination that the obstacle is not a monitoring target, output of the alarm signal and / or the notification by the notification device is prohibited. The periphery monitoring device according to claim 4.

6. The detection means is configured to include an imaging device that captures images of the surroundings of the work machine and a distance measuring sensor, The processor: When the distance measuring sensor detects an obstacle in the monitored area, a time required for the work machine to reach the obstacle is calculated; When the arrival time is longer than a predetermined threshold, the output of the alarm signal and / or the alarm by the alarm issuing device is prohibited. The periphery monitoring device according to claim 4.

7. The detection means is configured to include an imaging device that captures images of the surroundings of the work machine and a distance measuring sensor, The processor: If the distance measuring sensor detects an obstacle in the monitored area, the type of the obstacle is determined; If the obstacle is an object installed at the work site, output of the alarm signal and / or the alarm by the alarm issuing device is prohibited. The periphery monitoring device according to claim 4.

8. The processor: determining an overlapping state between the obstacle detected by the detection means in the monitoring target area and the monitoring target area; When the predetermined position of the obstacle and the monitoring target area do not overlap, output of the alarm signal and / or the alarm by the alarm issuing device is prohibited. The periphery monitoring device according to claim 1 .

Images