Periphery monitoring device for work machine

Abstract

To provide a periphery monitoring device for a work machine capable of improving productivity while ensuring safety of the work machine.SOLUTION: A periphery monitoring device SMA for a work machine comprises a plurality of object detection devices ODD that detects an object around the work machine, and a calibration unit 211 that calibrates the coordinate systems of these object detection devices ODD. The calibration unit 211 sequentially executes a plurality of calibration algorithms for calibration in increasing order of processing time or manual operation until the calibration of the coordinate systems of the object detection devices ODD is completed.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a peripheral monitoring system for a work machine.

Background Art

[0002] Conventionally, a surrounding display system and a work machine equipped with the same have been known (see Patent Document 1). The surrounding display system described in Patent Document 1 includes a controller that generates a composite image based on composite conditions and outputs the composite image to a display device.

[0003] In a plurality of images captured by a plurality of cameras, the controller respectively identifies the positions of calibration markers having a predetermined shape for use in calibration processing. Further, the controller calculates the coordinates in the vehicle body coordinate system of the feature points of the calibration marker for each of the plurality of images. Further, the controller calculates the coordinate differences of the feature points of the calibration marker in the plurality of images, determines whether calibration processing is necessary based on the coordinate differences, and outputs the determination result to the display device.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Since a work machine includes, for example, a work attachment that moves or transports loads or loaded objects, the positions and postures of a plurality of object detection devices mounted on the work machine and detecting surrounding objects may vary. Therefore, the work machine needs calibration for fusing the detection results of the plurality of object detection devices at the work site.

[0006] In the conventional surrounding display system described in Patent Document 1, in the above-described calibration process, there is a risk that the burden on on-site workers increases, or it takes an unexpectedly long time, resulting in a decrease in productivity.

[0007] The present disclosure provides a peripheral monitoring device for a work machine that can improve productivity while ensuring the safety of the work machine.

Means for Solving the Problems

[0008] One aspect of the present disclosure includes a plurality of object detection devices that detect objects around a work machine, and a calibration unit that calibrates the coordinate systems of the plurality of object detection devices. The calibration unit sequentially executes a plurality of calibration algorithms for the calibration in ascending order of processing time or manual operation until the calibration is completed, and provides a peripheral monitoring device for a work machine.

Advantages of the Invention

[0009] According to the above aspect of the present disclosure, it is possible to provide a peripheral monitoring device for a work machine that can improve productivity while ensuring the safety of the work machine.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of a peripheral monitoring device for a working machine according to the present disclosure will be described with reference to the drawings.

[0012] [Embodiment 1] FIG. 1 is a side view of a crane 1 which is an example of a working machine. FIG. 2 is a schematic rear view of the crane 1 shown in FIG. 1. FIG. 3 is a block diagram of the devices mounted on the crane 1 shown in FIG. 1.

[0013] Although details will be described later, the peripheral monitoring device SMA of the working machine in the present embodiment is characterized in that it has the following configuration. The peripheral monitoring device SMA of the working machine includes a plurality of object detection devices ODD for detecting objects existing around the working machine such as the crane 1, and a calibration unit 211 for calibrating the coordinate systems of those object detection devices ODD. The calibration unit 211 sequentially executes a plurality of calibration algorithms for calibration in ascending order of less processing time or manual operation until the calibration of the coordinate system of the object detection device ODD is completed.

[0014] The working machines on which the peripheral monitoring device SMA is mounted include, in addition to the crane 1, for example, hydraulic excavators, road machines, foundation construction machines, transport and handling machines, forklifts, wheel loaders, dump trucks, and the like. Further, working machines such as the crane 1 and the hydraulic excavator include, for example, work attachments for moving or transporting loads or cargos. The work attachment includes, for example, a jib, a wire, and a lifting tool, or a boom, an arm, an end attachment, and a hydraulic actuator, and the like.

[0015] First, an example of the configuration of a crane 1, which is an example of a working machine, will be described below. Then, the details of the peripheral monitoring device SMA of the working machine according to the present embodiment will be described.

[0016] As shown in FIG. 1, the crane 1 is a so-called mobile crawler crane. Specifically, the crane 1 includes a crawler-type lower traveling body 2 that can self-propel, and an upper revolving body 3 that is rotatably mounted on the lower traveling body 2.

[0017] Hereinafter, the front-rear, left-right directions as viewed from the operator of the crane 1 will be described as the front-rear, left-right directions of the crane 1. In addition, unless otherwise specified, in principle, the front-rear direction of the crane 1 will be described on the assumption that the lower traveling body 2 is in a state where the front-rear direction coincides with that of the upper revolving body 3 (reference posture). Also, the vertical direction in the state where the crane 1 is placed on a horizontal plane may be referred to as the vertical direction.

[0018] A boom 4 is attached to the front side of the upper revolving body 3 so as to be able to rise and fall. A counterweight 5 for balancing the weight of the boom 4 and the suspended load is attached to the rear part of the upper revolving body 3. A cabin 6 where an operator sits to operate the crane 1 is arranged at the front right part of the upper revolving body 3.

[0019] The raising and lowering operation of the boom 4 is performed by winding or unwinding a wire rope (raising and lowering rope) 7 by a raising and lowering winch (not shown). One end of a hoisting rope 8 is connected to a hook 10 at the tip (upper end) of the boom 4, and the hook 10 is suspended from the tip of the boom 4. The other end of the hoisting rope 8 is wound around a hoisting winch (not shown) on the upper revolving body 3, and the hook 10 moves up and down by driving the hoisting winch.

[0020] On the main frame 11 located on the lower surface side of the upper revolving body 3, a first measuring device mounting portion 9 is provided at a position avoiding the slewing bearing 12. And on the first measuring device mounting portion 9, a reference sensor 13 (for example, the first LiDAR 51a which is a 360° measurable full-circle LiDAR 51 shown in FIG. 2) as an object detection device ODD for detecting an object around the crane 1 is fixed (for example, fastened and fixed with screws). Note that the reference sensor 13 may include one or more cameras 52 shown in FIG. 2.

[0021] The first LiDAR 51a as the reference sensor 13 is capable of measuring 360° around the sensor central axis extending in the vertical direction of the lower surface of the main frame 11. And the reference sensor 13 irradiates light of a predetermined irradiation pattern at predetermined intervals while moving the laser light from the space generated between the lower traveling body 2 and the upper revolving body 3 to the external space of the crane 1, and widely measures the position information of surrounding objects (not only the position information of one point of the surrounding objects, but also the three-dimensional information including the position and shape (posture) of the surrounding objects). Also, the sensor central axis is a rotation axis parallel to the axis of the slewing center of the upper revolving body 3. Also, a plurality of the first measuring device mounting portions 9 may be provided.

[0022] Here, the predetermined irradiation pattern refers to the way of moving the position where the light is irradiated and the frequency of detecting the irradiated light. For example, as one pattern, there is a way of moving the irradiation position of the light vertically to the upper end portion of the measurement range, and after moving to the upper end portion of the measurement range, moving it a predetermined distance left and right and then moving it to the lower end portion of the measurement range repeatedly (irradiating in a zigzag in the vertical direction). Also, as another pattern, there is a way of moving the irradiation position of the light horizontally to the right end portion of the measurement range, and after coming to the right end portion, moving it a predetermined distance up and down and then moving it horizontally to the left end portion of the measurement range repeatedly (irradiating in a zigzag in the horizontal direction). As another pattern, when the position where the light is irradiated moves in the horizontal direction, there is also a way of moving it horizontally, and when it moves in the vertical direction, moving it obliquely with respect to the vertical direction. Thus, there are various irradiation patterns.

[0023] Further, a plurality of optional sensors 14 (for example, the second and third LiDARs 51b and 51c and one or more cameras 52 shown in FIG. 2) as object detection devices ODD for detecting objects around the crane 1 are detachably installed on the side surface of the upper slewing body 3 and the side surface, lower surface, or upper surface of the counterweight 5. This optional sensor 14 may include a LiDAR 51 that measures the position information of objects around the crane 1 over a wide range (not only the position information of a single point of the surrounding objects but also three-dimensional information including the position and shape (posture) of the surrounding objects). This LiDAR 51 irradiates the external space with laser light in a predetermined irradiation pattern different from that of the reference sensor 13 and measures the position information of the surrounding objects. Note that the irradiation patterns of the LiDAR 51 as the reference sensor 13 and the LiDAR 51 as the optional sensor 14 may be the same.

[0024] This optional sensor 14 is detachably arranged, for example, at an arbitrary position on the upper slewing body 3 or the counterweight 5 of the crane 1 by a magnet, and can be easily moved only by hand without relying on tools. Thereby, even if the place (the place the operator wants to see) where the operator wants to measure changes according to the situation at the site, the installation location can be freely changed. Note that the means for detachably attaching the optional sensor 14 to the upper slewing body 3 or the like is not limited to a magnet. For example, elastic deformable gripping members such as spring members are accommodated in a plurality of holes (mounting portions) previously provided in the counterweight 5 and the upper slewing body 3, and the mounting shaft integrated with the optional sensor 14 is pushed into the hole (mounting portion) until there is a feeling of attachment (until it clicks), so that the mounting shaft of the optional sensor 14 is elastically clamped by the gripping member in the hole (mounting portion). In this case, the optional sensor 14 can push its mounting shaft into the clamping member only by hand (without using tools), and can also pull out its mounting shaft from the clamping member.

[0025] The reference sensor 13 is arranged at a position near the rear end of the main frame 11 of the upper swing body 3 and on the center line extending in the front-rear direction passing through the swing center of the upper swing body 3. Taking this sensor central axis (rotation axis center) as the reference position of the reference sensor 13, the case where the posture line indicating the posture of the reference sensor 13 is located along the center line extending in the front-rear direction passing through the swing center of the upper swing body 3 is defined as the reference posture of the reference sensor 13. The position of this reference sensor 13 is a predetermined position and is recorded in the storage unit 212 described later. Note that the reference sensor 13 is not limited to the position shown in FIG. 1, and can be fixed at any position as long as the fixed position is a predetermined position and is the position recorded in the storage unit 212 described later.

[0026] As shown in FIG. 3, the peripheral monitoring device SMA of the working machine according to the present embodiment includes, for example, a plurality of object detection devices ODD that detect objects around the crane 1 as the working machine, and a calibration unit 211 that calibrates the coordinate systems of the plurality of object detection devices ODD. The calibration unit 211 is constituted by, for example, a controller 21 mounted on the upper swing body 3 of the crane 1.

[0027] Here, the plurality of object detection devices ODD include, for example, the aforementioned reference sensor 13 or the LiDAR 51 or camera 52 as the arbitrarily installed sensor 14. The LiDAR 51 or camera 52 as the reference sensor 13 can be used as the calibration reference for the LiDAR 51 or camera 52 as the arbitrarily installed sensor 14 to be calibrated, for example. Also, the LiDAR 51 or camera 52 as the arbitrarily installed sensor 14 after calibration can be used as the calibration reference for the LiDAR 51 or camera 52 before calibration, for example.

[0028] Further, the peripheral monitoring device SMA of the working machine may include, for example, an input device 22, a sound output device 23, and a display device 24. These input device 22, sound output device 23, and display device 24 are installed, for example, around the driver's seat where the operator sits in the cabin 6 of the upper swing body 3. The input device 22 is composed of, for example, a switch, a button, a lever, a pedal, a touch panel, a keyboard, a mouse, etc. The sound output device 23 is composed of, for example, a buzzer, a speaker, etc. The display device 24 is composed of, for example, a liquid crystal display, an organic EL display, etc.

[0029] The controller 21 is composed of, for example, a CPU (Central Processing Unit) etc., and controls the operations of each part of the crane 1. The controller 21 includes the function of an ECU (Electronic Control Unit) and is arranged on the upper swing body 3. Specifically, the controller 21 operates the crane 1 based on an operation input from the operator's input device 22 etc., develops various programs pre-stored in the storage unit 212, reads out various data, and executes various processes using the developed programs and the read-out various data.

[0030] The controller 21 has the functions of a calibration unit 211, a storage unit 212, and an output unit 213, which are realized, for example, by the CPU executing various programs. Note that each part of these controllers 21 may be realized by, for example, common hardware, or may be realized by a plurality of different hardwares.

[0031] The calibration unit 211 sequentially executes a plurality of calibration algorithms for the calibration in ascending order of processing time or manual operation until the calibration of the object detection device ODD is completed. The storage unit 212 stores various data and programs including a plurality of calibration algorithms, for example, in a non-volatile storage device constituting the controller 21 or a non-volatile storage device connected to the controller 21. The output unit 213 synthesizes and outputs the detection results of the object by a plurality of object detection devices ODD.

[0032] FIG. 4 is a block diagram showing details of the calibration unit 211 in FIG. 3. FIG. 5 is a flowchart showing the flow of the calibration process by the calibration unit 211 in FIGS. 3 and 4. FIG. 6 is a schematic plan view showing an arrangement example of the calibration object in the calibration process of FIG. 5.

[0033] The calibration unit 211 has, for example, as shown in FIG. 4, a target extraction unit 211a, a feature extraction unit 211b, a deviation amount determination unit 211c, and a coordinate conversion unit 211d. Each part of these calibration units 211 also represents a function realized by executing various programs by, for example, the CPU of the controller 21.

[0034] Since the working machine includes, for example, a work attachment that moves or transports loads and loaded objects, the positions and postures of a plurality of object detection devices ODD mounted on the working machine and detecting surrounding objects may vary. Specifically, when the position of the object detection device ODD due to, for example, vibration during operation of the working machine such as the crane 1 or during disassembly and assembly of the crane 1 occurs, an operator such as an operator or maintenance personnel of the working machine such as the crane 1 starts the flow of the calibration process shown in FIG. 5. Before starting the flow of the calibration process, for example, the operator arranges calibration objects in the overlapping detection range A12, which is the overlapping detection range of the plurality of object detection devices ODD, as shown in FIG. 6.

[0035] Specifically, in the example shown in FIG. 6, the detection range A1 of the first camera 52a as the object detection device ODD and the detection range A2 of the second camera 52b as the object detection device ODD share the overlapping detection range A12. And in this overlapping detection range A12, as calibration objects 37, 38, 39, for example, a person such as a field worker is arranged. The number of people as the calibration objects 37, 38, 39 is not particularly limited. Also, when calibrating two LiDARs 51 as the object detection device ODD or calibrating the LiDAR 51 and the camera 52, the calibration objects 37, 38, 39 can be arranged in the overlapping detection range, which is the range where their detection ranges overlap, in the same manner as the example shown in FIG. 6.

[0036] An operator who calibrates the coordinate systems of multiple object detection devices ODD places calibration objects 37, 38, and 39 in the overlapping detection range, and then inputs an instruction to start the calibration process to the controller 21 via, for example, the input device 22. When an instruction to start the calibration process is input via the input device 22, for example, the controller 21 starts the processing flow shown in FIG. 5 and determines whether calibration is necessary by the calibration unit 211 (process P01).

[0037] In this process P01, the calibration unit 211 acquires, for example, from multiple object detection devices ODD having an overlapping detection range A12 as shown in FIG. 6, the detection results of objects including the detection results of the calibration objects 37, 38, and 39 by the target extraction unit 211a. Further, the target extraction unit 211a extracts the detection results of the calibration objects 37, 38, and 39 arranged in the overlapping detection range A12 from the detection results of each acquired object detection device ODD.

[0038] Thereafter, the calibration unit 211 extracts feature points by the feature extraction unit 211b using, for example, the detection results of the objects 37, 38, and 39 by each object detection device ODD extracted by the target extraction unit 211a. Further, the calibration unit 211 calculates, for example, the displacement amount of the feature points extracted from the detection results of the objects 37, 38, and 39 by each object detection device ODD by the displacement amount determination unit 211c.

[0039] The displacement amount determination unit 211c determines, for example, that calibration is unnecessary (NO) in process P01 when the calculated displacement amount is within the allowable range previously stored in the storage unit 212. In this case, the calibration unit 211 ends the calibration process flow shown in FIG. 5. Also, the displacement amount determination unit 211c determines, for example, that calibration is necessary (YES) in process P01 when the calculated displacement amount is outside the allowable range previously stored in the storage unit 212.

[0040] In this case, the calibration unit 211 sets the number of repetitions N of the first calibration algorithm to 1 (process P11) and executes the first calibration algorithm (process P12). This first calibration algorithm takes less processing time compared to the second calibration algorithm described later. On the other hand, this first calibration algorithm has a lower calibration accuracy compared to, for example, the second calibration algorithm described later.

[0041] FIG. 7 is a diagram for explaining an example of the first calibration algorithm. In the example shown in FIG. 7, both the object detection device ODD serving as the calibration reference and the object detection device ODD to be calibrated are LiDAR 51. In this case, the calibration unit 211 can execute the following first calibration algorithm by the target extraction unit 211a, the feature extraction unit 211b, the deviation amount determination unit 211c, and the coordinate conversion unit 211d.

[0042] The target extraction unit 211a extracts the detection results of the objects 37, 38, and 39 arranged in the overlapping detection ranges from the LiDAR 51 serving as the calibration reference and the object to be calibrated. Based on the detection results of the extracted objects 37, 38, and 39, the feature extraction unit 211b generates a bounding box BB by regarding the three-dimensional objects other than the ground surface as the persons who are the calibration objects 37, 38, and 39. Further, the feature extraction unit 211b automatically extracts the bottom centroids BP1, BP2, and BP3 of the objects 37, 38, and 39 as feature points.

[0043] The deviation amount determination unit 211c calculates the position and area of the first triangle having the bottom centroids BP1, BP2, and BP3, which are the feature points extracted from the detection results of the LiDAR 51 serving as the calibration reference, as vertices. Also, the deviation amount determination unit 211c calculates the position and area of the second triangle having the bottom centroids BP1, BP2, and BP3, which are the feature points extracted from the detection results of the LiDAR 51 to be calibrated, as vertices.

[0044] Furthermore, the deviation amount determination unit 211c calculates, for example, the ratio of the area of the overlapping portion of the first triangle and the second triangle to the area of the first triangle as the deviation amount. The coordinate conversion unit 211d converts the coordinate system of the LiDAR 51 to be calibrated so as to reduce the deviation amount by overlapping the first triangle and the second triangle. Thus, the first calibration algorithm (process P12) ends.

[0045] Note that the calibration unit 211 can use the above-described first calibration algorithm, for example, in the process P01 of determining the necessity of the above-described calibration. That is, when the deviation amount, which is the ratio of the area of the overlapping portion of the first triangle and the second triangle to the area of the first triangle, is outside a preset allowable range, for example, less than 90%, the deviation amount determination unit 211c may determine that calibration is necessary.

[0046] After the end of the first calibration algorithm (process P12), the calibration unit 211 determines whether the calibration is completed (process P13). In this process P13, the calibration unit 211 executes the same process as the process P01 of determining the necessity of the above-described calibration using the object detection result by the calibrated LiDAR 51. If the deviation amount calculated by the deviation amount determination unit 211c is within the allowable range, it is determined that the calibration is completed (YES), and the flow of the calibration process shown in FIG. 5 is ended.

[0047] On the other hand, in the process P13, if the deviation amount calculated by the feature extraction unit 211b is outside the allowable range, the calibration unit 211 determines that the calibration is not completed (NO), and determines whether the number of repetitions N of the first algorithm is equal to or greater than the threshold Th1 (process P14). Here, the threshold Th1 is set to an arbitrary number, for example, 3 times, and is stored in the storage unit 212 in advance.

[0048] In this process P14, when the calibration unit 211 determines that the number of repetitions N of the first calibration algorithm is less than the threshold Th1 (NO), it adds 1 to the number of repetitions N of the first calibration algorithm (process P15), and executes the first calibration algorithm again (process P12). Then, when it is determined in process P13 that the iterative calibration is not completed (NO), the addition of the number of repetitions N (process P15) and the execution of the first calibration algorithm (process P12) are repeated until the number of repetitions N reaches the threshold Th1.

[0049] In this case, in process P14, the calibration unit 211 determines that the number of repetitions N of the first calibration algorithm is equal to or greater than the threshold Th1 (YES), and sets the number of repetitions N of the second calibration algorithm to 1 (process P21). Here, the calibration unit 211 may notify, for example, an operator who calibrates a plurality of object detection devices ODD via the sound output device 23 and the display device 24 that the number of repetitions N of the first calibration algorithm has reached the threshold Th1 and the second calibration algorithm is to be executed.

[0050] After that, the calibration unit 211 executes the second calibration algorithm (process P22). This second calibration algorithm takes more processing time (longer processing time) compared to the aforementioned first calibration algorithm. On the other hand, this second calibration algorithm has, for example, higher calibration accuracy compared to the aforementioned first calibration algorithm.

[0051] FIG. 8 is a diagram for explaining an example of the second calibration algorithm. In the example shown in FIG. 8, both the object detection device ODD serving as the calibration reference and the object detection device ODD to be calibrated are LiDAR 51. In this case, the calibration unit 211 can execute, for example, a second calibration algorithm using a known ICP (Iterative Closest Point).

[0052] As a result, the calibration unit 211 automatically estimates a plurality of feature points CP of the objects 37, 38, 39 based on the detection results of the object detection device ODD serving as the calibration reference, a plurality of feature points CP of the objects 37, 38, 39 based on the detection results of the object detection device ODD to be calibrated, and the corresponding points of each feature point CP. Further, based on the estimation result, the calibration unit 211, for example, converts the coordinate system of the LiDAR 51 to be calibrated. Thus, the second calibration algorithm (process P22) ends.

[0053] In this second calibration algorithm, for example, when the number of objects included in the detection result of the object detection device ODD increases, the processing time tends to increase. Also in the second calibration algorithm, the base centroids BP1, BP2, BP3 of the objects 37, 38, 39 and the areas of the triangles having those as vertices may be calculated to calculate the deviation amount between the detection results by the plurality of object detection devices ODD.

[0054] After the end of the second calibration algorithm (process P22), the calibration unit 211 determines whether the calibration is completed (process P23). In this process P23, the calibration unit 211 executes the same process as the process P01 for determining the necessity of the aforementioned calibration using the detection result of the object by the calibrated LiDAR 51. If the deviation amount calculated by the deviation amount determination unit 211c is within the allowable range, it is determined that the calibration is completed (YES), and the flow of the calibration process shown in FIG. 5 is ended.

[0055] On the other hand, in process P23, if the deviation amount calculated by the deviation amount determination unit 211c is outside the allowable range, it is determined that the calibration is not completed (NO), and it is determined whether the repetition number N of the second algorithm is equal to or greater than the threshold value Th2 (process P24). Here, the threshold value Th2 is set to an arbitrary number of times, for example, 3 times, and is stored in the storage unit 212 in advance.

[0056] In this process P24, when the calibration unit 211 determines that the number of repetitions N of the second calibration algorithm is less than the threshold Th2 (NO), it adds 1 to the number of repetitions N of the second calibration algorithm (process P25), and executes the second calibration algorithm (process P22) again. After that, when it is determined in process P33 that the iterative calibration is not completed (NO), the addition of the number of repetitions N (process P25) and the execution of the second calibration algorithm (process P22) are repeated, and it is assumed that the number of repetitions N has reached the threshold Th2.

[0057] In this case, the calibration unit 211 determines in process P24 that the number of repetitions N of the second calibration algorithm is equal to or greater than the threshold Th2 (YES). Further, the calibration unit 211 executes an error notification (process P02) to notify, for example, an operator who calibrates the plurality of object detection devices ODD via the sound output device 23 and the display device 24 that the number of repetitions N of the second calibration algorithm has reached the threshold Th2 and the calibration is completed. After that, the calibration unit 211 ends the calibration process flow shown in FIG. 5.

[0058] Note that in the example shown in FIG. 5, as the plurality of calibration algorithms, the first calibration algorithm and the second calibration algorithm are sequentially executed. However, after the end of process P24, the third calibration algorithm may be executed in the same manner as from process P21 to process P25, and further the fourth calibration algorithm may be executed. That is, the number of the plurality of calibration algorithms sequentially executed by the calibration unit 211 in the calibration process is arbitrary.

[0059] As described above, the peripheral monitoring device SMA of the working machine according to the present embodiment includes a plurality of object detection devices ODD that detect objects around the crane 1 which is a working machine, and a calibration unit 211 that calibrates the coordinate systems of the object detection devices ODD. The calibration unit 211 sequentially executes a plurality of calibration algorithms for calibrating the object detection devices ODD in ascending order of processing time or manual operation until the calibration of the object detection devices ODD is completed.

[0060] With such a configuration, according to the peripheral monitoring device SMA of the working machine of the present embodiment, when calibrating the object detection device ODD, it is possible to preferentially execute a calibration algorithm that finishes processing in a shorter time from among a plurality of calibration algorithms. Therefore, when the calibration of the object detection device ODD is completed by the first calibration algorithm, the time required for calibrating the object detection device ODD can be shortened.

[0061] Further, when the object detection device ODD cannot be configured by the first calibration algorithm, the calibration accuracy of the object detection device ODD can be further improved by sequentially executing calibration algorithms that take more processing time. Therefore, according to the peripheral monitoring device SMA of the working machine of the present embodiment, it is possible to improve the efficiency when calibrating a plurality of object detection devices ODD, ensure the safety of the working machine such as the crane 1, and improve productivity.

[0062] Also, in the peripheral monitoring device SMA of the working machine of the present embodiment, the calibration unit 211 calculates the deviation amount between the detection results by the plurality of object detection devices ODD based on the detection results of the same objects 37, 38, 39 by the plurality of object detection devices ODD, and determines the necessity or completion of calibration based on the deviation amount.

[0063] With such a configuration, the peripheral monitoring device SMA of the working machine of the present embodiment can determine the necessity or completion of calibration based on the detection results of the same objects 37, 38, 39 by the plurality of object detection devices ODD.

[0064] Also, in the peripheral monitoring device SMA of the working machine of the present embodiment, when the calibration does not complete even after repeating the first calibration algorithm, which is the calibration algorithm being executed among the plurality of calibration algorithms, a predetermined number of times, the calibration unit 211 executes the second calibration algorithm, which is the next calibration algorithm among the plurality of calibration algorithms.

[0065] With such a configuration, even if the calibration is not completed by the first calibration algorithm with the least processing time or manual operation in the peripheral monitoring device SMA of the working machine according to the present embodiment, it is possible to complete the calibration with the first calibration algorithm executed after the second time. Thereby, compared with the case where the second calibration algorithm with more processing time or manual operation than the first calibration algorithm is immediately executed when the calibration is not completed by the first calibration algorithm, the processing time or manual operation can be reduced.

[0066] Further, in the peripheral monitoring device SMA of the working machine according to the present embodiment, the plurality of calibration algorithms executed by the calibration unit 211 include a first calibration algorithm. In the first calibration algorithm, a coordinate transformation is performed to match a polygon having, as vertices, the base centroids BP1, BP2, BP3, which are points obtained by projecting the centroid positions of the same three or more objects 37, 38, 39 detected by each of the plurality of object detection devices ODD, onto the ground surface, between at least two of the plurality of object detection devices ODD.

[0067] With such a configuration, the peripheral monitoring device SMA of the working machine according to the present embodiment can perform the calibration process of the plurality of object detection devices ODD automatically and in a shorter time compared with methods such as ICP. As a result, the efficiency of the calibration process when calibrating the plurality of object detection devices ODD can be improved, and the labor of the operator can be reduced.

[0068] Further, as shown in FIG. 3, the peripheral monitoring device SMA of the working machine according to the present embodiment includes a sound output device 23 and a display device 24 as output units that synthesize and output the detection results of objects by the plurality of object detection devices.

[0069] With such a configuration, the peripheral monitoring device SMA of the working machine according to the present embodiment can detect objects around the working machine such as the crane 1 by a plurality of object detection devices ODD after calibration. Further, the detection results of the plurality of object detection devices ODD can be combined and information on the surrounding objects can be provided to the operator of the working machine such as the crane 1 by output devices such as the sound output device 23 and the display device 24. Therefore, according to the peripheral monitoring device SMA of the working machine of the present embodiment, the safety of the working machine such as the crane 1 can be ensured.

[0070] As described above, according to the present embodiment, it is possible to provide a peripheral monitoring device SMA for a working machine that can improve the efficiency when calibrating a plurality of object detection devices ODD, improve productivity while ensuring the safety of the working machine.

[0071] [Embodiment 2] Next, Embodiment 2 of the peripheral monitoring device for a working machine according to the present disclosure will be described with reference to FIGS. 9 and 10. FIG. 9 is a diagram for explaining an example of the first calibration algorithm in FIG. 5. FIG. 10 is a diagram for explaining an example of the second calibration algorithm in FIG. 5.

[0072] In the peripheral monitoring device SMA of the working machine according to the foregoing Embodiment 1, both the object detection device ODD serving as the calibration reference and the object detection device ODD to be calibrated were the LiDAR 51. In contrast, in the peripheral monitoring device SMA of the working machine according to the present embodiment, the object detection device ODD serving as the calibration reference and the object detection device ODD to be calibrated are the camera 52 and the LiDAR 51.

[0073] Also, the content of the first calibration algorithm (processing P12) and the second calibration algorithm (processing P22) shown in FIG. 5 of the peripheral monitoring device SMA of the working machine according to the present embodiment is different from that of the peripheral monitoring device SMA of the working machine according to the foregoing Embodiment 1. Since other configurations of the peripheral monitoring device SMA of the working machine according to the present embodiment are the same as those of the peripheral monitoring device SMA of the working machine according to the foregoing Embodiment 1, the same parts are denoted by the same reference numerals and the description thereof is omitted.

[0074] The peripheral monitoring device SMA of the working machine according to this embodiment executes, for example, the first calibration algorithm shown in FIG. 5 by the calibration unit 211 (process P12), similar to the peripheral monitoring device SMA of the working machine according to Embodiment 1. In this process P12, the calibration unit 211 acquires, for example, the detection results of objects including the detection results of the calibration objects 37 and 38 arranged in the overlapping detection range from the camera 52 and the LiDAR 51 having the overlapping detection range by the target extraction unit 211a.

[0075] Furthermore, the target extraction unit 211a extracts the detection results of the calibration objects 37 and 38 arranged in the overlapping detection range from the acquired detection results of the camera 52 and the LiDAR 51. Specifically, for example, as shown in the upper left of FIG. 9, the target extraction unit 211a recognizes the objects 37 and 38 from the detection result DR1 of the camera 52 and generates a bounding box BB surrounding the objects 37 and 38. Also, for example, as shown in the lower left of FIG. 9, the target extraction unit 211a generates a bounding box BB surrounding the three-dimensional object remaining after removing the detection results of other objects including the ground surface from the detection result DR2 of the LiDAR 51.

[0076] Next, the feature extraction unit 211b extracts, for example, as shown in the upper right of FIG. 9, the coordinates of the center of the lower end of the rectangular bounding box BB generated based on the detection result DR1 of the camera 52 as the feature point CP. Similarly, the feature extraction unit 211b extracts, for example, as shown in the lower right of FIG. 9, the coordinates of the center of the lower end surface of the cuboid bounding box BB generated from the detection result of the LiDAR 51 as the feature point CP.

[0077] Thereafter, the deviation amount determination unit 211c calculates the deviation amount based on the coordinates of the feature point CP extracted based on the detection result DR1 of the camera 52 and the coordinates of the feature point CP extracted based on the detection result of the LiDAR 51. Further, the coordinate conversion unit 211d converts the coordinates of the object detection device ODD to be calibrated, among the LiDAR 51 and the camera 52, so as to reduce the calculated deviation amount and match the coordinates of the feature point CP.

[0078] This first calibration algorithm has a shorter processing time compared to methods such as ICP, for example. However, since it cannot accurately measure the size, shape, and position of a person as calibration objects 37 and 38, the calibration accuracy is slightly inferior. Note that by changing the arrangement of the calibration objects 37 and 38 and repeatedly executing the first calibration algorithm, it is possible to improve the calibration accuracy. Also, the calibration unit 211 may perform a process P01 for determining the necessity of calibration and processes P13 and P23 for determining the completion of calibration based on the deviation amount calculated as described above.

[0079] Also, the peripheral monitoring device SMA of the working machine according to the present embodiment executes, for example, the second calibration algorithm shown in FIG. 5 by the calibration unit 211 (process P22), similar to the peripheral monitoring device SMA of the working machine in the first embodiment. In this process P22, the calibration unit 211 acquires, for example, as shown in FIG. 10, the detection results DR1 of the camera 52 and the detection results DR2 of the LiDAR 51 by the target extraction unit 211a, and extracts the objects 37 and 38 respectively.

[0080] Furthermore, the feature extraction unit 211b displays, for example, as shown in FIG. 10, the objects 37 and 38 extracted based on the detection result DR1 of the camera 52 and the objects 37 and 38 extracted based on the detection result DR2 of the LiDAR 51 on the display device 24 respectively. Furthermore, the feature extraction unit 211b receives an input of feature points CP by an operator who configures the camera 52 and the LiDAR 51 via an input device 22 such as a mouse.

[0081] Specifically, the operator operates the input device 22 such as a mouse while looking at the images of the objects 37 and 38 displayed on the display device 24. Then, the operator manually selects the corresponding feature points CP for both the object 37 and 38 extracted based on the detection result DR1 of the camera 52 and the object 37 and 38 extracted based on the detection result DR2 of the LiDAR 51, and manually inputs them to the feature extraction unit 211b.

[0082] Thereafter, the deviation amount determination unit 211c calculates the deviation amount between the detection result of the camera 52 and the detection result of the LiDAR 51 based on the corresponding feature point CP. Thereafter, the coordinate conversion unit 211d converts the coordinates of the object detection device ODD to be calibrated among the camera 52 and the LiDAR 51 so as to reduce the deviation amount and match the corresponding feature point CP.

[0083] As described above, in the peripheral monitoring device SMA of the working machine according to the present embodiment, the plurality of calibration algorithms executed by the calibration unit 211 include a second calibration algorithm that performs coordinate conversion by manually associating the feature points CP of the same objects 37 and 38 detected by each of the plurality of object detection devices ODD.

[0084] According to the second calibration algorithm of the present embodiment, by manually operating the mouse as the input device 22 and selecting the feature point CP on the image displayed on the display device 24, the corresponding feature point CP can be specified relatively accurately. On the other hand, compared with the above-described first calibration algorithm, this second calibration algorithm has a longer processing time (the processing time is long) and a greater burden on the manual operation of the operator.

[0085] However, the peripheral monitoring device SMA of the working machine according to the present embodiment includes, as in the first embodiment, a plurality of object detection devices ODD that detect objects around the working machine such as the crane 1, and a calibration unit 211 that calibrates the coordinate systems of the plurality of object detection devices ODD. And the calibration unit 211 sequentially executes a plurality of calibration algorithms for calibration in ascending order of less processing time or less manual operation until the calibration of the object detection device ODD is completed.

[0086] Therefore, according to the peripheral monitoring device SMA of the working machine of the present embodiment, as in the first embodiment described above, the efficiency in calibrating the plurality of object detection devices ODD can be improved, and while ensuring the safety of the working machine such as the crane 1, the productivity can be improved.

[0087] In addition, in the peripheral monitoring device SMA of the working machine according to the present embodiment, the plurality of calibration algorithms executed by the calibration unit 211 include a calibration algorithm that performs coordinate transformation to match the point at the center of the lower end of the bounding box BB surrounding the same objects 37 and 38 detected by each of the plurality of object detection devices ODD between at least two of the plurality of object detection devices ODD.

[0088] With such a configuration, according to the peripheral monitoring device SMA of the working machine of the present embodiment, the calibration unit 211 can automatically execute the calibration algorithm, and the burden on the operator performing the calibration can be reduced. In addition, by the calibration unit 211 executing the above calibration algorithm, the processing time of the calibration process can be shortened compared with methods such as ICP.

[0089] In the peripheral monitoring device SMA of the working machine according to the above-described Embodiment 1 and Embodiment 2, the calibration unit 211 executes the process P01 for determining whether calibration is necessary as shown in FIG. 5, but this process P01 can be omitted.

[0090] FIG. 11 is a flowchart showing a modification of the calibration process in FIG. 5. In the example shown in FIG. 11, the calibration unit 211 sets the number of repetitions N of the first calibration algorithm to 1 (process P11) without first executing the process P01 for determining whether calibration is necessary. Thereafter, in the same manner as the process flow shown in FIG. 5, the first calibration algorithm (process P12) and the second calibration algorithm (process P22) are sequentially executed until the calibration of the object detection device ODD is completed.

[0091] The process P01 for determining whether calibration is necessary performs the process up to the calculation of the deviation amount in the same manner as the first calibration algorithm except for coordinate transformation. Therefore, in the example shown in FIG. 11, by effectively utilizing the calculation of the deviation amount and performing coordinate transformation based on the deviation amount, the first calibration algorithm is executed. Thereby, when calibration of the object detection device ODD is necessary, the process P01 for determining whether calibration is necessary can be omitted, and the efficiency of the calibration process can be improved.

[0092] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments. Various modifications, substitutions, etc. can be applied to the above-described embodiments without departing from the scope of the present invention. Also, the features described separately can be combined as long as there is no technical contradiction.

[0093] For example, in the above-described embodiment, the LiDAR and the camera were described as the object detection devices, but the object detection device may include other sensors capable of detecting objects around the working machine.

[0094] Also, a calibration board on which a calibration pattern such as a cross pattern is displayed may be used as the calibration object arranged in the overlapping detection range of a plurality of object detection devices when executing the first calibration algorithm similar to that in Embodiment 2 in the second calibration algorithm. By using the calibration board as the calibration object in this way, for example, feature points such as the center of the cross pattern and the corners of the board can be clearly defined. Therefore, even when automatically extracting feature points by image processing, the measurement tolerance can be reduced.

[0095] Also, when there are many objects similar to the calibration object around the working machine, the calibration algorithm that automatically performs calibration may be difficult to calibrate. In this case, the calibration unit may be able to preferentially execute the second calibration algorithm by manual operation described in Embodiment 2 above, for example, according to user settings.

[0096] Also, the calibration unit may refer to, for example, the absolute value of the deviation amount calculated by the deviation amount determination unit or its improvement history, and select an appropriate calibration algorithm from a plurality of calibration algorithms based on the absolute value of the deviation amount and the improvement history information, and sequentially execute it.

Explanation of Signs

[0097] 1 Crane (working machine) 13 Reference sensor (object detection device) 14 Arbitrary installation sensor (object detection device) 211 Correction Unit 213 Output Unit 37 Object 38 Object 39 Object 51 LiDAR (Object Detection Device) 52 Camera (Object Detection Device) BB Bounding Box BP1 Foot Center of Gravity (Point where the center of gravity is projected onto the ground surface) BP2 Foot Center of Gravity (Point where the center of gravity is projected onto the ground surface) BP3 Foot Center of Gravity (Point where the center of gravity is projected onto the ground surface) CP Feature Point DR1 Detection Result DR2 Detection Result ODD Object Detection Device SMA Peripheral Monitoring Device for Working Machine

Claims

1. A plurality of object detection devices for detecting objects around a working machine, and a calibration unit for calibrating the coordinate systems of the plurality of object detection devices, wherein the calibration unit sequentially executes a plurality of calibration algorithms for the calibration in ascending order of processing time or manual operation until the calibration is completed, a peripheral monitoring device for a working machine.

2. The calibration unit calculates a deviation amount between the detection results of the plurality of object detection devices based on the detection results of the same object by the plurality of object detection devices, and determines the necessity or completion of the calibration based on the deviation amount, The peripheral monitoring device for a working machine according to Claim 1.

3. When the calibration is not completed even if the calibration algorithm being executed in the plurality of calibration algorithms is repeated a predetermined number of times, the calibration unit executes the next calibration algorithm in the plurality of calibration algorithms, The peripheral monitoring device for a working machine according to Claim 1.

4. The plurality of calibration algorithms include a calibration algorithm for performing coordinate transformation by manually associating feature points of the same object detected by each of the plurality of object detection devices, The peripheral monitoring device for a working machine according to Claim 1.

5. The plurality of calibration algorithms include a calibration algorithm for performing coordinate transformation to match polygons having as vertices points obtained by projecting the centroid positions of the same three or more objects detected by each of the plurality of object detection devices onto the ground surface, between at least two of the plurality of object detection devices, The peripheral monitoring device for a working machine according to Claim 1.

6. The plurality of calibration algorithms include a calibration algorithm for performing coordinate transformation to match points at the center of the lower end of a bounding box surrounding the same object detected by each of the plurality of object detection devices, between at least two of the plurality of object detection devices, The peripheral monitoring device for a working machine according to Claim 1.

7. having an output unit for synthesizing and outputting the detection results of the objects by the plurality of object detection devices, The peripheral monitoring device for a working machine according to Claim 1.

Images