ENVIRONMENTAL MONITORING DEVICE FOR WORKING MACHINE

Abstract

An environmental monitoring device for a work machine, comprising a plurality of object detection devices configured to detect an object around the work machine, and a calibration part configured to perform calibration of coordinate systems of the plurality of object detection devices, wherein the calibration part is configured to sequentially execute a plurality of calibration algorithms for calibration in ascending order of processing time and number of manual operations until the calibration is completed.

Description

TECHNICAL FIELD

[0001] The present disclosures relate to environmental monitoring systems for work machines. BACKGROUND

[0002] Conventionally, an environmental display system and a work machine equipped therewith are known (see Patent Literature (PTL) 1). The environmental display system described in PTL 1 includes a controller that generates a composite image based on a composite condition and outputs it to a display device.

[0003] The controller specifies a position of a calibration mark having a predetermined shape to be used for calibration processing in multiple images captured by multiple cameras. The controller calculates coordinates of feature points of the calibration marks in a vehicle body coordinate system for each of the multiple images. The controller calculates coordinate differences of the feature points of the calibration marks in the multiple images, determines whether calibration processing is required based on the coordinate difference, and outputs the determination result to the display device. CITATION LISTPATENT LITERATURE

[0004] [PTL 1] Japanese Unexamined Patent Publication No. 2020-147962 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0005] For example, since a work machine includes an attachment for moving or conveying a load, the positions and postures of multiple object detection devices mounted on the work machine that detect surrounding objects may vary. Therefore, the work machine must be calibrated to effectively combine the detection results of multiple object detection devices on the construction site.

[0006] In a known environmental display system described in the prior art, the above-mentioned calibration process increases the burden on on-site personnel or requires an unexpectedly long period of time, which may reduce productivity.

[0007] The present disclosure provides an environmental monitoring device for a work machine capable of improving productivity while ensuring safety of the work machine. MEANS TO SOLVE THE PROBLEMS

[0008] One aspect of the present disclosure provides an environmental monitoring device for a work machine, comprising a plurality of object detection devices configured to detect an object around the work machine, and a calibration part configured to perform calibration of coordinate systems of the plurality of object detection devices, wherein the calibration part is configured to sequentially execute a plurality of calibration algorithms for calibration in ascending order of processing time and number of manual operations until the calibration is completed. EFFECTS OF THE INVENTION

[0009] According to the above aspect of the present disclosure, it is possible to provide an environmental monitoring device for a work machine capable of improving productivity while ensuring safety of the work machine. BRIEF DESCRIPTION OF THE DRAWINGS [ Fig. 1] Fig. 1 is a side view of a crane as an example of a working machine. [ Fig. 2] Fig. 2 is a rear view of the Fig. 1 shown crane. [ Fig. 3] Fig. 3 is a block diagram of an apparatus used in the Fig. 1 is mounted on the crane shown. [ Fig. 4] Fig. 4 is a block diagram showing details of a Fig. 3 shows the calibration part. [ Fig. 5] Fig. 5 is a flowchart showing an example of calibration processing by the Fig. 3 and Fig. 4 represents the calibration part shown. [ Fig. 6] Fig. 6 is a schematic plan view showing an example of arrangement of objects in calibration processing according to Fig. 5 shows. [ Fig. 7] Fig. Figure 7 is a drawing showing an example of a first calibration algorithm used in Fig. 5 is shown. [ Fig. 8] Fig. Figure 8 is a drawing showing an example of a second calibration algorithm used in Fig. 5 is shown. [ Fig. 9] Fig. 9 is a drawing showing an example of the first calibration algorithm used in Fig. 5 is shown. [ Fig. 10] Fig. 10 is a drawing showing an example of the second calibration algorithm used in Fig. 5 is shown. [ Fig. 11] Fig. Figure 11 is a flow chart illustrating a modification of the calibration process described in Fig. 5 is shown. DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of an environmental monitoring device for work machines according to the present disclosure will be described with reference to the drawings. EMBODIMENT 1

[0011] Fig. 1 is a side view of a crane 1 as an example of a working machine. Fig. 2 is a rear view of the Fig. 1 shown crane 1. Fig. 3 is a block diagram of the device used in the Fig. 1 shown crane 1 is mounted.

[0012] Although details will be described later, the environmental monitoring device for a work machine SMA according to the present embodiment has the following configuration as its most distinctive feature. The environmental monitoring device for the work machine SMA includes a plurality of object detection devices ODD for detecting objects present around the work machine, such as the crane 1, and a calibration part 211 for calibrating the coordinate system of the object detection devices ODD. The calibration part 211 sequentially executes a plurality of calibration algorithms in an ascending order of processing time and a number of manual operations until the calibration of the coordinate system of the object detection device ODD is completed.

[0013] The work machine on which the environmental monitoring device SMA is mounted includes, in addition to the crane 1, for example, a hydraulic excavator, a road machine, a civil engineering construction machine, a material handling machine, a forklift, a wheel loader, a dump truck, etc. The work machine, such as the crane 1 and the hydraulic excavator, includes, for example, a work attachment for moving or conveying a cargo or load. The work attachment includes, for example, a crane arm, a rope and guying, or a boom, an arm, an end attachment, and a hydraulic actuator.

[0014] Hereinafter, an example of a configuration of the crane 1 as an example of a work machine will be described, and then the environment monitoring device for the work machine SMA of the present embodiment will be described in detail.

[0015] As in Fig. As shown in Figure 1, the crane 1 is what is referred to as a mobile crawler crane. Specifically, the crane 1 includes a crawler-type lower traveling body 2 capable of traveling, and an upper slewing body 3 mounted on the lower traveling body 2 capable of slewing.

[0016] In the following description, the longitudinal and transverse directions of the crane 1, as viewed from an occupant, are referred to as the longitudinal and transverse directions of the crane 1. Unless otherwise stated, the longitudinal direction of the crane 1 is described assuming that the lower traveling body 2 is in a state where the longitudinal direction of the crane 1 is basically consistent with that of the upper slewing body 3 (referred to as the reference position). Furthermore, the vertical direction when the crane 1 is on a horizontal plane may also be referred to as the perpendicular direction.

[0017] A boom 4 capable of lifting is mounted on the front of the upper slewing body 3. A counterweight 5 is attached to the rear of the upper slewing body 3 to balance the weight of the boom 4 and a suspended load. A cabin 6 is located on the right front side of the upper slewing body 3, in which an operator sits to control the crane 1.

[0018] A lifting operation of the boom 4 is performed by winding or unwinding a wire rope (hoist rope) 7 by a hoist winch (not shown). One end of a hoist rope 8 is connected to a hook 10 at the upper end of the boom 4, and the hook 10 is suspended from the upper end of the boom 4. The other end of the hoist rope 8 is wound around a winch (not shown) on the upper swing body 3, and the hook 10 is moved up and down by driving the winch.

[0019] A main frame 11 located at the bottom of the upper pivot body 3 is provided with a first measuring device mounting part 9 at a position to avoid a pivot bearing 12. A reference sensor 13 (e.g., the first LiDAR 51a, which is an omnidirectional LiDAR 51 capable of measuring 360°, as shown in Fig. 2) as an object detection device ODD for detecting objects around the crane 1 is attached to the first measuring device mounting part 9 (e.g., fastening with screws). The reference sensor 13 may be one or more of the Fig. 2 shown cameras 52.

[0020] The first LiDAR 51a as the reference sensor 13 can measure 360° around a sensor center axis extending vertically on the lower surface of the main frame 11. The reference sensor 13 radiates light with a predetermined irradiation pattern at predetermined intervals while moving laser light from a space formed between the lower traveling body 2 and the upper slewing body 3 to the exterior of the crane 1, and measures position information (three-dimensional information including a position and shape (posture) of surrounding objects, as well as position information of a single point of the surrounding object) of surrounding objects over a wide range. The sensor center axis is a rotation axis parallel to an axis of a rotation center of the upper slewing body 3. Further, a plurality of the first measuring device mounting parts 9 may be provided.

[0021] Here, the predetermined irradiation pattern refers to a method for moving a position to be irradiated with light and a frequency for detecting the irradiated light. For example, the position to be irradiated with light is moved vertically to the upper end of the measurement range, and when the position has moved to the upper end of the measurement range, the position is moved horizontally by a predetermined distance, and the position is repeatedly moved to the lower end of the measurement range (irradiation is performed in a zigzag direction in the vertical direction). As another pattern, the position is moved horizontally parallel to the right end of the measurement range, and when the position has reached the right end, the position is moved vertically by a predetermined distance, and the position is repeatedly moved parallel to the left end of the measurement range (irradiation in a zigzag direction).In another pattern, the position to be irradiated is moved horizontally parallel to the left and right, and when the position is moved vertically, the position is moved diagonally to the vertical direction. As described above, there are various irradiation patterns.

[0022] In addition, several optionally installed sensors 14 (e.g. second and third LiDAR 51b, 51c and one or more cameras 52, which are in Fig. 2) as object detection devices (ODDs) for detecting objects around the crane 1 are detachably mounted on the side surface of the upper slewing body 3 and the side surface, bottom surface, or top surface of the counterweight 5. The optionally installed sensors 14 may include LiDAR 51 for widely measuring position information (three-dimensional information including the position and shape (posture) of the surrounding object, as well as the position information of a single point of the surrounding object) of objects around the crane 1. The LiDAR 51 irradiates the outside space with laser light with a predetermined irradiation pattern different from that of the reference sensor 13 and measures position information of the surrounding objects. The irradiation pattern of the LiDAR 51 as the reference sensor 13 and the LiDAR 51 as the optionally installed sensor 14 may be the same.

[0023] The optionally installed sensor 14 is detachably mounted at any position of the upper slewing body 3 or the counterweight 5 of the crane 1, for example, by a magnet, and can be easily moved by hand without relying on tools. Thus, a worker can freely change the installation position even if a location where the worker wants to measure (the location where the worker wants to view) changes depending on the situation of a construction site. A means for detachably attaching the optionally installed sensor 14 to the upper slewing body 3 or the like is not limited to the magnet.For example, an elastically deformable gripping element, such as a spring element, may be housed in a plurality of holes (attachment parts) previously installed in the counterweight 5 or the upper swing body 3. The attachment piece integral with the optionally installed sensor 14 is pushed into the hole (attachment part) until a tactile sensation indicating attachment is felt (until a click), so that the attachment piece of the optionally installed sensor 14 is elastically held in the hole (attachment part) by the gripping element. In this case, the optionally installed sensor 14 can insert the attachment piece into the gripping element only by hand (without using tools), and the attachment piece can be pulled out of the gripping element.

[0024] The reference sensor 13 is arranged at a position near the rear end of the main frame 11 of the upper rotary body 3 and on a center line extending in the longitudinal direction through the rotation center of the upper rotary body 3. This sensor center axis (rotation axis) is regarded as the reference position of the reference sensor 13, and when a position line indicating a position of the reference sensor 13 is positioned along the center line extending in the longitudinal direction through the rotation center of the upper rotary body 3, it is regarded as the reference position of the reference sensor 13. The position of the reference sensor 13 is a predetermined position and is stored in a memory 212, which will be described later. The reference sensor 13 is not limited to the Fig. 1, but can be fixed in any position as long as the fixed position is a predetermined position and is recorded in the memory 212 described later.

[0025] As in Fig. 3, the environmental monitoring device for the work machine SMA of the present embodiment includes, for example, the plurality of object detection devices ODD for detecting objects around the crane 1 as a work machine, and the calibration part 211 for calibrating the coordinate systems of the plurality of object detection devices ODD. The calibration part 211 includes, for example, a controller 21 mounted on the upper swing body 3 of the crane 1.

[0026] The plurality of object detection devices ODD here include, for example, LiDAR 51 or a camera 52 as the aforementioned reference sensor 13 or optionally installed sensor 14. The LiDAR 51 or the camera 52 as the reference sensor 13 can be used as a calibration standard of the LiDAR 51 to be calibrated or the camera 52 as the optionally installed sensor 14. The LiDAR 51 or the camera 52 as the optionally installed sensor 14 after calibration can be used as a calibration standard, for example, for the LiDAR 51 or the camera 52 before calibration.

[0027] The environmental monitoring device for the work machine SMA may include, for example, an input device 22, a sound output device 23, and a display device 24. The input device 22, the sound output device 23, and the display device 24 are installed, for example, around a driver's seat on which the operator sits in the cab 6 of the upper revolving body 3. The input device 22 includes, for example, switches, buttons, levers, pedals, touch panels, keyboards, mice, and the like. The sound output device 23 includes, for example, a buzzer, a speaker, and the like. The display device 24 includes, for example, a liquid crystal display, an organic EL display, and the like.

[0028] The controller 21 includes, for example, a central processing unit (CPU) and controls the operation of each part of the crane 1. The controller 21 includes a function of an electronic control unit (ECU) and is arranged in the upper slewing body 3. Specifically, the controller 21 controls the crane 1 based on the operation inputs from the operator input device 22, develops various programs stored in advance in the memory 212, reads various data, and executes various processing using the developed programs and the read various data.

[0029] The controller 21 has, for example, functions of the calibration part 211, the memory 212, and an output part 213, which are achieved by executing various programs by a CPU. Each part of the controller 21 can be realized, for example, by a common hardware or a plurality of different hardware.

[0030] The calibration part 211 executes a plurality of calibration algorithms in an ascending order of the amount of processing time or the number of manual operations until the calibration of the object detection device ODD is completed. The memory 212 stores, for example, various data and programs including a plurality of calibration algorithms in a non-volatile memory device that constitutes the controller 21 or is connected to the controller 21. The output part 213, for example, synthesizes and outputs the object detection results by the plurality of object detection devices ODD.

[0031] Fig. 4 is a block diagram showing details of a Fig. 3 shown calibration part 211. Fig. 5 is a flowchart showing an example of calibration processing by the Fig. 3 and Fig. 4 represents the calibration part 211 shown. Fig. 6 is a schematic plan view showing an example of the arrangement of calibration objects in the calibration processing shown in Fig. 5 is shown.

[0032] The calibration part 211 includes, for example, a target extraction part 211a, a feature extraction part 211b, a deviation amount determination part 211c, and a coordinate transformation part 211d, as shown in Fig. 4. Each part of the calibration part 211 represents a function that is achieved, for example, by executing various programs by the CPU of the controller 21.

[0033] For example, since the work machine includes a work attachment for moving or transporting a cargo or load, the positions and postures of several object detection devices (ODD) mounted on the work machine and detecting surrounding objects may vary. In particular, workers such as operators and maintenance personnel of a work machine such as the crane 1 start the sequence of Fig. 5, for example, after disassembly and assembly of the crane 1 or when displacement of the object detection device ODD occurs due to vibration during operation of the crane 1. Before the calibration process is started, workers place the calibration object, for example, in an overlapping detection area A12 in which detection areas of several object detection devices ODD overlap, as shown in Fig. 6 shown.

[0034] In particular, the Fig. 6, the detection area A1 of the first camera 52a as the object detection device ODD and the detection area A2 of the second camera 52b as the object detection device ODD form the overlapping detection area A12. In the overlapping detection area A12, people, such as on-site workers, are placed as the calibration objects 37, 38, and 39. The number of people as the calibration objects 37, 38, and 39 is not particularly limited. Even in the case of calibrating two LiDARs 51 as object detection devices ODD or the LiDAR 51 and the camera 52, the calibration objects 37, 38, and 39 can be arranged in the overlapping detection area where their detection areas overlap, as in the case shown in Fig. Example shown in Figure 6.

[0035] A worker who calibrates the coordinate systems of the plurality of object detection devices ODD places the calibration objects 37, 38 and 39 in the overlapping detection area and then inputs an instruction to start calibration processing to the controller 21, for example, via the input device 22. When the instruction to start calibration processing is input, for example, via the input device 22, the controller 21 starts the calibration processing in Fig. 5, and the calibration part 211 determines whether calibration is required or not (process P01).

[0036] In this process P01, the calibration part 211 includes detection results of objects, including detection results of calibration objects 37, 38 and 39 from a plurality of object detection devices ODD having the overlapping detection range A12, as shown in Fig. 6, for example, by the target extraction part 211a. Furthermore, the target extraction part 211a extracts detection results of calibration objects 37, 38, and 39 arranged in the overlapping detection area A12 from the obtained detection results of each object detection device ODD.

[0037] Next, the calibration part 211 extracts feature points using detection results of the objects 37, 38, and 39 by each object detection device ODD, for example, through the feature extraction part 211b. Further, the calibration part 211, for example, the deviation amount determination part 211c, calculates the amount of deviation of the feature points extracted from the detection results of the objects 37, 38, and 39 by the object detection devices ODD.

[0038] For example, if the calculated deviation amount is within the allowable range stored in the memory 212, the deviation amount determination part 211c determines that calibration is not required in the process P01 (NO). In this case, the calibration part 211 terminates the process shown in Fig. 5. For example, when the calculated deviation amount is outside the allowable range stored in the memory 212, the deviation amount determining part 211c determines that calibration is required in the process P01 (YES).

[0039] In this case, the calibration part 211 sets the repetition number N of the first calibration algorithm to one (process P11) and executes the first calibration algorithm (process P12). The first calibration algorithm requires less processing time than the second calibration algorithm described later.

[0040] Conversely, the calibration accuracy of the first calibration algorithm is lower than, for example, that of the second calibration algorithm described later.

[0041] Fig. Figure 7 is a drawing describing an example of the first calibration algorithm. Fig. 7, the object detection device ODD of the calibration reference and the object detection device ODD of the calibration target are both LiDAR 51. In this case, the calibration part 211 may execute the following first calibration algorithm through the target extraction part 211a, the feature extraction part 211b, the deviation amount determination part 211c, and the coordinate transformation part 211d.

[0042] The target extraction part 211a extracts detection results of the objects 37, 38, and 39 located in the overlapping detection range from the LiDAR 51 of the calibration reference and the calibration target. Based on the detection results of the extracted objects 37, 38, and 39, the feature extraction part 211b defines three-dimensional objects other than the ground surface as people, which are the calibration objects 37, 38, and 39, and generates a bounding box BB. Furthermore, the feature extraction part 211b extracts the foot balance points BP1, BP2, and BP3 of the objects 37, 38, and 39 as feature points through automatic calculation.

[0043] The deviation amount determination part 211c calculates the position and area of a first triangle whose vertices are the foot balance points BP1, BP2, and BP3, which are feature points extracted from the detection results of the LiDAR 51 of the calibration reference. The deviation amount determination part 211c calculates the position and area of a second triangle whose vertices are the foot balance points BP1, BP2, and BP3, which are feature points extracted from the detection results of the LiDAR 51 of the calibration target.

[0044] Further, the deviation amount determination part 211c calculates, for example, a ratio between an area of a portion where the first triangle and the second triangle intersect and the area of the first triangle as the deviation amount. The coordinate transformation part 211d converts the coordinate system of the LiDAR 51 to be calibrated to reduce the amount of deviation, for example, by superimposing the first triangle and the second triangle. Thus, the first calibration algorithm (process P12) ends.

[0045] For example, the calibration part 211 may use the first calibration algorithm in process P01 to determine the need for calibration. That is, the deviation amount determination part 211c may determine that calibration is necessary when the deviation amount, which is the ratio between the area of the part where the first triangle and the second triangle overlap, and the area of the first triangle is outside the predetermined tolerance range, for example, less than 90%.

[0046] After the first calibration algorithm (process P12) ends, the calibration part 211 determines whether the calibration is completed (process P13). In this process P13, the calibration part 211 executes the same process as the process P01 to determine the necessity of calibration using the object detection result by the LiDAR 51 after the calibration. If the deviation amount calculated by the deviation amount determination part 211c is within the allowable range, it is determined that calibration is completed (YES), and the Fig. The calibration process shown in Figure 5 ends.

[0047] Conversely, at process P13, if the deviation amount calculated by the feature extraction part 211b is outside the allowable range, the calibration part 211 determines that calibration has not been completed (NO) and determines whether the repetition number N of the first algorithm is equal to or greater than the threshold Th1 (process P14). Here, the threshold Th1 is set to a configurable repetition number, for example, 3 times, and stored in the memory 212 in advance.

[0048] If the calibration part 211 determines at process P14 that the repetition number N of the first calibration algorithm is less than the threshold value Th1 (NO), it adds one to the repetition number N of the first calibration algorithm (process P15) and executes the first calibration algorithm again (process P12). Then, at process P13, it determines that the repetition calibration has not been completed (NO), and the addition of the repetition number N (process P15) and the execution of the first calibration algorithm (process P12) are repeated so that the repetition number N reaches the threshold value Th1.

[0049] In this case, the calibration part 211 determines in process P14 that the repetition number N of the first calibration algorithm is equal to or greater than the threshold value Th1 (YES), and sets the repetition number N of the second calibration algorithm to 1 (process P21). Here, the calibration part 211 can notify a worker who calibrates a plurality of object detection devices ODD via the sound output device 23 and the display device 24 that the repetition number N of the first calibration algorithm has reached the threshold value Th1 and that the second calibration algorithm is being executed.

[0050] Thereafter, the calibration part 211 executes the second calibration algorithm (process P22). This second calibration algorithm requires more processing time (longer processing time) than the first calibration algorithm. Conversely, this second calibration algorithm has, for example, a higher calibration accuracy than the first calibration algorithm.

[0051] Fig. Figure 8 is a drawing describing an example of the second calibration algorithm. Fig. 8, the object detection device ODD of the calibration reference and the object detection device ODD of the calibration target are both LiDAR 51. In this case, the calibration part 211 may, for example, execute the second calibration algorithm using a known ICP (Iterative Closest Point).

[0052] Thus, the calibration part 211 automatically estimates a plurality of feature points CP of the objects 37, 38, and 39 based on the detection result of the object detection device ODD of the calibration reference, a plurality of feature points CP of the objects 37, 38, and 39 based on the detection result of the object detection device ODD of the calibration target, and corresponding points of each feature point CP. Further, the calibration part 211 converts, for example, the coordinate system of the LiDAR 51 to be calibrated based on the estimation result. Thus, the second calibration algorithm (process P22) ends.

[0053] With this second calibration algorithm, the processing time tends to increase as, for example, the number of objects included in the detection result of the object detection device ODD increases. Note that even with the second calibration algorithm, the amount of deviation between the detection results of the object detection devices ODD can be calculated by calculating the areas of the foot balance points BP1, BP2, and BP3 of the objects 37, 38, and 39 and the triangles they have as vertices.

[0054] After the second calibration algorithm (process P22) ends, the calibration part 211 determines whether the calibration is completed (process P23). In this process P23, the calibration part 211 uses the detection result of the object by the LiDAR 51 after calibration to perform the same process as in process P01 to determine whether calibration is necessary. Then, if the deviation amount calculated by the deviation amount determination part 211c is within the allowable range, it determines that the calibration is completed (YES), and the Fig. The calibration process shown in Figure 5 ends.

[0055] Conversely, in process P23, if the deviation amount calculated by the deviation amount determination part 211c is outside the allowable range, it is determined that calibration has not been completed (NO), and it is determined whether the repetition number N of the second algorithm is equal to or greater than the threshold Th2 (process P24). Here, the threshold Th2 is set to a configurable number of times, for example, three times, and stored in advance in the memory 212.

[0056] If the calibration part 211 determines in process P24 that the repetition number N of the second calibration algorithm is less than the threshold Th2 (NO), it adds 1 to the repetition number N of the second calibration algorithm (process P25) and executes the second calibration algorithm again (process P22). Then, in process P33, it determines that the repetition calibration is not completed (NO), and the addition of the repetition number N (process P25) and the execution of the second calibration algorithm (process P22) are repeated so that the repetition number N reaches the threshold Th2.

[0057] In this case, the calibration part 211 determines in process P24 that the repetition number N of the second calibration algorithm is not less than the threshold value Th2 (YES). Furthermore, the calibration part 211 executes an error notification (process P02) to inform the worker calibrating the plurality of object detection devices ODD, for example, via the sound output device 23 and the display device 24, that the repetition number N of the second calibration algorithm has reached the threshold value Th2 and the calibration is completed. Thereafter, the calibration part 211 completes the Fig. 5 shows the calibration process.

[0058] In the Fig. In the example shown in FIG. 5, the first calibration algorithm and the second calibration algorithm are executed sequentially as the plural calibration algorithms, but a third calibration algorithm or a fourth calibration algorithm may be executed after the end of process P24, as in process P21 to process P25. That is, the number of plural calibration algorithms that the calibration part 211 sequentially executes in the calibration process is configurable.

[0059] As described above, the environmental monitoring device for the work machine SMA of the present embodiment includes a plurality of object detection devices ODDs for detecting objects around the crane 1, which is the work machine, and the calibration part 211 for calibrating the coordinate systems of the object detection devices ODDs. The calibration part 211 sequentially executes a plurality of calibration algorithms for calibrating the object detection devices ODDs in an ascending order of a processing time or a number of manual operations until the calibration of the object detection devices ODDs is completed.

[0060] According to the environmental monitoring device for the work machine SMA of the present embodiment, when the calibration of the object detection device ODD is completed, the calibration algorithm that completes processing in a shorter time can be preferentially executed. Therefore, when the calibration of the object detection device ODD is completed by the first calibration algorithm, the time required for the calibration of the object detection device ODD can be shortened.

[0061] When the object detection device ODD cannot be configured by the first calibration algorithm, the calibration accuracy of the object detection device ODD can be further improved by sequentially executing the calibration algorithm, which requires more processing time. Therefore, according to the environmental monitoring device for the work machine SMA of the present embodiment, efficiency in performing calibration of the plurality of object detection devices ODD can be improved, and productivity can be improved while ensuring the safety of the work machine such as the crane 1.

[0062] In the environmental monitoring device for the work machine SMA of the present embodiment, the calibration part 211 further calculates a deviation amount between the detection results of the plurality of object detection devices ODD based on the detection results of the same object 37, 38, and 39 by the plurality of object detection devices ODD, and determines necessity or completion of the calibration based on the deviation amount.

[0063] With this configuration, the environmental monitoring device for the work machine SMA of the present embodiment can determine the necessity or completion of calibration based on the detection results of the same object 37, 38, and 39 by the plurality of object detection devices ODD.

[0064] Further, in the environmental monitoring device for the work machine SMA of the present embodiment, the calibration part 211 executes the second calibration algorithm, which is a next algorithm among the plurality of calibration algorithms, when the calibration is not completed after the first calibration algorithm, which is a current calibration algorithm, among the plurality of calibration algorithms is repeated a predetermined number of times.

[0065] With this configuration, even if the calibration is not completed by the first calibration algorithm, which requires the least processing time or manual operation, the environmental monitoring device for the work machine SMA of the present embodiment can complete the calibration by the first calibration algorithm executed in the second and subsequent times. Thus, when the calibration is not completed by the first calibration algorithm, the processing time or manual operation can be reduced, compared to immediately executing the second calibration algorithm, which requires more processing time or manual operation than the first calibration algorithm, when the calibration is not completed by the first calibration algorithm.

[0066] In the environmental monitoring device for the work machine SMA of the present embodiment, the plurality of calibration algorithms executed by the calibration part 211 include the first calibration algorithm. The first calibration algorithm performs coordinate transformations that align polygons between at least two of the plurality of object detection devices, each of the polygons having vertices that are points on the ground surface obtained by projecting foot balance points BP1, BP2, and BP3 of three or more identical objects 37, 38, and 39 detected by each of the plurality of object detection devices ODD.

[0067] With this configuration, the environmental monitoring device for the work machine SMA of the present embodiment can automatically perform calibration processing of the multiple object detection devices ODDs in a short time compared to a method such as ICP. As a result, the efficiency of calibration processing when performing the calibration of the multiple object detection devices ODDs can be improved, and the labor burden of the workers can be reduced.

[0068] As in Fig. 3, the environmental monitoring device for the work machine SMA of the present embodiment further includes a sound output device 23 and a display device 24 as output parts for synthesizing and outputting the detection results of the object by the plurality of object detection devices.

[0069] With this configuration, the environmental monitoring device for the work machine SMA of the present embodiment can detect objects around the work machine, such as the crane 1, through the plurality of object detection devices ODD after calibration. Furthermore, the detection results of the plurality of object detection devices ODD can be synthesized to provide information about surrounding objects to the operator of the work machine, such as the crane 1, using output devices such as the sound output device 23 and the display device 24. Therefore, the environmental monitoring device for the work machine SMA of the present embodiment can ensure the safety of the work machine, such as the crane 1.

[0070] As described above, according to the present embodiment, it is possible to provide the environmental monitoring device for the work machine SMA, which can improve the efficiency of calibration of the multiple object detection devices ODD and increase productivity while ensuring the safety of the work machine. [EMBODIMENT 2]

[0071] Next, Embodiment 2 of the environment monitoring device for the work machine according to this disclosure will be described with reference to Fig. 9, Fig. 10A and Fig. 10B. Fig. 9 is a drawing showing an example of the Fig. 5 describes the first calibration algorithm. Fig. 10A and Fig. 10B are drawings showing an example of the Fig. Describe the second calibration algorithm shown in Figure 5.

[0072] In the environmental monitoring device for the work machine SMA of Embodiment 1, the object detection device ODD of the calibration reference and the object detection device ODD of the calibration target are both LiDAR 51. However, in the environmental monitoring device for the work machine SMA of the present embodiment, the object detection device ODD of the calibration reference and the object detection device ODD of the calibration target are a camera 52 and LiDAR 51.

[0073] The environment monitoring device for the work machine SMA of the present embodiment differs from the environment monitoring device SMA of Embodiment 1 in the contents of the first calibration algorithm (process P12) and the second calibration algorithm (process P22) described in Fig. 5. Since other configurations of the environmental monitoring device for the work machine SMA of the present embodiment are the same as those of the environmental monitoring device SMA of Embodiment 1, similar parts are denoted by the same reference numerals and description is omitted.

[0074] The environment monitoring device for the work machine SMA of the present embodiment carries out the Fig. 5 by the calibration part 211, for example, in the same manner as the environmental monitoring device SMA of Embodiment 1 (process P12). In this process P12, the calibration part 211 includes detection results of objects, including detection results of calibration objects 37 and 38 located in the overlapping detection range of the camera 52 and the LiDAR 51 with the overlapping detection range, for example, by the target extraction part 211a.

[0075] Further, the target extraction part 211a extracts detection results of calibration objects 37 and 38 arranged in the overlapping detection area from the obtained detection results of the camera 52 and the LiDAR 51. Specifically, the target extraction part 211a detects, as shown in Fig. 9 top left, the objects 37 and 38 from the detection result DR1 of the camera 52 and creates bounding boxes BB that surround the objects 37 and 38. As shown in Fig. 9, the target extraction part 211a, for example, generates bounding boxes BB surrounding three-dimensional objects remaining after the detection result of other objects including the ground surface is removed from the detection result DR2 of the LiDAR 51.

[0076] As in Fig. Next, as shown in the upper right of Figure 9, the feature extraction part 211b extracts, for example, the coordinates of the lower end center of a rectangular bounding box BB generated based on the detection result DR1 of the camera 52 as the feature point CP. Similarly, as shown in Fig. 9, the feature extraction part 211b extracts, for example, the coordinates of the lower end center of the rectangular bounding box BB generated from the detection result of the LiDAR 51 as the feature point CP.

[0077] Next, the deviation amount determination part 211c calculates the deviation amount based on the coordinates of the feature point CP extracted based on the detection result DR1 of the camera 52 and the coordinates of the feature point CP extracted based on the detection result of the LiDAR 51. Furthermore, the coordinate transformation part 211d converts the coordinates of the object detection device ODD to be calibrated from the LiDAR 51 and the camera 52 to reduce the calculated deviation amount and align the coordinates of the feature point CP.

[0078] This first calibration algorithm requires less processing time than a method such as ICP, but the calibration accuracy is somewhat inferior because it cannot accurately measure the size, shape, and position of a person as the calibration objects 37 and 38. The calibration accuracy can be improved by repeatedly executing the first calibration algorithm while changing the arrangement of the calibration objects 37 and 38. Furthermore, the calibration part 211 can perform processes P01 for determining the necessity of calibration and processes P13 and P23 for determining the completion of calibration based on the deviation amount calculated as described above.

[0079] The environment monitoring device for the work machine SMA of the present embodiment carries out the Fig. 5 by the calibration part 211, for example, in the same manner as the environmental monitoring device for the work machine SMA of Embodiment 1 (process P22). In this process P22, for example, the calibration part 211 obtains the detection result DR1 of the camera 52 as shown in Fig. 10A, and the detection result DR2 of the LiDAR 51 by the target extraction part 211a, as shown in Fig. 10B, and extracts objects 37 and 38, respectively.

[0080] In addition, the feature extraction part 211b displays the objects 37 and 38 extracted based on the detection result DR1 of the camera 52 and the objects 37 and 38 extracted based on the detection result DR2 of the LiDAR 51 on the display device 24, for example, as shown in Fig. 10A and Fig. 10B. Furthermore, the feature extraction part 211b receives the input of the feature point CP from a worker who configures the camera 52 and the LiDAR 51 via an input device 22 such as a mouse.

[0081] Specifically, the worker operates the input device 22, such as a mouse, while viewing the images of the objects 37 and 38 displayed on the display device 24. Then, the worker manually selects the corresponding feature points CP for both the objects 37 and 38 extracted based on the detection result DR1 of the camera 52 and the objects 37 and 38 extracted based on the detection result DR2 of the LiDAR 51, and manually inputs them into the feature extraction part 211b.

[0082] Then, the deviation amount determination part 211c calculates the deviation amount between the detection result of the camera 52 and the detection result of the LiDAR 51 based on the corresponding feature points CP. Then, the coordinate transformation part 211d converts the coordinates of the object detection device ODD to be calibrated between the camera 52 and the LiDAR 51 to reduce the deviation amount and align the corresponding feature points CP.

[0083] As described above, in the environmental monitoring device for the work machine SMA of the present embodiment, the plurality of calibration algorithms executed by the calibration part 211 include the second calibration algorithm for performing coordinate transformations by manually matching the feature points CP of the same object 37 and 38 detected by each of the plurality of object detection devices ODD.

[0084] According to the second calibration algorithm of the present embodiment, by manually operating the mouse as the input device 22 and selecting the feature points CP on the image displayed on the display device 24, the corresponding feature points CP can be specified relatively accurately. Conversely, this second calibration algorithm requires more processing time (longer processing time) than the first calibration algorithm described above, and the burden of manual operation by the operator is large.

[0085] However, the environmental monitoring device for the work machine SMA according to the present embodiment includes a plurality of object detection devices ODD for detecting objects around the work machine, such as the crane 1, and the calibration part 211 for calibrating the coordinate systems of the plurality of object detection devices ODD, as in the case of Embodiment 1. The calibration part 211 sequentially executes the plurality of calibration algorithms for calibration in an ascending order of an amount of processing time or a number of manual operations until the calibration of the object detection device ODD is completed.

[0086] Therefore, according to the environmental monitoring device for the work machine SMA according to the present embodiment, as in the case of Embodiment 1, the efficiency of calibration of the plurality of object detection devices ODD can be improved and the productivity can be increased while ensuring the safety of the work machine such as the crane 1.

[0087] Furthermore, in the environmental monitoring device for the work machine SMA according to the present embodiment, the plurality of calibration algorithms executed by the calibration part 211 include a calibration algorithm for performing coordinate transformations by which lower end centers of bounding boxes BB between at least two of the plurality of object detection devices ODD are aligned, wherein each of the bounding boxes BB surrounds the same object 37, 38 detected by each of the plurality of object detection devices ODD.

[0088] With such configurations, according to the environmental monitoring device for the work machine SMA according to the present embodiment, the calibration algorithm can be automatically executed by the calibration part 211, and the burden on the worker performing the calibration can be reduced. Furthermore, by executing the above-described calibration algorithm by the calibration part 211, the processing time of the calibration process can be shortened compared to a technique such as ICP.

[0089] In the environment monitoring device for the working machine SMA of Embodiment 1 and Embodiment 2 as shown in Fig. 5, the calibration part 211 first performs the process P01 to determine the necessity of calibration, but this process P01 may be omitted.

[0090] Fig. 11 is a flowchart showing a modification example of the calibration process of Fig. 5 shows. In the Fig. 11, the calibration part 211 sets the repetition number N of the first calibration algorithm to 1 without executing the process P01 for determining the necessity of the first calibration (process P11). Then, similar to the Fig. 5, the first calibration algorithm (process P12) and the second calibration algorithm (process P22) are executed sequentially until the calibration of the object detection device ODD is completed.

[0091] The process P01 for determining the need for calibration performs processing up to the calculation of the deviation amount in the same way as the first calibration algorithm, except for coordinate transformations. Therefore, in the Fig.In the example shown in Figure 11, the calculation of the deviation amount is effectively utilized to execute the first calibration algorithm by performing coordinate transformations based on the deviation amount. Thus, when calibration of the object detection device ODD is required, the process P01 for determining the need for calibration can be omitted, and the efficiency of the calibration process can be improved.

[0092] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments. Various variations and modifications can be applied to the above-described embodiments without departing from the scope of the present invention. Furthermore, the separately described features can be combined as long as there is no technical conflict.

[0093] For example, in the embodiment described above, the LiDAR and the camera were described as object detection devices, but the object detection device may include other sensors capable of detecting objects around the work machine.

[0094] Furthermore, in the second calibration algorithm, a calibration object to which the first calibration algorithm is applied similarly to Embodiment 2 and which is arranged in the overlapping detection area of the plurality of object detection devices can be used. Thus, by using the calibration plate as the calibration object, feature points such as a center of a cross pattern and corners of a plate can be clearly defined. Therefore, even when feature points are automatically extracted by image processing, the measurement tolerance can be reduced.

[0095] In the calibration algorithm for automatically performing calibration, calibration may be difficult when there are many objects similar to the calibration object around the work machine. In this case, the calibration part may preferably execute the second calibration algorithm through manual operation as described in Embodiment 2, for example, through user setting.

[0096] In addition, for example, the calibration part may refer to the absolute value of the deviation amount calculated by the deviation amount determination part and the improvement history of the deviation, select an appropriate calibration algorithm from the plurality of calibration algorithms based on the absolute value of the deviation amount and the improvement history information, and execute the calibration algorithms sequentially. DESCRIPTION OF REFERENCE SYMBOLS 1 crane (working machine) 13 Reference sensor (object detector) 14 optionally installed sensor (object detector) 211 Calibration part 213 Output section 37 objects 38 objects 39 objects 51 LiDAR (object detector) 52 Camera (object detector) BB bounding box BP1 Foot equilibrium point (the point at which the equilibrium point is projected onto the ground surface) BP2 Foot equilibrium point (the point where the equilibrium point is projected onto the ground surface) BP3 Foot equilibrium point (the point at which the equilibrium point is projected onto the ground surface) CP feature point DR1 detection result DR2 detection result ODD object detection device SMA environmental monitoring device for a work machine QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2020-147962

[0004]

Claims

[1] Environmental monitoring device for a work machine, comprising: a plurality of object detection devices configured to detect an object around the work machine; and a calibration part configured to perform calibration of coordinate systems of a plurality of object detection devices, wherein the calibration part is configured to sequentially execute a plurality of calibration algorithms for calibration in ascending order of processing time and number of manual operations until the calibration is completed. [2] The environmental monitoring device for the work machine according to claim 1, wherein the calibration part is configured to calculate a deviation amount between detection results by the plurality of object detection devices based on detection results of the same object by the plurality of object detection devices, and to determine whether the calibration is necessary or whether the calibration is completed based on the deviation amount. [3] The environmental monitoring device for the work machine according to claim 1, wherein the calibration part is configured to execute a next calibration algorithm from the plurality of calibration algorithms when the calibration is not completed after a current calibration algorithm from the plurality of calibration algorithms is repeated a predetermined number of times. [4] The environmental monitoring device for the work machine according to claim 1, wherein the plurality of calibration algorithms include a calibration algorithm for performing coordinate transformations by manually assigning feature points of the same object detected by each of the plurality of object detection devices. [5] The environmental monitoring device for the work machine according to claim 1, wherein the plurality of calibration algorithms include a calibration algorithm for performing coordinate transformations by which polygons are aligned between at least two of the plurality of object detection devices, each of the polygons having vertices that are points on a ground surface obtained by projecting an equilibrium point of three or more same objects detected by each of the plurality of object detection devices. [6] The environmental monitoring device for the work machine according to claim 1, wherein the plurality of calibration algorithms comprises a calibration algorithm for performing coordinate transformations that align lower end centers of bounding boxes between at least two of the plurality of object detection devices, each of the bounding boxes surrounding a same object detected by each of the plurality of object detection devices. [7] The environmental monitoring device for the work machine according to claim 1, comprising an output part configured to synthesize and output the detection results of the object by the plurality of object detection devices.

Images