Object Detection System

Abstract

Provided is a data aggregation method by which near-miss incidents data can be concretely extracted by comprehensively displaying a detected situation of an entire site. The present invention has: a communication unit 30 for receiving detection data indicating that an object detection sensor 19 mounted to a vehicle body of a construction machine 100 has detected an object and receiving position information of the vehicle body; a recording unit 34 for recording therein at least the position information; and a detection result output unit 32 for, on the basis of the detection data and the position information, outputting, to an external monitor 40, the distribution density of positions and frequencies at which the object detection sensor 19 has detected the object.

Description

[Technical field]

[0001] The present invention relates to an object detection system applied to a work site of a construction machine. [Background technology]

[0002] Construction machinery, as disclosed in Patent Document 1, for example, is equipped with sensors that detect people and objects around the vehicle, and has a function to warn the operator and restrict the movement of the vehicle when the sensors detect a person or object in close proximity to the vehicle.

[0003] Furthermore, by accumulating information such as the direction, location, time and operation performed at the time when a person or object was detected, it is possible to output near-miss trends in report format for on-site managers. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6805883 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with only the information that the vehicle body has detected surrounding objects, it is difficult to imagine near-misses that have occurred at the site, and it is difficult to propose specific improvement methods.The present invention aims to provide an object detection system for construction machinery that can easily grasp where and under what circumstances near-misses have occurred at the site where the construction machinery is operating. [Means for solving the problem]

[0006] In order to solve the above problems, the object detection system of the present invention has a communication unit that receives detection data indicating that object detection sensors attached to the bodies of multiple construction machines have detected an object and vehicle body position information, a recording unit that records at least the position information of the multiple construction machines, and a detection result output unit that consolidates the vehicle body positions when the object detection sensors of the multiple construction machines detected an object within a collection period based on the detection data and the position information, and outputs the density of the detection locations, which is the frequency of the locations where the object was detected, to an external monitor as a distribution density within a specified range. Effect of the Invention

[0007] With the above configuration, it is possible to display the location and frequency of object detection on an external monitor in a schematic and visual manner, in addition to the single information that the vehicle has detected surrounding objects, etc. Therefore, it is possible to easily grasp where and in what circumstances near misses have occurred at the site, making it easier for site managers to propose improvements to safety.

[0008] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. Furthermore, the objects, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the structure of a construction machine having a camera and an object detection sensor. [Diagram 2] FIG. 2 is a block diagram showing the functional configuration of a camera and an object detection sensor of the construction machine. [Diagram 3] FIG. 1 is a functional block diagram of an object detection system implemented in a server. [Figure 4] 6 is a flowchart showing a process of storing video data captured by a camera of a construction machine in a recording device. [Diagram 5] 6 is a flowchart showing a process of transmitting data from a construction machine to an object detection system. [Figure 6] An example of a report screen displayed on an external monitor. [Figure 7] 13 is an example of a video list screen associated with object detection positions on a distribution density window. [Figure 8] An example of a screen showing one of the videos in the video list in Figure 7 being played. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] FIG. 1 shows a construction machine 100 having a camera and an object detection sensor to which the present invention is applied.

[0012] The construction machine 100 is equipped with an engine 1 as a prime mover and a main pump 2 driven by the engine 1, and the lower traveling body 3, upper rotating body 4, and front device 5 each operate independently using the hydraulic oil delivered by the main pump 2.

[0013] The lower traveling structure 3 controls the drive of a pair of tracks 6 (only one side is shown in FIG. 1) by a traveling hydraulic motor 7.

[0014] The upper rotating body 4 is provided so as to be able to rotate relative to the lower traveling body 3, and the upper rotating body 4 is driven and controlled by rotating a rotating hydraulic motor 8.

[0015] The front device 5 is mounted on the upper rotating body 4, and is composed of a boom 9, a boom cylinder 10 for driving the boom 9, an arm 11, an arm cylinder 12 for driving the arm 11, a bucket 13, and a bucket cylinder 14 for driving the bucket 13. Each cylinder expands and contracts with hydraulic oil sent from the main pump 2, and drives the upper rotating body 4, a rotating shaft 15 of the boom 9, a rotating shaft 16 of the boom 9, a rotating shaft 16 of the arm 11, and a rotating shaft 17 of the arm 11 and the bucket 13, respectively, to perform work such as excavation and leveling.

[0016] FIG. 2 is a block diagram showing the functional configuration of the camera and object detection sensor mounted on the construction machine 100.

[0017] A number of cameras 18 for monitoring the surroundings and object detection sensors 19 are mounted outside the driver's seat of the construction machine 100, and a monitor control controller 20, a vehicle control controller 21, a monitor 22, and a recording device 23 are mounted inside the driver's seat.

[0018] Each controller and the recording device 23 can communicate with each other via an in-vehicle network 24 such as a CAN. The cameras 18 are connected to a monitor control controller 20, and the object detection sensor 19 is connected to a vehicle body control controller 21. The monitor control controller 20 synthesizes the surrounding images input from the cameras 18, and outputs them to a monitor 22 as an image in which the images from each camera 18 are divided, or as a surround view in which the boundaries between adjacent cameras 18 are blended. The monitor 22 displays the input composite image, and further outputs the image displayed on the monitor 22 to a recording device 23.

[0019] A communication terminal 25 is connected to the recording device 23, which stores the input image on the monitor 22 and the vehicle body information transmitted through the in-vehicle network 24, and connects to the Internet via the communication terminal 25, enabling both uploading to the server 26 and access to the recording device 23 from the server 26. The communication terminal 25 also obtains location information from a GPS (not shown).

[0020] The vehicle body controller 21 transmits the detection signal of the object detection sensor 19 to the recording device 23 via the in-vehicle network 24. Upon receiving the detection signal, the recording device 23 stores the video images and vehicle body data before and after the detection for a preset period of time. In addition, the vehicle body position at the time of object detection acquired by the communication terminal 25 is also simultaneously stored and linked to the video images and vehicle body data. Note that the vehicle body data referred to here includes engine RPM, input values ​​of various levers, position information, time, and other operating information obtained from sensors installed on the construction machine.

[0021] The vehicle body controller 21 is also a controller that performs hydraulic control of the machine, and acquires the amount of operation of a swing lever 27 and a travel lever 28 operated by the operator. It also acquires the locked or unlocked state of a gate lock device 29 that permits or prohibits the operation of the vehicle body.

[0022] 3 is a block diagram showing the functional configuration of the object detection system implemented in the server 26. Note that hereinafter, the object detection system will be simply referred to as the "system."

[0023] The system has a communication unit 30, a video processing unit 31, a detection result output unit 32, a camera control unit 33, and a recording unit 34. The communication unit 30 transmits and receives data to and from the communication terminal 25 on the construction machine 100 side via a network 39.

[0024] The video processing unit 31 processes the video captured by the camera 18 of the construction machine 100 and received by the communication unit 30, stores the video as necessary, and transmits it to the detection result output unit 32.

[0025] In response to the communication unit 30 receiving the detection data, the detection result output unit 32 records the detection position of the object corresponding to the detection data, and calculates a distribution density indicating the position and frequency of object detection based on the accumulated data. In addition, the detection result output unit 32 outputs the distribution density information indicating the position and frequency to an external monitor 40 connected via a network 39 and causes it to be displayed. The display of the distribution density indicating the position and frequency will be described in detail later.

[0026] The camera control unit 33 controls the activation of the camera 18 of the construction machine 100. Specifically, the activation of the camera 18 is triggered by the release of the gate lock device 29 of the construction machine 100.

[0027] The recording unit 34 stores data 35 transmitted to the communication unit 30 via the communication terminal 25 of the construction machine 100. The received data 35 is stored for each model (35A, 35B, ...). Each data 35 includes time series data 37 and constant data 38. The time series data 37 includes data related to recorded video, operation information, detection sensor information, etc. within a predetermined time before and after the point in time when the object detection sensor 19 detects an object, etc. The predetermined time before and after the point in time can be set to, for example, 30 seconds before and after the point in time, but it does not have to be determined in advance and can be set to any time, such as the time from when the object detection sensor 19 starts to detect an object to when it ends. The constant data 38 includes data such as the time when the object detection sensor 19 detects an object, the vehicle position, etc.

[0028] FIG. 4 shows a flowchart when the server 26 stores the image data and the vehicle body data.

[0029] First, when the vehicle body is powered on, the recording device 23 is powered on and preparations for recording are completed (steps A1, A2).

[0030] The camera control unit 33 judges whether the gate lock device 29 of the vehicle body is released or not (step A3). If the gate lock device 29 is released, the camera 18 is started to start photographing and recording (step A4).

[0031] Next, the communication unit 30 judges whether or not a detection signal of an object or the like is received from the object detection sensor 19 of the vehicle body from the vehicle body control controller 21 (step A5).

[0032] When a detection signal is received from the object detection sensor 19, the communication unit 30 further receives from the recording device 23 the video data extracted from the video data generated by the monitor controller 20 from a preset time T1 seconds before (step A6). The communication unit 30 also stores the vehicle body data at the time when the object detection sensor 19 detected an object or the like.

[0033] Then, the communication unit 30 waits until a predetermined time T2 seconds has elapsed since the object detection sensor 19 detected an object, etc. After T2 seconds have elapsed, the recording unit 34 records the video data up to that point (steps A7 and A8).

[0034] After recording the video data in step A8, it is determined whether the power supply to the vehicle body is turned off (step A9). If the power supply is not turned off, the process returns to step A3, and the process of determining whether a detection signal has been received and generating and storing data is repeated. If the gate lock device 29 is locked in step A3 or if a detection signal has not been received in step A5, the process returns to step A9.

[0035] If it is determined in step A9 that the vehicle power supply is OFF, the power supply of the recording device 23 is turned OFF (step A10), and the control flow ends. At this time, although the vehicle power supply is OFF, the recording device 23 is driven by a separate battery power supply (not shown), and the recording device 23 does not turn OFF its own power supply until the final data saving is completed.

[0036] 5 shows a flowchart when the communication terminal 25 transmits the video data and the vehicle body data recorded in the recording device 23 to the server 26. This control is executed in parallel with the data generation flow of FIG.

[0037] First, when the vehicle body is powered on, the communication terminal 25 is powered on (step B1).

[0038] Next, the communication terminal 25 judges whether or not it is capable of communicating with the server 26 (step B2). If communication is not possible, it retries, and does not perform subsequent control until communication is possible.

[0039] When the communication terminal 25 and the server 26 are able to communicate with each other, the communication terminal 25 refers to the recording device 23 and determines whether or not there is unsent data in the recording device 23 (step B3). If there is no unsent data, the process returns to step B2.

[0040] If unsent data is stored in the recording device 23, it is further determined whether there is one or multiple pieces of unsent data (step B4). If only one piece of data is stored, that data is transmitted to the server 26 (step B5). If two or more pieces of unsent data are stored, the oldest data among them is transmitted to the server 26 (step B6). This is because, as will be described later, the stored video data is displayed in chronological order in the video list, and if the oldest data is not transmitted first, the order of the videos in the video list will be changed, which may make handling cumbersome.

[0041] After the transmission process in steps B5 and B6, in either case, it is determined whether the transmission process performed in each step has been reliably completed (step B7).

[0042] If the data transmission is not complete, it is determined whether the communication terminal 25 is able to communicate with the server 26 (step B8). If the communication is possible, the process returns to step B7 and waits until the data transmission is complete. If the communication is not possible, the process returns to step B2 and waits until the communication is possible.

[0043] If it is determined in step B7 that data transmission is complete, it is determined whether the vehicle power supply is turned off (step B9). If the power supply is not turned off, the process returns to step B2, and the process of checking and transmitting unsent data is repeated.

[0044] If it is determined that the vehicle power supply is OFF, the communication terminal 25 is powered OFF (step B10), and the control flow ends. At this time, the vehicle power supply is OFF, but the communication terminal 25 is driven by a separate battery power supply (not shown), and the communication terminal 25 does not power itself OFF until the final data transmission is completed.

[0045] FIG. 6 shows an example of a report screen that the system displays on an external monitor.

[0046] 6 is output to the external monitor 40 by the detection result output unit 32 of the system based on the vehicle body data received by the communication unit 30 from the communication terminal 25 of the construction machine 100, and is displayed on the screen of the external monitor 40. An operating machine list window 41, a distribution density window 43, a detection time window 45, a detection date window 46, and a detection direction window 47 are displayed on the report screen.

[0047] The list of model information 36 explained in Fig. 3 is displayed in the operating machine list window 41. A changeover switch 42 is assigned to each machine, and the machine with the changeover switch 42 enabled is treated as the target of aggregation, and the received data 35 corresponding to that machine is referenced from the recording unit 34.

[0048] The detection result output unit 32 integrates the vehicle positions when the object detection sensor 19 detected an object or the like during the tallying period from the received data 35 during the tallying period, and draws a distribution density 44 on the map in the distribution density window 43, the color of which becomes darker in proportion to the density of the detection location. At this time, link information (link) with video data taken in a predetermined range indicating the distribution density 44 is added to the distribution density 44. This links the distribution density 44 with the video within that range, and as described later, by clicking any point on the distribution density 44, the video within the range (area) indicating the distribution density 44 can be viewed. The density of the detection location here indicates the frequency at which an object or the like was detected at a location (position) where the object was detected during the tallying period (frequency of the position). The distribution density can be generated by, for example, giving some value to a predetermined radius around the position where the object or the like was detected, and determining the shade of the color by the sum of the values ​​given to each position after tallying all the data.

[0049] The distribution density 44 may be displayed in a manner that allows the user to intuitively grasp the magnitude of the distribution density by using color shading, the size of a figure, or a combination of these.

[0050] The detection time window 45 refers to the time when an object, etc. was detected during the collection period, and displays the ratio for each time period. The detection date window 46 refers to the time when an object, etc. was detected during the collection period, and displays the number of detections per day and a breakdown of the operations being performed at the time of detection. The detection direction window 47 refers to the direction of the object detection sensor 19 that transmitted a detection signal to the recording device 23 during the collection period, and displays the ratio for each direction. Note that in the above, the time when the recording device 23 received the detection signal may be used as the base time.

[0051] FIG. 7 is a diagram showing the configuration of a moving image list screen linked from the distribution density 44.

[0052] By clicking on any point on the distribution density 44 in Fig. 6, it is possible to move to a list of videos recorded within a certain range centered on the clicked point. This list is generated by the video processing unit 31 of the system. A still image of the time when an object or the like was detected is displayed in the thumbnail field 48. The date and time of occurrence field 49 displays the date and time of the event. The event field 50 displays the type of object detection sensor 19 that sent the detection signal at that time and the details of the event. The operation field 51 displays the operation being performed at that time. The model field 52 displays the model in which the recording device 23 is installed. By clicking on any line in the video list, it is possible to move to the video playback screen.

[0053] FIG. 8 is a diagram showing the configuration of a video playback screen linked from the video list screen in FIG.

[0054] The information listed in the video list is displayed as video title 53 at the top of the video playback screen. A site map similar to that in distribution density window 43 is displayed in map window 54, and a machine icon 55 is displayed at the location referenced from the vehicle's position data. Recorded video is displayed in video window 56. The video can be played / stopped using play button 57 and stop button 58, and the playback position can be adjusted using seek bar 59. Operation window 62 displays operation information such as engine RPM, driving operation (driving lever operation), turning operation (turn lever operation), and unlocking / locking of the gate lock device in the form of a graph. The graph is synchronized with the video data, and the playback position 63 on the graph slides according to the position of seek bar 59.

[0055] FIG. 8 shows a state in which the video showing the "rear approach warning" in FIG.

[0056] This embodiment makes it possible to extract and compile near-miss events with a high degree of accuracy at construction sites where construction machinery equipped with object detection systems is operating, and to provide reports that allow users to accurately review the situation at the site.

[0057] According to the embodiment of the present invention described above, the following advantageous effects are obtained.

[0058] (1) The object detection system of the present invention includes a communication unit that receives video data captured by a camera attached to the body of a construction machine, detection data indicating that an object has been detected by an object detection sensor attached to the body, and position information of the body, a recording unit that records at least the video data and the position information, and a detection result output unit that outputs to an external monitor the position and frequency distribution density of the object detected by the object detection sensor based on the detection data and the position information.

[0059] With the above configuration, it is possible to display the distribution density of the location and frequency of object detection in a schematic and visual manner, in comparison with the single information that the vehicle body has detected surrounding objects, etc. Therefore, it is possible to easily grasp where and in what circumstances near misses have occurred at the site, making it easier for site managers to propose improvements to safety.

[0060] (2) The detection result output unit displays on the external monitor the distribution density of the location and frequency, as well as the distribution of the time period when the object detection sensor detected the object, the date and type of vehicle operation when the object detection sensor detected the object, and a breakdown of the direction in which the object detection sensor detected the object. This allows a wider variety of information to be displayed on the external monitor, enabling on-site managers to take safety measures from various perspectives.

[0061] (3) The detection result output unit generates a distribution density of the position and frequency for each of the multiple vehicle bodies, displays a list of the multiple vehicle bodies on an external monitor, and enables selection of a vehicle for which the distribution density of the position and frequency is displayed on the external monitor. This makes it possible to collect data from multiple operating machines, improving the accuracy of the calculated distribution density of the position and frequency.

[0062] (4) The detection result output unit calculates the distribution density of the location and frequency for any period. This allows for flexible response, such as shortening the collection period to obtain short-term information for periods when there were many near misses, and lengthening the collection period to obtain long-term information for periods when there were few near misses.

[0063] (5) The apparatus further includes a video processing unit for processing video data, and the video processing unit stores the video data received by the communication unit within a predetermined time before and after the object detection sensor detects an object in association with the point on the display of the distribution density of positions and frequency at which the object detection sensor detected the object. This allows the user to click on any point on the display of the distribution density of positions and frequency to view the video taken at that point, making it easy to recognize the on-site environment.

[0064] (6) The system further includes a camera control unit that starts the camera when the gate lock of the construction machine is released. This makes it possible to reliably operate the camera while the construction machine is in operation.

[0065] The present invention is not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to delete a part of the configuration of each embodiment, or to add or replace another configuration. [Industrial Applicability]

[0066] The present invention is used at a construction site where a construction machine equipped with an object detection system is operated. [Explanation of symbols]

[0067] 18 camera, 19 object detection sensor, 26 server (object detection system), 29 gate lock device, 30 communication unit, 31 video processing unit, 32 detection result output unit, 33 camera control unit, 34 recording unit, 40 external monitor

Claims

1. a communication unit that receives detection data indicating that an object has been detected by an object detection sensor attached to the vehicle body of each of the plurality of construction machines, and position information of the vehicle body; A recording unit that records at least the position information of the plurality of construction machines; and a detection result output unit that integrates the vehicle positions when the object detection sensors of the multiple construction machines detected the object within a collection period based on the detection data and the position information, and outputs the density of the detection locations, which is the frequency of the locations where the object was detected, to an external monitor as a distribution density within a specified range.

2. 2. The object detection system of claim 1, The communication unit receives video data captured by a camera attached to a body of the construction machine, The recording unit records the video data, The object detection system according to claim 1, wherein the detection result output unit links the distribution density to the video data and outputs the video data to the external monitor.

3. 2. The object detection system of claim 1, The object detection system according to claim 1, wherein the detection result output unit displays, on the external monitor, the distribution density as well as a distribution of time periods when the object detection sensor detected the object.

4. 2. The object detection system of claim 1, The object detection system according to claim 1, wherein the detection result output unit further displays, on the external monitor, the date when the object detection sensor detected the object and the type of operation of the vehicle body.

5. 2. The object detection system of claim 1, The object detection system according to claim 1, wherein the detection result output unit further displays, on the external monitor, a breakdown of the direction in which the object detection sensor detected the object.

6. 2. The object detection system of claim 1, The object detection system is characterized in that the detection result output unit generates the distribution density for each of the multiple vehicle bodies, displays a list of the multiple vehicle bodies on the external monitor, and makes it possible to select the vehicle body for which the distribution density is displayed on the external monitor.

7. 2. The object detection system of claim 1, The object detection system, wherein the detection result output unit generates the distribution density for an arbitrary period of time.

8. 2. The object detection system of claim 1, The communication unit receives video data captured by a camera attached to a body of the construction machine, The recording unit records the video data, A moving image processing unit that processes the moving image data is further provided. The object detection system is characterized in that the video processing unit stores the video data received by the communication unit within a predetermined time before and after the object detection sensor detects the object, in association with the point on the distribution density display at which the object detection sensor detected the object.

9. 2. The object detection system of claim 1, The communication unit receives video data captured by a camera attached to a body of the construction machine, The recording unit records the video data, The object detection system further comprises a camera control unit that activates the camera when a gate lock of the construction machine is released.

Images