System for detecting position of object around work machine, and program for detecting position of object around work machine

Abstract

To provide a system which is a periphery detection object position detection system applicable to a work machine, and can detect a position of a detection object captured by an imaging device with high accuracy even in a work machine which does not have a sensor or a laser radar dedicated to distance measurement.SOLUTION: A work machine periphery detection object position detection system includes: a detection part C1 for detecting a position of a detection object W in the periphery of a work machine by detection processing using a plurality of sizes of detection frames F performed with respect to an image imaged by an imaging device I mounted on the work machine H; a detection region restriction part C2 for restricting a detection region regarding a priority detection object in a predetermined region in the image, based on a detection result by the detection part C1; and a detection coordinate calculation part C3 for calculating a position of the priority detection object based on the detection result by the detection part C1 in a detection region E restricted by the detection region restriction part C2.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a system for detecting a detection object around a work machine such as a construction machine, and a program for detecting the position of a detection object around the work machine. [Background technology]

[0002] For example, in a work machine (construction machine) having a structure specialized for various tasks, such as a hydraulic excavator having a lower traveling body and an upper rotating body rotatably attached to the lower traveling body, an operator can move the vehicle forward, backward, or turn by getting into the control room and operating a traveling operation part such as an operating lever. When performing such operations, the operator usually grasps the situation around the vehicle by visually checking or by viewing an image taken by an on-board camera installed at an appropriate place on the vehicle on a monitor installed in the cab, and confirms that there are no people or obstacles in the direction of the vehicle's movement.

[0003] In addition, in order to prevent accidents, it is also important to check surrounding conditions that are invisible or difficult to see by displaying camera images on a monitor and to accurately grasp the position of objects (detection targets) in the images displayed on the monitor. Therefore, it is possible to configure the work machine to have a sensor that measures the distance to the detection target captured by the camera, but implementing a sensor dedicated to distance measurement directly leads to increased costs.

[0004] Meanwhile, a large proportion of vehicles such as ordinary passenger cars are equipped with equipment (active safety) for preventing traffic accidents. As an example of active safety, a technology is known that warns the driver of obstacles existing around the vehicle by performing recognition using a distance measuring device such as a laser radar, or recognition using an image processing technology that analyzes images from an on-board camera (for example, Patent Document 1 below). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2011-48485 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, as described above, when taking into consideration the cost and the size and performance of the controller mounted on the work machine, the reality is that it is difficult to apply such active safety technology to a work machine.

[0007] Therefore, the present applicant is proceeding with the development of a system capable of detecting the position of a detection object captured by an imaging device with high accuracy even in a work machine that does not have a dedicated sensor or laser radar for measuring distance.Specifically, the applicant has developed and is proceeding with the practical application of a detection object position detection system around a work machine that exhibits a function of calculating the position of a detection object based on the processing result of a weighting process in which the coordinates of a detection frame when a person captured by an imaging device is image-recognized are used as weights, and even if the image recognition results vary, it is possible to avoid a situation in which the variation in the image recognition results is directly reflected in the calculation result, and detects the position of the detection object with high accuracy and calculates the distance from the imaging device to the detection object with high accuracy.

[0008] However, when two people are detected in an image, for example when it is desired to detect the position of the person relatively closer to the imaging device, the coordinates of the detection frame of the person relatively farther from the imaging device are also used in the weighting process, which may result in an error in the detection value of the position of the person closer to the imaging device, and a position farther than the actual distance may be calculated as the detected position of the person closer to the imaging device.

[0009] The present invention has been made with attention to these problems, and its main objective is to provide a surrounding detection object position detection system applicable to work machines, which is a system that can detect the position of a detection object captured by an imaging device with high accuracy even in work machines that do not have a dedicated sensor or laser radar for measuring distance, and further, a system that can calculate the distance from the imaging device to any one of the detection objects with high accuracy even when multiple detection objects are detected in the captured image. [Means for solving the problem]

[0010] In other words, the work machine surroundings monitoring system of the present invention, which is capable of detecting the position of a detection object around a work machine, comprises a detection unit that detects candidate positions of a detection object in an image captured by an imaging device attached to the work machine by using detection frames of multiple sizes for the image, a detection area limiting unit that limits a detection area for a priority detection object that is detected with priority based on the detection result by the detection unit to within a specified area in the image, and a detection coordinate calculation unit that calculates the position of the priority detection object as detection coordinates based on the detection result by the detection unit within the limited detection area limited by the detection area limiting unit. The work machine periphery monitoring system according to the present invention is characterized in that it employs, as a detection area limiting unit, a reference coordinate calculation unit that calculates reference coordinates that serve as the reference for the limited detection area for a reference detection frame that serves as a reference among a plurality of detection frames in the image based on the detection results by the detection unit; a limited detection area calculation unit that calculates a limited detection area based on the reference coordinates; a comparison coordinate calculation unit that calculates comparison coordinates for each of the non-reference detection frames that are detection frames other than the reference detection frame among the plurality of detection frames in the image based on the detection results by the detection unit; and a priority detection frame detection unit that extracts, from the non-reference detection frames, detection frames whose comparison coordinates calculated by the comparison coordinate calculation unit are within the limited detection area as priority detection frames.

[0011] With such a system for detecting the position of a detected object around a work machine according to the present invention, even if a plurality of detection objects are detected using a detection frame, one of the detection objects is set as a priority object and a detection frame related to the priority detection object is extracted, so that detection frames related to detection objects other than the priority detection object are not treated as valid data when calculating the position of the priority detection object as detection coordinates, and it is possible to detect the position of the priority detection object captured by the imaging device with high accuracy. Specifically, with the system for detecting the position of a detected object around a work machine according to the present invention, a reference detection frame that serves as a reference for a limited detection area is used to calculate reference coordinates that serve as the reference for a limited detection area, out of a plurality of detection frames in the image, and a limited detection area is calculated based on the reference coordinates, so that a limited detection area can be set in the detection object image with a relatively simple calculation process. In the system for detecting the position of a detection object around a work machine according to the present invention, comparison coordinates are calculated for each of the non-reference detection frames, which are detection frames other than the reference detection frame, among the multiple detection frames in the image, and when calculating the position of the priority detection object, only the priority detection frames having comparison coordinates within the limited detection area are used together with the reference detection frame, so that the position of the priority detection object can be detected without being affected by the coordinate values ​​of the non-reference detection frames for detection objects other than the priority detection object.

[0012] A suitable example of the "reference coordinates of the reference detection frame" in the present invention can be the central X coordinate of the reference detection frame. Furthermore, it is preferable that the "comparison coordinates of the non-reference detection frames" are similar to the "reference coordinates of the reference detection frame", and a suitable example can be the central X coordinate of the non-reference detection frame.

[0013] Furthermore, in the work machine periphery monitoring system according to the present invention, if the detection coordinate calculation unit calculates the position of the priority detection object in the detection target image as detection coordinates based on the processing results of weighting according to the count numbers (density numbers) of detection object identifying coordinates capable of identifying the position of the priority detection object from the reference detection frame, detection object identifying coordinates capable of identifying the position of the priority detection object from the priority detection frame having comparison coordinates within the limited detection area, and these detection object identifying coordinates, for each of a plurality of regions obtained by dividing the detection target image in the video, then even if the detection results using the detection frames vary, the variation can be reduced by weighting processing that increases the weight as the count number increases, and the detection coordinates that identify the position of the priority detection object in the detection target image can be calculated with high accuracy. Note that when a single detection object is detected in the video by the detection unit, the detection object becomes the priority detection object.

[0014] A preferred configuration of the weighting processing unit in the present invention includes a large area weighting processing unit that performs weighting for each large area obtained by dividing the entire detection target image into a plurality of areas according to the count number of the detection target object identification coordinates, and a small area weighting processing unit that performs weighting for each small area obtained by further dividing each large area into a plurality of areas according to the count number of the detection target object identification coordinates. In this case, if the detection coordinates are calculated based on the processing results by the large area weighting processing unit and the small area weighting processing unit, the detection target image can be divided into smaller areas, and the detection coordinates can be calculated with high accuracy that further reflects the density of the detection target object identification coordinates.

[0015] A specific example of weighting in the weighting processing section is weighted averaging based on the count number of the detection target object specific coordinates.

[0016] Furthermore, when detecting a detection target in the vicinity of a work machine, it is sufficient to detect a detection target (person) in a standing posture, and in particular to identify the position of the person's feet. Therefore, in the present invention, the coordinates corresponding to the bottom side of the rectangular detection frame are set as the priority detection target identification coordinates.

[0017] The present invention also includes a configuration for quickly calculating the real-time detection coordinates as the priority detection coordinates, but in order to calculate detection coordinates taking into account the time-series movement of the same detection object in the detection target image, it is preferable to configure the system to calculate the final detection coordinates by taking a moving average of the real-time detection coordinates and multiple detection coordinates including the most recent past.In particular, when the real-time detection coordinates are in a close position within a specified range from the work machine, that is, when there is a person in the vicinity of the work machine, quick response is required, so by adopting a configuration for changing the weight of the moving average (dynamically switching the weight) according to the real-time detection coordinates, it becomes possible to immediately output as a calculation result a dangerous situation in which there is a person in a close position within a specified range from the work machine.

[0018] If the system for detecting the position of a detection object around a work machine according to the present invention is configured to measure the distance from an imaging device to a priority detection object based on calculated detection coordinates, it becomes possible to calculate the distance to the priority detection object with high accuracy based on the position of the priority detection object in the target image, which is a calculation result that reduces the variability in the image recognition results using a detection frame (as a specific example, the position of the bottom edge of the detection frame).

[0019] Further, a detection program for detecting a position of a detected object in the vicinity of a work machine according to the present invention is a software program executed by a computer as a detection system for detecting the position of a detected object in the vicinity of a work machine, and includes a detection step of detecting position candidates of a detected object in the image captured by an imaging device attached to the work machine using detection frames of multiple sizes, a detection area limiting step of limiting a detection area for a priority detection object to be detected with priority based on a detection result by the detection step to within a predetermined area in the image, and a detection coordinate calculation step of calculating a position of the priority detection object in the image as detection coordinates based on the detection result by the detection step within the limited detection area limited in the detection area limiting step, wherein the detection area limiting step includes a reference coordinate calculation step of calculating reference coordinates that are the reference for the limited detection area for a reference detection frame that is the reference among the multiple detection frames in the image based on the detection result by the detection step, a limited detection area calculation step of calculating a limited detection area based on the reference coordinates, and Steps The method includes a comparison coordinate calculation step of calculating comparison coordinates for each of non-reference detection frames, which are detection frames other than the reference detection frame, among a plurality of detection frames in the video, based on the detection results by the above-mentioned detection method, and a priority detection frame detection step of extracting, from among the non-reference detection frames, detection frames whose comparison coordinates calculated in the comparison coordinate calculation step are within a limited detection area, as priority detection frames.

[0020] According to such a program for detecting the position of a detection object around a work machine, when calculating the position of a priority detection object, it is possible to exclude detection results (coordinates of the detection frame) relating to detection objects other than the priority detection object, and it is possible to calculate the position of the priority detection object with high accuracy. Effect of the Invention

[0021] According to the present invention, even if two or more detection objects are detected in a plurality of captured images, only the coordinate values ​​of the detection frame that contributed to the detection of the priority detection object can be used as valid data, and a detection object position detection system around a work machine capable of detecting the position of the priority detection object with high accuracy can be provided. By applying the present invention to a work machine that does not have a dedicated sensor or laser radar, it is possible to identify the position of a detection object around the work machine with high accuracy while avoiding high costs, and it is a technology with excellent practicality. [Brief description of the drawings]

[0022] [Figure 1] 1 is a diagram illustrating an example of application of a system for detecting the position of a detection target around a work machine according to an embodiment of the present invention; [Diagram 2] 2 is an overall configuration diagram of a detection system for detecting the position of a detection target around a work machine according to the embodiment; [Diagram 3] 13 is a diagram showing an example of a detection object image immediately after execution of a detection coordinate calculation process in the embodiment; FIG. [Figure 4] 5A and 5B are diagrams illustrating processing performed by a detection unit on a detection target image in the embodiment. [Diagram 5] 5 is a flowchart showing a detection program for detecting a position of a detection target around a work machine according to the embodiment; [Figure 6] 5A and 5B are diagrams illustrating processing contents of a reference coordinate calculation unit for a detection target image in the embodiment. [Figure 7] 5A and 5B are diagrams illustrating processing performed by a limited detection area calculation unit on a detection target image in the embodiment. [Figure 8] 5A and 5B are diagrams illustrating processing performed by a comparison coordinate calculation unit for a detection target image in the embodiment. [Figure 9] 4 is a diagram corresponding to FIG. 3 and showing an image in which accurate detection coordinates cannot be calculated in the embodiment; [Figure 10] 5A to 5C are diagrams for explaining the processing content of a large area weighting unit in the embodiment. [Figure 11] 5A to 5C are diagrams for explaining the processing content of a small region weighting unit in the embodiment. [Figure 12] 5 is a diagram showing an equation used in a detection coordinate calculation process in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. As shown in FIG. 1, the detection system X for detecting a position of a detected object around a work machine according to this embodiment can be used to monitor the periphery of a work machine H such as a hydraulic excavator (also called a backhoe). The detection system X for detecting a position of a detected object around a work machine includes a controller C capable of capturing an image of an imaging device I attached to the work machine H and outputting the image to an output destination. The imaging device I used in this embodiment is a general vehicle-mounted camera. The position at which the imaging device I is attached to the work machine H is not particularly limited as long as it is a position at which it can capture an image of a range (blind spot) or a range that is difficult to see from an operator in the cockpit H1. In this embodiment, the imaging device I is attached to a position at which it can capture an image of a predetermined range behind the hydraulic excavator H. Note that an imaging device capable of providing a 360-degree bird's-eye view of the periphery of the work machine from the sky above the operator's cabin can also be applied. In FIG. 1, the devices (the imaging device I and the display unit D described later) used in implementing the detection system X for detecting a detected object around a work machine are each exaggerated and typically shown.

[0024] As shown in Figures 1 and 2, the controller C is equipped with a detection unit C1 that detects position candidates of a detection object in an image captured by an imaging device I using a detection frame F (see Figure 3), a detection area limiting unit C2 that limits position candidates for a priority detection object to be detected with priority based on the detection result by the detection unit C1, and a detection coordinate calculation unit C3 that calculates the position of the priority detection object as detection coordinates based on the position candidates limited (specified) by the detection area limiting unit C2.

[0025] The detection object in this embodiment is a person W in the vicinity of the work machine. The imaging device I captures a video (image) including the detection object in the surrounding environment of the work machine H and captures it as image data. The detection object image V (video screen) detected by the detection unit C1 using the detection frame F may be the image data itself (unprocessed raw data), or it may be image data whose data size has been reduced for processing. FIG. 3 shows an example of the detection object image V immediately after execution of a detection coordinate calculation step S3 described later.

[0026] 4, the detection unit C1 detects an area where a person W exists while scanning the detection target image V using a rectangular detection frame F so as to thoroughly search the entire image. The size of the detection frame F in the detection unit C1 may be changed according to the expected height, body type, etc. of the person W to be detected. In this embodiment, for example, HOG (Histograms of Oriented Gradients) is applied as a feature amount used to detect the person W in the detection target image, and for example, SVM (Support Vector Machine) is applied as a learning method used for learning the detection.

[0027] The detection logic of the detection unit C1 in this embodiment (detection step S1; see FIG. 5) is as follows. Note that FIG. 5 is a flowchart of a work machine periphery detection program, which is a software program executed by a computer mounted on the work machine H as a work machine periphery detection system X for detecting the position of a detected object. This computer is either provided as a component of the controller C, or is the controller C itself.

[0028] First, the detection target image V at the start of the detection process is searched in the detection frame F to detect position candidates of the person W, which is the detection target. Next, the detection target image V is reduced by a specified ratio, and the same size detection frame F is used to search again to detect position candidates of the person W. Then, the detection target image V is reduced a specified number of times, or the detection target image V is terminated when it becomes smaller than the detection frame F. By performing the above processing, multiple position candidates are detected for each person W, and the position and size of the detection frame F in the detection target image V are different for each candidate. FIG. 4 shows a schematic diagram of the detection target image V when multiple position candidates are detected for the same person W (multiple detection frames F are displayed). As shown in the figure, the position candidates of the person W may be detected in a detection frame F that is closely attached to the person W (a detection frame F of the same size as the image of the person W), or in a frame one size larger than the detection frame that is closely attached to the person W.

[0029] 4, when two or more persons (two persons W1 and W2 in the figure) are shown in the detection target image V, multiple position candidates are detected for each of the persons W1 and W2, and the position and size of the detection frame F in the detection target image V differ for each of the persons W1 and W2. The figure shows a state in which position candidates based on four detection frames F are detected for each of the persons W1 and W2 (a total of eight position candidates based on the detection frames F).

[0030] In this embodiment, the person W1 closer to the imaging device I is set as a detection target (priority detection target) that is given priority over the person W2 farther away. Therefore, it is desirable that the position candidates based on the four detection frames F for the person W2 farther away from the imaging device I do not affect (are excluded from) calculation of the coordinates of the person W1, which is the priority detection target.

[0031] Therefore, in the work machine surrounding detection object position detection system X according to this embodiment, the detection area for the priority detection object is limited to a predetermined area in the image V by the detection area limiting unit C2 based on the detection result by the detection unit C1 (detection area limiting step S2; see FIG. 5).

[0032] The detection area limiting section C2 includes a reference coordinate calculation section C21, a limited detection area calculation section C22, a comparison coordinate calculation section C23, and a priority detection frame extraction section C24.

[0033] As shown in FIG. 6, the reference coordinate calculation unit C21 specifies a reference detection frame FA as a reference among a plurality of detection frames F in the video V based on the detection result by the detection unit C1, and calculates reference coordinates as a reference of a limited detection area for the reference detection frame FA. In this embodiment in which the person W1 closest to the imaging device I is set as the priority detection object, the reference coordinate calculation unit C21 sets the detection frame F closest to the imaging device I (the detection frame F shown by a relatively thick line in FIG. 6, which is the detection frame F closest to the imaging device I in the position candidates of the plurality of detection frames F in the detection target image V) as the reference detection frame FA, and calculates reference coordinates as a reference of a limited detection area for the reference detection frame FA (reference coordinate calculation step S21). As a specific calculation process in the reference coordinate calculation unit C21, in this embodiment, a process is applied in which the center X coordinate of the reference detection frame FA (the X coordinate shown by the straight line with the symbol "FSa" in FIG. 6) is calculated as a reference coordinate by the following formula (11). Center X coordinate of reference detection frame = X coordinate of reference detection frame + detection frame width of reference detection frame × 0.5 ... Equation (11)

[0034] Here, the position of the detection frame F within the detection target image V can be found as coordinate values ​​by a predetermined calculation. Therefore, the coordinate size of the detection frame F can be identified from the detection result by the detection unit C1, and the position coordinates (X coordinate) and detection frame width (length of the detection frame in the X-axis direction) of all detection frames F including the reference detection frame FA can also be identified. In this embodiment, the position of the bottom left corner (bottom left end) of the rectangular detection frame F is set as the X coordinate of the detection frame F. Also, the bottom left corner of the video screen (detection target image V) is set as the origin of the XY coordinates.

[0035] The limited detection area calculation unit C22 calculates the limited detection area based on the coordinates of the reference detection frame FA. In this embodiment, the minimum and maximum X coordinates of the detection area are calculated by the following formulas (12) and (13) based on the X coordinate and detection frame width of the reference detection frame FA, and the area between the minimum and maximum X coordinates is calculated as the limited detection area E as shown in Fig. 7 (limited detection area calculation step S22). Minimum X coordinate of detection area = X coordinate of reference detection frame + detection frame width of reference detection frame × 0.25 ... formula (12) Maximum X coordinate of detection area = X coordinate of reference detection frame + detection frame width of reference detection frame × 0.75 ... Equation (13)

[0036] In Fig. 7, the minimum X coordinate and the maximum X coordinate of the detection area are respectively indicated by dashed lines. Through such processing, as shown in Fig. 7, the area (area painted in gray) that is equal to or greater than the minimum X coordinate of the detection area and equal to or less than the maximum X coordinate of the detection area, i.e., the area within a range of a predetermined dimension in each of the positive and negative directions of the X axis from the center X coordinate FSa of the reference detection frame FA, is set as the limited detection area E.

[0037] The comparison coordinate calculation unit C23 calculates comparison coordinates for each of the non-reference detection frames F, which are detection frames F other than the reference detection frame FA, among the multiple detection frames F in the video V, based on the detection results by the detection unit C1. In this embodiment, for the detection frames F other than the reference detection frame FA among the detection frames F (seven detection frames F in total), the process calculates the center X coordinate of each detection frame F (the X coordinate indicated by the dashed line marked with the symbol "FSb" in FIG. 8) as the comparison coordinate (comparison coordinate calculation step S23), and this can be calculated using the following equation (14). Center X coordinate of non-reference detection frame = X coordinate of non-reference detection frame + detection frame width of non-reference detection frame × 0.5 ... Equation (14)

[0038] The priority detection frame extraction unit C24 extracts a priority detection frame FB, which is a detection frame F that is located within the limited detection region E, based on the comparison coordinates calculated by the comparison coordinate calculation unit C23. That is, the detection region limiting unit C2 in this embodiment has a priority detection frame extraction unit C24 that extracts, as a priority detection frame FB, a detection frame F whose comparison coordinates calculated by the comparison coordinate calculation unit C23 are located within the limited detection region E among the non-reference detection frames F. In the example shown in FIG. 8, the four non-reference detection frames F on the left side of the screen V among the seven non-reference detection frames F have center X coordinates FSb, which are comparison coordinates, less than the minimum X coordinate value of the limited detection region E, and the three non-reference detection frames F in the center of the screen V have center X coordinates FSb, which are comparison coordinates, greater than or equal to the minimum X coordinate value and less than or equal to the maximum X coordinate value of the limited detection region E. Therefore, in the example shown in FIG. 8, the three non-reference detection frames F in the center of the screen V are extracted as priority detection frames FB (priority detection frame extraction step S24). It can be identified from the drawing that the priority detection frames FB having comparison coordinates within the limited detection area E are all detection frames F used to detect the person W1, who is the priority detection object.

[0039] The above is the processing content (detection area limiting step S2) in the detection area limiting unit C2. By performing such processing, the detection frame F relating to detection objects other than the priority detection object can be eliminated so as not to affect the calculation processing by the detection coordinate calculation unit C3.

[0040] The detection coordinate calculation unit C3 includes a detection target identification coordinate setting unit C31 that uses the reference detection frame FA and the priority detection frame FB to set detection target identification coordinates FL that can identify the position of the person W, which is the priority detection target in the detection target image V, from each of these detection frames FA, FB (see FIG. 2). The detection coordinate calculation unit C3 does not use a non-reference detection frame F having comparison coordinates outside the limited detection area E. Therefore, the detection target identification coordinate setting unit C31 does not set the detection target identification coordinates FL based on the non-reference detection frame F having comparison coordinates outside the limited detection area E.

[0041] The position of each detection frame F in the detection target image V can be obtained as a coordinate value by a predetermined calculation as described above. In this embodiment, a rectangular detection frame F is applied in the detection unit C1, and as shown in Fig. 3, the detection target identification coordinate setting unit C31 is configured to set the position of the bottom side F1 of the detection frame F as the detection target identification coordinate FL. That is, the detection target identification coordinate setting unit C31 of this embodiment calculates the position of the feet of the person W1, who is the priority detection target in the detection target image V, as the detection target identification coordinate FL.

[0042] The reference detection frame FA and the priority detection frame FB, whose reference coordinates and comparison coordinates are within the limited detection region E, include not only the detection frame F that is in close contact with the priority detection object W1, but also the detection frame F that is not in close contact with the priority detection object W1, as described above. In such a case, when the detection coordinate calculation unit C3 treats the detection result detected with the detection frame F that is not in close contact with the priority detection object W1 as equivalent to the detection result detected with the detection frame F that is in close contact with the priority detection object W1 like the former, performs arithmetic averaging, and calculates the position of the priority detection object W1 in the detection target image V as the detection coordinate, the position of the priority detection object W1 according to the calculation result becomes an inaccurate position far from the actual position of the priority detection object W1. For example, as can be seen from FIG. 9, there is a variation in the position of the detection object identification coordinate FL (the position of the priority detection object W1 in the detection target image V, which is the lower side F1 of the detection frame F in this embodiment) calculated and set by the detection object identification coordinate setting unit C31. When calculating the position of the priority detection object W1 in the detection target image V as detection coordinates (representative coordinates) from such multiple position candidates F1 (coordinates of the lower side F1 of the detection frame F), if the arithmetic average of the multiple position candidates F1 is performed and the position of the priority detection object W1 in the detection target image V is calculated as the detection coordinates, the position FL' of the priority detection object W1 according to the calculation result is affected by the coordinates of the detection frame F of a large size as shown in the figure, and is calculated to be closer (closer to the camera I) than the coordinate FL (actual distance) that should be calculated, as shown by the dashed line in the figure. Conversely, there is a case where the detection object is detected with a detection frame F of a smaller size than the actual detection object, and in this case, the calculated distance is further back (farther from the camera I) than the actual distance. As described above, when a process of searching the detection target image V with the detection frame F to detect multiple detection target candidates is adopted, the calculation result is not stable if it is an arithmetic average.

[0043] In order to eliminate such a problem, in this embodiment, a predetermined weighting process is performed by the weighting process unit C32 in the detection coordinate calculation unit C3 that calculates the position of the priority detection object W1 as a detection coordinate. The predetermined weighting process by the weighting process unit C32 is a process performed following the process of setting the detection object specific coordinate FL by the detection object specific coordinate setting unit C31.

[0044] 2, the weighting processing unit C32 of this embodiment has a large area weighting processing unit C33 that performs a predetermined weighting for each large area obtained by dividing the entire detection target image into a plurality of areas according to the count number (density) of the detection target object identification coordinates FL (lower side F1 of the detection frame F) that can identify the position of the priority detection target object W1 in the detection frame F, and a small area weighting processing unit C34 that performs a predetermined weighting for each small area obtained by further dividing each large area into a plurality of areas according to the count number (density) of the detection target object identification coordinates FL (lower side F1 of the detection frame F). The processing content of the detection coordinate calculation unit C3 using such large area weighting processing unit C33 and small area weighting processing unit C34 will be described below.

[0045] FIG. 10(a) shows an example of horizontal lines representing the detection object identification coordinates FL (the position of the bottom side F1 of the detection frame F, the coordinate position of the feet of the person W1) calculated and set by the detection object identification coordinate setting unit C31 for a detection object image with a width of 720 pixels and a height of 480 pixels. FIG. 10(a) shows five horizontal lines, assuming that five detection object identification coordinates FL have been detected, by dividing the detection object image V into multiple large regions that are equal parts divided into 10 regions in the height direction. FIG. 10(b) is a histogram showing how many detection object identification coordinates FL belong to each large region (the count number of detection object identification coordinates FL). It can be seen from the figure that one detection object identification coordinate FL belongs to large region 1, and four detection object identification coordinates FL belong to large region 3.

[0046] Fig. 11(a) is a diagram showing a state where large region 3, which had the largest count number of detection object identification coordinates FL among the multiple large regions shown in Fig. 10(a), is further divided into eight small regions (48 pixels in height) by dividing it equally in the height direction. Fig. 11(b) is a histogram showing how many detection object identification coordinates FL belong to each small region (the number of counts of detection object identification coordinates FL), and it can be seen from this figure that three detection object identification coordinates FL belong to small region 3 and one detection object identification coordinate FL belongs to small region 4. Note that the number of large regions and the number of small regions are not limited to 10 and 8, respectively, and may be changed to appropriate values.

[0047] The weighting processing unit C32 of this embodiment uses these pieces of information to perform weighting processing to take a weighted average, increasing the weight near the coordinates where the detection object specifying coordinates FL are dense, and decreasing the weight when the detection object specifying coordinates FL are far from the coordinates where they are dense. Specifically, weighting processing is performed to multiply the count number of the detection object specifying coordinates FL that can be grasped from the histogram by some power (for example, cube). The power of the weight is determined as a trade-off with the computation cost, and the larger the multiplier, the stronger the feature.

[0048] The calculation and setting process of the detection object specific coordinate FL by the detection object specific coordinate setting unit C31, and the detection coordinate calculation process by the detection coordinate calculation unit C3 involving weighting processing by the weighting processing unit C32 (detection coordinate calculation step S3; see Figure 5) are performed by calculation processing using the equations shown in Figure 12.

[0049] First, the representative coordinates of each large area are calculated using equation (1) shown in the figure. Specifically, each large area is divided into small areas, and the number of detection object identification coordinates FL (foot coordinates) belonging to each small area is multiplied by a certain power to determine the weight. The arithmetic mean of the detection object identification coordinates FL (foot coordinates) belonging to each small area is determined as the representative coordinates of the small area, and a weighted average is calculated from these. The calculation result is the representative coordinates of each large area. In this way, small area weighting processing is performed by the small area weighting processing unit C34 when calculating the representative coordinates of each large area.

[0050] Next, the detection coordinates (detection target object specified coordinates FL) of the current detection target image (current detection target image) are calculated using equation (2) shown in FIG. 12. Specifically, the number of feet coordinates belonging to each large area is multiplied by a certain power, and the result is used as the weight. The representative coordinates of each large area calculated using equation (1) are used to calculate a weighted average. The calculation result is the detection target object specified coordinates FL (real-time detection coordinates) of the current detection target image. In this way, large area weighting processing is performed by the large area weighting processing unit C33 when calculating the detection target object specified coordinates FL of the current detection target image.

[0051] Here, from the relationship between the mounting position and mounting angle of the imaging device I and the coordinates on the detection target image V, it is possible to calculate the distance from the imaging device I to the priority detection target W1 based on the detection target specific coordinates FL (foot coordinates) on the detection target image V. In this embodiment, a table is prepared in advance in which the distance from the imaging device I to the detection target W is associated with each coordinate on the detection target image V, and the system is set to instantly calculate the distance from the imaging device I to the detection target W from the detection target specific coordinates FL on the detection target image V by referring to the table.

[0052] Next, the detection coordinates are calculated using equation (3) shown in FIG. 12. Specifically, the detection coordinates of the current detection target image calculated using equation (2) and past detection coordinates (detection coordinates for multiple times including the most recent past) are used to calculate the detection coordinates by taking a moving average. However, if there is a person near the work machine H, a quick response is required, so the weight of the moving average is changed according to the detection coordinates of the current detection target image (see equation (3) in FIG. 12). In the example of equation (3), the dangerous distance from the imaging device I is set to 2 m, but this distance may be changed as appropriate. Furthermore, in the example of equation (3), the weight values ​​of the current image and the nth previous image are clearly indicated, but the weight values ​​are not limited to this. In the example of equation (3), the number of moving averages is set to 5 (an equation using one real-time detection coordinate and five detection coordinates for four times including the most recent past), but the number of moving averages is not limited to this. In this manner, in this embodiment, when calculating the detection coordinates, a method is adopted in which, depending on the distance (detection coordinates) calculated by the weighted average, a process is selected in which no weighting is applied to the moving average when the detection coordinates are greater than the danger distance, and a process in which the weighting is gradually decreased as the moving average processing progresses when the detection coordinates are equal to or less than the danger distance, and when an early warning is required, responsiveness is prioritized over accuracy.

[0053] In this embodiment, the distance measuring unit C4 of the controller C measures (specifies) the distance from the work machine H (specifically, the imaging device I) to the priority detection object W1 by referring to the above table based on the calculation result (detection coordinates) by the detection coordinate calculation unit C3 (distance measuring step S4; see FIG. 5). The distance measuring unit C4 of this embodiment outputs the measurement result (distance to the priority detection object W1) as an image to the monitor D. In this embodiment, as an example of outputting the measurement result to the monitor D, as shown in FIG. 3, a process is adopted in which the measurement result (distance to the priority detection object W1) is displayed by a horizontal line FL that is longer in the width direction than any of the detection frames F. In particular, by displaying the horizontal line FL on the monitor D in a color different from that of the detection frame F, the operator can intuitively grasp the distance to the priority detection object W1. Note that the distance measuring unit C4 may be provided with a function of outputting sound or light via an appropriate speaker or light emitter to notify or warn that a person is in a position close to the work machine H.

[0054] Thus, according to the work machine surroundings detection object position detection system X of this embodiment, when calculating detection coordinates, the above-mentioned weighting process is performed in which the weight is increased for an area with a larger number of detection object identification coordinates FL (foot coordinates), so that the calculation result in which the position of the priority detection object W1 in the detection target image V is calculated as detection coordinates reflects the processing content weighted by the density of the detection frame F. As a result, even if there is variation in the position candidates of the priority detection object W1 around the work machine in image recognition using the detection frame F, the variation in the detection coordinates identifying the position of the priority detection object W1 in the detection target image V is reduced, and the detection coordinates for the priority detection object W1 in the detection target image V can be calculated with high accuracy.

[0055] In particular, according to the work machine surroundings detection object position detection system X of this embodiment, even if two or more detection objects W1, W2 are detected using the detection frame F, one of the detection objects is set as the priority detection object W1 and the detection frame F related to the priority detection object W1 is extracted, so that the detection frame F related to detection objects other than the priority detection object W1 is not treated as valid data when calculating the position of the priority detection object W1 as a detection coordinate, and it is possible to detect with high accuracy the position of the priority detection object W1 captured in the imaging device I. Specifically, according to the work machine surroundings detection object position detection system X of this embodiment, a reference coordinate FSa serving as the reference for the limited detection area E is calculated using a reference detection frame FA serving as a reference among a plurality of detection frames F in the image V, and the limited detection area E is calculated based on the reference coordinate FSa, so that the limited detection area E can be set in the detection object image V by a relatively simple calculation process. In the work machine surrounding detection object position detection system X according to this embodiment, a comparison coordinate FSb is calculated for each of the non-reference detection frames F among the multiple detection frames F in the image V, and a detection frame F having a comparison coordinate FSb within the limited detection area E among the non-reference detection frames F is extracted as a priority detection frame FB, and when calculating the position of the priority detection object W1, only the reference detection frame FA and the extracted priority detection frame FB are used, so that the position of the priority detection object W1 can be detected without being affected by the coordinate values ​​of the non-reference detection frames F related to detection objects other than the priority detection object W1.

[0056] Furthermore, according to the work machine surroundings detection object position detection system X of this embodiment, the final detection coordinates are calculated by taking a moving average of the real-time detection coordinates and the multiple detection coordinates including the most recent past detection coordinates, so that the final detection coordinates can be calculated with high accuracy considering the time-series movement of the same priority detection object W1 in the detection target image V, compared to a mode in which the real-time detection coordinates are the final detection coordinates. In particular, by adopting a configuration in which the weight of the moving average is changed (the weight is dynamically switched) according to the real-time detection coordinates, when a person who is the nearest priority detection object W1 is away from the work machine H, the weight of the moving average is set relatively large, so that the final detection coordinates can be detected with high accuracy, whereas when the real-time detection coordinates are in a close position within a predetermined range from the work machine H, a dangerous situation in which a person is in a close position within a predetermined range from the work machine H can be immediately output as a calculation result, and detection processing that prioritizes responsiveness can be performed.

[0057] Although the embodiment of the present invention has been described above, the present invention is not limited to the configuration of the above embodiment. For example, in the above embodiment, a hydraulic excavator is exemplified as a work machine, but the peripheral detection target position detection system of the present invention can be applied to work machines other than hydraulic excavators (machines that exhibit special performance for specific purposes, such as construction machines, agricultural and forestry work machines, or various vehicles other than ships and airplanes).

[0058] In the above-described embodiment, when multiple detection objects are detected in a detection target image by detection processing using a detection frame, the detection object closest to the work machine is set as a priority detection object and only the position of the priority detection object is detected. However, it is also possible to adopt a configuration in which a detection object other than the detection object closest to the work machine is set as a priority detection object and only the position of the priority detection object is detected, or a configuration in which a priority is determined for multiple detection objects and the above-described detection area limiting processing and detection coordinate calculation processing are performed for each detection object (for example, a first priority detection object, a second priority detection object, a third priority detection object, etc.). In this case, first, the detection object closest to the work machine is set as a first priority detection object, and a priority detection frame FB for the first priority detection object is extracted by the priority detection frame extraction unit C24 of the detection area specification unit C2. Next, the priority detection frame FB for the first priority detection object is removed from the detection frame F, and the detection object next closest to the first priority detection object from the work machine is set as the second priority detection object, and the priority detection frame FB for the second priority detection object is similarly extracted. By repeating this process according to the number of detection objects, it is possible to extract a priority detection frame for each detection object. Then, it is also possible to configure the system to measure (identify) the distance from the work machine (specifically, the imaging device) for all or any of the detection objects based on the processing result (calculation result) by the detection coordinate calculation unit.

[0059] In addition, when multiple detection objects are detected in an image, it is possible to perform a grouping process based on a predetermined coordinate (e.g., X coordinate) of each detection frame, and calculate the coordinates of each detection object based on the processing results.

[0060] Coordinates other than the central X-axis coordinate (for example, the central coordinates, X coordinate, Y coordinate, central Y coordinate, etc. of the reference frame or non-reference frame) may be applied as the reference coordinates of the reference frame or the comparison coordinates of the non-reference frame. Furthermore, when calculating the limited detection area based on the reference coordinates of the reference frame, the coefficient values ​​used in the calculation formula using the reference coordinates (in the above embodiment, "0.25" in (Equation 12) and "0.75" in (Equation 13)) may be changed as appropriate. This allows the range of the limited detection area to be selected and changed as appropriate.

[0061] The processing contents shown in Figures 4, 6 to 8 (the processing contents of the detection unit for the detection target image, the processing contents of the reference coordinate calculation unit, the processing contents of the limited detection area calculation unit, and the processing contents of the comparison coordinate calculation unit) are not actually displayed on the display unit D of the work machine, but may be set to be displayed on the display unit D if desired.

[0062] In addition, in the above-described embodiment, an example was given of an aspect having both a large area weighting processing unit and a small area weighting processing unit, but it is also possible to adopt an aspect having only one of the weighting processing units (for example, an aspect in which the representative coordinates of the large area are obtained by arithmetic averaging, although this reduces accuracy), or to adopt an aspect in which weighting is performed for each of the areas obtained by further dividing the small area into a plurality of areas according to the count number of the coordinates identifying the detected object.

[0063] A plurality of imaging devices may be attached to one (one) work machine, and the detection area limiting process may be performed individually for each image captured by each imaging device, or the detection coordinate calculation process may be performed with weighting. Alternatively, the current images captured by each imaging device and taken at the same time may be collectively subjected to the detection area limiting process, or the detection coordinate calculation process may be performed with weighting. A plurality of monitors visible to the operator may also be set.

[0064] The number of divisions into the large region and the number of divisions into the small regions in the above-described embodiment can be changed as appropriate.

[0065] Furthermore, the weighting in the weighting processing section may be weighting different from the weighted average based on the count number of the detection target object specific coordinates.

[0066] In the present invention, when multiple detection targets are detected in a detection target image by detection processing using a detection frame, the position of each detection target can be detected by a detection coordinate calculation process involving the above-mentioned weighting process.

[0067] In the above embodiment, an example was given of the detection target object identification coordinates being coordinates corresponding to the bottom side of a rectangular detection frame, but the detection target object identification coordinates may also be coordinates corresponding to a side or corner other than the bottom side of the detection frame.

[0068] As features used to detect a detection target object in a detection target image, features other than HOG (Histograms of Oriented Gradients), such as CoHOG, LBP, edgelet, Haar, Joint-Haar, etc., can be applied, and as a learning method used for detection training, a method other than SVM (Support Vector Machine), such as boosting, random trees, k-NN, Perceptron, Passive Aggressive, AROW, Confidence weighted, etc. can be applied.

[0069] As the display unit, in addition to a monitor, a head-up display (HUD) or a head-mounted display (HMD) may be used.

[0070] Furthermore, the detection object in the present invention is not limited to a person. Even if the detection object is a living thing other than a human (such as an animal), or a small construction machine or a work support robot (for example, a disaster support robot, a ground drone, etc.), the position of the detection object can be detected by the detection object position detection system around a work machine and the detection object position detection program around a work machine according to the present invention.

[0071] Furthermore, the specific configuration of each part is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0072] C1: Detection section C2: Detection area limiting section C21: Reference coordinate calculation section C22: Limited detection area calculation section C23: Comparison coordinate calculation section C24: Priority detection frame extraction section C3: Detection coordinate calculation section C31: Weighting processing section C32: Large area weighting processing section C33: Small area weighting processing section E…Limited detection area FA: Reference detection frame FB: Priority detection frame FSa…reference coordinates FSb…Comparison coordinates H: Work machine I…Imaging device X: Work machine surrounding detection object position detection system

Claims

1. A work machine surrounding detection object position detection system capable of detecting a position of a detection object around a work machine, comprising: a detection unit that detects position candidates of a detection target object in an image captured by an imaging device attached to the work machine by using detection frames of a plurality of sizes for the image; a detection area limiting unit that limits a detection area for a priority detection object that is to be preferentially detected based on a detection result by the detection unit to within a predetermined area in the image; a detection coordinate calculation unit that calculates a position of the priority detection object as a detection coordinate based on a detection result by the detection unit within the limited detection area limited by the detection area limiting unit, The detection area limiting unit is a reference coordinate calculation unit that calculates reference coordinates that serve as a reference for the limited detection area for a reference detection frame that serves as a reference among the plurality of detection frames in the image based on a detection result by the detection unit; a limited detection area calculation unit that calculates the limited detection area based on the reference coordinates; a comparison coordinate calculation unit that calculates comparison coordinates for each of the non-reference detection frames, which are detection frames other than the reference detection frame, among the plurality of detection frames in the video based on a detection result by the detection unit; a priority detection frame detection unit that extracts, from the non-reference detection frames, a detection frame whose comparison coordinates calculated by the comparison coordinate calculation unit are within the limited detection area, as a priority detection frame.

2. 2. The system for detecting a detected object position around a work machine according to claim 1, wherein the detection coordinate calculation unit calculates a position of the priority detection object in the detection target image as detection coordinates based on a processing result in which, for each of a plurality of regions obtained by dividing a detection target image in the video, weighting is performed in accordance with a count number of detection target identification coordinates that can identify the position of the priority detection object from the reference detection frame and the priority detection frame having the comparison coordinates within the limited detection area.

3. 3. The system for detecting the position of a detection target around a work machine according to claim 1, wherein the reference coordinate is a central X coordinate of the reference detection frame, and the comparison coordinate is a central X coordinate of the non-reference detection frame.

4. 4. A detection system for detecting the position of a detection object in the vicinity of a work machine according to claim 1, wherein the system is configured to measure a distance from the imaging device to the priority detection object based on the calculated detection coordinates.

5. A software program for causing a computer to function as a work machine surroundings detection object position detection system for detecting the position of a detection object around a work machine, a detection step of detecting position candidates of a detection target object in an image captured by an imaging device attached to the work machine using detection frames of a plurality of sizes for the image; a detection area limiting step of limiting a detection area for a priority detection object to be preferentially detected based on a detection result by the detection step to within a predetermined area in the image; a detection coordinate calculation step of calculating a position of a priority detection object in an image as a detection coordinate based on a detection result by the detection step within a limited detection area limited by the detection area limiting step, The detection area limiting step includes: a reference coordinate calculation step of calculating reference coordinates serving as a reference for the limited detection area for a reference detection frame serving as a reference among the plurality of detection frames in the image based on a detection result by the detection step; a limited detection area step of calculating the limited detection area based on the reference coordinates; a comparison coordinate calculation step of calculating comparison coordinates for each of the non-reference detection frames, which are detection frames other than the reference detection frame, among the plurality of detection frames in the image based on a detection result in the detection step; and a priority detection frame detection step of extracting, from the non-reference detection frames, a detection frame whose comparison coordinates calculated in the comparison coordinate calculation step are within the limited detection area, as a priority detection frame.

Images