Work vehicle display system and work vehicle display method
The display system for work vehicles enhances obstacle detection by prioritizing high-accuracy markers, ensuring operators focus on crucial information amidst multiple detections.
Patent Information
- Application Number
- JP2021533111
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-07-18
- Filing Date
- 2020-07-17
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-07-17
AI Technical Summary
Existing systems display multiple detected obstacles simultaneously, overwhelming the operator and making it difficult to identify necessary information quickly.
A display system for work vehicles that prioritizes displaying obstacles with the highest detection accuracy, using multiple cameras to generate an overhead image with markers indicating the certainty of the obstacle's position, reducing the number of displayed markers to focus on critical information.
Reduces the likelihood of overlooking important obstacle detection information by highlighting the most accurate detections, allowing operators to prioritize nearby and critical obstacles.
Smart Images

Figure 0007590325000001 
Figure 0007590325000002 
Figure 0007590325000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a work vehicle display system and a work vehicle display method. This application claims priority to Japanese Patent Application No. 2019-133074, filed in Japan on July 18, 2019, the contents of which are incorporated herein by reference. [Background technology]
[0002] Patent Document 1 discloses technology related to a perimeter monitoring system that detects people around a work vehicle. According to the technology described in Patent Document 1, the perimeter monitoring system uses image processing to identify whether or not a person is included in an image. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2017-151815 A Summary of the Invention [Problem to be solved by the invention]
[0004] In the process of detecting obstacles such as people, multiple obstacles may be detected from one area. In this case, if all obstacles detected by the detection process are displayed on the screen, a large number of detection results will be displayed, and it may be difficult to immediately identify the necessary information.
[0005] An object of the present disclosure is to provide a work vehicle display system and a work vehicle display method that can reduce the possibility of overlooking necessary information related to obstacle detection. [Means for solving the problem]
[0006] According to a first aspect of the present disclosure, a display system for a work vehicle includes an image acquisition unit that acquires an image of the area around the work vehicle, a detection unit that detects obstacles around the work vehicle, and a display control unit that, when a plurality of obstacles are detected by the detection unit, generates a signal to display an obstacle in an identifiable manner at a position corresponding to an obstacle that has a high detection accuracy indicating the likelihood of the obstacle being detected among the plurality of obstacles in an area corresponding to the captured image. Effect of the Invention
[0007] According to the above aspect, the display system can reduce the possibility of overlooking necessary information related to obstacle detection. [Brief description of the drawings]
[0008] [Figure 1] 1 is a schematic diagram showing a configuration of a work vehicle according to a first embodiment. [Diagram 2] 2 is a diagram showing the imaging ranges of a plurality of cameras provided on the work vehicle according to the first embodiment. FIG. [Diagram 3] FIG. 2 is a diagram showing the internal configuration of a driver's cab according to the first embodiment. [Figure 4] 1 is a schematic block diagram showing a configuration of a control device according to a first embodiment. [Diagram 5] 5 is a flowchart showing the operation of the control device according to the first embodiment. [Figure 6] 4 is an example of a display image according to the first embodiment. [Figure 7] 4 is an example of a display image according to the first embodiment. [Figure 8] 4 is an example of a display image according to the first embodiment. [Figure 9] 4 is an example of a display image according to the first embodiment. [Figure 10] 10 is a flowchart showing an operation of a control device according to another embodiment. [Figure 11] 13 is an example of a display image according to another embodiment. [Figure 12]13 is an example of a display image according to another embodiment. [Figure 13] 13 is a diagram illustrating an example of correspondence between a frame region of a display image and a captured image in another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] First embodiment Hereinafter, the embodiments will be described in detail with reference to the drawings.
[0010] <<Configuration of work vehicles>> FIG. 1 is a schematic diagram showing the configuration of a work vehicle according to a first embodiment. The work vehicle 100 operates at a construction site and works on a construction target such as earth and sand. The work vehicle 100 according to the first embodiment is, for example, a hydraulic excavator. The work vehicle 100 includes a running body 110, a rotating body 120, a work machine 130, and a cab 140. The running body 110 supports the work vehicle 100 so that the work vehicle 100 can run. The running body 110 is, for example, a pair of left and right caterpillar tracks. The rotating body 120 is supported by the running body 110 so as to be capable of rotating about a rotation center. The work machine 130 is hydraulically driven. The work machine 130 is supported on the front part of the rotating body 120 so as to be drivable in the vertical direction. The operator's cab 140 is a space where an operator boards and operates the work vehicle 100. The operator's cab 140 is provided on the left front part of the rotating body 120. Here, the part of the rotating body 120 to which the work machine 130 is attached is referred to as the front part. Furthermore, with respect to the rotating body 120, the part opposite the front part is referred to as the rear part, the left part as the left part, and the right part as the right part.
[0011] <<Configuration of rotating body>> A plurality of cameras 121 are provided on the rotating body 120 to capture images of the surroundings of the work vehicle 100. Fig. 2 is a diagram showing the imaging ranges of the plurality of cameras provided on the work vehicle according to the first embodiment. Specifically, the rotating body 120 is provided with a left rear camera 121A that images a left rear range Ra of the periphery of the rotating body 120, a rear camera 121B that images a rear range Rb of the periphery of the rotating body 120, a right rear camera 121C that images a right rear range Rc of the periphery of the rotating body 120, and a right front camera 121D that images a right front range Rd of the periphery of the rotating body 120. Note that the imaging ranges of the multiple cameras 121 may partially overlap each other. The imaging ranges of the multiple cameras 121 cover the entire circumference of the work vehicle 100 excluding the left front range Re visible from the cab 140. Note that, although the cameras 121 according to the first embodiment capture images of the left rear, rear, right rear, and right front of the revolving body 120, other embodiments are not limited to this. For example, the number and imaging ranges of the cameras 121 according to other embodiments may be different from the examples shown in FIGS. 1 and 2.
[0012] The left rear camera 121A, as shown in the rear range Rb in Fig. 2, images the left side area and the left rear area of the revolving body 120, but may image either one of these areas. Similarly, the right rear camera 121C, as shown in the right rear range Rc in Fig. 2, images the right side area and the right rear area of the revolving body 120, but may image either one of these areas. Similarly, the right front camera 121D, as shown in the right front range Rd in Fig. 2, images the right front area and the right side area of the revolving body 120, but may image either one of these areas.
[0013] <<Configuration of the work machine>> The work machine 130 includes a boom 131 , an arm 132 , a bucket 133 , a boom cylinder 134 , an arm cylinder 135 , and a bucket cylinder 136 .
[0014] The base end of the boom 131 is attached to the rotating body 120 via a boom pin P1. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is attached to the tip of the boom 131 via an arm pin P2. The bucket 133 includes a blade for excavating earth and sand, and a storage portion for storing the excavated earth and sand. A base end portion of the bucket 133 is attached to a tip portion of the arm 132 via a bucket pin P3.
[0015] The boom cylinder 134 is a hydraulic cylinder for operating the boom 131. A base end of the boom cylinder 134 is attached to the rotating body 120. A tip end of the boom cylinder 134 is attached to the boom 131. The arm cylinder 135 is a hydraulic cylinder for driving the arm 132. A base end of the arm cylinder 135 is attached to the boom 131. A tip end of the arm cylinder 135 is attached to the arm 132. The bucket cylinder 136 is a hydraulic cylinder for driving the bucket 133. A base end of the bucket cylinder 136 is attached to the arm 132. A tip end of the bucket cylinder 136 is attached to a link member connected to the bucket 133.
[0016] <Driver's cab configuration> FIG. 3 is a diagram showing the internal configuration of the operator's cab according to the first embodiment. A driver's seat 141, an operating device 142, and a control device 143 are provided in the driver's cab 140.
[0017] The operation device 142 is an interface for driving the traveling body 110, the rotating body 120, and the work machine 130 by manual operation by an operator. The operation device 142 includes a left operation lever 1421, a right operation lever 1422, a left foot pedal 1423, a right foot pedal 1424, a left traveling lever 1425, and a right traveling lever 1426.
[0018] The left operating lever 1421 is provided on the left side of the driver's seat 141. The right operating lever 1422 is provided on the right side of the driver's seat 141.
[0019] The left operation lever 1421 is an operation mechanism for performing the rotation operation of the revolving body 120 and the pulling / pushing operation of the arm 132. Specifically, when the operator of the work vehicle 100 tilts the left operation lever 1421 forward, the arm 132 performs a pushing operation. When the operator of the work vehicle 100 tilts the left operation lever 1421 backward, the arm 132 performs a pulling operation. When the operator of the work vehicle 100 tilts the left operation lever 1421 to the right, the revolving body 120 rotates to the right. When the operator of the work vehicle 100 tilts the left operation lever 1421 to the left, the revolving body 120 rotates to the left. Note that in other embodiments, when the left operation lever 1421 is tilted in the forward / rearward direction, the revolving body 120 rotates to the right or left, and when the left operation lever 1421 is tilted in the left / right direction, the arm 132 performs a dumping operation or an excavation operation.
[0020] The right operating lever 1422 is an operating mechanism for performing an excavation / dump operation of the bucket 133 and a raising / lowering operation of the boom 131. Specifically, when the operator of the work vehicle 100 tilts the right operating lever 1422 forward, a lowering operation of the boom 131 is performed. When the operator of the work vehicle 100 tilts the right operating lever 1422 rearward, a raising operation of the boom 131 is performed. When the operator of the work vehicle 100 tilts the right operating lever 1422 to the right, a dump operation of the bucket 133 is performed. When the operator of the work vehicle 100 tilts the right operating lever 1422 to the left, an excavation operation of the bucket 133 is performed.
[0021] The left foot pedal 1423 is disposed on the left side of the floor surface in front of the driver's seat 141. The right foot pedal 1424 is disposed on the left side of the floor surface in front of the driver's seat 141. The left travel lever 1425 is pivotally supported by the left foot pedal 1423 and configured so that the tilt of the left travel lever 1425 and the depression of the left foot pedal 1423 are linked. The right travel lever 1426 is pivotally supported by the right foot pedal 1424 and configured so that the tilt of the right travel lever 1426 and the depression of the right foot pedal 1424 are linked.
[0022] The left foot pedal 1423 and the left travel lever 1425 correspond to the rotational drive of the left track of the traveling body 110. Specifically, when the operator of the work vehicle 100 tilts the left foot pedal 1423 or the left travel lever 1425 forward, the left track rotates in the forward direction. Conversely, when the operator of the work vehicle 100 tilts the left foot pedal 1423 or the left travel lever 1425 backward, the left track rotates in the reverse direction.
[0023] The right foot pedal 1424 and the right travel lever 1426 correspond to the rotational drive of the right track of the running body 110. Specifically, when the operator of the work vehicle 100 pushes the right foot pedal 1424 or the right travel lever 1426 forward, the right track rotates in the forward direction. Conversely, when the operator of the work vehicle 100 pushes the right foot pedal 1424 or the right travel lever 1426 backward, the right track rotates in the reverse direction.
[0024] The control device 143 is an input / output device, and can display, for example, an overhead image of the surrounding situation of the work vehicle 100 captured by the multiple cameras 121. The input means of the control device 143 according to the first embodiment is a hard key. Note that in other embodiments, a touch panel, a mouse, a keyboard, or the like may be used as the input means.
[0025] Configuration of the control device FIG. 4 is a schematic block diagram showing the configuration of the control device according to the first embodiment. The control device 143 is a computer that includes a processor 210, a main memory 230, a storage 250, and an interface 270.
[0026] Storage 250 is a non-transitory tangible storage medium. Examples of storage 250 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), and a semiconductor memory. Storage 250 may be an internal medium directly connected to the bus of control device 143, or may be an external medium connected to control device 143 via interface 270 or a communication line. Storage 250 stores a program for implementing monitoring of the surroundings of work vehicle 100.
[0027] The program may be for realizing a part of the functions to be performed by the control device 143. For example, the program may be for realizing the functions by combining with other programs already stored in the storage 250 or by combining with other programs implemented in other devices. In another embodiment, the control device 143 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor may be realized by the integrated circuit.
[0028] The storage 250 also stores person dictionary data D1 for detecting people. The person dictionary data D1 is, for example, dictionary data of feature amounts extracted from each of a plurality of known images in which people appear. Examples of the feature amounts include HOG (Histograms of Oriented Gradients) and CoHOG (Co-occurrence HOG).
[0029] By executing the program, the processor 210 functions as an image acquisition unit 211, a feature extraction unit 212, a detection unit 213, an image generation unit 214, a marker placement unit 215, a presence determination unit 216, and a display control unit 217. In addition, by executing the program, a storage area for a marker storage unit 231 is secured in the main memory 230.
[0030] The image acquisition unit 211 acquires images from the multiple cameras 121. The images acquired from each camera 121 are frame images that are updated at a predetermined frame rate. The feature extraction unit 212 extracts features from the frame images acquired by the image acquisition unit 211. Examples of the features extracted by the feature extraction unit 212 include HOG features and CoHOG features.
[0031] The detection unit 213 detects a person in the frame image based on the feature extracted by the feature extraction unit 212 and the person dictionary data D1 stored in the storage 250. Examples of a person detection method include pattern matching and object detection processing based on machine learning. A person is an example of an obstacle. In the first embodiment, the detection unit 213 detects a person using the feature of an image, but is not limited to this. For example, in other embodiments, the detection unit 213 may detect a person based on a measurement value of an infrared sensor. The detection unit 213 specifies the detection accuracy for each detected person. The detection accuracy is an accuracy indicating the certainty of a person, and is expressed as, for example, the similarity or likelihood of the feature.
[0032] The image generating unit 214 generates an overhead image of a work site viewed from above in plan, with the work vehicle 100 at the center, by transforming and combining the multiple frame images acquired by the image acquiring unit 211. That is, the image generating unit 214 generates an overhead image by transforming each of the multiple frame images into an overhead coordinate system and combining the transformed frame images. The image generating unit 214 may also generate an overhead image by cutting out a portion of each transformed frame image and combining the cut-out frame images. An image of the work vehicle 100 viewed from above in plan is attached in advance to the center of the overhead image generated by the image generating unit 214.
[0033] The marker placement unit 215 places a marker indicating the position of the person at a position corresponding to the detection position of the person by superimposing it on the overhead image generated by the image generation unit 214. Note that the marker placement unit 215 according to another embodiment may place the marker within a predetermined range from the detection position. Examples of the shape of the marker include a circle, an ellipse, a regular polygon, and a polygon. Note that the marker is an example of a display capable of identifying an obstacle. In another embodiment, the display capable of identifying an obstacle may be, for example, a display capable of identifying an obstacle, such as displaying text indicating a person, changing the color of the person, or changing the shape of the person. The marker placement unit 215 records the positions of the placed markers in the marker storage unit 231. The marker storage unit 231 stores the positions of the markers placed on the overhead image.
[0034] The marker placement unit 215 changes the color of the marker depending on the positional relationship between the detection position and a warning range set around the work vehicle 100. For example, the marker placement unit 215 changes the color of the marker to red when the detection position is within the warning range, and changes the color of the marker to yellow when the detection position is outside the warning range. The warning range may be set as a circle centered around the turning center of the work vehicle 100, for example. The color of the marker and the relationship between the position and the color of the marker are not limited to this. For example, the marker placement unit 215 according to another embodiment may set the marker to a color other than red and yellow. Furthermore, the marker placement unit 215 according to another embodiment may set the color of the marker to red when the detection position is within the warning range, set the color of the marker to yellow when the detection position is outside the warning range and within the caution range, and set the color of the marker to gray when the detection position is outside the caution range. Furthermore, the color of the marker may be red when the detection position is within the warning range or the caution range. Furthermore, the color of the marker may be red regardless of the detection position. The caution range may be, for example, a circle centered on the turning center of the work vehicle 100, and may be set as a range outside the warning range.
[0035] Furthermore, the marker placement unit 215 may highlight an area including the position where the person is detected together with the placement of the marker in the overhead image. The area is determined by the imaging ranges of the multiple cameras 121. The area may be determined by the imaging range of one camera 121, or may be determined by the imaging ranges of at least two cameras 121. The area may be narrower than the imaging range. An example of highlighting is displaying the frame of the area with a thick red frame. The example of highlighting is not limited to this, and may be a color other than red. The highlighting may be performed by blinking the frame of the area.
[0036] Also, the marker placement unit 215 may highlight an area including a detected position only when the detected position is within a warning range or a caution range. In this case, a marker is placed on the detected person and an area corresponding to the detected position is highlighted, so that the operator can easily understand the person and direction to be careful of. In other words, the control device 143 can alert the operator.
[0037] The presence determining unit 216 determines whether or not a person is present inside a caution range or a warning range that is set with the work vehicle 100 at the center, based on the detected position in the overhead coordinate system.
[0038] The display control unit 217 generates a surveillance image by attaching an icon corresponding to the determination result of the presence determination unit 216 to the overhead image on which the marker is placed. The display control unit 217 outputs the generated surveillance image to a touch panel. The icon has a different appearance depending on, for example, whether a person is not present within the caution range, whether a person is present within the caution range but outside the warning range, or whether a person is present within the warning range. Examples of the appearance of the icon include color, transparency, size, etc.
[0039] <<Operation of the control device>> FIG. 5 is a flowchart showing the operation of the control device according to the first embodiment. When the control device 143 starts the surroundings monitoring process, the image acquisition unit 211 acquires frame images from the multiple cameras 121 (step S1). Next, the control device 143 executes a person detection process P1 and a display control process P2 in parallel based on the acquired frame images.
[0040] The person detection process P1 includes the following steps S2 and S3. The feature extraction unit 212 extracts features from the frame image acquired in step S1 (step S2). Next, the detection unit 213 reads the person dictionary data D1 from the storage 250, and identifies the detection position where the person exists in the frame image by comparing the extracted features with the person dictionary data D1 (step S3). The person detection process P1 requires a larger amount of calculation than the display control process P2.
[0041] The display control process P2 includes the following steps S4 to S9. The image generating unit 214 generates an overhead image based on the frame image acquired in step S1 (step S4). Next, the presence determining unit 216 determines whether or not a person is present inside the caution range or warning range based on the position of the marker stored in the marker storage unit 231 (step S5).
[0042] The display control unit 217 generates a surveillance image by drawing a marker on the overhead image generated in step S4, the position and size of which are stored in the marker storage unit 231, and attaching an icon corresponding to the determination result of step S7 to a position of the overhead image corresponding to the left front range Re (step S6). The display control unit 217 outputs the generated surveillance image to a touch panel (step S7).
[0043] Next, the display control unit 217 determines whether the person detection process P1 is completed (step S8). If the person detection process P1 is not completed (step S8: NO), the image acquisition unit 211 acquires the next frame image from the multiple cameras 121 (step S9) and returns the process to step S4.
[0044] When the person detection process P1 and the display control process P2 are completed, the presence determination unit 216 determines whether or not a person is present inside the warning range for each frame image that was the detection target in the person detection process P1, based on the detection position related to the result of the detection process in step S3 (step S10). For a frame image in which a person is present inside the warning range (step S10: YES), the marker placement unit 215 identifies one detection position related to the person whose detection result indicates the highest detection accuracy within the warning range (step S11).
[0045] For a frame image in which a person does not exist within the warning range (step S10: NO), the presence determination unit 216 determines whether or not a person exists within the attention range based on the detection position resulting from the detection process of step S3 (step S12). For a frame image in which a person exists within the attention range (step S12: YES), the marker placement unit 215 identifies one detection position within the attention range for the person whose detection result indicates the highest detection accuracy (step S13). For a frame image in which no person exists within the warning range or attention range (step S12: NO), the marker placement unit 215 identifies the person whose detection result indicates the highest detection accuracy (step S14).
[0046] Then, the marker placement unit 215 places a marker at the detection position of the person identified in any of steps S10 to S14 for each frame image that is the detection target in the person detection process P1. That is, the marker placement unit 215 records information about the marker at the position corresponding to the detection position in the marker storage unit 231 (step S15), and the control device 143 ends the periphery monitoring process.
[0047] Note that the marker disappears after a certain time has passed since it was placed. In other words, the marker placement unit 215 deletes the information of the marker from the marker storage unit 231 when a certain time has passed since the marker was placed. Note that the marker placement unit 215 according to another embodiment may delete the marker when the marker becomes equal to or larger than a predetermined size. At this time, the marker placement unit 215 may determine the color of the marker based on the relationship between the detection position and the warning range. This makes it possible to keep the number of markers in each region corresponding to multiple frame images in the overhead image to one or less.
[0048] The person detection process P1 may be performed in parallel for a plurality of frame images, or may be performed sequentially for each frame image. If a marker related to a person extracted from another frame image is already present at the detection position, the marker placement unit 215 may not place a marker at the detection position. This prevents markers from being placed in a redundant manner, making the markers difficult to see. When the person detection process P1 is performed sequentially for each frame image, the processes from step S10 to step S15 are performed only for the frame image that was the target of the person detection process P1. 5 is an example, and in other embodiments, not all steps may be executed. For example, in other embodiments, any of step S2, steps S5 to S9, and steps S10 to S13 may not be executed.
[0049] <<Example of display image>> 6 to 9 are examples of display images according to the first embodiment. 6 to 9, the display image includes a monitoring image G1 generated by the display control unit 217 and a single camera image G2. The display image also includes a work mode indicator G32, an SMR (service meter reading) indicator G33, a fuel level indicator G34, a hydraulic oil temperature indicator G35, and an engine water temperature indicator G36. The monitoring image G1 includes an overhead image G11, an area frame G12 indicating a caution area, an area frame G13 indicating a warning area, a plurality of markers G14, and an icon G15. The single camera image G2 is a frame image captured by one of the plurality of cameras 121.
[0050] 6 to 9 show a detection position E that is not displayed in the display image but is not specified in the processing of steps S10 to S14. Also, in FIGS. 6 to 9, frame areas F1, F2, F3, and F4 that are not displayed in the display image are shown. Frame F1 is a frame area corresponding to the left rear range Ra captured by the left rear camera 121A. Frame F2 is a frame area corresponding to the rear range Rb captured by the rear camera 121B. Frame F3 is a frame area corresponding to the right rear range Rc captured by the right rear camera 121C. Frame F4 is a frame area corresponding to the right front range Rd captured by the right front camera 121D. Note that the overhead image G11 is not generated in the forward and left front frame areas F0. This is because the operator can directly view the front and left front of the cab 140.
[0051] FIG. 6 is an example of a display image in which a frame image captured by the right front camera 121D is displayed on the single camera image G2 when one or more people are present within the warning range of the work vehicle. According to FIG. 6, when one or more people are detected within the warning range, a marker G14 is attached only to the detection position related to the person with the highest detection accuracy. This allows the operator to grasp the person who should be particularly careful among people in the close range. Note that FIG. 6 is an example of a case where one or more people are present within the warning range, but the same result as above will be obtained even if there is no person within the warning range and one or more people are present within the attention range. Note that the one or more people may be one person or multiple people.
[0052] Fig. 7 shows an example of a display image in which a frame image captured by the right rear camera 121C is displayed in the single camera image G2 when one or more people are present outside the attention range. According to Fig. 7, when one or more people are detected outside the attention range, a marker G14 is attached only to the detection position of the person with the highest detection accuracy. This allows the operator to grasp the person who should be particularly careful when there is no person in the vicinity. Note that the one or more people may be one person or multiple people.
[0053] FIG. 8 is an example of a display image in which one or more people are present within the attention range, and an object is present both within the attention range and outside the attention range, and a frame image captured by the left rear camera 121A is displayed on the single camera image G2. According to FIG. 8, when an object is erroneously detected as a person, one or more people are detected within the attention range, and a person is detected outside the attention range, the marker G14 is attached only to the detection position related to the person with the highest detection accuracy within the attention range. When comparing a person and an object, the person generally has a higher detection accuracy, so the marker G14 is attached to the position related to the person, not the object. This allows the operator to grasp the person who should be particularly careful.
[0054] Moreover, since the marker G14 is attached only to the person with the highest detection accuracy among the multiple detected people, it is possible to prevent the marker G14 from being attached to a position related to a false detection. Even if a person is detected outside the attention range, the marker G14 is attached only to the person within the attention range, thereby preventing the marker from being bothersome and enabling the user to immediately identify the necessary information. Note that FIG. 8 shows an example in which one or more people and objects exist within the attention range, but the same result as above is obtained even when one or more people and objects exist within the warning range. Note that the one or more people may be one person or multiple people.
[0055] FIG. 9 is an example of a display image in which a frame image captured by the right front camera 121D is displayed on the single camera image G2 when one or more people are present within the warning range and one or more people are present outside the attention range. According to FIG. 9, when one or more people are present within the warning range and one or more people are present outside the attention range, the marker G14 is attached only to the detection position related to the person with the highest detection accuracy within the warning range. Even if the person outside the attention range has a higher detection accuracy than the person within the warning range, the marker G14 is attached only to the person with the highest detection accuracy within the warning range. This allows the operator to grasp the person who should be particularly careful when multiple people are detected. In addition, the nuisance of the marker can be prevented, and the necessary information can be immediately identified. Note that the one or more people may be one person or multiple people.
[0056] When the key of the work vehicle is turned on, a frame image captured by the rear camera 121B is displayed as the single camera image G2. The single camera image G2 can be switched to a frame image captured by another camera 121 by the input means of the control device 143. A frame image captured by a camera 121 other than the rear camera 121B may be used as the single camera frame image G2 displayed when the key is turned on. Furthermore, when a person is detected, the single camera image G2 may be automatically switched to a frame image of the camera 121 in which the person is captured.
[0057] Actions and Effects In this way, when multiple obstacles are detected in the detection process of obstacles captured in the captured image, the control device 143 according to the first embodiment displays a display image in which markers are placed at positions related to a predetermined number of obstacles with the highest detection accuracy. This makes it possible to reduce the number of markers in the display image, and thus reduces the possibility of overlooking necessary information related to obstacle detection.
[0058] Furthermore, when multiple obstacles are detected within a predetermined range centered on the work vehicle 100, the control device 143 according to the first embodiment displays a display image in which markers are placed at positions related to a predetermined number of obstacles with the highest detection accuracy within that range. This allows the notification of nearby obstacles to be prioritized even when obstacles farther away from the work vehicle have a higher detection accuracy than obstacles in the vicinity of the work vehicle. Note that other embodiments are not limited to this. For example, the control device 143 according to other embodiments may display a display image in which markers are placed at positions related to a predetermined number of obstacles with the highest detection accuracy, regardless of the caution range and warning range.
[0059] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes are possible. In other embodiments, the order of the above-mentioned processes may be changed as appropriate. In addition, some of the processes may be executed in parallel.
[0060] In the above-described embodiment, the work vehicle 100 has been described as a hydraulic excavator, but in other embodiments, the work vehicle 100 can be applied to various work vehicles such as a dump truck, a wheel loader, etc. For example, the work vehicle 100 can be applied to a display system for a dump truck, a wheel loader, or other work vehicles.
[0061] In addition, in the above-described embodiment, one control device 143 is described as being installed in a work vehicle, but in other embodiments, a part of the configuration of the control device 143 may be arranged in another control device, and a display system consisting of two or more control devices may be realized. Note that the one control device 143 shown in the above-described embodiment is also an example of a display system.
[0062] The control device 143 in the above-described embodiment has been described as being installed in a work vehicle, but in other embodiments, some or all of the configuration of the control device 143 may be installed outside the work vehicle.
[0063] Further, although the control device 143 according to the embodiment described above detects people, it may also detect obstacles other than people. Furthermore, although the control device 143 according to the embodiment described above does not include a left front camera that captures the left front range Re, in other embodiments, a left front camera that captures the left front range Re may be included. In this case, the control device 143 can generate an overhead image that captures the entire periphery of the work vehicle 100. On the other hand, even in this case, the control device 143 does not need to detect people from the frame image of the left front camera for the person detection process P1. This is because the amount of calculation required for the person detection process P1 is large, and by not performing the person detection process P1 for a range that is less likely to be monitored by the control device 143, an increase in the overall calculation load is prevented.
[0064] Furthermore, the control device 143 according to the embodiment described above specifies one detection position with the highest detection accuracy for each frame image, but is not limited to this. For example, in other embodiments, a predetermined number of detection positions, two or more, may be specified in order of decreasing detection accuracy. Also, for example, in other embodiments, a predetermined number of detection positions with high detection accuracy may be specified for all frame images (left rear range Ra, rear range Rb, right rear range Rc, right front range Rd), or a predetermined number of detection positions with high detection accuracy may be specified for each combination of frame images (for example, the combination of the left rear range Ra and the rear range Rb, and the combination of the right rear range Rc and the right front range Rd).
[0065] Furthermore, the control device 143 according to the embodiment described above alerts the operator by displaying the detection result, but in other embodiments, the control device may further control the work vehicle 100 based on the detection result of the control device 143. For example, when a marker is placed in the warning area, the control device may control the work vehicle 100 to prohibit driving, turning, and operation of the work machine. In this case, by reducing the number of markers as in the embodiment described above, it is possible to prevent the work vehicle 100 from being controlled to stop due to erroneous detection.
[0066] Although the control device 143 according to the embodiment described above has a display and displays a display image on the display, other embodiments are not limited to this. For example, the control device 143 according to other embodiments may not have a display and may transmit a signal to display a display image on a display separate from the control device 143. The signal is a signal for displaying a display image in which a marker is placed at a position related to an obstacle. Note that the display system may be realized by a display separate from the control device 143 and two or more control devices each having a part of the configuration of the control device 143 described above.
[0067] In the above-described embodiment, the warning range and the caution range have been described as circles centered on the turning center of the work vehicle, but in other embodiments, they may be ranges other than the circle centered on the turning center of the work vehicle 100. For example, they may be ranges obtained by expanding a circle centered on the turning center of the work vehicle 100 by a predetermined distance in the direction of the front, rear, left, or right. They may also be circles centered on parts other than the turning center of the work vehicle 100. They may also be ranges determined from the operating range of the work implement 130. They may also be ranges formed by an ellipse, a polygon, or straight lines and curves.
[0068] Furthermore, in the above-described embodiment, the display has been described as being installed on the work vehicle 100, but in other embodiments, the display may be installed outside the work vehicle 100. For example, the display may be provided at a location away from the work site, and the control device 143 may transmit a signal to cause the display to display an image via a network such as the Internet.
[0069] In the above embodiment, a warning range and a caution range are set, but in other embodiments, either one of the warning range and the caution range may be set, or three or more ranges may be set.
[0070] In the above-described embodiment, the marker G14 is added to the overhead image G11, but this is not limited thereto. For example, in another embodiment, the marker G14 may be added to both the overhead image G11 and the single camera image G2, or the marker G14 may be added only to the single camera image G2. In this case, the marker G14 may be added to a predetermined number of detection positions with high detection accuracy for both the overhead image G11 and the single camera image G2. Also, the marker G14 may be added only to a predetermined number of detection positions with high detection accuracy for one of the overhead image G11 and the single camera image G2, and the marker G4 may be added to all detection positions for the other image.
[0071] In another embodiment, the display image may include only one of the overhead image G11 and the single camera image G2. In this case, the marker G14 is attached to the overhead image G11 or the single camera image G2 included in the display image. In another embodiment, the single camera images G2 from a plurality of cameras 121 may be displayed side by side. In this case, the marker G14 may be attached to one single camera image G2, or to two or more single camera images.
[0072] Here, an example will be described in which the control device 143 generates a display image including the single camera image G2 to which a marker G14 is added. 10 is a flowchart showing the operation of a control device according to another embodiment. In this embodiment, the control device 143 executes steps S104, S106, and S115 instead of steps S4, S6, and S15 of the first embodiment. In step S104, the image generating unit 214 of the control device 143 generates a single camera image based on the frame image acquired in step S1. In step S106, the display control unit 217 of the control device 143 draws a marker on the single camera image generated in step S104 at the position stored in the marker storage unit 231, and further attaches an icon to the single camera image according to the determination result of step S5, thereby generating a display image. The icon is placed in a position in the single camera image where there is little need to monitor, such as an area showing part of the work vehicle 100, thereby preventing the icon from obscuring a person in a position where there is a high need to monitor the control device 143. In step S115, the marker placement unit 215 of the control device 143 places a marker at the detection position related to the result of the detection process in step S3. That is, the marker placement unit 215 records, in the marker storage unit 231, information on the marker at the detection position.
[0073] FIG. 11 is an example of a display image according to another embodiment. According to FIG. 11, the display image includes two single camera images G2 captured by different cameras 121. Each single camera image G2 is provided with one marker G14. In the example shown in FIG. 11, a predetermined number of detection positions with high detection accuracy are identified and the marker G14 is attached to each of the two single camera images G2, but this is not limited to the above. For example, in another embodiment, a predetermined number of detection positions with high detection accuracy may be identified and the marker G14 may be attached to one of the two single camera images G2, and the marker G14 may not be attached to the other. In another embodiment, a predetermined number of detection positions with high detection accuracy may be identified and the marker G14 may be attached to one of the two single camera images G2, and the marker G14 may be attached to all detection positions of the other. In another embodiment, the display image may include three or more single camera images G2.
[0074] FIG. 12 is an example of a display image according to another embodiment. The imaging ranges of the frame areas F1, F2, F3, and F4 may overlap. The hatched portion in FIG. 12 is an overlapping portion of the imaging range that is not displayed on the actual screen. For example, the imaging range of the right rear camera 121C corresponding to the frame area F3 and the imaging range of the right front camera 121D corresponding to the frame area F4 may partially overlap with each other in the frame areas F3 and F4. If the detection accuracy of a person detected in the overlapping portion of the frame areas F3 and F4 is higher than the detection accuracy of a person detected outside the overlapping portion, the marker G14 may be arranged at the position of the other person detected outside the overlapping portion. In addition, when the detection accuracy of a person detected in the overlapping portion of the imaging range is higher than that of a person detected outside the overlapping portion, the marker G14 may be arranged at a predetermined number of detection positions among the multiple people detected in the adjacent frame regions, for example, frame region F3 and frame region F4, in descending order of detection accuracy. In this case, it is possible to prevent the marker G14 from being arranged only on the people detected in the overlapping portion. Although the above description has been given taking the example of the overlapping portion of frame areas F3 and F4, the above processing may be performed on any combination of adjacent frame areas among frame areas F1, F2, F3, and F4. In addition, determining that a person has been detected in an overlapping area can be done, for example, by determining for each frame image an area where the imaging ranges of adjacent cameras overlap, and determining whether a person has been detected in that overlapping area.
[0075] Furthermore, there are cases where the imaging range of each camera 121 is wider than the corresponding frame areas F1, F2, F3, and F4. Fig. 13 shows an example of the correspondence between the frame areas of a display image and captured images in another embodiment. As shown in Fig. 13, when multiple people are detected in the frame image of the rear camera 121B, the people detected outside the frame area OF may be excluded, and a marker may be placed at the position of the person with high detection accuracy in the frame image excluding the outside frame area OF. This prevents the marker G14 from being placed on the person detected outside the frame area OF, and the marker G14 is placed in the specified area where monitoring is highly necessary, so that necessary information can be immediately determined. Note that the outside frame area OF is a predetermined area in the frame image, and may be, for example, an area other than a part of the frame image cut out when generating the overhead image, that is, outside the area of the overhead image that is not displayed in the overhead image. Also, when multiple people are detected in a frame image, the people detected outside the attention range may be excluded, and a marker may be placed at the position of the person with high detection accuracy in the frame image excluding the people outside the attention range. This prevents the marker G14 from being placed on the person detected outside the attention range, and attaches the marker G14 to the person within the specified attention range, making it possible to immediately identify the necessary information. In the above embodiment, the lines of the frame regions F1, F2, F3, and F4 are not displayed in the display image, but this is not limiting. For example, in other embodiments, the lines of the frame regions F1, F2, F3, and F4 may be displayed in the display image. [Industrial Applicability]
[0076] According to the above disclosure, the display system can reduce the possibility of overlooking necessary information related to obstacle detection. [Explanation of symbols]
[0077] REFERENCE SIGNS LIST 100 work vehicle 110 travelling body 120 rotating body 121 camera 130 work machine 140 driver's cab 141 driver's seat 143 control device 211 image acquisition unit 212 feature extraction unit 213 detection unit 214 image generation unit 215 marker placement unit 216 presence determination unit 217 display control unit
Claims
1. an image acquisition unit that acquires an image of the surroundings of the work vehicle; A detection unit that detects obstacles around the work vehicle; a presence determination unit that determines whether or not the obstacle is present within a predetermined range centered on the work vehicle; a display control unit that generates, when a plurality of obstacles are detected by the detection unit, a signal for displaying an obstacle in an identifiable manner at a position of an obstacle having a high detection accuracy indicating a likelihood of the obstacle being detected among the plurality of obstacles detected in the area corresponding to the captured image; Equipped with When a plurality of obstacles are detected within the predetermined range, the display control unit causes the obstacle to be identifiable at a position corresponding to an obstacle having a high detection accuracy within the predetermined range among the plurality of obstacles, and when no obstacle is detected within the predetermined range, causes the display control unit causes the obstacle to be identifiable at a position corresponding to an obstacle having a high detection accuracy outside the predetermined range among the plurality of obstacles. Work vehicle display system.
2. An image acquisition unit that acquires an image of the surroundings of a work vehicle; A detection unit that detects obstacles around the work vehicle; a display control unit that generates, when a plurality of obstacles are detected by the detection unit, a signal for displaying an obstacle in an identifiable manner at a position of an obstacle having a high detection accuracy indicating a likelihood of the obstacle being detected among the plurality of obstacles detected in the area corresponding to the captured image; and Equipped with The image acquisition unit acquires a plurality of captured images of an area around the work vehicle, When the detection accuracy of a first obstacle detected in a portion where imaging ranges of adjacent images of the plurality of captured images overlap is higher than the detection accuracy of a second obstacle detected in a portion different from the overlapping portion, the display control unit generates a signal for identifiably displaying an obstacle at a position related to the first obstacle as well as a position related to the second obstacle. Work vehicle display system.
3. an image generating unit that generates an overhead image of the surroundings of the work vehicle based on a plurality of captured images; The image acquisition unit acquires an image of an area around the work vehicle, The detection unit executes a process of detecting the obstacle for each of the plurality of captured images, The display control unit generates a signal for displaying, on the overhead image, a display that can identify an obstacle at a position related to an obstacle having a high detection accuracy among the plurality of detected obstacles, for each region of the overhead image corresponding to the plurality of captured images.
3. A display system for a work vehicle according to claim 1 or 2.
4. The display control unit excludes an obstacle detected in a predetermined area of the captured image, and generates a signal for displaying an indication that enables the obstacle to be identified on the overhead image at a position related to an obstacle having a high detection accuracy among a plurality of obstacles other than the excluded obstacle. The display system for a work vehicle according to claim 3 .
5. an image generating unit that generates a single camera image that is an image of the surroundings of the work vehicle based on the captured image, The display control unit generates a signal for displaying, on the single camera image, a display that can identify an obstacle at a position related to an obstacle that has a high detection accuracy among the plurality of detected obstacles. A display system for a work vehicle according to any one of claims 1 to 4.
6. The display control unit generates a signal for identifiably displaying an obstacle at a position related to an obstacle having the highest detection accuracy among the plurality of detected obstacles. A display system for a work vehicle according to any one of claims 1 to 5.
7. The display control unit generates a signal for identifiably displaying the obstacles at positions related to a predetermined number of the detected obstacles in order of the highest detection accuracy. A display system for a work vehicle according to any one of claims 1 to 5.
8. A step of acquiring an image of the surroundings of the work vehicle; A step of executing a process of detecting an obstacle captured in the captured image; A step of determining whether or not the obstacle is present within a predetermined range centered on the work vehicle; a step of generating a signal for displaying an obstacle in an identifiable manner at a position of an obstacle having a high detection accuracy indicating a likelihood of the obstacle being detected among the plurality of obstacles in the area corresponding to the captured image when the plurality of obstacles are detected in the detection process; Equipped with In the step of generating the signal, when a plurality of obstacles are detected within the predetermined range, the obstacle is identifiably displayed at a position corresponding to an obstacle having a high detection accuracy within the predetermined range among the plurality of obstacles, and when no obstacle is detected within the predetermined range, the obstacle is identifiably displayed at a position corresponding to an obstacle having a high detection accuracy outside the predetermined range among the plurality of obstacles. How to display work vehicles.
Citation Information
Patent Citations
Surrounding monitoring equipment of working machine
JP2008179940A
Surrounding monitoring device for work machine
JP2013066010A
Device for detecting obstacle around vehicle
JP2017074871A
Device for detecting obstacle around vehicle
JP2017074873A
Periphery monitoring system for work machine
JP2017151815A
