Work assistance system

By integrating attitude detection and imaging devices on the working machinery and combining image processing technology, a solution to intuitively set the working area on the working machinery such as hydraulic excavators is realized, solving the problems of difficulty in setting and low efficiency in the existing technology. , improves the operating efficiency and accuracy of the working machinery.

CN118900944APending Publication Date: 2024-11-05HITACHI CONSTRUCTION MACHINERY CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202380027275.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-31
Filing Date
2023-03-03
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

There are difficulties in setting the rotation range and operation area of ​​existing working machinery, especially in hydraulic excavators, because the blind spots caused by the driver's seat and bucket structures cause the view to be blocked, making it difficult to achieve accurate alignment, and at the same time, the fixed marks are fixed in the operation. Frequent adjustments are required when the area changes, which reduces the working efficiency.

Method used

Using the operation support system, including attitude detection device, camera device and control device, the appearance of the work machine is captured through the camera, the image area is extracted, and the image coordinates are converted into actual spatial coordinates, allowing the operator to touch the display. Intuitively set the work area.

Benefits of technology

The work area is set intuitively and easily, and the misconfiguration problems caused by blind spots are avoided, and the working efficiency and accuracy of the work machinery are improved, so as to adapt to changes in the work site.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118900944A_ABST
    Figure CN118900944A_ABST
Patent Text Reader

Abstract

A work assistance system is provided with a front work machine provided in a hydraulic shovel, an angle sensor that detects attitude information of the front work machine, and a control device that performs work assistance of the hydraulic shovel on the basis of a work area that is set and stored in advance. The hydraulic excavator is characterized by further comprising a camera capable of shooting the appearance of the hydraulic excavator on a working site and a touch display capable of specifying any position in an image shot by the camera, and the control device extracts an image area occupied by the hydraulic excavator from the image shot by the camera. On the basis of the number of pixels in the longitudinal direction of the hydraulic shovel in the extracted image region, the attitude information of the hydraulic shovel, and the actual size of the hydraulic shovel, the position in the image specified by the touch display is converted to the coordinate value of the actual space coordinate, and the work region is set such that the position of the converted coordinate value becomes the end point. As a result, it is possible to intuitively and easily set a desired work area.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a work support system. Background Art

[0002] In working machines represented by hydraulic excavators, as a technology for improving working efficiency by setting a predetermined working area and performing working support based on the working area, for example, structures described in Patent Documents 1 and 2 are known.

[0003] Patent document 1 discloses a slewing control device for a working machine, which includes: a slewing operating lever for setting the slewing speed of an upper slewing body relative to a lower traveling body of the working machine; an electric motor having a stator and a rotor, and driving the upper slewing body to slew relative to the lower traveling body by rotating the rotor relative to the stator; an angular velocity detection mechanism built into the electric motor, detecting the angular velocity of the rotor relative to the stator as the motor angular velocity; and an electric motor control mechanism for controlling the rotation speed of the electric motor so that the angular velocity detected by the angular velocity detection mechanism is The motor angular velocity becomes a size corresponding to the rotation speed set by the rotation operating lever; a rotation angle calculation mechanism, which calculates the rotation angle of the upper rotating body relative to the lower traveling body based on the time integral value of the motor angular velocity detected by the angular velocity detection mechanism; a rotation range setting mechanism, which sets the rotation range of the upper rotating body relative to the lower traveling body; and a stop control mechanism, which stops the operation of the electric motor under the control of the electric motor control mechanism to prevent the rotation angle from exceeding the rotation range set by the rotation range setting mechanism.

[0004] Patent document 2 discloses an operation assisting device that assists the operation of a working machine. The operation assisting device includes a motion control unit that controls the movement of the working machine in accordance with information represented by a mark image corresponding to a mark at a working site in an image captured by a camera device mounted on the working machine.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2011-52383

[0008] Patent Document 2: Japanese Patent Application Publication No. 2013-151830 Summary of the invention

[0009] Problems to be solved by the invention

[0010] However, in the prior art described in Patent Document 1, the operator is required to quantitatively grasp the angle until the rotation stop position in order to input a predetermined threshold value, and it is difficult to set the rotation range as desired. In addition, even when the upper rotating body is moved to the rotation stop position for setting, part of the working machine becomes a shield to block the field of view, so there is a problem that it is difficult to achieve correct alignment. In particular, in a hydraulic excavator, when it is desired to set the working area such as the working height and working radius according to the current position of the machine, the blind spot generated by the ceiling of the driver's seat and the bucket is large, and the working area may be set to an undesirable position.

[0011] In addition, in the prior art described in Patent Document 2, a fixed mark is set at the work site, and the entry-prohibited area is set according to the information indicated by the mark, and the area is determined using the marks set around it, thereby setting a work area suitable for the site environment. However, when the work area changes or when moving between work sites occurs, the mark needs to be re-placed each time, which may reduce the work efficiency.

[0012] The present invention has been made in view of the above-mentioned circumstances, and an object of the present invention is to provide a work support system capable of intuitively and easily setting a desired work area.

[0013] Means for solving problems

[0014] The present application includes multiple means for solving the above-mentioned problems. As an example, a work support system includes: a work device, which is arranged on a work machine; a posture detection device, which detects the posture information of the work device; and a control device, which performs work support for the work machine based on a pre-set and stored work area. The work support system also includes: a camera device, which can capture the appearance of the work machine at the work site; and an input device, which can specify any position in the image captured by the camera device. The control device extracts the image area occupied by the work machine from the image captured by the camera device, and based on the number of pixels in the length direction of the work machine in the extracted image area, the posture information of the work machine and the actual size of the work machine, transforms the position in the image specified by the input device into a coordinate value of the actual space coordinate, so as to set the work area in such a way that the position of the transformed coordinate value becomes an endpoint.

[0015] Effects of the Invention

[0016] According to the present invention, it is possible to provide a work support system that can intuitively and easily set a desired work area. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1It is a side view schematically showing the appearance of a hydraulic excavator as an example of a working machine.

[0018] Figure 2 This is a diagram showing the appearance of a tablet computer as an example of a field environment information acquisition device.

[0019] Figure 3 This is a functional block diagram schematically showing the overall configuration of the work support system.

[0020] Figure 4 This is a diagram showing an example of an image captured by a tablet computer.

[0021] Figure 5 : is a flowchart showing the processing contents of the mechanical image extraction unit.

[0022] Figure 6 FIG. 1 is a diagram showing a situation in which an operator selects a region of an image displayed on the touch display.

[0023] Figure 7 It is a flowchart showing the processing contents of the coordinate conversion unit.

[0024] Figure 8 This is a flowchart showing the processing contents of the area setting unit.

[0025] Fig. 9 It is a diagram showing an example of the area setting screen.

[0026] Fig.10 This is a diagram showing a display example of a normal state of a monitor indicating the presence or absence of work support.

[0027] Fig.11 It is a diagram showing a display example of a monitor in a proximity state indicating the presence or absence of work support.

[0028] Fig.12 It is a diagram showing a display example in a disengaged state of a monitor indicating the presence or absence of work support.

[0029] Fig.13 The diagrams are for explaining the difference in the appearance of the hydraulic excavator caused by the difference in the imaging direction.

[0030] Fig.14 The diagrams are for explaining the difference in the appearance of the hydraulic excavator caused by the difference in the imaging direction.

[0031] Fig.15 This is a diagram showing an example of imaging direction guidance by the imaging support unit. DETAILED DESCRIPTION

[0032] Below, refer to Figure 1 to Figure 15In the present embodiment, a hydraulic excavator having a front working machine is described as an example of a working machine, but the present invention can also be applied to other working machines.

[0033] Figure 1 is a side view schematically showing the appearance of a hydraulic excavator as an example of a working machine according to the present embodiment. Figure 2 2 is a diagram showing the appearance of a tablet computer as an example of a field environment information acquisition device. Figure 3 This is a functional block diagram schematically showing the overall configuration of the work support system.

[0034] exist Figure 1 In the embodiment, a hydraulic excavator 1 as a working machine is roughly composed of an articulated front working machine 1A and a vehicle body 1B.

[0035] The vehicle body 1B includes a lower traveling body 1e that is driven by the left and right traveling hydraulic motors 3e and 3f, and an upper revolving body 1d that is rotatably provided on the upper portion of the lower traveling body 1e and is driven to rotate by a revolving hydraulic motor 5a. Figure 1 In the figure, only one travel hydraulic motor 3f is shown, and the other travel hydraulic motor 3e is denoted by a reference numeral in brackets.

[0036] The front working machine 1A is a working device composed of a plurality of driven parts (a boom 1a, an arm 1b, and a bucket 1c) that rotate in the vertical direction. The base end of the boom 1a is rotatably supported at the front of the upper revolving body 1d by means of a boom pin. In addition, the arm 1b is rotatably connected to the front end of the boom 1a by means of an arm pin. In addition, the bucket 1c is rotatably connected to the front end of the arm 1b by means of a bucket pin and a bucket link.

[0037] The boom 1a is rotationally driven by a boom cylinder 2a, the arm 1b is rotationally driven by an arm cylinder 2b, and the bucket 1c is rotationally driven by a bucket cylinder 2c.

[0038] Angle sensors (attitude sensors) 4a, 4b, 4c, 4d are provided at the connection part between the upper swing body 1d and the boom 1a, the connection part between the boom 1a and the arm 1b, the connection part between the arm 1b and the bucket 1c (for example, the bucket link), and the connection part between the lower travel body 1e and the upper swing body 1d, respectively, for detecting the relative angle of the boom 1a with respect to the upper swing body 1d, the relative angle of the arm 1b with respect to the boom 1a, the relative angle of the bucket 1c with respect to the arm 1b, and the swing angle of the upper swing body 1d with respect to the lower travel body 1e. Here, the angle sensors 4a, 4b, 4c, 4d constitute attitude detection devices for detecting the attitude of the working device. It should be noted that the angle sensors 4a, 4b, 4c, 4d can be replaced by inertial measurement units (IMU: Inertial Measurement Unit) for detecting the relative angles with respect to the reference plane. The upper swing body 1d is provided with a GNSS 13 (Global Navigation Satellite System) as a position detection device for acquiring position information (position coordinates) of the hydraulic excavator 1 at the work site. Figure 1 Only the antenna is shown in the figure) and a gyro sensor 14 for acquiring information on the direction and tilt angle of the hydraulic excavator 1.

[0039] An engine, a hydraulic pump, an electronic control valve, etc. (not shown) are mounted on the body frame of the upper revolving body 1d, and the movement of the hydraulic excavator 1 is controlled by controlling the flow and direction of the pressure oil discharged from the hydraulic pump driven by the engine and supplied to multiple hydraulic actuators (i.e., travel hydraulic motors 3e, 3f, revolving hydraulic motor 5a, boom cylinder 2a, arm cylinder 2b, bucket cylinder 2c, etc.) through the electronic control valve.

[0040] An operating room 1f for an operator to ride in the hydraulic excavator 1 is provided beside the front working machine 1A in front of the upper swing body 1d. A plurality of electric operating levers (not shown) are provided inside the operating room 1f for the operator to operate the hydraulic excavator 1 while the operator is seated in the driver's seat. When the operator operates the operating levers, the amount of operation of each lever is controlled by the control device 9 (see Figure 3 ) is converted into control command values ​​of each hydraulic actuator 2a, 2b, 2c, 3e, 3f, 5a and sent to the electronic control valve, and the electronic control valve controls the flow rate and direction of the pressure oil supplied to each hydraulic actuator 2a, 2b, 2c, 3e, 3f, 5a according to the control command value. That is, the operator operates the hydraulic excavator 1 such as the working action of the front working device 1A, the rotation action of the upper swing body 1d, and the travel action of the lower travel body 1e by operating the operating lever.

[0041] In addition, if Figure 3As shown, the operating room 1f is provided with a control device 9 for controlling the overall operation of the hydraulic excavator 1, a monitor 10 as an input / output device, a buzzer 11 as a notification device, and an operating switch 12 as a setting device, which can provide information prompts to the operator and enable the operator to make various settings for the hydraulic excavator 1.

[0042] Figure 2 The environmental information acquisition device shown is, for example, a tablet-type information terminal (hereinafter referred to as a tablet 20 ), and includes a touch display 25 and a camera 22 as input and output devices.

[0043] In addition, if Figure 3 As shown, the tablet computer 20 is equipped with a control device 21 for controlling the operation of the tablet computer 20 , and a gyro sensor 23 and an orientation sensor 24 as an imaging direction detection device for detecting the orientation of the tablet computer 20 (ie, the imaging direction of the camera 22 ).

[0044] The control device 9 of the hydraulic excavator 1 and the control device 21 of the tablet computer 20 include communication devices 26 and 27 , respectively, and can perform wireless communication with each other.

[0045] Here, the hydraulic excavator 1 (working machine) in the present embodiment has a work support function that notifies the operator of a preset working range (also referred to as a working area) through a notification device such as a monitor 10 and a buzzer 11 when a part of the hydraulic excavator such as the front working device 1A exceeds the preset working range (also referred to as a working area). The working range is defined by, for example, a turning radius, a turning angle, a height, etc. In the present embodiment, the working range is set based on information obtained by the tablet computer 20.

[0046] like Figure 3 As shown, the work support system is constituted by the functional units and related configurations of the control device 9 of the hydraulic excavator 1 and the control device 21 of the tablet computer 20 .

[0047] As functional units of the work support system, the control device 9 of the hydraulic excavator 1 includes a machine direction acquisition unit 91, a machine posture calculation unit 92, a machine dimension storage unit 93, an area setting unit 94, an area storage unit 95, and a work support determination unit 96. In addition, as functional units of the work support system, the control device 21 of the tablet computer 20 includes an image storage unit 210, a machine image extraction unit 211, a teaching data storage unit 212, a position specifying unit 213, a coordinate conversion unit 214, an imaging direction acquisition unit 215, and an imaging support unit 216.

[0048] Although not shown in the figure, the control device 9 and the control device 21 are hardware equivalent to a computer, each having a CPU (Central Processing Unit) as a processing device, a program executed by the processing device, and a storage device (e.g., a semiconductor memory such as a ROM or RAM or a hard disk drive, etc.) storing data required for executing the program, etc. That is, for example, in the control device 9 and the control device 21, the mechanical dimension storage unit 93, the area storage unit 95, the teaching data storage unit 212, etc. are implemented by a storage device such as a hard disk drive, and other functional units are implemented by executing the program in the CPU, etc.

[0049] Hereinafter, the details of each functional unit of the control device 9 and the control device 21 will be described according to the processing flow of the work support system.

[0050] First, in the tablet computer 20 , the camera 22 captures an image of the hydraulic excavator 1 at the work site so as to include the exterior appearance of the hydraulic excavator 1 .

[0051] Figure 4 The figure shows an example of an image captured by a tablet computer.

[0052] The camera 22 of the tablet computer 20 is set to a certain direction of the working range. For example, when the working range in the turning radius and height direction of the hydraulic excavator 1 is set, Figure 4 As shown, the hydraulic excavator 1 is photographed from the side. In addition, when setting the working range for the rotation angle, for example, the tablet computer 20 (or a function equivalent to the camera 22) can be mounted on a drone, etc., and the hydraulic excavator 1 can be photographed from a bird's-eye view. The image photographed by the camera 22 is stored in the image storage unit 210, output to the machine image extraction unit 211, and output to the position specifying unit 213 via the machine image extraction unit 211.

[0053] The machine image extraction unit 211 of the control device 21 extracts the image region occupied by the hydraulic excavator 1 in the input image acquired from the image storage unit 210 , and outputs the coordinates thereof.

[0054] Figure 5 : is a flowchart showing the processing contents of the mechanical image extraction unit.

[0055] like Figure 5 As shown, the mechanical image extraction unit 211 first obtains the latest image from the images stored in the image storage unit 210 as an input image (step S100 ). It should be noted that the input image obtained by the mechanical image extraction unit 211 can also be configured to be arbitrarily selected by the user by operating the touch display 25 .

[0056] Next, an image region of a predetermined size is extracted from the input image acquired in step S100 (step S110 ).

[0057] Next, the image feature amount of the image area extracted in step S110 is calculated (step S120). The image feature amount is calculated by constructing an appropriate algorithm in advance according to the color and shape of the working machine (here, the hydraulic excavator 1) to be extracted.

[0058] Next, the image feature calculated in step S120 is compared with the teaching data stored in advance in the teaching data storage unit 212 to determine whether the hydraulic excavator 1 (working machine) is included in the area extracted in step S110 (step S130 ).

[0059] It should be noted that the teaching data stored in the teaching data storage unit 212 and used in step S120 can be in various forms such as various posture states related to the hydraulic excavator 1, image data of the hydraulic excavator 1 (working machine) taken from various viewpoints, its feature value data, or a recognition function (classifier) ​​for the feature value constructed based on the above data.

[0060] In addition, the teaching data used in the comparison process of step S120 can be selected. The mechanical posture calculation unit 92 of the control device 9 obtains the detection results from the angle sensors 4a, 4b, 4c, and 4d as posture sensors (posture detection devices) as the current posture information (current posture) of the hydraulic excavator 1, and outputs it to the mechanical image extraction unit 211, etc. Then, for example, based on the current posture of the hydraulic excavator 1 obtained by the mechanical posture calculation unit 92, the teaching data used in the comparison process of step S120 is selected, thereby improving the extraction accuracy of the image of the hydraulic excavator 1, the image processing speed, etc.

[0061] It should be noted that, when the same comparison process is performed on an image captured in the past, the posture information of the hydraulic excavator 1 at the time of capturing the image is stored together with the image.

[0062] Next, it is determined whether the comparison process of step S130 for the entire area is completed (step S140). If the result of the determination is negative, an image area different from the image area extracted until the previous cycle is newly extracted from the input image acquired in step S100 (step S110), and the processes of steps S120 and S130 are performed on the extracted image area. That is, the processes of steps S110 to S130 are repeated until the result of the determination in step S140 is positive.

[0063] In addition, when the determination result of step S140 is yes, that is, when the processing of steps S110 to S130 is completed for all areas of the input image acquired in step S100, the coordinates of the image area determined to be the machine image (hydraulic excavator 1) are output to the position specifying unit 213 and the coordinate conversion unit 214 (step S150), and the processing is completed. It should be noted that when there are multiple image areas determined to include the hydraulic excavator 1 (working machine), the image area determined to be closest to the teaching data in the determination processing of step S130 is selected and its coordinates are output.

[0064] The position specifying unit 213 of the control device 21 displays the image sent from the image storage unit 210 via the machine image extracting unit 211 on the touch display 25 and presents it to the operator, so that the operator selects the end point position of a predetermined area of ​​the image on the touch display 25 .

[0065] Figure 6 FIG. 1 is a diagram showing a situation in which an operator selects a region of an image displayed on a touch display.

[0066] like Figure 6 As shown, the input image 56 sent from the image storage unit 210 is displayed on the touch display 25 of the tablet computer 20, and the image area 50 extracted by the mechanical image extraction unit 211 is shown for the input image 56. In this state, the operator can touch the input image 56 with his finger or the like (in order to Figure 6 The operator's hand 57 (illustrated and schematically shown in the figure, such as a touch pen) indicates the position where the work area is desired to be set, thereby drawing the area end point 51, and drawing the work area 52 in such a manner that the area end point 51 becomes the boundary position in the height direction and the rotation radius direction, based on the image area of ​​the hydraulic excavator 1 (work machine). As a selection method for the operator to select the area end point 51, for example, when the purpose of work support is to prevent the hydraulic excavator 1 from contacting the obstacle 53, the position is specified on the touch display 25 so that the area end point 51 is located between the hydraulic excavator 1 and the obstacle 53. When the setting position of the work area 52 is drawn at the desired position, the operator presses the confirmation button 54 and proceeds to the next process (processing of the coordinate conversion unit 214). In addition, when the operator does not set the area end point 51 (i.e., the work area setting), he presses the cancel button 55 to end the process.

[0067] It should be noted that in the present embodiment, an example is given in which a boundary is set in a manner surrounding the hydraulic excavator 1 and the interior thereof is set as a working area, but the present invention is not limited to this. For example, the working area may be set in a manner in which a boundary is set in a manner surrounding an obstacle and the interior thereof is set as a range outside the working area.

[0068] The coordinate conversion unit 214 of the control device 21 converts the work area set by the operator on the touch display 25 into a value in the real space (a coordinate value of the real space coordinate).

[0069] Figure 7 It is a flowchart showing the processing contents of the coordinate conversion unit.

[0070] like Figure 7 As shown, the coordinate conversion unit 214 first obtains the current posture (posture information) of the hydraulic excavator 1 from the mechanical posture calculation unit 92, and obtains the dimension values ​​of each part of the hydraulic excavator 1 pre-stored in the mechanical dimension storage unit 93 of the control device 9 (step S200). At this time, when the position is specified using an image captured in the past, the posture information of the hydraulic excavator 1 at the time of image capture is stored together with the image, and the posture information is obtained while reading the image captured in the past.

[0071] Next, the coordinate conversion unit 214 calculates the height of the hydraulic excavator 1 and the length in the front direction (the length direction of the front working device 1A viewed from above the swing axis) based on the posture information and the dimension values ​​obtained in step S200 (step S210).

[0072] Next, the coordinates of the area output from the machine image extraction unit 211 (the image area occupied by the hydraulic excavator 1 ) and the area output from the position specification unit 213 (the predetermined area set by the operator) are acquired as pixel values ​​(step S220 ).

[0073] Next, by using the ratio of the actual size of the hydraulic excavator 1 obtained in step S210 and the pixel value of the hydraulic excavator 1 obtained in step S220, the size of the designated working area is converted from the pixel value to the value in the actual space (step S230), the calculation result is output to the area setting unit 94 of the control device 9 (step S240), and the processing is ended.

[0074] The area setting unit 94 of the control device 9 of the hydraulic excavator 1 sets an area where the work support function of the hydraulic excavator 1 is enabled based on the information of the work area transmitted from the coordinate conversion unit 214 of the control device 21 of the tablet computer 20 .

[0075] Figure 8 This is a flowchart showing the processing contents of the area setting unit.

[0076] like Figure 8As shown, the area setting unit 94 obtains the reception standby state of information from the tablet computer 20 (step S300), and determines whether there is a reception from the tablet computer 20 (step S310). Steps S300 and S310 show the state of waiting for the tablet computer 20 to communicate with the hydraulic excavator 1. If the determination result in step S300 is no, the processing of steps S300 and S310 is repeatedly executed until the determination result becomes yes, that is, until the information of the working area is received from the tablet computer 20.

[0077] In addition, if the result of determination in step S300 is YES, that is, if information has been received from the tablet computer 20, the operator is notified (notified) that the setting (change) of the work area has been performed on the tablet computer 20 (step S320). The notification to the operator is performed, for example, by displaying an icon or a message on the monitor 10 or by sounding a notification sound from the buzzer 11.

[0078] Next, it is determined whether the display of the area setting screen on the monitor 10 or the like is selected by operating the switch 12 or the like (see the following description). Fig.10 )(Step S330). It is also considered that the setting information of the working area is received at the timing of the action of the hydraulic excavator 1. Therefore, when the judgment result in step S330 is no, the processing of steps S320 and S330 is repeated and the operator is continuously notified until the judgment result becomes yes, that is, the operator recognizes the notification and selects the screen of the monitor 10 to jump to the area setting screen.

[0079] If the result of determination in step S330 is YES, the display of the monitor 10 is switched to the area setting screen (step S340 ), and it is determined whether the area setting is selected (step S350 ).

[0080] Fig. 9 It is a diagram showing an example of the area setting screen.

[0081] like Fig. 9 As shown, the area setting screen displayed on the monitor 10 includes display of the working area effective state, display of the working area setting value, display of the working area type options to be set, display of confirmation and cancellation of the setting content, etc.

[0082] As the contents indicating the effective state of the working area, for example, a line 82 indicating the effective state in the swing radius direction and a line 83 indicating the effective state in the height direction are displayed together with an icon 81 indicating the hydraulic excavator 1. When the working area is set (enabled) only in the swing radius direction of the hydraulic excavator 1, as shown in FIG. Fig. 9As shown in the example, the line 82 is highlighted (activated) to indicate that the working area in the swing radius direction is valid, and the line 83 is frozen to indicate that the working area in the height direction is not set (invalidated). Similarly, when the working area is set (validated) only in the height direction of the hydraulic excavator 1, the line 83 is highlighted and the line 82 is frozen. In addition, when the working area is set (validated) in both the swing radius direction and the height direction of the hydraulic excavator 1, the lines 82 and 83 are highlighted, and when the working area is not set in both the swing radius direction and the height direction of the hydraulic excavator 1, the lines 82 and 83 are frozen.

[0083] As the contents of the setting values ​​of the working area transmitted from the tablet computer 20, the setting value 84 of the height direction and the setting value 85 of the working radius direction (swing radius direction) are displayed. The operator of the hydraulic excavator 1 determines how to set (validate) the working area by referring to these setting values ​​84 and 85. That is, the working area (refer to the setting value 84 and 85) intuitively set by the operator (photographer) on the display 25 of the tablet computer 20 in order to avoid obstacles 14, etc. is set. Figure 6 ), the upper boundary portion is set to a setting value 84 in the height direction, and the side boundary portion is set to a setting value 85 in the working radius direction. Since the operator of the hydraulic excavator 1 selectively sets the validity / invalidation of the boundary of the working area set by the setting values ​​84 and 85 in the monitor 10, the desired working area can be set intuitively and easily.

[0084] As the content indicating the set work area type (for example, direction) option, for example, a radio button type selection button 86 is displayed. Fig. 9 As shown in the example, when the item for setting the working area only in the swing radius direction of the hydraulic excavator 1 ("Set working area (swing radius)") is selected, the line 82 indicating the effective state in the swing radius direction is highlighted and the line 83 indicating the effective state in the height direction is frozen.

[0085] As a part for confirming and canceling the setting content, a confirmation button 87 and a cancel button 88 are displayed. After selecting the set working area, by selecting the confirmation button 87, the setting of the working area is adapted and the working support function becomes effective. In addition, by selecting the cancel button 88, the setting content of the working area is abandoned.

[0086] It should be noted that the image captured by the tablet computer 20 may be received and displayed on the area setting screen including a position diagram set as the work area. That is, by displaying the captured image and the position of the area, the operator can easily recognize the set position of the work area.

[0087] return Figure 8 .

[0088] like Figure 8 As shown, when the judgment result in step S350 is yes, that is, when the area setting unit 94 selects the area setting, the value of the selected area is stored in the area storage unit 95 (step S351), and the notification of the area setting change to the monitor 10 is stopped (step S360), and the processing is ended.

[0089] In addition, when the judgment result in step S350 is negative, that is, when no area is set, all the information of the area stored in the area storage unit 95 at that time is deleted (step S352), the notification of the area setting change to the monitor 10 is stopped (step S360), and the processing is terminated.

[0090] The work support determination unit 96 determines whether work support is required based on the area information stored in the area storage unit 95 and the current attitude (attitude information) of the hydraulic excavator 1 calculated by the machine attitude calculation unit 92, and notifies the operator through the monitor 10 and the buzzer 11 if support is required.

[0091] Figure 10 to Figure 12 It is a diagram showing a display example of a monitor indicating the presence or absence of work support.

[0092] Fig.10 It is a figure which shows the display example in a normal state.

[0093] like Fig.10 As shown, when the work area is set, a display 191 indicating the state of the work support function is displayed on the monitor 10. When the work is performed in the set area and does not go out of the work area, only a display 192 indicating the effective state of the set work area is displayed. Fig.10 In FIG. 1 , a case where only the working area in the direction of the turning radius is set is illustrated.

[0094] Fig.11 It is a diagram showing a display example in the proximity state.

[0095] like Fig.11 As shown, when it is determined that the operator is close to the boundary of the work area and is likely to leave the work area, the display mode of display 192 is changed to display 193 using a warning color such as yellow or red to warn the operator. At this time, it is desirable to display messages 194 and 195 for warning the operator.

[0096] Fig.12 It is a diagram showing a display example in the disengaged state.

[0097] like Fig.12 As shown, when the operator has left the working range, the display 193 is flashed to display 196, thereby providing a stronger warning to the operator, and messages 197 and 198 are displayed to notify the operator of leaving the working area.

[0098] In this way, when there is a possibility of departure from the preset work area or when departure from the work area has already occurred, the work support function notifies the operator of the fact, thereby enabling efficient work.

[0099] It should be noted that in the present embodiment, support is provided based on notifications from the monitor 10 and the buzzer 11, but it may also be configured, for example, to provide an electronic control valve and utilize the command value from the work support determination unit 96 to restrict the movement of the hydraulic excavator 1 such as deceleration, stopping, etc. corresponding to the approach or separation state.

[0100] The camera direction acquisition unit 215 of the control device 21 of the tablet computer 20 calculates and obtains the camera direction of the camera 22 of the tablet computer 20 based on the information (orientation information) from the orientation sensor 24 provided on the tablet computer 20 and the information (tilt information) from the gyro sensor 14, and outputs it to the camera support unit 216.

[0101] The mechanical direction acquisition unit 91 of the control device 9 of the hydraulic excavator 1 calculates and acquires the direction (orientation) and tilt angle of the hydraulic excavator 1 based on the information (position information, orientation information) from the GNSS 13 installed in the hydraulic excavator 1 and the information (tilt information) from the gyro sensor 14, and outputs it to the camera support unit 216 of the control device 21 of the tablet computer 20.

[0102] The camera support unit 216 of the control device 21 of the tablet computer 20 displays navigation on the touch display 25 based on the information about the camera 22 of the tablet computer 20 sent from the camera direction acquisition unit 215 and the information about the direction (orientation) of the hydraulic excavator 1 sent from the mechanical direction acquisition unit 91 of the control device 9 of the hydraulic excavator 1, so that the camera direction of the tablet computer 20 becomes an appropriate direction, thereby supporting the operation of the tablet computer 20.

[0103] Fig.13 and Fig.14 This is a diagram that explains the difference in size (appearance) of a hydraulic excavator caused by differences in camera directions. Fig.13 and Fig.14 , a situation where the hydraulic excavator 1 is viewed from above is illustrated. Fig.13 The image plane of the camera is parallel to the direction of the hydraulic excavator. Fig.14 This diagram shows a case where the imaging plane of the camera is not parallel to the direction of the hydraulic excavator.

[0104] like Fig.13 As shown, when the hydraulic excavator 1 is photographed by the camera 22 at a certain photographing position 101 with a field of view angle 102, if the direction 105 of the hydraulic excavator 1 is parallel to the photographing plane 103 of the camera 22, the ratio of the actual size 106 of the hydraulic excavator 1 to the distance 107 from the bucket end of the hydraulic excavator 1 to the obstacle 104 is the same as the ratio of the length 108 and the length 109 respectively illuminated on the photographing plane 103. That is, if the actual size 106 of the hydraulic excavator 1 is known, the distance 107 to the obstacle 104 can be estimated using the ratio of the number of pixels of corresponding parts of the photographed image.

[0105] On the other hand, Fig.14 As shown in FIG. 1 , when the direction 105 of the hydraulic excavator 1 is not parallel to the imaging plane 103 of the camera 22, the ratio of the actual size 106 of the hydraulic excavator 1 to the distance 107 from the bucket end of the hydraulic excavator 1 to the obstacle 104 is different from the ratio of the length 112 and the length 113 respectively irradiated on the imaging plane 103. That is, when the distance 107 to the obstacle 104 is estimated using the ratio of the number of pixels of the corresponding parts of the captured image, the accuracy of the estimation result is reduced. Therefore, it is necessary to perform navigation in such a way that the direction of the hydraulic excavator 1 (that is, the direction of the front working machine 1A) is parallel to the imaging plane of the camera 22 when shooting.

[0106] Fig.15 This is a diagram showing an example of navigation of the imaging direction by the imaging support unit.

[0107] The camera support unit 216 calculates the relative angle between the tablet computer 20 and the front working machine 1A based on the information from the camera direction acquisition unit 215 and the machine direction acquisition unit 91. If the relative angle is greater than a predetermined threshold, the camera support unit 216 calculates the relative angle between the tablet computer 20 and the front working machine 1A. Fig.15 As shown, navigation is performed to tilt the camera 22 of the tablet computer 20 in a specific direction (for example, a message 1102 is displayed to prompt adjustment of the camera direction). In addition, when the relative angle is lower than a predetermined threshold, a message (not shown) is displayed to prompt shooting, and the photographer presses the shooting button 1101 to shoot. It should be noted that navigation is performed not only for the yaw direction of the hydraulic excavator 1, but also for each direction such as the roll direction and the pitch direction according to the camera position.

[0108] The effects of the present embodiment configured as described above will be described.

[0109] In the present embodiment, in a work support system including a work device (e.g., a front work device 1A) provided on a work machine (e.g., a hydraulic excavator 1), a posture detection device (e.g., angle sensors 4a, 4b, 4c, 4d) for detecting posture information of the work device, and control devices 9, 21 for performing work support of the work machine based on a pre-set and stored work area, there is also a camera device (e.g., a camera 22) capable of photographing the appearance of the work machine at the work site and an input device (e.g., a touch display 25) capable of specifying any position within the image photographed by the camera device. The control device extracts the image area occupied by the work machine from the image photographed by the camera device, and based on the number of pixels in the length direction of the work machine in the extracted image area, the posture information of the work machine, and the actual size of the work machine, transforms the position in the image specified by the input device into a coordinate value of an actual space coordinate, and sets the work area in such a way that the position of the transformed coordinate value becomes an endpoint. Since it is constructed in the above manner, the desired work area can be set intuitively and easily.

[0110] <Note>

[0111] It should be noted that the present invention is not limited to the above-mentioned embodiments, and includes various modifications and combinations within the scope of the gist thereof. In addition, the present invention is not limited to having all the structures described in the above-mentioned embodiments, and also includes a structure in which a part of the structure is deleted. In addition, the above-mentioned various structures, functions, etc. can also be implemented by partially or entirely designing them, for example, with integrated circuits. In addition, the above-mentioned various structures, functions, etc. can also be implemented in software by interpreting and executing a program that implements each function on a processor.

[0112] Description of Reference Numerals

[0113] 1…Hydraulic excavator, 1A…Front working machine, 1B…Car body, 1a…Boom, 1b…Arm, 1c…Bucket, 1d…Upper swing body, 1e…Lower travel body, 1f…Operation room, 2a…Boom cylinder, 2b…Arm cylinder, 2c…Bucket cylinder, 3e, 3f…Travel hydraulic motor, 4a, 4b, 4c, 4d…Angle sensor (attitude sensor), 5a…Swing hydraulic motor, 9, 21…Control device, 10…Monitor, 11…Buzzer, 12…Operation switch, 14…Gyro sensor, 20…Tablet computer, 22…Camera, 23…Gyro sensor, 24…Direction sensor, 25…Touch display, 26, 27…Communication device, 50…Image area, 51…Area end point, 52…Work area, 53…Obstacle, 54…Confirm button, 55…Cancel button, 56…Input image, 57…Hand , 81…icon, 82, 83…line, 84, 85…set value, 86…select button, 87…confirm button, 88…cancel button, 91…machine direction acquisition unit, 92…machine attitude calculation unit, 93…machine size storage unit, 94…area setting unit, 95…area storage unit, 96…work support determination unit, 101…imaging position, 102…angle of view, 103…imaging plane, 104…obstacle, 106…actual size, 107…distance, 191, 192, 193, 196…display, 194, 195, 197, 198…message, 210…image storage unit, 211…machine image extraction unit, 212…teaching data storage unit, 213…position designation unit, 214…coordinate conversion unit, 215…imaging direction acquisition unit, 216…imaging support unit, 1101…shooting button, 1102…message

Claims

1. A work support system, comprising: A working device, which is provided on the working machine; A posture detection device, which detects posture information of the working device; as well as A control device that performs work support for the work machine based on a pre-set and stored work area, The operation support system is characterized in that it further includes: A camera device capable of photographing the appearance of the working machine at the working site; as well as an input device capable of specifying an arbitrary position within the image captured by the camera device, The control device extracts an image area occupied by the working machine from the image captured by the camera device, Based on the number of pixels in the length direction of the working machine in the extracted image area, the posture information of the working machine and the actual size of the working machine, the position in the image specified by the input device is converted into a coordinate value of the actual space coordinate, The work area is set so that the position of the converted coordinate value becomes an end point.

2. The work support system according to claim 1, characterized in that: The control device calculates distances or angles in two directions with respect to the working machine based on the coordinates of the position designated by the input device. A range formed using the calculated distances or angles in the two directions is set as the work area.

3. The work support system according to claim 1, characterized in that: The control device stores a plurality of teaching data groups classified according to the posture of the working device as teaching data for extracting an image area occupied by the working machine from the image, Based on the current posture information of the working device, one teaching data is selected from the plurality of teaching data groups for use in extracting the image region.

4. The work support system according to claim 1, characterized in that: Also includes: A position detection device for detecting the position coordinates of the working machine; A camera direction detection device, which detects the camera direction of the camera device; as well as a notification device for notifying an operator of the working machine of information, The control device determines whether the camera direction of the camera device is the camera direction of the position specified by the input device and suitable for the working area based on the relative angle between the position coordinates of the working machine and the camera direction of the camera device. If it is determined that the camera direction is inappropriate, the control device notifies the notification device to prompt it to become an appropriate camera direction.

Citation Information

Patent Citations

  • Calibration operation assisting system

    CN112639226A

  • Shovel and autonomous aerial vehicle flying around shovel

    US20200370282A1

  • Work-implement external-shape measurement system, work-implement external-shape display system, work-implement control system and work machine

    US20210156121A1

  • Calibration work support system

    US20220018098A1