Job support system

Abstract

A work support system comprising a front work tool provided on a hydraulic shovel, an angle sensor for detecting attitude information of the front work tool, and a control device for providing work support for the hydraulic shovel based on a preset and stored work area, further comprises a camera capable of capturing the external appearance of the hydraulic shovel at the work site and a touch display capable of specifying any location within the image captured by the camera, wherein the control device extracts an image area occupied by the hydraulic shovel from the image captured by the camera, converts a location within the image specified on the touch display into a coordinate value of actual spatial coordinates based on the number of pixels in the longitudinal direction of the hydraulic shovel in the extracted image area, attitude information of the hydraulic shovel, and the actual dimensions of the hydraulic shovel, and sets a work area such that the location of the converted coordinate value is an end point. By doing so, an intended work area can be set intuitively and easily.

Description

Technology Field

[0001] The present invention relates to a work support system. Background Technology

[0002] In the case of a work machine including a hydraulic shovel, a technology for improving work efficiency by setting a predetermined work area and providing work support based on that work area is known, for example, as described in Patent Documents 1 and 2.

[0003] Patent Document 1 comprises: a pivot operating lever for setting the pivot speed of an upper pivot relative to a lower traveling body of a working machine; an electric motor having a stator and a rotor, which drives the upper pivot relative to the lower traveling body by rotating the rotor relative to the stator; an angular velocity detection means embedded in the electric motor and detecting the angular velocity of the rotor relative to the stator as the motor angular velocity; an electric motor control means for controlling the rotational speed of the electric motor such that the motor angular velocity detected by the angular velocity detection means corresponds to a magnitude of the pivot speed set by the pivot operating lever; a pivot angle calculation means for calculating the pivot angle of the upper pivot relative to the lower traveling body based on the time integral value of the motor angular velocity detected by the angular velocity detection means; a pivot range setting means for setting the pivot range of the upper pivot relative to the lower traveling body; and a stop control means for stopping the operation of the electric motor so that the pivot angle does not exceed the pivot range set by the pivot range setting means, under the control of the electric motor by the electric motor control means. A slewing control device for a work machine equipped with a work machine is disclosed.

[0004] Patent Document 2 discloses a driving assistance device that assists in the operation of a work machine, and comprises a motion control unit that controls the movement of the work machine according to information indicated by a marker image corresponding to a marker at a work site within an image captured by an imaging device mounted on the work machine. Prior art literature

[0005] Japanese Patent Publication No. 2011-52383 and Japanese Patent Publication No. 2013-151830 The problem to be solved

[0006] However, in the prior art described in Patent Document 1, in order to input a predetermined threshold, the operator needs to quantitatively determine the angle to the turning stop position, making it difficult to set the turning range as intended. Furthermore, even when setting the turning stop position by moving the upper turning body, there is a problem that accurate position alignment is difficult because part of the working machine becomes a shield and the view is blocked. In particular, in the case of a hydraulic shovel, when attempting to set the working area, such as the working height or working radius, according to the machine's current position, the blind spot caused by the ceiling of the operator's seat or the bucket is large, and there is a concern that the working area will be set to an unintended position.

[0007] In addition, in the prior art described in Patent Document 2, a fixed marker is installed at the work site, a prohibited entry area is established based on the information indicated by the marker, and a work area suitable for the site environment is established by using markers installed around it to determine the area. However, if the work area changes or if movement between work sites occurs, it is necessary to reinstall the marker each time, which raises concerns about a decrease in work efficiency.

[0008] The present invention has been made in consideration of the above and aims to provide a work support system that allows an intended work area to be set intuitively and easily. means of solving the problem

[0009] The present invention includes a plurality of means for solving the above problem, but as an example, it is a work support system equipped with a work device provided in a work machine, a posture detection device for detecting posture information of the work device, and a control device for performing work support of the work machine based on a pre-set and stored work area, and further equipped with an imaging device capable of capturing the exterior of the work machine at a work site and an input device capable of specifying any location within an image captured by the imaging device, wherein the control device extracts an image area occupied by the work machine from an image captured by the imaging device, and based on the number of pixels in the longitudinal direction of the work machine in the extracted image area, the posture information of the work machine, and the actual dimensions of the work machine, converts the location within the image specified by the input device into a coordinate value of actual spatial coordinates, and sets the work area such that the location of the converted coordinate value becomes an end point. Effects of the invention

[0010] According to the present invention, a work support system can be provided that allows an intended work area to be set intuitively and easily. Brief explanation of the drawing

[0011] FIG. 1 is a side view schematically illustrating the appearance of a hydraulic shovel, which is an example of a working machine. Figure 2 is a drawing illustrating the appearance of a tablet, which is an example of a field environment information acquisition device. Figure 3 is a functional block diagram schematically illustrating the overall configuration of the work support system. Figure 4 is a diagram illustrating an example of an image captured by a tablet. Figure 5 is a flowchart showing the processing contents of the machine image extraction unit. Figure 6 is a diagram illustrating an operator selecting an area of ​​an image displayed on a touch display. Figure 7 is a flowchart showing the processing details of the coordinate transformation unit. Figure 8 is a flowchart showing the processing contents of the area setting section. Figure 9 is a drawing illustrating an example of a region setting screen. FIG. 10 is a drawing illustrating an example of a display in the normal state of a monitor indicating whether or not there is work support. FIG. 11 is a drawing illustrating an example of a display in the access status of a monitor indicating whether or not there is work support. FIG. 12 is a drawing illustrating an example of a display in an out-of-bounds state of a monitor indicating whether or not there is work support. Figure 13 is a diagram illustrating the difference in the way a hydraulic shovel is seen due to the difference in the imaging direction. Figure 14 is a diagram illustrating the difference in the way a hydraulic shovel is seen due to the difference in the imaging direction. FIG. 15 is a drawing illustrating an example of navigation of the shooting direction by an imaging support unit. Specific details for implementing the invention

[0012] Hereinafter, an embodiment of the present invention will be described with reference to FIGS. 1 to 15. In addition, in this embodiment, a hydraulic shovel equipped with a front working mechanism is described as an example of a working machine, but the present invention can be applied to other working machines as well.

[0013] FIG. 1 is a side view schematically illustrating the exterior of a hydraulic shovel, which is an example of a work machine according to the present embodiment, and FIG. 2 is a drawing illustrating the exterior of a tablet, which is an example of a field environment information acquisition device. In addition, FIG. 3 is a functional block diagram schematically illustrating the overall configuration of a work support system.

[0014] In FIG. 1, the hydraulic shovel (1), which is a working machine, is roughly composed of a multi-jointed front working machine (1A) and a vehicle body (1B).

[0015] The vehicle body (1B) is equipped with a lower driving body (1e) that is driven by left and right driving hydraulic motors (3e, 3f), and an upper turning body (1d) that is pivotally provided on the upper part of the lower driving body (1e) and is driven to turn by a turning hydraulic motor (5a). In addition, in FIG. 1, only one driving hydraulic motor (3f) is shown, and the other driving hydraulic motor (3e) is indicated by a symbol in parentheses.

[0016] The front working device (1A) is a working device configured by connecting a plurality of driven members (boom (1a), arm (1b), bucket (1c)) that each rotate in a vertical direction, and the base of the boom (1a) is rotatably supported through a boom pin at the front part of the upper swivel body (1d). In addition, an arm (1b) is rotatably connected to the front end of the boom (1a) through an arm pin. In addition, a bucket (1c) is rotatably connected to the front end of the arm (1b) through a bucket pin and a bucket link.

[0017] The boom (1a) is pivotally driven by the boom cylinder (2a), the arm (1b) is pivotally driven by the arm cylinder (2b), and the bucket (1c) is pivotally driven by the bucket cylinder (2c).

[0018] In the connection portion between the upper pivot body (1d) and the boom (1a), the connection portion between the boom (1a) and the arm (1b), the connection portion between the arm (1b) and the bucket (1c) (e.g., bucket link), and the connection portion between the lower driving body (1e) and the upper pivot body (1d), angle sensors (attitude sensors) (4a, 4b, 4c, 4d) are provided to detect the relative angle of the boom (1a) with respect to the upper pivot body (1d), the relative angle of the arm (1b) with respect to the boom (1a), the relative angle of the bucket (1c) with respect to the arm (1b), and the pivot angle of the upper pivot body (1d) with respect to the lower driving body (1e), respectively. Here, the angle sensors (4a, 4b, 4c, 4d) constitute an attitude detection device that detects the attitude of the working device. Additionally, the angle sensors (4a, 4b, 4c, 4d) can each be replaced with an inertial measurement unit (IMU) that detects a relative angle with respect to a reference plane. Furthermore, the upper swivel body (1d) is equipped with a GNSS (13) (Global Navigation Satellite System; only the antenna is shown in FIG. 1) as a position detection device for acquiring position information (position coordinates) of the hydraulic shovel (1) at the work site, and a gyroscope sensor (14) for acquiring information on the direction or inclination angle of the hydraulic shovel (1).

[0019] On the body frame of the upper swivel body (1d), an engine, a hydraulic pump, an electronic control valve, etc., which are not shown are mounted, and the operation of the hydraulic shovel (1) is controlled by controlling the flow rate and direction of the hydraulic fluid discharged from the hydraulic pump driven by the engine and supplied to a plurality of hydraulic actuators (i.e., driving hydraulic motor (3e, 3f), swivel hydraulic motor (5a), boom cylinder (2a), arm cylinder (2b), bucket cylinder (2c), etc.) using the electronic control valve.

[0020] Next to the front work device (1A) at the front of the upper slewing body (1d), a driver's cabin (1f) is provided for a driver operating the hydraulic shovel (1). Inside the driver's cabin (1f), a plurality of electric operating levers (not shown) are installed for the operator to operate the hydraulic shovel (1) while seated in the driver's seat. When the operator operates the operating levers, the amount of operation of each lever is converted into a control command value for each hydraulic actuator (2a, 2b, 2c, 3e, 3f, 5a) by the control device (9) (see FIG. 3) and sent to an electronic control valve, and the electronic control valve controls the flow rate and direction of the hydraulic fluid supplied to each hydraulic actuator (2a, 2b, 2c, 3e, 3f, 5a) according to the control command value. That is, the operator performs the operation of the hydraulic shovel (1), such as the work operation of the front work machine (1A), the turning operation of the upper turning body (1d), and the driving operation of the lower driving body (1e), by operating the operating lever.

[0021] In addition, as shown in FIG. 3, the driver's cab (1f) is equipped with a control device (9) that controls the overall operation of the hydraulic shovel (1), a monitor (10) as an input / output device, a buzzer (11) as a notification device, and an operating switch (12) as a setting device, so that information can be presented to the operator and various settings of the hydraulic shovel (1) can be performed by the operator.

[0022] The environmental information acquisition device shown in FIG. 2 is, for example, a tablet-type information terminal (hereinafter referred to as tablet (20)) and is equipped with a touch display (25) or a camera (22) as an input / output device.

[0023] Additionally, as shown in FIG. 3, the tablet (20) is equipped with a control device (21) for controlling the operation of the tablet (20), and a gyroscope sensor (23) or an orientation sensor (24) as an imaging direction detection device for detecting the direction of the tablet (20) (i.e., the imaging direction by the camera (22)).

[0024] The control device (9) of the hydraulic shovel (1) and the control device (21) of the tablet (20) are each equipped with a communication device (26, 27), so that they can communicate wirelessly with each other.

[0025] Here, the hydraulic shovel (1) (working machine) in this embodiment has a work support function that notifies the operator of the fact through a notification device, such as a monitor (10) or a buzzer (11), when a part of the hydraulic shovel, such as a front working machine (1A), exceeds a preset working range (also called a working area). The working range is defined, for example, by a turning radius, turning angle, height, etc. In this embodiment, the working range is configured to be set based on information obtained from a tablet (20).

[0026] As shown in FIG. 3, a work support system is configured by each functional part and related configuration of the control device (9) of the hydraulic shovel (1) and the control device (21) of the tablet (20).

[0027] The control device (9) of the hydraulic shovel (1) has a machine direction acquisition unit (91), a machine posture calculation unit (92), a machine dimension memory unit (93), an area setting unit (94), an area memory unit (95), and a work support judgment unit (96) as functional parts related to the work support system. Additionally, the control device (21) of the tablet (20) has an image memory unit (210), a machine image extraction unit (211), a teacher data memory unit (212), a position designation unit (213), a coordinate transformation unit (214), an image direction acquisition unit (215), and an image support unit (216) as functional parts related to the work support system.

[0028] Although not shown, the control unit (9) and the control unit (21) are each computer-equivalent hardware having a CPU (Central Processing Unit) as a processing unit, a program executed by the processing unit, and a memory unit (e.g., semiconductor memory such as ROM or RAM, or a hard disk drive, etc.) in which data required for the execution of the program is stored. That is, for example, in the control unit (9) and the control unit (21), the machine dimension memory unit (93), the area memory unit (95), the teacher data memory unit (212), etc. are realized by a memory unit such as a hard disk drive, and other functional units are realized by the execution of a program, etc. in the CPU.

[0029] Below, details of each functional part of the control device (9) and the control device (21) are described according to the flow of processing of the work support system.

[0030] First, the tablet (20) takes a picture using a camera (22) to include the appearance of the hydraulic shovel (1) at the work site.

[0031] Figure 4 is a diagram illustrating an example of an image captured by a tablet.

[0032] The shooting direction by the camera (22) of the tablet (20) is determined by which direction of the working range is set. For example, when setting the working range for the turning radius or height direction of the hydraulic shovel (1), shooting is performed from the side of the hydraulic shovel (1) as shown in FIG. 4. Also, when setting the working range for the turning angle, for example, the tablet (20) (or a function equivalent to the camera (22)) may be mounted on a drone, and the hydraulic shovel (1) may be shot from an overhead view. The image captured by the camera (22) is stored in the image memory unit (210), output to the machine image extraction unit (211), and is also output to the position designation unit (213) through the machine image extraction unit (211).

[0033] The machine image extraction unit (211) of the control device (21) extracts an image area occupied by a hydraulic shovel (1) among the input images acquired from the image memory unit (210) and outputs the coordinates thereof.

[0034] Figure 5 is a flowchart showing the processing contents of the machine image extraction unit.

[0035] As illustrated in FIG. 5, the machine image extraction unit (211) first acquires the newest image among the images stored in the image memory unit (210) as an input image (step S100). Additionally, the input image acquired by the machine image extraction unit (211) may be configured so that the user can arbitrarily select it through the operation of the touch display (25).

[0036] Next, an image region of a predetermined size is extracted from the input image acquired in step S100 (step S110).

[0037] Next, image feature quantities of the image region extracted in step S110 are calculated (step S120). For the calculation of image feature quantities, an appropriate algorithm is established in advance according to the color or shape of the work machine (here, hydraulic shovel (1)) to be extracted.

[0038] Next, a comparison is performed between the image feature quantity calculated in step S120 and the teacher data previously stored in the teacher data memory unit (212), and a determination is made as to whether the hydraulic shovel (1) (working machine) is included in the area extracted in step S110 (step S130).

[0039] In addition, the teacher data stored in the teacher data memory unit (212) and used in step S120 can take various forms, such as various posture states regarding the hydraulic shovel (1), image data of the hydraulic shovel (1) (working machine) taken at various points in time, or the feature data, or an identification function (classifier) ​​for the feature data constructed based on them.

[0040] Additionally, it is possible to select the teacher data used in the comparison processing of step S120. The machine attitude calculation unit (92) of the control device (9) acquires the detection result from the angle sensor (4a, 4b, 4c, 4d), which is an attitude sensor (attitude detection device), as the current attitude information (current attitude) of the hydraulic shovel (1) and outputs it to the machine image extraction unit (211), etc. Then, for example, by selecting the teacher data used in the comparison processing of step S120 based on the current attitude of the hydraulic shovel (1) acquired by the machine attitude calculation unit (92), it is possible to improve the image extraction precision and image processing speed of the hydraulic shovel (1).

[0041] In addition, when the same comparison processing is performed on an image taken in the past, the posture information of the hydraulic shovel (1) at the time of shooting is stored together with the image.

[0042] Next, it is determined whether the comparison processing of Step S130 for the entire area is completed (Step S140). If the determination result is "No," a new image area different from the image area extracted up to the previous cycle is extracted from the input image acquired in Step S100 (Step S110), and the processing of Steps S120 and S130 is performed on the extracted image area. That is, the processing of Steps S110 to S130 is repeated until the determination result in Step S140 becomes "Yes."

[0043] In addition, if the judgment result in step S140 is "yes," that is, if the processing of steps S110 to S130 is completed for all areas of the input image acquired in step S100, then the coordinates of the image area determined to be a machine image (hydraulic shovel (1)) are output to the position designation unit (213) and the coordinate conversion unit (214) (step S150), and the processing is terminated. In addition, if there are multiple image areas determined to include a hydraulic shovel (1) (working machine), in the judgment processing of step S130, the image area determined to be closest to the teacher data is selected, and its coordinates are output.

[0044] The position designation unit (213) of the control device (21) displays an image sent from the image memory unit (210) through the machine image extraction unit (211) on the touch display (25) to present it to the operator, and allows the operator to select the position of an end point of a predetermined area of ​​the image on the touch display (25).

[0045] Figure 6 is a diagram illustrating an operator selecting an area of ​​an image displayed on a touch display.

[0046] As shown in FIG. 6, an input image (56) sent from an image memory unit (210) is displayed on the touch display (25) of the tablet (20), and an image area (50) extracted by the machine image extraction unit (211) for the input image (56) is displayed. In this state, the operator indicates the location where the work area is to be set with a finger, etc. (the operator's hand (57) or a touch pen, etc., which is schematically illustrated in FIG. 6 for explanation), and the area end point (51) is drawn, and the work area (52) is drawn so that the area end point (51) becomes the boundary position for the height direction and the turning radius direction based on the image area of ​​the hydraulic shovel (1) (work machine). As a method for selecting the area end point (51) by the operator, for example, if the purpose is to provide work support such as preventing contact between the hydraulic shovel (1) and the obstacle (53), the position is designated on the touch display (25) so that the area end point (51) is located between the hydraulic shovel (1) and the obstacle (53). If the set position of the work area (52) is drawn at the intended position, the operator presses the decision button (54) and proceeds to the next process (processing by the coordinate conversion unit (214)). In addition, if the operator does not set the area end point (51) (i.e., set the work area), the operator presses the cancel button (55) to end the process.

[0047] In addition, in this embodiment, the case in which the interior is set as a work area by setting a boundary to surround the hydraulic shovel (1) is described as an example, but it is not limited to this, and for example, the work area may be configured to be set by setting a boundary to surround an obstacle, etc., and setting the interior as a range excluded from the work area.

[0048] The coordinate conversion unit (214) of the control device (21) converts the work area set by the operator on the touch display (25) into a value in actual space (coordinate value of actual space coordinates).

[0049] Figure 7 is a flowchart showing the processing details of the coordinate transformation unit.

[0050] As shown in FIG. 7, the coordinate transformation unit (214) first obtains the current posture (posture information) of the hydraulic shovel (1) from the machine posture calculation unit (92), and at the same time obtains the dimension values ​​of each part of the hydraulic shovel (1) that are previously stored in the machine dimension memory unit (93) of the control device (9) (step S200). At this time, when positioning is performed using an image taken in the past, the posture information of the hydraulic shovel (1) at the time of taking the image is used, which is stored together with the image, and the posture information is obtained while reading the image taken in the past.

[0051] Next, the coordinate transformation unit (214) calculates the height of the hydraulic shovel (1) and the length of the front direction (the longitudinal direction of the front work device (1A) when viewed from above on the pivot axis) based on the posture information and dimension values ​​obtained in step S200 (step S210).

[0052] Next, the coordinates of the area output from the machine image extraction unit (211) (the image area occupied by the hydraulic shovel (1)) and the area output from the positioning unit (213) (a predetermined area set by the operator) are each acquired as pixel values ​​(step S220).

[0053] Next, the size of the designated work area is converted from the pixel value to the value in actual space by using the ratio of the actual dimensions of the hydraulic shovel (1) obtained in step S210 and the pixel value of the hydraulic shovel (1) obtained in step S220 (step S230), the result of the calculation is output to the area setting unit (94) of the control device (9) (step S240), and the processing is terminated.

[0054] The area setting unit (94) of the control device (9) of the hydraulic shovel (1) sets the area that is enabled as a work support function of the hydraulic shovel (1) based on the work area information transmitted from the coordinate conversion unit (214) of the control device (21) of the tablet (20).

[0055] Figure 8 is a flowchart showing the processing contents of the area setting section.

[0056] As shown in FIG. 8, the area setting unit (94) takes a waiting state for receiving information from the tablet (20) (step S300) and determines whether there has been reception from the tablet (20) (step S310). Steps S300 and S310 indicate a state of waiting for communication between the tablet (20) and the hydraulic shovel (1), and if the result of the determination in step S300 is "No," the processing of steps S300 and S310 is repeated until the result of the determination becomes "Yes," that is, until information on the work area is received from the tablet (20).

[0057] In addition, if the result of the judgment in step S300 is "yes," that is, if information is received from the tablet (20), a notification (notification) is made to the operator that the work area has been set (changed) on the tablet (20) (step S320). The notification to the operator is made, for example, by displaying an icon or message on the monitor (10) or by sounding a notification sound from the buzzer (11).

[0058] Next, it is determined whether the display on the monitor (10), etc., of the area setting screen (see Fig. 10 later) is selected by the operation switch (12), etc. (step S330). Since it is possible to receive the work area setting information at the time when the hydraulic shovel (1) is operating, if the result of the determination in step S330 is "No," the processing of steps S320 and S330 is repeated until the result of the determination becomes "Yes," that is, until the operator recognizes the notification and selects to transition to the area setting screen of the monitor (10), and the notification to the operator is continued.

[0059] Additionally, if the result of the judgment in step S330 is "yes," the display of the monitor (10) is moved to the area setting screen (step S340), and whether the area setting is selected is determined (step S350).

[0060] Figure 9 is a drawing illustrating an example of a region setting screen.

[0061] As shown in FIG. 9, the area setting screen displayed on the monitor (10) includes an indication of the valid status of the work area, an indication of the setting value of the work area, an indication of the selection of the type of work area to be set, and an indication of determining or canceling the setting content.

[0062] As for indicating the effective state of the work area, a line (82) indicating the effective state in the turning radius direction and a line (83) indicating the effective state in the height direction are displayed together with an icon (81) representing a hydraulic shovel (1). When the work area is set (enabled) only for the turning radius direction of the hydraulic shovel (1), as exemplified in FIG. 9, the line (82) is highlighted (activated) to indicate that the work area in the turning radius direction is effective, and the line (83) is deactivated to indicate that the work area in the height direction is not set (invalidated). Likewise, when the work area is set (enabled) only for the height direction of the hydraulic shovel (1), the line (83) is highlighted while the line (82) is deactivated. Additionally, when setting (enabling) a work area in both the turning radius direction and the height direction of the hydraulic shovel (1), lines (82) and (83) are highlighted, and when not setting a work area in both the turning radius direction and the height direction of the hydraulic shovel (1), lines (82) and (83) are disabled, etc.

[0063] As for the setting values ​​of the work area transmitted from the tablet (20), the setting value in the height direction (84) and the setting value in the work radius direction (turning radius direction) (85) are displayed. The operator of the hydraulic shovel (1) determines how to set (enable) the work area by referring to these setting values ​​(84, 85). That is, among the work area (see FIG. 6) intuitively set by the operator (photographer) on the display (25) of the tablet (20) with the intention of avoiding obstacles (14), etc., the upper boundary part is set as the setting value in the height direction (84), and the lateral boundary part is set as the setting value in the work radius direction (85). Since the operator of the hydraulic shovel (1) selectively sets the validity and invalidation of the boundaries of the work area set as the setting values ​​(84, 85) on the monitor (10), the intended work area can be set intuitively and easily.

[0064] As a selection of the type of work area to be set (e.g., direction), a radio button type selection button (86) is displayed. As illustrated in FIG. 9, when an item to set the work area only for the turning radius direction of the hydraulic shovel (1) ("Set work area (turn radius)") is selected, a line (82) indicating the effective state of the turning radius direction is highlighted in conjunction with this, and a line (83) indicating the effective state of the height direction is deactivated.

[0065] A confirmation button (87) or a cancellation button (88) is displayed for determining or canceling the settings. After selecting the work area to be set, selecting the confirmation button (87) applies the settings of the work area and enables the work support function. Additionally, selecting the cancellation button (88) destroys the settings of the work area.

[0066] Additionally, the configuration may be such that an image captured by a tablet (20) is received and displayed within the area setting screen, including a view of the location set as the work area. That is, by displaying the captured image and the location of the area, it is possible to make it easier for the operator to recognize the location of the work area.

[0067] Return to Figure 8.

[0068] As shown in FIG. 8, the area setting unit (94), when the result of the judgment in step S350 is "yes", that is, when the area setting is selected, stores the value of the selected area in the area memory unit (95) (step S351), stops the notification of the area setting change to the monitor (10) (step S360), and terminates the processing.

[0069] Additionally, if the result of the judgment in step S350 is "No," that is, if no area is set, all information about the area stored in the area memory unit (95) at that time is erased (step S352), the notification of the area setting change to the monitor (10) is stopped (step S360), and the processing is terminated.

[0070] The work support determination unit (96) determines whether work support is needed based on the information of the area stored in the area memory unit (95) and the current posture (posture information) of the hydraulic shovel (1) calculated by the machine posture calculation unit (92), and if support is needed, notifies the operator by the monitor (10) or buzzer (11).

[0071] FIGS. 10 to 12 are drawings illustrating examples of monitor displays indicating whether or not work support is available.

[0072] FIG. 10 is a drawing illustrating an example of a display in a normal state.

[0073] As shown in FIG. 10, when a work area is set, a display (191) indicating the status of the work support function is displayed on the monitor (10). When work is being performed within the set area and there is no risk of deviation outside the work area, only a display (192) indicating the valid status of the set work area is displayed. In addition, FIG. 10 illustrates a case where only the work area in the direction of the turning radius is set.

[0074] FIG. 11 is a drawing illustrating an example of a display in an access state.

[0075] As shown in FIG. 11, when it is determined that there is a high probability of approaching the boundary of the work area and deviating outside the work area, the display mode of the display (192) is changed to a display (193) using a warning color such as yellow or red, for example, to alert the operator. At this time, it is preferable to display a message (194, 195) for alerting the operator.

[0076] FIG. 12 is a drawing illustrating an example of a display in an out-of-bounds state.

[0077] As shown in FIG. 12, when deviating from the work range, the indicator (193) is flashed to become the indicator (196) to provide stronger alert to the operator, and a message (197, 198) is displayed to notify that the operator is deviating from the work area.

[0078] In this way, the work support function enables efficient work performance by notifying the operator of the possibility of deviating from the preset work area or when the work area has been deviated.

[0079] In addition, in this embodiment, support is provided by notification from the monitor (10) or buzzer (11), but, for example, an electronic control valve may be provided, and the operation of the hydraulic shovel (1) may be restricted, such as by deceleration or stopping according to the state of approach or deviation, by a command value from the work support judgment unit (96).

[0080] The imaging direction acquisition unit (215) of the control device (21) of the tablet (20) calculates and acquires the imaging direction by the camera (22) of the tablet (20) based on information (directional information) from the orientation sensor (24) provided in the tablet (20) and information (tilt information) from the gyroscope sensor (14), and outputs it to the imaging support unit (216).

[0081] The machine direction acquisition unit (91) of the control device (9) of the hydraulic shovel (1) calculates and acquires the direction (direction) and tilt angle of the hydraulic shovel (1) based on information (position information, direction information) from the GNSS (13) provided in the hydraulic shovel (1) and information (tilt information) from the gyro sensor (14), and outputs it to the image support unit (216) of the control device (21) of the tablet (20).

[0082] The image support unit (216) of the control device (21) of the tablet (20) supports the operation of the tablet (20) by displaying navigation on the touch display (25) so that the image direction of the tablet (20) becomes an appropriate direction, based on information regarding the image direction by the camera (22) of the tablet (20) sent from the image direction acquisition unit (215) and information regarding the direction (orientation) of the hydraulic shovel (1) transmitted from the machine direction acquisition unit (91) of the control device (9) of the hydraulic shovel (1).

[0083] FIGS. 13 and 14 are drawings illustrating the difference in the size (viewing method) of a hydraulic shovel due to the difference in the imaging direction. FIGS. 13 and 14 illustrate the case where the hydraulic shovel (1) is viewed from above, FIG. 13 shows the case where the imaging plane of the camera and the direction of the hydraulic shovel are parallel, and FIG. 14 shows the case where the imaging plane of the camera and the direction of the hydraulic shovel are not parallel.

[0084] As shown in FIG. 13, when a hydraulic shovel (1) is photographed from a certain imaging position (101) using a camera (22) of a field of view (102), if the direction (105) of the hydraulic shovel (1) and the imaging plane (103) of the camera (22) are parallel, the ratio of the actual dimensions (106) of the hydraulic shovel (1) to the distance (107) from the bucket end point of the hydraulic shovel (1) to the obstacle (104) is equal to the ratio of the length (108) and the length (109) that are respectively projected onto the imaging plane (103). That is, if the actual dimensions (106) of the hydraulic shovel (1) are known, it is possible to estimate the distance (107) to the obstacle (104) by using the ratio of the number of pixels at the corresponding location of the captured image.

[0085] Meanwhile, as shown in FIG. 14, when the direction (105) of the hydraulic shovel (1) and the imaging plane (103) of the camera (22) are not parallel, the ratio of the actual dimensions (106) of the hydraulic shovel (1) and the distance (107) from the bucket end point of the hydraulic shovel (1) to the obstacle (104) is different from the ratio of the length (112) and the length (113) that are respectively projected on the imaging plane (103). That is, when estimating the distance (107) to the obstacle (104) using the ratio of the number of pixels at the corresponding location of the captured image, the accuracy of the estimation result is reduced. Therefore, when taking a picture, it is necessary to perform navigation of the picture so that the direction of the hydraulic shovel (1) (i.e., the direction of the front work machine (1A)) and the imaging plane of the camera (22) become parallel.

[0086] FIG. 15 is a drawing illustrating an example of navigation of the shooting direction by an imaging support unit.

[0087] The imaging support unit (216) calculates the relative angle between the tablet (20) and the front work machine (1A) based on information from the imaging direction acquisition unit (215) and the machine direction acquisition unit (91). If the relative angle is greater than or equal to a predetermined threshold, navigation is performed to tilt the camera (22) of the tablet (20) in a specific direction, as shown in FIG. 15 (e.g., displaying a message (1102) prompting adjustment of the shooting direction). Additionally, if the relative angle is less than the predetermined threshold, a message prompting shooting is displayed (not shown), and shooting is performed by the photographer pressing the shooting button (1101). Furthermore, navigation may be performed not only in the yaw direction of the hydraulic shovel (1), but also in each direction such as the roll direction or pitch direction depending on the imaging position.

[0088] The effects of the present embodiment configured as described above will be explained.

[0089] In the present embodiment, a work support system comprising a work device (e.g., a front work device (1A)) provided on a work machine (e.g., a hydraulic shovel (1)), a posture detection device (e.g., an angle sensor (4a, 4b, 4c, 4d)) for detecting posture information of the work device, and a control device (9, 21) for performing work support of the work machine based on a pre-set and stored work area, further comprises an imaging device (e.g., a camera (22)) capable of capturing the appearance of the work machine at the work site, and an input device (e.g., a touch display (25)) capable of specifying any location within an image captured by the imaging device, wherein the control device extracts an image area occupied by the work machine from an image captured by the imaging device, converts a location within the image specified by the input device into a coordinate value of actual spatial coordinates based on the number of pixels in the longitudinal direction of the work machine in the extracted image area, posture information of the work machine, and actual dimensions of the work machine, and the converted coordinate value Since the work area is configured to be set so that the location becomes an end point, the intended work area can be set intuitively and easily.

[0090] Bookkeeping

[0091] Furthermore, the present invention is not limited to the embodiments described above, and includes various modifications or combinations within the scope that do not deviate from the gist thereof. Additionally, the present invention is not limited to having all the configurations described in the embodiments described above, but also includes cases where some of the configurations are omitted. Furthermore, each of the above configurations, functions, etc. may be realized by designing some or all of them, for example, as an integrated circuit. Additionally, each of the above configurations, functions, etc. may be realized as software by a processor interpreting and executing a program that realizes each function. Explanation of the symbols

[0092] 1: Hydraulic Shovel 1A: Front implement 1B: Chassis 1a: Boom 1b: Female 1c: Bucket 1d: Upper slewing body 1e: Lower driving body 1f: Cab 2a: Boom Cylinder 2b: Female cylinder 2c: Bucket cylinder 3e, 3f: Driving hydraulic motors 4a, 4b, 4c, 4d: Angle sensors (attitude sensors) 5a: Slewing hydraulic motor 9, 21: Control unit 10: Monitor 11: Buzzer 12: Control switch 14: Gyro sensor 20: Tablet 22: Camera 23: Gyro sensor 24: Orientation sensor 25: Touch display 26, 27: Communication device 50: Image area 51: Area end point 52: Work area 53: Obstacles 54: Confirm button 55: Cancel button 56: Input image 57: Hand 81: Icon 82, 83: Line 84, 85: Setting values ​​86: Select button 87: Confirm button 88: Cancel button 91: Machine direction acquisition unit 92: Machine attitude calculation unit 93: Machine dimension memory unit 94: Area setting unit 95: Area Memory Unit 96: Task Support Decision Unit 101: Imaging Position 102: Angle of View 103: Imaging surface 104: Obstacle 106: Actual dimensions 107: Distance 191, 192, 193, 196: Display 194, 195, 197, 198: Message 210: Image memory unit 211: Machine image extraction unit 212: Teacher Data Memory Unit 213: Location Unit 214: Coordinate transformation unit 215: Imaging direction acquisition unit 216: Imaging Support Unit 1101: Shooting Button 1102: Message

Claims

Claim 1 A work support system comprising a work device provided on a work machine, a posture detection device for detecting posture information of the work device, and a control device for performing work support of the work machine based on a pre-set and stored work area, and further comprising an imaging device capable of capturing the exterior of the work machine at a work site and an input device capable of specifying an arbitrary location within an image captured by the imaging device, wherein the control device extracts an image area occupied by the work machine from an image captured by the imaging device, converts a location within the image specified by the input device into a coordinate value of actual spatial coordinates based on the number of pixels in the longitudinal direction of the work machine in the extracted image area, posture information of the work machine, and actual dimensions of the work machine, and sets the work area such that the location of the converted coordinate value becomes an end point. Claim 2 delete Claim 3 delete Claim 4 delete

Images