Control method of work support system, control program of work support system

Abstract

This invention can correctly assist the operator's operation even when the sensor is not positioned correctly. A control method for a work assistance system (1) that assists the operator's operation through a machine guidance function, wherein the work assistance system includes: a sensor unit (11) held in a movable part (5) of an engineering machine (2) to acquire posture information using the sensor; and a portable information terminal device (13) that acquires the posture information acquired by the sensor unit through data communication with the sensor unit and notifies the operator of information to assist the operator's operation based on the posture information. The control method for the work assistance system includes: a step of acquiring a photographic result by acquiring a photographic result of the movable part equipped with the sensor unit; an error calculation step of performing image processing on the photographic result and calculating the installation error of the sensor unit relative to a reference mounting position; and a correction step of correcting the posture information based on the installation error.

Description

Technical Field

[0001] This invention relates to a control method and a control program for a work assistance system, which can be applied, for example, to a hydraulic excavator. Background Technology

[0002] Previously, the company provided construction machinery equipped with machine-guided assembly capabilities (the so-called ICT construction machinery).

[0003] Machine guidance here refers to the technology that uses surveying techniques such as total stations (TS) and GNSS (Global Navigation Satellite System) to support the operation of construction machinery. Based on this machine guidance, the operator's work can be appropriately assisted, improving work efficiency, safety, and accuracy.

[0004] Regarding this type of ICT construction machine, Patent Document 1 discloses a structure for correcting deviations in the stroke length of the hydraulic cylinder.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2014-137343 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] However, in the machine guidance function, the original engineering machinery is introduced after the installation of sensors, etc.

[0010] However, even with the sensor installed and configured in this way, it is predicted that the sensor may not be configured in the correct position. Furthermore, it is predicted that if the sensor needs to be replaced during maintenance, the sensor may also be misconfigured.

[0011] In ICT deployment, when sensors are not configured in the correct locations, it may be difficult to properly assist operators.

[0012] This invention was developed with the above considerations in mind, and its purpose is to provide a control method and control program for a job assistance system that can correctly assist the operator's operation even when the sensor is not configured in the correct position.

[0013] Methods for solving problems

[0014] To address the aforementioned issues, the first technical solution of the present invention provides a control method for a job assistance system, which assists the operator's work through a machine guidance function, wherein...

[0015] The job assistance system includes:

[0016] A sensor unit, which is held in the movable part of the working machinery, acquires posture information using sensors; and

[0017] A portable information terminal device acquires the posture information obtained by the sensor unit through data communication, and notifies the operator of information based on the posture information to assist the operator's operation.

[0018] The control method of the job assistance system includes:

[0019] A step for acquiring imaging results by acquiring imaging results of the movable part equipped with the sensor unit;

[0020] The image processing steps for the captured images include calculating the installation error of the sensor unit relative to the reference mounting position; and...

[0021] The correction steps for correcting the posture information based on the installation error.

[0022] According to the configuration of the first technical solution, by calculating the installation error of the sensor unit relative to the reference mounting position and correcting the posture information, the operator's operation can be correctly assisted even if the sensor is not configured in the correct position.

[0023] Furthermore, the second technical solution of the present invention provides a control program for a work assistance system, which executes predetermined processing steps through the execution of an arithmetic processing circuit, wherein...

[0024] The job assistance system includes:

[0025] A sensor unit, which is held in the movable part of the working machinery, acquires posture information using sensors; and

[0026] A portable information terminal device acquires the posture information obtained by the sensor unit through data communication, and notifies the operator of information based on the posture information to assist the operator's operation.

[0027] The processing steps include:

[0028] A step for acquiring imaging results by acquiring imaging results of the movable part equipped with the sensor unit;

[0029] The image processing steps for the captured images include calculating the installation error of the sensor unit relative to the reference mounting position; and...

[0030] The correction steps for correcting the posture information based on the installation error.

[0031] According to the configuration of the second technical solution, by calculating the installation error of the sensor unit relative to the reference mounting position and correcting the posture information, the operator's operation can be correctly assisted even if the sensor is not configured in the correct position.

[0032] Invention Effects

[0033] According to the present invention, even if the sensor is not configured in the correct position, it can still properly assist the operator's operation. Attached Figure Description

[0034] Figure 1 This is a diagram illustrating the work assistance system according to the first embodiment of the present invention.

[0035] Figure 2 yes Figure 1 A block diagram of the work assistance system.

[0036] Figure 3 This is a flowchart showing the processing steps of the arithmetic unit.

[0037] Figure 4 It is used for explanation Figure 3 A diagram showing the processing steps.

[0038] Figure 5 This diagram illustrates the misalignment of the sensor section.

[0039] Figure 6 This diagram illustrates the state in which icons related to operation assistance are displayed on the portable information terminal device according to the second embodiment of the present invention.

[0040] Figure 7 This is a diagram showing a mobile terminal device when operating the camera area icon of this second embodiment.

[0041] Figure 8 This is a diagram showing the state of the object being photographed in the photographable area of ​​this second embodiment.

[0042] Figure 9 This is a diagram showing a portable information terminal device when operating the silhouette icon of the third embodiment of the present invention.

[0043] Figure 10 This is a diagram illustrating a portable information terminal device used when operating the shooting example icon of the fourth embodiment of the present invention.

[0044] Figure 11 This is a diagram showing a portable information terminal device when operating the silhouette selection icon according to the fifth embodiment of the present invention.

[0045] Figure 12 This is a diagram showing the state of displaying operation assistance-related icons in the portable information terminal device according to the sixth embodiment of the present invention.

[0046] Figure 13 This is a diagram showing a portable information terminal device when operating the rotation axis prediction icon of the sixth embodiment.

[0047] Figures 14(a) to (c) are images showing the photographic state when the position of the forearm is changed according to the sixth embodiment.

[0048] Figure 15 This is a diagram illustrating a hypothetical rotation axis in the sixth embodiment.

[0049] Symbol Explanation

[0050] 1: Operation assistance system; 2: Hydraulic excavator; 2A: Subject being photographed; 3: Main body; 4: Boom; 5: Arm; 5A: Rotation shaft; 5B: Rotation center (first reference point); 6: Bucket; 6A: Rotation shaft; 6B: Rotation center (second reference point); 11: Sensor unit; 12, 23, 35: Communication unit; 13: Portable information terminal device; 14: Notification unit; 21: Detection unit; 22, 34: Computation unit; 31: Display unit; 32: Imaging unit; 3 3: Operation unit; 40: Shooting assistance information; 41: Shooting area information; 42, 44, 46: Notification information; 43, 45: Demonstration shooting images; 47: First selected image; 48: Second selected image; C: Circle; F: Initial screen; F1: Shooting area icon; F2: Silhouette icon; F3: Shooting example icon; F4: Silhouette selection icon; F5: First reference point estimation icon; F6: First reference point calculation icon; L: Line; P4: Center point of the circle. Detailed Implementation

[0051] (First Implementation) Figure 1 This is a diagram illustrating the work assistance system 1 according to the first embodiment of the present invention. Figure 2 It's a block diagram.

[0052] The operation assistance system 1 assists the operator of the hydraulic excavator 2, which is a construction machine (operational machine), in their work through a machine guidance function.

[0053] Here, the hydraulic excavator 2 has a boom 4, a forearm 5, and a bucket 6 sequentially mounted on its self-propelled body 3 via tracks. Furthermore, the work assistance system 1 is not limited to hydraulic excavators; it can be widely used in various engineering machinery, such as those used in foundation improvement and civil engineering / construction work.

[0054] The work assistance system 1 includes a sensor unit 11, a communication unit 12, a portable information terminal device 13, and a notification unit 14.

[0055] Here, the sensor unit 11 is installed on the movable part of the hydraulic excavator 2, namely the boom 5, and uses the sensor to acquire posture information and output it to the communication unit 12. Here, the posture information is information that can detect the posture of the boom 5, which is the movable part. In this embodiment, the angle information of the boom 5 relative to the horizontal direction is used.

[0056] In addition, the sensor unit can be installed on the boom 4, forearm 5, and bucket 6 respectively, or it can be installed on the boom 4 or bucket 6, and can be installed in various parts as needed.

[0057] Therefore, as Figure 2 As shown, the sensor unit 11 operates powered by a battery (not shown) and acquires three-dimensional acceleration and angular velocity information using the detection unit 21 equipped with a sensor. Furthermore, the information detected by the detection unit 21 is processed by the arithmetic unit 22 to detect posture information, which is then transmitted to the communication unit 12 via wireless communication through the communication unit 23. By operating powered by a battery and transmitting posture information wirelessly, the sensor unit 11 can be easily installed at a desired location without requiring dedicated power supply and data communication cables. This allows the work assistance system 1 to easily integrate machine guidance functions into existing machine tools, thereby improving machine guidance capabilities.

[0058] More specifically, the sensor in the detection unit 21 can be an IMU (Intinal Measuring Unit) sensor, and the wireless communication in the communication unit 23 can be BLUETOOTH (registered trademark). In addition, various structures capable of detecting posture information can be widely used in the detection unit 21, and various structures capable of data communication can also be widely used in the wireless communication.

[0059] The communication unit 12 is installed in the main body of the hydraulic excavator 2. It collects the posture information obtained by the sensor unit 11 through data communication with the sensor unit 11 and outputs it to the portable information terminal device 13. Conversely, it acquires data output from the portable information terminal device 13 and outputs it to the notification unit 14.

[0060] The notification unit 14 is a structure that notifies the operator of operational information from the driver's seat of the hydraulic excavator 2. In this embodiment, it is formed by an image display device. The information assisting the operator can utilize various information that aids the operator's operation, such as the current construction position relative to the construction target. However, in this embodiment, the angle of the boom 5 relative to a reference direction (e.g., the horizontal direction) is used, thereby enabling simple and highly accurate confirmation of the boom 5's tilt. Furthermore, the notification unit 14 can notify the operator of operational information via sound or alarm tones, and can also be combined with a portable information terminal device 13.

[0061] The portable information terminal device 13 is a so-called smartphone or tablet terminal that calculates information to assist the operator's operation based on the posture information of the sensor unit 11 obtained through the communication unit 12.

[0062] More specifically, the portable information terminal device 13 includes a display unit 31, a camera unit 32, an operation unit 33, a computing unit 34, and a communication unit 35.

[0063] Here, the display unit 31 is formed by an image display panel such as a liquid crystal display panel, displaying various image information of the portable information terminal device 13. The operation unit 33 is formed by a touch panel or the like disposed on the display unit 31, detecting various operations of the operator. The imaging unit 32 responds to the operator's operations under the control of the computing unit 34 and acquires imaging results.

[0064] The communication unit 35 inputs and outputs posture information, information to assist the operator's operation, etc., between itself and the communication unit 12 via wireless data communication.

[0065] The arithmetic unit 34 is an arithmetic processing circuit that executes the application software of the job assistance system 1. It displays various image information on the display unit 31, and then switches the operation of the portable information terminal device 13 through the operation unit 33, thereby switching the operation of the job assistance system 1.

[0066] When the arithmetic unit 34 controls the actions of each part in this way and the operator instructs the assembly process, it executes... Figure 3 The processing steps shown calculate the installation error of the sensor unit 11 relative to the reference mounting position and register data for correcting the posture information detected by the sensor unit 11.

[0067] The arithmetic unit 34 corrects the posture information input from the communication units 12 and 35 based on the registered data, and sends out the corrected posture information as information to assist the operator's operation.

[0068] Specifically, when the processing of the assembly begins, the arithmetic unit 34 instructs the operator, through the display unit 31, to capture images from the sensor unit 11 used for error calculation, and records the image information (SP1-SP2-SP3) of the capture results obtained via the imaging unit 32 (image acquisition step). Here, the imaging unit 32 displays the image information on the display unit 31, for example... Figure 4 As shown, the acquisition of the imaging result is indicated based on a certain size of the rotation shafts 5A and 6A at both ends of the movable part, i.e., the boom 5, which includes the sensor unit 11, so that the installation error of the sensor unit 11 can be fully detected. In this example, the above-mentioned imaging result can be obtained by using the imaging unit 32 provided in the portable information terminal device 13 to photograph the hydraulic excavator 2 (e.g., the movable part such as the boom 5).

[0069] Next, the arithmetic unit 34 receives a condition specification (SP4) for an ideal mounting position (reference mounting position) as a detection reference for mounting errors. Here, the ideal mounting position is the mounting position in which the sensor unit 11 is correctly positioned and installed. The arithmetic unit 34 displays, for example, selectable ideal mounting positions on the display unit 31, and receives the condition specification through selection by the operator corresponding to the sensor unit 11. The arithmetic unit 34 performs image processing on the shooting results based on the condition specification and sets the detection reference for mounting errors. Figure 4 This is an example of setting the straight line L1, connecting the rotation centers 5B and 6B of rotating shafts 5A and 6A, as the reference for detecting installation errors by specifying this condition. Furthermore, in this... Figure 4 In the figure, the horizontal line of the sensor unit 11 is used to represent the detection reference for the posture information.

[0070] Next, the arithmetic unit 34 performs image processing on the captured image, detects the mounting angle of the sensor unit 11 relative to the straight line L1 of the ideal mounting position, and thereby calculates the mounting error (SP5) of the sensor unit 11 relative to the ideal mounting position (error calculation step). Figure 4 In the example, the sensor unit 11 is formed in a rectangular shape when viewed from above, and the correct mounting position is that the long side L2 of the rectangle is parallel to the straight line L1. Therefore, the mounting angle of the sensor unit 11 relative to the straight line L1 is 0 degrees, enabling accurate detection of the forearm 5's posture information.

[0071] In contrast, Figure 5 Through with Figure 4 The comparison shows an example where the sensor unit 11 is installed at an angle with a misalignment. In this case, the arithmetic unit 34 detects the angle θ2 of the long side L2 relative to the straight line L1.

[0072] The arithmetic unit 34 registers the angle θ2 of the long side L2 relative to the straight line L1 as data for correcting the posture information detected by the sensor unit 11.

[0073] Therefore, the arithmetic unit 34 corrects the posture information input from the communication units 12 and 35 based on the registered data (SP6) (correction step), and sends out the corrected posture information as information to assist the operator's operation (SP7).

[0074] Here, the posture information of the sensor unit 11, configured according to angle θ2, is detected by angle θ3. When the angle detected is set to θ1 when configured in the correct installation position, angles θ1, θ2, and θ3 can be expressed by the relationship θ1 = θ2 + θ3. Therefore, the arithmetic unit 34 corrects the posture information by adding the angle θ3 of the posture information input from the communication units 12 and 35 to the angle θ2 of the registered data.

[0075] Based on the above structure, by calculating the installation error of the sensor unit relative to the reference mounting position and correcting the posture information, the operator's operation can be correctly assisted even if the sensor is not configured in the correct position.

[0076] Furthermore, it can be assembled using an information mobile terminal and by an operator, and can also be assembled through simple operations such as taking pictures and selecting the ideal installation location.

[0077] Furthermore, by eliminating the need to modify the mounting position of the sensor unit 11, the installation of the sensor unit 11 can be performed simply, and the correct posture information can be provided appropriately by utilizing the communication function of the information mobile terminal.

[0078] (Second Implementation) Next, using Figure 6 , Figure 7 The second embodiment of the present invention will be described. In this second embodiment, the structure is configured such that the photographer (or operator) using the camera unit 32 captures at least the following: Figure 4 Before the target object is photographed by the hydraulic excavator 2, which has sensor unit 11 installed at both ends of the forearm 5 (movable part) with rotating shafts 5A and 6A, shooting assistance information 40 is displayed on the display unit 31 of the portable information terminal device 13. The shooting assistance information 40 has the function of assisting in the shooting method when the photographer (or operator) is not familiar with the shooting method of the target object.

[0079] Furthermore, here, the photographer (or operator) touches the display unit 31 (omitted in the illustration) to operate the function extension icons (which are provided as the operation unit 33). Above the display unit 31, an initial shooting screen F is displayed, consisting of a shooting area icon F1, a silhouette icon F2, a shooting example icon F3, and a silhouette selection icon F4. In this initial screen, messages indicating the selection of icons F1 to F4, which function as the operation unit 33, may also be displayed as needed.

[0080] like Figure 7 As shown, the shooting assistance information 40 in this second embodiment is configured to include, for example, shooting area information 41 displayed on the display unit 31 and notification information 42 related to the shooting area information 41 displayed on the same display unit 31.

[0081] The captureable area information 41 is a roughly rectangular mark (symbol) that provides an image prompt to the photographer (operator) indicating the area of ​​the captureable subject. It is displayed on the display unit 31 when the photographer (or operator) touches the capture area icon F1 on the initial screen F. The captureable area information 41 is set to, for example, the area excluding the outer edge of the display unit 31 (e.g., the display area occupying approximately 80% of the maximum display area). Furthermore, the shape of the captureable area information 41 is not limited to a rectangle; it can be set to any shape such as a trapezoid or parallelogram. Additionally, when the photographer (or operator) touches the capture area icon F1, the silhouette icon F2, the shooting example icon F3, and the silhouette selection icon F4 are not displayed.

[0082] Notification message 42 can display text such as "Please take a picture within the dotted line area (photographable area information 41)" and is displayed in conjunction with the display of photographable area information 41. Notification message 42 is displayed below photographable area information 41 on display unit 31. Furthermore, the state in which the photographer (or operator) has placed the subject 2A within photographable area information 41 (see reference) is also displayed. Figure 8 Under the touch operation, the shooting icon (illustration omitted) is displayed on the display unit 31, thereby acquiring the shooting result. Then, the calculation unit 34 performs the processing of receiving the conditions specified for the ideal installation position.

[0083] Thus, in the second embodiment, the display unit 31 displays the imageable area information 41, and the shooting method is easily understood and communicated using the notification information 42. As a result, the object to be photographed by the hydraulic excavator 2 is captured within the imageable area information 41. Therefore, when photographing the object, image distortion is minimized, further improving the accuracy of the posture information (and more accurately assisting the operator's operation).

[0084] That is, the imaging unit 32 is generally composed of an imaging element and a lens (e.g., a biconvex lens) located on the subject (object) side of the imaging element, but it is less prone to distortion of the peripheral portion of the captured image caused by the lens. By employing the image capture area information 41, the accuracy of the aforementioned posture information can be improved. In addition, by displaying notification information 42 on the display unit 31, the photographer (operator) can easily grasp the shooting method.

[0085] (Third Embodiment) Next, the third embodiment of the present invention will be described. In this third embodiment, the structure is configured such that the demonstration shooting information 43 of the hydraulic excavator 2 composed of silhouette images is used instead of the shooting area information 41 used in the second embodiment described above. That is, as Figure 9 As shown, the shooting assistance information 40 in this third embodiment is configured to include demonstration shooting information 43 and notification information 44 related to the demonstration shooting information 43.

[0086] The demonstration shooting information 43 can use a silhouette image representing the required parts of the hydraulic excavator 2 (here, the entire hydraulic excavator 2). When the photographer (or operator) touches the operation silhouette icon F2 in the initial screen, it is displayed on the display unit 31 (for example, the area of ​​the display unit 31 excluding the outer edge portion mentioned above). In addition, when the photographer (or operator) touches the operation silhouette icon F2, the shooting area icon F1, the shooting example icon F3, and the silhouette selection icon F4 are not displayed.

[0087] The notification message 44 can display text such as "Please take a picture while overlapping with the silhouette" and is displayed in conjunction with the demonstration shooting information 43. The notification message 44 is displayed below the demonstration shooting information 43 on the display unit 31. Furthermore, when the photographer (or operator) touches the shooting icon while the subject is approximately overlapping with the demonstration shooting information 43, the shooting result is acquired, and then the calculation unit 34 performs processing to receive the conditions specified for the ideal mounting position.

[0088] Thus, in the third embodiment, the display unit 31 displays the demonstration shooting information 43, and the shooting method is easily understood using the notification information 44. As a result, the subject of the hydraulic excavator 2 (the aforementioned required part) is photographed in a manner that substantially overlaps with the demonstration shooting information 43. Even with this configuration, it has the advantage of further improving the accuracy of the aforementioned posture information.

[0089] That is, the imaging unit 32 is generally composed mainly of an imaging element and a lens (e.g., a biconvex lens) located on the subject (object) side of the imaging element. However, it is less prone to distortion of the peripheral portion of the captured image caused by the lens. By employing the demonstration shooting information 43, the accuracy of the aforementioned posture information can be further improved. In addition, by displaying the notification information 44 on the display unit 31, the photographer (operator) can easily grasp the shooting method. Furthermore, in this third embodiment, the demonstration shooting information 43 is displayed on the display unit 31, but it can also be configured such that when the demonstration shooting information 43 is displayed on the display unit 31, the camera area information 41 used in the second embodiment is also added and displayed.

[0090] (Fourth Embodiment) Next, the fourth embodiment of the present invention will be described. In this fourth embodiment, compared with the structure of the third embodiment described above, it is configured such that only demonstration shooting information 45 for a predetermined time is displayed on the display unit 31.

[0091] like Figure 10 As shown, the shooting assistance information 40 in this fourth embodiment is configured to include demonstration shooting information 45 and notification information 46 related to the demonstration shooting information 45. The demonstration shooting information 45 can apply example images of shooting without silhouette processing of the demonstration shooting information 43 used in the third embodiment. When the photographer (or operator) touches the shooting example icon F3 in the initial screen, the display unit 31 only displays a predetermined time (e.g., 5 to 10 seconds). After the predetermined time has elapsed, the display unit 31 automatically switches to a screen showing the subject that can be photographed. Furthermore, when the photographer (or operator) touches the shooting example icon F3, the shooting area icon F1, the silhouette icon F2, and the silhouette selection icon F4 are not displayed.

[0092] Notification message 46 can display text such as "Please take a picture as shown in this example" in conjunction with the display of demonstration shooting information 45. Notification message 46 is displayed below demonstration shooting information 45 on display unit 31. Furthermore, when the photographer (or operator) determines that the subject being photographed is approximately the same as that in demonstration shooting information 45, they touch the shooting icon to obtain the shooting result. Then, the calculation unit 34 performs processing to receive the conditions specified for the ideal installation position.

[0093] Thus, in the fourth embodiment, the demonstration shooting information 45 is displayed on the display unit 31, and the shooting is notified in a way that is easy to understand using the notification information 46. Therefore, the same effect as the third embodiment described above can be achieved. Furthermore, the demonstration shooting information 45 automatically transforms into a screen showing the subject to be photographed after the aforementioned predetermined time. However, it can also be configured such that the screen showing the subject can be photographed via touch operation of the screen recovery icon (not shown) displayed on the display unit 31 (which is provided as the operation unit 33).

[0094] (Fifth Embodiment) Next, the fifth embodiment of the present invention will be described. In this fifth embodiment, the exemplary shooting information 43 used as a silhouette image in the third embodiment described above is configured to be selectable from a plurality of options.

[0095] In this case, by having the photographer (operator) touch the silhouette selection icon F4 on the initial screen, multiple candidate images of sample shooting information 43 as silhouette images are displayed on the display unit 31. For example, in this configuration, the first selection image 47 and the second selection image 48 are displayed on the display unit 31 in a vertical arrangement as selection candidates by operating the silhouette selection icon F4. Alternatively, the configuration can be linked to the display of each selection image 47 and 48, displaying text (notification information) such as "Please select an image close to the subject being photographed" on the display unit 31.

[0096] Furthermore, detailed illustrations are omitted here. For example, in each of the selected images 47 and 48, when the photographer (operator) selects the first selected image 47 (when the operation part 33 corresponding to the first selected image 47 is touched), the silhouette image corresponding to the first selected image 47 is the same as that described in the third embodiment above. Figure 9 The same situation is displayed on the display unit 31. That is, this means that when the photographer (operator) selects one of the multiple selection candidates (each selection image 47, 48) (e.g., the first selection image 47), the silhouette image corresponding to the selected first selection image 47 is displayed on the display unit 31.

[0097] Furthermore, when the photographer (operator) touches the shooting icon while the subject is approximately overlapping with the silhouette image corresponding to the first selected image 47, the shooting result is acquired, and the calculation unit 34 performs processing to receive the conditions specified for the ideal installation position. Even with this configuration, the same effect as in the third embodiment described above can be obtained.

[0098] (Sixth Implementation) Next, using Figures 12-15The sixth embodiment of the present invention will be described. In this sixth embodiment, when the photographer is photographing the subject (the aforementioned required part of the hydraulic excavator 2), even if the rotation axis 5a (rotation center 5B) of the forearm 5, which is a movable part, cannot be seen from the photographer's side, the aforementioned reference mounting position is hypothetically set.

[0099] First, by having the photographer touch the aforementioned function expansion icons, an initial screen Fa, consisting of the shooting area icon F1, silhouette icon F2, shooting example icon F3, silhouette selection icon F4, and first reference point estimation icon F5, is displayed above the display unit 31. Furthermore, in this... Figure 12 In the initial screen Fa shown, a message indicating the selection of the first reference point prediction icon F5, which serves as the function of the operation unit 33, may also be displayed as needed.

[0100] Here, the photographer confirms whether the rotation axis 5A (rotation center 5B) of the forearm 5 can be identified from the photographer's side. If the rotation axis 5A (rotation center 5B) cannot be seen from the photographer's side of the subject being photographed (the aforementioned required part of the hydraulic excavator 2) due to some reason (e.g., poor visibility from near the rotation axis 5A), touch the first reference point estimation icon (rotation axis estimation icon) F5. Figure 13 The display unit 31 indicates the first reference point of the touch operation, after the F5 icon. Figure 13 In the display unit 31, icons F1 to F4 are not displayed, and notification information 49 is displayed on the lower side. Notification information 49 can display text such as "Please fix the mobile terminal in a way that the subject is included in the frame and move only the forearm to take multiple shots at the same time".

[0101] Next, the photographer takes the first shot. As shown in Figure 14(a), the photographer photographs the aforementioned required part of the hydraulic excavator 2, namely the subject 2B. Here, the subject 2B is the entire hydraulic excavator 2. Then, with the portable information terminal device 13 set to a state that does not move from the state shown in Figure 14(a), the photographer takes the second shot (see Figure 14(b)) with the movable part, the forearm 5, slightly lowered. Furthermore, with the portable information terminal device 13 set to a state that does not move from the state shown in Figure 14(b), the photographer takes the third shot (see Figure 14(c)) with the movable part, namely the forearm 5, slightly moved forward. In addition, after the third shot is taken, the first reference point calculation icon F6 is displayed directly below the first reference point estimation icon F5.

[0102] Thus, in the sixth embodiment, only the forearm 5 of the boom 4, forearm 5, and bucket 6 is movable. The forearm 5 has a rotation center 5B that remains unchanged when the forearm 5 is operated, and a rotation center 6B that is located near the connection between the forearm 5 and the bucket 6 and changes position when the forearm 5 is operated. Here, the rotation centers 5B and 6B of this sixth embodiment correspond to the first and second reference points described in the claims below, respectively. Hereinafter, the rotation center 5B will be described as the first reference point 5B and the rotation center 6B as the second reference point 6B.

[0103] Next, the photographer touches the first reference point calculation icon F6. The touch operation of the first reference point calculation icon F6 forms a value in the display unit 31. Figure 15 The image is a composite image of subjects 2B, 2C, and 2D overlapping as shown. Here, regarding the second reference point 6B of the bucket 6 located near the connection between the forearm 5 and the bucket 6, the second reference point 6B corresponding to subject 2B is designated as the first imaginary point P1, the second reference point 6B corresponding to subject 2C is designated as the second imaginary point P2, and the second reference point 6B corresponding to subject 2D is designated as the third imaginary point P3.

[0104] The calculation unit 34 uses the equation of a circle to calculate the center point P4 of circle C, which passes through the second reference point 6B (i.e., each imaginary point P1 to P3) of the three locations obtained by operating the boom 5 and photographing the hydraulic excavator 2. The calculation unit 34 performs the process of imaginarily determining the center point P4 of circle C as the first reference point 5B of boom 5.

[0105] Next, the arithmetic unit 34 proceeds to the processing of specifying the conditions for the aforementioned reference mounting position. Here, similar to the first embodiment described above, the arithmetic unit 34 executes the process of setting the straight line L connecting the first reference point 5B and the second reference point 6B as the aforementioned reference mounting position. Then, the arithmetic unit 34 calculates the mounting error of the sensor unit 11 relative to the aforementioned reference mounting position, corrects the posture information, and executes the process of sending the corrected posture information as information to assist the operator's operation to the hydraulic excavator 2.

[0106] Furthermore, in the case where the first reference point 5B is hypothetically determined as in the sixth embodiment, no additional shooting of new shooting objects is required besides shooting the shooting objects 2B to 2D. It is sufficient to use any one of the three images of shooting objects 2B to 2D (e.g., the image of shooting object 2B) to hypothetically set the reference installation position, calculate the installation error, correct the posture information, and process the transmission of the corrected posture information to the hydraulic excavator 2. Additionally, the shooting objects 2B to 2D are objects used to shoot the state of the forearm 5 when it is temporarily stationary, but they can also be objects used to shoot the state of the forearm 5 when it is movable multiple times (e.g., more than three times) at regular intervals.

[0107] (Other implementation methods)

[0108] The above describes in detail the specific structures suitable for implementing the present invention. However, the present invention can be modified in various ways to modify the structures of the above embodiments without departing from the spirit of the present invention.

[0109] That is, in the above embodiments, the case of correcting posture information on the portable information terminal device side has been described, but the present invention is not limited thereto, and can also be performed on the hydraulic excavator side.

[0110] Furthermore, in the above embodiments, the case of providing information to assist the operator's operation by tilting the forearm 5 has been described, but the present invention is not limited thereto, and can be widely applied to cases where information to assist the operator's operation is provided based on the required mobility up to the construction target.

[0111] In addition, while the above embodiments provide information to assist the operation of the hydraulic excavator 2, they may also provide information to assist the operation of other working machinery, such as agricultural machinery.

[0112] Furthermore, in the sixth embodiment described above, an example was given of the rotation center (first reference point) 5B of the forearm 5 that is presumably not identifiable from the photographer's side. However, it can also be applied to situations where, for example, the sensor unit is also installed (fixed) on the boom 4 or the bucket 6, the rotation center located on the lower end side of the boom 4, which is a movable part, cannot be identified from the photographer's side for some reason (e.g., it is hidden by the operator's cab of the hydraulic excavator 2), or the rotation center 6B of the bucket 6, which is a movable part, cannot be identified from the photographer's side.

[0113] Furthermore, in the sixth embodiment described above, the first reference point 5B is inferred using images of three subjects 2B to 2D, but images of four or more subjects can also be used to infer the first reference point 5B. Additionally, in the sixth embodiment described above, as shown in FIG14, the forearm 5 is moved in stages and images are captured, but images of three or more subjects moving continuously can also be captured.

Claims

1. A control method for a job assistance system, which assists the operator's work through machine guidance function, characterized in that, The job assistance system includes: A sensor unit, which is held in the movable part of the working machinery, acquires posture information using sensors; and A portable information terminal device acquires the posture information obtained by the sensor unit through data communication, and notifies the operator of information based on the posture information to assist the operator's operation. The control method of the job assistance system includes: A step for acquiring imaging results by acquiring imaging results of the movable part equipped with the sensor unit; The image processing steps for the captured results include calculating the installation error of the sensor unit relative to the reference mounting position. as well as The correction steps for correcting the posture information based on the installation error.

2. The control method for the work assistance system according to claim 1, characterized in that, The shooting result can be obtained by using the shooting unit in the portable information terminal device to shoot the movable part. Before using the camera to photograph the movable part, shooting assistance information is displayed on the display of the portable information terminal device.

3. The control method for the work assistance system according to claim 2, characterized in that, The shooting assistance information includes at least one of the shooting area information displayed on the display and the demonstration shooting information of the operating machinery displayed on the display.

4. The control method for the work assistance system according to claim 3, characterized in that, The display unit displays notification information related to the cameraable area information or the demonstration shooting information.

5. The control method for the work assistance system according to claim 3 or 4, characterized in that, The demonstration image is a silhouette image.

6. The control method for the work assistance system according to claim 3 or 4, characterized in that, When the demonstration image is displayed on the display unit, it is shown on the display unit for a specified time only when the demonstration image is operated and displayed on the display unit.

7. The control method for the work assistance system according to claim 5, characterized in that, By activating the silhouette selection icon displayed on the display unit, multiple candidate silhouette images are displayed on the display unit. The selected silhouette image is displayed on the display unit by selecting one of a plurality of selection candidates.

8. The control method for the work assistance system according to claim 1, characterized in that, The movable part has a first reference point whose position does not change when the movable part is operated, and a second reference point whose position changes when the movable part is operated. The work assistance system includes a calculation unit that imaginarily determines the center point of a circle passing through three or more second reference points obtained by operating the movable part and photographing the required part of the work machine as the first reference point.

9. The control method for the work assistance system according to claim 8, characterized in that, The calculation unit sets the straight line connecting the first reference point and the second reference point, which are in a state that cannot be seen from the photographer's side of the machine being photographed, as the hypothetical reference mounting position.

10. A control program for a work assistance system, which executes predetermined processing steps through the execution of an arithmetic processing circuit, characterized in that, The job assistance system includes: A sensor unit, which is held in the movable part of the working machinery, acquires posture information using sensors; and A portable information terminal device acquires the posture information obtained by the sensor unit through data communication, and notifies the operator of information based on the posture information to assist the operator's operation. The processing steps include: A step for acquiring imaging results by acquiring imaging results of the movable part equipped with the sensor unit; The image processing steps for the captured results include calculating the installation error of the sensor unit relative to the reference mounting position. as well as The correction steps for correcting the posture information based on the installation error.

Images