Surrounding monitoring device for industrial machinery

Abstract

To provide a monitoring device for work machinery that can properly monitor the area around the work machinery. [Solution] A monitoring device for work machinery that monitors the surroundings of work machinery having work equipment for performing work at a construction site comprises an environmental information acquisition device that acquires environmental information about the surroundings of the work machinery, a determination unit that uses a machine learning-based object recognition model to perform detection and determination of objects present around the work machinery based on the environmental information, and a storage unit that stores multiple different object recognition models. The determination unit selects and uses different object recognition models for determinations based on first environmental information including information about the work equipment of the work machinery and determinations based on second environmental information that does not include information about the work equipment of the work machinery.

Description

Technical Field

[0001] The present invention relates to a surrounding monitoring device for a work machine.

Background Art

[0002] In civil engineering work, construction work, demolition work, etc., various work machines such as a hydraulic excavator having a front work machine are used. In such work machines, there is known a technique for determining the presence or absence and type of an object around the excavator using environmental information representing the situation around the excavator.

[0003] For example, in Patent Document 1, a learned model in which machine learning has been performed is updated to an additional learned model in which additional learning has been performed based on teacher information generated from environmental information around the excavator, and the updated learned model is used to perform determination regarding an object around the excavator based on the environmental information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Generally, in the area surrounding a work machine (for example, the front work machine of a hydraulic excavator), the terrain and surrounding objects change frequently in conjunction with operations such as excavation and loading. Furthermore, the work machine is visible in the video footage of the area surrounding the work machine, and its position and shape also change frequently in conjunction with the operation. On the other hand, in areas other than the area surrounding the work machine, the terrain and surrounding objects change less than in the area surrounding the work machine. In addition, in video footage of areas other than the area surrounding the work machine, the work machine is either not visible or only visible at the edge of the screen, so the impact of the work machine's reflection is minimal. Thus, in the case of work machines, the environment often differs greatly between the area surrounding the work machine and the area outside of it.

[0006] However, in the conventional technology described above, when applying machine learning-based judgment to a work machine, the characteristic of the work machine that the environment differs between the video footage acquired from the area around the work machine and the video footage acquired from the area outside the work machine is not taken into consideration. Therefore, if the same trained model is used for judgment in both the area around the work machine and the area outside the work machine, appropriate notification may not be provided, potentially reducing the operator's work efficiency.

[0007] The present invention has been made in view of the above, and aims to provide a surrounding monitoring device for work machinery that can perform appropriate monitoring of the surroundings of the work machinery. [Means for solving the problem]

[0008] The present invention includes several means for solving the above problems, but to give one example, in a work machine surroundings monitoring device for a work machine having a work machine for performing work at a construction site, the device comprises an environmental information acquisition device that acquires environmental information of the work machine, a determination unit that uses a machine learning-based object recognition model to perform detection determinations regarding objects present around the work machine based on the environmental information, and a storage unit that stores multiple different object recognition models, wherein the determination unit uses different object recognition models for detection determinations based on first environmental information that includes information related to the work machine of the work machine, and for detection determinations based on second environmental information that does not include information related to the work machine of the work machine. [Effects of the Invention]

[0009] According to the present invention, it is possible to properly monitor the surroundings of the work machine. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic side view showing the overall configuration of a hydraulic excavator, which is an example of a work machine. [Figure 2] This is a functional block diagram showing a surrounding monitoring device that monitors the area around a work machine. [Figure 3] This is a flowchart showing the procedure for selecting an object recognition model. [Figure 4] This flowchart shows the processing steps involved in the learning and update process. [Figure 5] This diagram illustrates the different types of terrain height (upward direction) in terrain information. [Figure 6] This diagram illustrates the different types of terrain height (horizontal direction) in terrain information. [Figure 7] This diagram illustrates the different types of terrain height (downward direction) in terrain information. [Modes for carrying out the invention]

[0011] Hereinafter, one embodiment of the present invention will be described with reference to the drawings. In this embodiment, a crawler-type hydraulic excavator will be used as an example of a work machine used in various tasks such as civil engineering, construction, and demolition work, but the present invention is not limited to this, and can be applied to other work machines such as wheel loaders and wheel excavators with front work equipment.

[0012] <Working equipment: Hydraulic excavator 1> An overview of a hydraulic excavator, shown as an example of a work machine to which the present invention applies, will be described.

[0013] Figure 1 is a schematic side view showing the overall configuration of a hydraulic excavator, which is an example of a work machine according to this embodiment.

[0014] In Figure 1, the hydraulic excavator 1 (working machine) is generally composed of a lower traveling body 10, an upper rotating body 20 that is rotatably mounted on the upper part of the lower traveling body 10, a front working machine 30 (working machine) mounted on the upper rotating body 20, and hydraulic actuators 13, 14, 22, 32, 34, and 36 that drive the lower traveling body 10, the upper rotating body 20, and the front working machine 30.

[0015] The lower traveling body 10 consists of a pair of left and right crawlers 11 and a crawler frame 12, and a pair of travel hydraulic motors 13, 14 (hydraulic actuators: only one is shown in Figure 1, and the other is indicated only by its symbol in parentheses) and a reduction mechanism (not shown) that independently drive and control each crawler 11.

[0016] The upper swing body 20 is provided with a swing hydraulic motor 22 (hydraulic actuator), a speed reduction mechanism (not shown) for reducing the rotation of the swing hydraulic motor 22, and a swing mechanism 23 for swing-driving the upper swing body 20 with respect to the lower traveling body 10 by the driving force of the swing hydraulic motor 22. Further, the upper swing body 20 is provided with an engine 21 as a prime mover, a hydraulic pump (not shown) driven by the engine 21, a hydraulic system for controlling the flow rate and direction of the pressure oil discharged from the hydraulic pump to drive and control the hydraulic actuators 13, 14, 22, 32, 34, 36, and a control device 40 for controlling the overall operation of the hydraulic excavator 1 including the hydraulic system.

[0017] In this embodiment, the case where the hydraulic pump is driven by the engine 21 is exemplified and described, but the present invention is not limited thereto. For example, the present invention can also be applied to a work machine that drives a hydraulic pump by an electric motor.

[0018] The front work implement 30 is a multi-joint work implement that performs various operations such as ground excavation and earth and sand transportation. It includes a boom 31 pivotally supported in the vertical direction in front of the upper swing body 20, an arm 33 pivotally supported at the tip of the boom 31 in the vertical direction, and a bucket 35 pivotally supported at the tip of the arm 33 in the vertical direction. Further, the front work implement 30 is provided with boom cylinders 32, arm cylinders 34, and bucket cylinders 36, which are hydraulic actuators, and the front work implement 30 is driven by these hydraulic actuators 32, 34, 36.

[0019] On the front, rear, left, and right of the upper part of the upper revolving body 20, there are respectively provided an image sensor 41 such as a camera for acquiring the surrounding video (image) of the hydraulic excavator 1 as environmental information, and a distance sensor 42 such as a LiDAR (Light Detection And Ranging) for acquiring the terrain around the hydraulic excavator 1 as terrain information. Here, the plurality of image sensors 41 constitute a surrounding video acquisition device 55 (environmental information acquisition device) for acquiring the environmental information around the hydraulic excavator 1 (working machine). Also, the plurality of distance sensors 42 constitute a terrain information acquisition device 50 for acquiring the terrain information around the hydraulic excavator 1 (working machine).

[0020] Among the plurality of image sensors 41 that constitute the surrounding video acquisition device 55, the image sensor 41 on the front side of the upper revolving body 20 (the front side of the operator sitting in the driver's seat), that is, the side in the direction where the front working machine 30 is provided and operates, is particularly referred to as the image sensor 41 on the front working machine 30 side to distinguish it from the other image sensors 41. Also, the other image sensors 41 are referred to as the image sensors 41 on the left, right, and rear sides. In the image sensor 41 on the front working machine 30 side, the front working machine 30 is included within the viewing angle (the acquired image or the detection range of an object). On the other hand, in the image sensors 41 on the left, right, and rear sides, the front working machine 30 is not included within the viewing angle (the acquired image or the detection range of an object), or is included only in a part of the end portion.

[0021] <System Configuration: Control Device 40> The system configuration of the surrounding monitoring device in the present embodiment will be described.

[0022] FIG. 2 is a functional block diagram showing a surrounding monitoring device for monitoring the surroundings of a working machine.

[0023] As shown in FIG. 2, the surrounding monitoring device is composed of each functional part realized by a control device 40, a terrain information acquisition device 50, a surrounding video acquisition device 55, and a driving support part 51.

[0024] The control device 40 consists of an arithmetic unit 52 such as a CPU (Central Processing Unit) and a storage device 53 such as memory.

[0025] The computing unit 52 includes an object recognition unit 54, a driving support determination unit 56, and a model learning unit 57, which are implemented as functional units related to the surrounding monitoring device.

[0026] Furthermore, the storage device 53 includes a model storage unit 58 and a learning data storage unit 59, which are implemented as functional units related to the surrounding monitoring device.

[0027] The surrounding image acquisition device 55 acquires images of the area around the hydraulic excavator 1 (the vehicle itself) and transmits the acquired images and mounting information of the image sensors 41, such as cameras, that constitute the surrounding image acquisition device 55 (information relating to the mounting position and direction of the image sensors 41 on the hydraulic excavator 1) to the object recognition unit 54 as environmental information. The terrain information acquisition device 50 measures the terrain around the hydraulic excavator 1 (the vehicle itself) and transmits the measurement results and mounting information of the distance sensors 42, such as LiDAR, that constitute the terrain information acquisition device 50 (information relating to the mounting position and direction of the distance sensors 42 on the hydraulic excavator 1) to the object recognition unit 54 as terrain information. Although the explanation has illustrated the case where the terrain information acquisition device 50 measures the terrain around the hydraulic excavator 1 to acquire terrain information, it is not limited to this, and for example, terrain information created in advance by surveying or the like may be acquired from outside the hydraulic excavator 1 and the acquired terrain information may be transmitted to the object recognition unit 54.

[0028] The model storage unit 58 stores multiple input object recognition models in response to a request from the model learning unit 57, and outputs the stored object recognition models to the model learning unit 57. The model storage unit 58 also outputs the stored object recognition models in response to a request from the object recognition unit 54.

[0029] An object recognition model is, for example, weighted parameter data trained using deep learning. In deep learning, training is performed using training data that takes into account the object and conditions to be detected, and the detection performance of the detection function is improved by adjusting the model's weights. Since the detection function performs detection by reading the object recognition model, the detection performance depends on the performance of the object recognition model. For example, by updating (replacing) the object recognition model used by the detection function from a conventional model to a model that has been trained according to the environment, it is possible to create a detection function that is suitable for the environment and obtain higher detection performance.

[0030] The learning data storage unit 59 stores the input learning data in response to a request from the model learning unit 57, and also outputs the stored learning data to the model learning unit 57.

[0031] The object recognition unit 54, as part of its object recognition processing, uses an object recognition model acquired from the model storage unit 58 to detect objects around the hydraulic excavator 1 (self-vehicle) based on the surrounding video (environmental information) acquired by the surrounding video acquisition device 55 and the terrain measurement results (terrain information) acquired by the terrain information acquisition device 50. The object recognition unit 54 then determines and calculates the presence or absence of an object on the video (image) of the object to be detected, the coordinates of the detected object, and the type of the detected object, and transmits this information to the driving support determination unit 56. When detecting an object, the object recognition unit 54 selects an object recognition model to be used for object detection from a plurality of different object recognition models stored in the model storage unit 58, based on the terrain information acquired from the terrain information acquisition device 50 and the mounting information (mounting position and mounting direction) of the image sensor 41 acquired from the surrounding video acquisition device 55.

[0032] The driver assistance determination unit 56, as part of the driver assistance determination process, determines whether driver assistance is necessary based on the coordinates and type of the detected object obtained from the object recognition unit 54, and outputs a notification command to the driver assistance unit 51 based on the determination result.

[0033] The driver support unit 51 consists of a monitor and a buzzer, and receives notification commands from the driver support judgment unit 56. Based on the acquired notification commands, it provides driver support by notifying the operator or workers around the vehicle of various information through displays on the monitor and buzzer activations.

[0034] The model learning unit 57 acquires video from the ambient video acquisition device 55, generates training data based on the acquired video, and outputs the generated training data to the training data storage unit 59 for storage. The model learning unit 57 also acquires training data from the training data storage unit 59 and performs additional training to generate a trained object recognition model, and outputs the generated object recognition model to the training data storage unit 59 for storage. When performing transfer learning, the base object recognition model is acquired from the training data storage unit 59. Here, transfer learning is, for example, when the performance of an already used trained object recognition model deteriorates for a specific untrained condition, the trained object recognition model is used as a base to additionally train data corresponding to that specific untrained condition. In other words, instead of learning everything from scratch, the training time can be reduced by utilizing the conventional object recognition model and additionally training only a portion of the data.

[0035] <Processing details: Object recognition processing, driver assistance judgment processing> This section explains the processing details for object recognition and driver assistance decision-making.

[0036] Figure 3 is a flowchart showing the procedure for selecting an object recognition model.

[0037] In Figure 3, the object recognition unit 54 of the control device 40 first determines whether the image to be detected is the image from the image sensor 41 on the front work machine 30 side (step S100). The object recognition unit 54 obtains mounting information of the image sensor 41 (information relating to the mounting position and mounting direction of the image sensor 41 on the hydraulic excavator 1) from the surrounding image acquisition device 55 and determines whether the mounting direction (shooting direction) of the image sensor 41 points towards the front work machine 30 side.

[0038] If the result of the determination in step S100 is NO, that is, if the mounting direction (shooting direction) of the image sensor 41 that captured the video to be detected is not towards the work machine side, the "left / right rear side model" is selected as the object recognition model and detection is performed (step S101). In other words, for video from directions other than the work machine side, detection is performed using an appropriate object recognition model such as the left / right rear side model.

[0039] Furthermore, if the result of the determination in step S100 is YES, it is determined that an object recognition model (referred to as the work machine side model) targeting the image of the front work machine 30 is necessary, and then, based on the terrain information from the terrain information acquisition device 50, it is determined whether the height direction of the terrain around the front work machine 30 (in other words, the height of the ground targeted by the front work machine 30) is upward, horizontal, or downward (step S110).

[0040] Figures 5 to 7 illustrate the different types of terrain height in terrain information.

[0041] Figure 5 shows the case where the terrain elevation is upward. When the terrain elevation is upward, for example, a hydraulic excavator 1 (working machine) is working on the lower side of a slope, and the slope of the embankment is reflected in the upper part of the image acquired by the image sensor 41 on the front working machine 30 side. Also, if the object to be detected is on the slope of the embankment, the object to be detected will be reflected in the upper part of the image compared to when the terrain is horizontal or downward, and the way the object is reflected in the image will be different from other terrain elevations (horizontal and downward), resulting in a unique appearance when the terrain is upward.

[0042] Figure 6 shows the case where the terrain elevation is horizontal. When the terrain elevation is horizontal, for example, this is the case when a hydraulic excavator 1 (working machine) is working on flat ground, and the image acquired by the image sensor 41 on the front work machine 30 does not show anything other than the ground being worked on.

[0043] Figure 7 shows the case where the terrain elevation is downward. When the terrain elevation is downward, for example, this is the case when a hydraulic excavator 1 (working machine) is working on the upper side of a slope or ditch, and the slope of the embankment or the ground further away from the embankment is captured in the lower part of the image acquired by the image sensor 41 on the front working machine 30. Also, if the object to be detected is on the slope of the embankment or on the ground surface further away from the embankment, the object to be detected will be captured in the lower part of the image compared to when the terrain is upward or horizontal, and the way the object is captured in the image will differ from other terrain elevations (upward and horizontal), resulting in a unique appearance specific to the downward direction.

[0044] Return to Figure 3.

[0045] As shown in Figures 5 to 7, the way terrain and objects to be detected appear in video footage changes significantly depending on the type of terrain. Therefore, it is desirable to perform object recognition processing using an object recognition model suitable for each type of terrain.

[0046] Therefore, in Figure 3, if the determination result in step S110 is "upward," that is, if it is determined that the terrain height is upward, then from among the multiple work machine side models, the "work machine side model (upward)" is selected as the object recognition model suitable for detecting upward terrain and detection is performed (step S111). In this way, in order to perform detection by an appropriate object recognition model for the image from the image sensor 41 on the front work machine 30 side and for the image where the terrain height is upward, detection is performed using the work machine side model (upward). The work machine side model (upward) is an object recognition model that has sufficiently learned images where the terrain height is upward and has confirmed a certain level of performance in evaluation of images where the terrain height is upward.

[0047] Furthermore, if the determination result in step S110 is "horizontal," that is, if the terrain height is determined to be horizontal, then from among the multiple work machine side models, the "work machine side model (horizontal)" is selected as the object recognition model suitable for detecting horizontal terrain and detection is performed (step S112). In this way, in order to perform detection using an appropriate object recognition model for the image from the image sensor 41 on the front work machine 30 side and for images where the terrain height is horizontal, detection is performed using the work machine side model (horizontal). The work machine side model (horizontal) is an object recognition model that has sufficiently learned images where the terrain height is horizontal and has confirmed a certain level of performance in evaluation of images where the terrain height is horizontal.

[0048] Furthermore, if the determination result in step S110 is "downward," that is, if it is determined that the terrain height is downward, then from among the multiple work machine side models, the "work machine side model (downward)" is selected as the object recognition model suitable for detecting downward terrain and detection is performed (step S113). In this way, in order to perform detection by an appropriate object recognition model for the image from the image sensor 41 on the front work machine 30 side and for the image where the terrain height is downward, detection is performed by the work machine side model (downward). The work machine side model (downward) is an object recognition model that has sufficiently learned images where the terrain height is downward and has confirmed a certain level of performance in evaluation of images where the terrain height is downward.

[0049] When any of steps S101, S111, S112, or S113 is completed, the driving assistance determination unit 56 then determines whether driving assistance is necessary based on the coordinates and type of the detected object obtained from the object recognition unit 54 (step S120). If the determination result is NO, the process ends.

[0050] Furthermore, if the result of the determination in step S120 is YES, the system outputs a notification command to the driver support unit 51 to perform driver support (step S121), and the process ends.

[0051] <Processing details: Learning and update processing> This section describes the processing details of the learning and updating process, which involves learning (additional learning) and updating the object recognition model. In updating the object recognition model, the frequency of model updates through learning of the object recognition model used for detection and judgment based on environmental information including information related to the front work equipment 30 of the hydraulic excavator 1 (working machine) (e.g., images of the work equipment side: first environmental information) is higher than the frequency of model updates through learning of the object recognition model used for detection and judgment based on environmental information that does not include information related to the work equipment 30 of the front work equipment (e.g., images of the left, right, and rear sides: second environmental information).

[0052] Figure 4 is a flowchart showing the processing steps of the learning and updating process.

[0053] In Figure 4, the control device 40 determines whether the target object recognition model is the work machine side model when updating the object recognition model (step S200).

[0054] If the result of the determination in step S200 is YES, that is, if the target object recognition model is the work machine side model, it is determined that learning and updating of the work machine side model is necessary, and it is determined whether or not it is a suitable time to perform learning of the work machine side model (step S210). For example, if the CPU (arithmetic unit 52) ​​is under heavy load due to vehicle operation or other processes, it is determined that the ECU (control unit 40) is not in a suitable state for learning and should not perform learning. On the other hand, if the CPU load is below a predetermined threshold value, it is determined that it is a suitable time to perform learning.

[0055] If the result of the determination in step S210 is NO, that is, if it is determined that it is not a suitable time to perform learning, the system will notify that learning was not performed as an updated result (step S260) and terminate the process.

[0056] Furthermore, if the determination result in step S210 is YES, that is, if it is determined that it is a suitable time to perform learning, the learning data storage unit 59 performs learning of the work machine side model (step S220) and performs evaluation of the learned work machine side model (step S230). The evaluation of the work machine side model, which is an object recognition model, is performed based on a group of evaluation images stored in advance in the vehicle (hydraulic excavator 1), a group of surrounding images acquired at the work site where the vehicle is operating, or a group of surrounding images acquired by an imaging device outside the vehicle. For example, by evaluating the object recognition model based on a group of surrounding images acquired by the vehicle, it can be confirmed that the learning model to be updated is suitable for the environment in which the vehicle is used.

[0057] Next, it is determined whether the performance of the work machine model is above the standard (step S240). If the result of the determination is NO, that is, if it is determined that the trained work machine model does not meet a certain performance standard, it is notified as an update result that a work machine model that is an object recognition model that meets a certain performance standard was not generated through training, and the work machine model was not updated (step S260), and the process ends.

[0058] Furthermore, if the determination result in step S240 is YES, that is, if it is determined that the learned work machine model meets the criteria and meets a certain level of performance, the work machine model is updated with the learned work machine model (step S250), and then the completion of learning is notified as the update result (step S260), and the process ends.

[0059] Furthermore, if the result of the determination in step S200 is NO, that is, if the object recognition model to be updated is not the work machine side model, it is determined that the left and right rear side models need to be updated, and the left and right rear side models are downloaded and acquired from a cloud server etc. 60 on the external network of the vehicle (hydraulic excavator 1) and stored in the learning data storage unit 59 (step S201).

[0060] Next, the model learning unit 57 performs an evaluation of the left and right rear models stored in the learning data storage unit 59 (step S202) and determines whether the performance of the left and right rear models is above the standard (step S203).

[0061] If the result of step S203 is NO, that is, if it is determined that the trained left and right rear models do not meet a certain performance standard, the system notifies that no left and right rear model object recognition model that meets a certain performance standard was generated through training, and that the left and right rear models were not updated (step S260), and the process terminates.

[0062] Furthermore, if the result of the determination in step S203 is YES, that is, if it is determined that the trained left and right rear models meet the criteria and satisfy a certain level of performance, the left and right rear models are updated with the trained left and right rear models (step S204), and then the completion of training is notified as an update result (step S260), and the process ends.

[0063] The effects of this embodiment, configured as described above, will now be explained.

[0064] Generally, in the area surrounding a work machine (for example, the front work machine of a hydraulic excavator), the terrain and surrounding objects change frequently in conjunction with operations such as excavation and loading. Furthermore, the work machine is visible in the video footage of the area surrounding the work machine, and its position and shape also change frequently in conjunction with the operation. On the other hand, in areas other than the area surrounding the work machine, the terrain and surrounding objects change less than in the area surrounding the work machine. In addition, in video footage of areas other than the area surrounding the work machine, the work machine is either not visible or only visible at the edge of the screen, so the impact of the work machine's reflection is minimal. Thus, in the case of work machines, the environment often differs greatly between the area surrounding the work machine and the area outside of it.

[0065] However, in conventional technologies, when applying machine learning-based judgment to work machines, the characteristic of work machines that the environment differs between the area around the work machine and the area outside the work machine is not taken into consideration. Therefore, if the same trained model is used for judgment in both the area around the work machine and the area outside the work machine, appropriate notifications may not be provided, potentially reducing the operator's work efficiency.

[0066] In contrast, this embodiment provides a monitoring device for a work machine that monitors the surroundings of a work machine (hydraulic excavator 1) having a work machine (front work machine 30) for performing work at a construction site. The device includes an environmental information acquisition device (surrounding video acquisition device 55) that acquires environmental information about the surroundings of the work machine, a determination unit (object recognition unit 54) that uses a machine learning-based object recognition model to perform detection determinations regarding objects present around the work machine based on the environmental information, and a storage unit (storage device 53) that stores multiple different object recognition models. The determination unit is configured to use different object recognition models for detection determinations based on first environmental information that includes information about the work machine's work machine, and for detection determinations based on second environmental information that does not include information about the work machine's work machine, thereby enabling appropriate monitoring of the surroundings of the work machine.

[0067] <Other> It should be noted that the present invention is not limited to the embodiments described above, and includes various modifications that do not depart from the spirit of the invention. Furthermore, for example, the present invention is not limited to having all the configurations described in the embodiments described above, but also includes configurations in which some of those configurations are omitted. In addition, it is possible to add or replace some of the configurations of one embodiment with the configurations of another embodiment.

[0068] For example, in this embodiment, mounting information for the image sensor 41 (information relating to the mounting position and mounting direction of the image sensor 41 on the hydraulic excavator 1) is obtained from the surrounding image acquisition device 55, but this is not limited to this. The mounting information may also be obtained via another system that has mounting information for the image sensor 41, such as an image processing unit that creates an overhead view of the hydraulic excavator 1.

[0069] Furthermore, although the system is configured to determine the imaging direction of the image sensor 41 based on the mounting information, it is not limited to this, and for example, the system may be configured to determine the imaging direction by detecting the presence or absence of a front work machine in the image using image recognition or the like.

[0070] Furthermore, the functions provided by driver assistance are not limited to notification; they may also perform vehicle control and log vehicle information.

[0071] Furthermore, the means of acquiring terrain information are not limited to distance sensors or surveying; for example, terrain information may be estimated based on the operation history of the work unit. For example, when the front work unit is working with its tip positioned below the plane along the bottom surface of the lower vehicle body, the terrain may be estimated to be downward. Also, when terrain information near the front work unit has been determined, if the position that was near the work unit shifts to the left or right rear of the vehicle body due to movement such as turning, the terrain information at the time when it was near the work unit may be used to estimate the terrain information near the left or right rear of the vehicle body after the movement.

[0072] Furthermore, each of the configurations related to the control device described above, as well as the functions and execution processes of each of those configurations, may be partially or entirely implemented in hardware (for example, by designing the logic for executing each function using an integrated circuit). Alternatively, the configuration of the control device described above may be a program (software) that is read and executed by an arithmetic processing unit (e.g., a CPU) to realize each of the functions related to the configuration of the control device. Information related to such a program can be stored, for example, in semiconductor memory (flash memory, SSD, etc.), magnetic storage devices (hard disk drives, etc.), and recording media (magnetic disks, optical disks, etc.). [Explanation of Symbols]

[0073] 1...Hydraulic excavator (working machine), 10...Lower traveling body, 11...Crawler, 12...Crawler frame, 13...Travel hydraulic motor, 14...Travel hydraulic motor, 20...Upper slewing body, 21...Engine, 22...Slewing hydraulic motor, 23...Slewing mechanism, 30...Front working machine, 31...Boom, 32...Boom cylinder, 33...Arm, 34...Arm cylinder, 35...Bucket, 36...Bucket cylinder, 40...Control device, 41...Image sensor, 42...Distance sensor, 50...Terrain information acquisition device, 51...Driving support unit, 52...Calculation unit, 53...Storage device, 54...Object recognition unit, 55...Surrounding image acquisition device, 56...Driving support judgment unit, 57...Model learning unit, 58...Model storage unit, 59...Learning data storage unit

Claims

1. In a monitoring device for work machinery that monitors the area around work machinery having work implements, An environmental information acquisition device that acquires environmental information about the surroundings of the aforementioned work machine, A determination unit that uses a machine learning-based object recognition model to detect and determine objects present around the work machine based on the environmental information, Equipped with, The surrounding monitoring device for a work machine is characterized in that the determination unit uses different object recognition models for detection determination based on first environmental information which includes information relating to the work equipment of the work machine, and for detection determination based on second environmental information which does not include information relating to the work equipment of the work machine.

2. In the surrounding monitoring device for a work machine according to claim 1, The update frequency of the object recognition model used for detection and determination based on the first environmental information is: An ambient monitoring device for work machinery, characterized in that it has a higher frequency of model updates than the learning frequency of the object recognition model used for detection and determination based on the second environmental information.

3. In the surrounding monitoring device for a work machine according to claim 2, A surrounding monitoring device for a work machine, characterized in that the work machine performs training of an object recognition model used for detection and determination based on the first environmental information.

4. In the surrounding monitoring device for a work machine according to claim 1, The surrounding monitoring device for a work machine is characterized in that the determination unit selects and uses an object recognition model to be used for detection and determination based on terrain information including height information of the terrain around the work machine.

5. In the surrounding monitoring device for a work machine according to claim 1, The surrounding monitoring device for a work machine is characterized in that, when the determination unit performs a detection determination based on the first environmental information which includes information related to the work machine of the work machine, it selects an object recognition model based on terrain information which includes height information of the terrain around the work machine.

Images