Construction machine, information processing device, program

Abstract

Provided is a technology capable of improving safety in the vicinity of a construction machine. A construction machine (100) according to an embodiment of the present invention includes a control device (110) that acquires information indicating a shape of a work object or information indicating a shape of the work object and an object in the vicinity of the work object, predicts a change in the shape of the work object or the work object and the object in the vicinity of the work object corresponding to an action of the construction machine (100) based on the information, and notifies an operator or an outside or restricts the action of the construction machine (100) based on a result of the prediction of the change.

Description

Technical Field

[0001] This invention relates to construction machinery, etc. Background Technology

[0002] Previously, there were technologies that, if an object (e.g., a person) was detected in the monitoring area around the construction machinery, the safety of the area around the construction machinery was ensured by notifying the operator or the outside world, or by restricting the movement of the construction machinery (see Patent Document 1).

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2017-101419 Summary of the Invention

[0004] The problem that the invention aims to solve However, there is a desire to further improve the safety around construction machinery.

[0005] Therefore, in view of the above issues, the aim is to provide a technology that can improve the safety of the area surrounding construction machinery.

[0006] Methods for solving problems To achieve the above objectives, in one embodiment of the present invention, a construction machine is provided, the construction machine having a control device, the control device acquiring information representing the shape of a work object or information representing the shape of the work object and objects surrounding the work object, predicting changes in the shape of the work object or objects surrounding the work object corresponding to the action of the construction machine based on the information, and notifying the operator or external parties based on the predicted change, or restricting the action of the construction machine.

[0007] Furthermore, in another embodiment of the present invention, an information processing device is provided, which acquires information representing the shape of a work object of a construction machine or information representing the shape of the work object and objects surrounding the work object, predicts changes in the shape of the work object or objects surrounding the work object corresponding to the movement of the construction machine based on the information, and notifies the operator or the outside of the construction machine based on the predicted change, or restricts the movement of the construction machine.

[0008] Furthermore, in another embodiment of the present invention, a program is provided that causes an information processing device to perform the following processing: acquiring information representing the shape of a work object of a construction machine or information representing the shape of the work object and objects surrounding the work object; predicting, based on the information, changes in the shape of the work object or the work object and objects surrounding the work object corresponding to the movement of the construction machine; and notifying, based on the predicted change, the operator or the outside of the construction machine, or restricting the movement of the construction machine.

[0009] Invention Effects The above-described implementation method can improve the safety of the area surrounding construction machinery. Attached Figure Description

[0010] Figure 1 This is a diagram illustrating an example of an operational support system.

[0011] Figure 2 This is a diagram illustrating an example of a structure related to the remote operation of construction machinery.

[0012] Figure 3 This is a diagram illustrating an example of the hardware structure of an information processing device.

[0013] Figure 4 This is a functional block diagram illustrating the first example of the functional structure of an operational support system.

[0014] Figure 5 This is a diagram illustrating an example of how security levels are calculated.

[0015] Figure 6 This is a flowchart illustrating the first example of the operation support system's processing.

[0016] Figure 7 This is a diagram showing an example of the area being observed.

[0017] Figure 8 This is a functional block diagram illustrating the functional structure of the operation support system, as shown in the second example.

[0018] Figure 9 This is a flowchart illustrating the second example of the operation support system's processing.

[0019] Figure 10 This is a side view showing an example of an excavator.

[0020] Figure 11 This is a top view showing an example of an excavator.

[0021] Figure 12 This is the first example of a diagram showing the change in the shape of the sand around the excavator caused by the excavator's digging action.

[0022] Figure 13 This is the second example of a diagram showing the change in the shape of the sand around the excavator caused by the excavator's digging action.

[0023] Figure 14 This is the third example of a diagram showing the change in the shape of the sand around the excavator caused by the excavator's digging action.

[0024] Figure 15 This is the fourth example of a diagram showing the changes in the shape of the sand around the excavator caused by the excavator's digging action.

[0025] Figure 16 This is a diagram illustrating an example of a continuous unloader.

[0026] Figure 17 This is a diagram illustrating an example of a continuous unloader.

[0027] Figure 18 This is a diagram illustrating an example of a continuous unloader.

[0028] Figure 19 This is a diagram illustrating an example of a continuous unloader.

[0029] Figure 20 This is a diagram illustrating an example of the change in shape of the bulk cargo around the scraping section caused by the scraping action of a continuous unloader. Detailed Implementation

[0030] The embodiments will now be described with reference to the accompanying drawings.

[0031] [Overview of the Operation Support System] refer to Figure 1 , Figure 2 The overview of the operation support system SYS involved in this embodiment will be described.

[0032] Figure 1 This is a diagram illustrating an example of the SYS operating support system. Figure 2 This is a diagram illustrating an example of a structure related to the remote operation of construction machinery 100.

[0033] like Figure 1 As shown, the operation support system SYS includes construction machinery 100, information processing device 200 and sensor group 300.

[0034] The operation support system SYS uses the information processing device 200 to cooperate with the construction machinery 100 to provide support related to the operation of the construction machinery 100.

[0035] The construction machinery 100 included in the operation support system SYS can be one or more machines.

[0036] The construction machinery 100 drives the working device 125 through the actuator 120 and uses the working device 125 to perform predetermined actions, thereby performing predetermined operations to change the shape of the work object in the construction site.

[0037] Construction machinery 100, for example, is the SVL excavator described later (see reference). Figure 10 , Figure 11 In this case, the work object of the construction machinery 100 is, for example, sand in a predetermined area within the construction site where the excavator SVL is deployed. Furthermore, the predetermined actions of the construction machinery 100 are, for example, digging or unloading actions by the excavator SVL, and the predetermined tasks of the construction machinery 100 are, for example, digging operations by the excavator SVL or loading operations of loading sand onto trucks. Additionally, the construction machinery 100 can also be a continuous unloader ULD (see reference below). Figures 16-19 In this case, the object of operation of the construction machinery 100 is, for example, bulk cargo M loaded inside the hold HD of a ship SP moored at the construction site (port area). Furthermore, in this case, the predetermined action of the construction machinery 100 is the scraping action of the continuous unloader ULD, and the predetermined operation of the construction machinery 100 is the unloading operation to transport the bulk cargo M to the shore.

[0038] The construction machinery 100 is equipped with a communication device 180, which can communicate with the information processing device 200 through a predetermined communication line NW.

[0039] The communication line NW may include, for example, a local area network (LAN) encompassing a predetermined area of ​​the construction site. Furthermore, the communication line NW may also include a wide area network (WAN). Wide area networks include, for example, mobile communication networks with base stations as endpoints, satellite communication networks utilizing communication satellites, and the Internet. Additionally, the communication line NW may also include, for example, short-range communication lines based on wireless communication standards such as WiFi or Bluetooth (registered trademark).

[0040] For example, the construction machinery 100 performs a predetermined action by manipulating its driven components according to the operation of the operator riding on it. The driven components of the construction machinery 100 are driven by actuators 120 and are movable parts of the construction machinery 100. Examples of the driven components of the construction machinery 100 include, for example, the lower walking body 1 (tracks 1CL, 1CR), upper slewing body 3, boom 4, stick 5, and bucket 6 of an excavator SVL (described later). Furthermore, examples of the driven components of the construction machinery 100 include, for example, the walking body 52, slewing body 55, and bucket elevator 59 (scraper 61) of a continuous unloader ULD (described later).

[0041] Furthermore, the construction machinery 100 can also be configured to be remotely operated from outside the construction machinery 100 (also known as "remote operation") instead of being operated by an operator riding on it, or, in addition to being configured to be operated by an operator riding on it, it can also be remotely operated from outside the construction machinery 100. In the case of remote operation of the construction machinery 100, the operator's cab of the construction machinery 100 (e.g., the cab 10 of an excavator SVL or the operator's cab 66 of a continuous unloader ULD) can be unmanned. Furthermore, if the construction machinery 100 is dedicated to remote operation, the operator's cab can be omitted. Hereinafter, the description will be based on at least one of the operator's operation including the operation of the operating device 130 by an operator in the operator's cab and remote operation by an external operator.

[0042] For example, such as Figure 2 As shown, remote operation includes operating the construction machinery 100 by inputting operation inputs to the actuator 120 of the construction machinery 100 through the remote operation support device 400. The remote operation support device 400 can communicate with the construction machinery 100 via a communication line NW. The remote operation support device 400 can be separate from the information processing device 200, or it can be the same as the information processing device 200.

[0043] The remote operation support device 400 may be installed, for example, in a management center that manages the operation of the construction machinery 100 from the outside. Furthermore, the remote operation support device 400 may also be a portable operating terminal, in which case the operator can remotely operate the construction machinery 100 while directly monitoring its operating status from its vicinity.

[0044] For example, the construction machinery 100 transmits an image (hereinafter referred to as "remote operation image") to the remote operation support device 400 via a communication device 180, which is used by the remote operation support device 400 to confirm the operating status of the construction machinery 100, including the area surrounding the work device 125. The remote operation image can be a captured image from a camera included in the peripheral monitoring sensor 140 (described later), or a processed image generated by processing the captured image. Furthermore, if the remote operation image is a processed image, the construction machinery 100 can transmit the captured image output by the camera to the remote operation support device 400 via the communication device 180, and the remote operation support device 400 can generate the remote operation image by processing the captured image received from the construction machinery 100. Thus, the remote operation support device 400 can display the remote operation image on its own display device (hereinafter referred to as "remote operation display device"). Therefore, the operator using the remote operation support device 400 can remotely operate the construction machinery 100 while simultaneously confirming the remote operation image displayed on the remote operation display device. Furthermore, the same information images (hereinafter referred to as "remote operation information images") displayed on the output device 160 (display device 162) of the operator's seat of the construction machinery 100 can also be displayed on the remote operation display device. Thus, the operator using the remote operation support device 400 can remotely operate the construction machinery 100 while simultaneously monitoring its various states by checking the content of the remote operation information images displayed on the remote operation display device. Then, the construction machinery 100 can activate the actuator 120 to drive the driven components based on the remote operation signal received from the remote operation support device 400 by the communication device 180, indicating the content of the remote operation.

[0045] Furthermore, remote operation may also include operating the construction machinery 100 via external voice or gesture input from people in the vicinity (e.g., workers). Specifically, the construction machinery 100 recognizes voice or gestures from nearby workers using a voice input device (e.g., a microphone) or gesture input device (e.g., a camera). Then, the construction machinery 100 can activate the actuator 120 to drive driven components based on the recognized voice or gestures.

[0046] Furthermore, the construction machinery 100 can also automatically operate the actuators without relying on the operator's actions. Thus, the construction machinery 100 can realize the function of automatically operating at least a part of the driven components, including the working device 125, which is the so-called "automatic operation function" or "machine control (MC) function".

[0047] Automatic operation functions may include, for example, semi-automatic operation functions (operation support type MC functions). A semi-automatic operation function is a function that automatically activates driven components other than the driven components of the workpiece based on operator input. In other words, a semi-automatic operation function is a function that automatically activates actuators 120 other than the actuator 120 of the workpiece based on operator input. Hereinafter, the operation of the driven components of the construction machinery 100 has the same meaning as the operation of the actuator 120 that drives those driven components. Furthermore, automatic operation functions may also include fully automatic operation functions (fully automatic type MC functions). A fully automatic operation function is a function that automatically activates at least a portion of multiple driven components without operator input. In the construction machinery 100, when the fully automatic operation function is active, the operator's seat of the construction machinery 100 can be unmanned. Furthermore, if the construction machinery 100 is dedicated to fully automatic operation, the operator's seat can be omitted. Moreover, semi-automatic or fully automatic operation functions may include, for example, rule-based automatic operation functions. Rule-based automatic operation is an automatic operation function that automatically determines the action content of the driven components of the automatic operation object according to predefined rules. Furthermore, semi-automatic or fully automatic operation functions may also include autonomous operation functions. Autonomous operation functions are automatic operation functions in which the construction machinery 100 autonomously makes various judgments and determines the action content of the driven components of the automatic operation object based on the judgment results.

[0048] Furthermore, the operation of the construction machinery 100 can be remotely monitored. In this case, a remote monitoring support device with the same functions as the remote operation support device 400 can be installed. The remote monitoring support device is, for example, an information processing device 200. Thus, the monitor, as the user of the remote monitoring support device, can monitor the operating status of the construction machinery 100 while viewing the surrounding image displayed on the display device of the remote monitoring support device. Furthermore, for example, if the monitor determines from a safety point of view that it is necessary, he / she can use the input device of the remote monitoring support device to make a predetermined input, thereby intervening in the operation performed by the operator of the construction machinery 100 or in automatic operation to bring the construction machinery 100 to an emergency stop.

[0049] The information processing device 200 communicates with the construction machinery 100 and cooperates with it to provide support related to the operation of the construction machinery 100.

[0050] The information processing device 200 may be a server or management terminal device, such as one installed in a management office at the construction site where the construction machinery 100 is located, or in a management center located at a different location from the construction site where the construction machinery 100 is located, to manage the operation of the construction machinery 100. The server device may be a local server, a cloud server, or an edge server. The management terminal device may be a fixed terminal device such as a desktop PC (Personal Computer), or a portable terminal device (i.e., a mobile terminal) such as a tablet, smartphone, or laptop. In the latter case, construction site workers, supervisors, and managers may carry the portable information processing device 200 while moving around the construction site. Furthermore, in the latter case, the operator may, for example, bring the portable information processing device 200 into the driver's seat of the construction machinery 100.

[0051] The information processing device 200 acquires data indicating the operating status from the construction machinery 100, for example. Thus, the information processing device 200 can grasp the operating status of the construction machinery 100 and monitor for any abnormalities. Furthermore, the information processing device 200 can, for example, display the data indicating the operating status of the construction machinery 100 via the display device 208 (described later) and allow the user to confirm it. Additionally, the information processing device 200 can, for example, enable a learning model to learn the operating status of the construction machinery 100 and generate a completed learning model to support the operation of the construction machinery 100.

[0052] Furthermore, the information processing device 200 can also send various data, such as programs or reference data used in the processing of the control device 110, to the construction machinery 100. As a result, the construction machinery 100 can use the various data downloaded from the information processing device 200 to perform various processes related to the operation of the construction machinery 100.

[0053] Sensor group 300 is installed at the construction site of construction machinery 100.

[0054] For example, when the operation support system SYS includes multiple construction machines 100, a sensor group 300 is provided for each construction machine 100. Furthermore, when the multiple construction machines 100 included in the operation support system SYS are operating at the same construction site, a single sensor group 300 can be shared by the multiple construction machines 100.

[0055] Sensor group 300 includes sensors 300-1 to 300-M (M: an integer greater than 2). Hereinafter, sensors 300-1 to 300-M are sometimes collectively referred to as sensor 300-X.

[0056] Sensors 300-1 to 300-M measure the state of objects at the construction site of construction machinery 100 and acquire measurement data representing that state. The objects at the construction site include, in addition to the work objects of construction machinery 100 (e.g., sand in a predetermined area of ​​the construction site of excavator SVL or bulk cargo M inside the hold HD of ship SP), personnel and other personnel present in or around the work objects. Furthermore, the objects at the construction site may also include other construction machinery such as excavators and bulldozers, or work vehicles such as sand transport trucks, in or around the work objects. The state of the objects includes their shape or characteristics.

[0057] Sensors 300-1 to 300-M include, for example, range sensors (also called "distance sensors"). Range sensors include, for example, LIDAR (Light Detecting and Ranging), millimeter-wave radar, ultrasonic sensors, and infrared sensors. Furthermore, sensors 300-1 to 300-M may include, for example, stereo cameras, TOF (Time of Flight) cameras, and other 3D cameras capable of acquiring distance (depth) data in addition to two-dimensional images. Moreover, range sensors and 3D cameras can be combined in sensors 300-1 to 300-M. Thus, sensor group 300 can acquire measurement data representing the shape of objects at the construction site surrounding construction machinery 100. Hereinafter, for convenience, sensors capable of acquiring measurement data representing the shape of objects, such as range sensors or 3D cameras, are sometimes referred to as "shape sensors."

[0058] Furthermore, sensors 300-1 to 300-M may include multi-wavelength beam splitters. Multi-wavelength beam splitters may include, for example, multispectral cameras or hyperspectral cameras. Thus, for example, sensor group 300 can acquire measurement data representing the characteristics of objects at the construction site, such as the hardness or moisture content of sand. Hereinafter, for convenience, sensors capable of acquiring measurement data representing the characteristics of objects, such as multi-wavelength beam splitters, are sometimes referred to as "characteristic sensors."

[0059] For example, sensors 300-1 to 300-M include multiple shape sensors. Furthermore, these multiple shape sensors can be positioned at different locations around the construction site surrounding the construction machinery 100, and their respective sensing ranges overlap with the sensing range of at least one other shape sensor. Thus, for example, even if occlusion prevents the acquisition of measurement data representing the shape of a portion of the object within the sensing range from being obtained from the measurement data of one shape sensor, measurement data representing the shape of the object within that range can still be obtained from other shape sensors. Therefore, sensor group 300 can more reliably acquire measurement data representing the shape of objects at the construction site of the construction machinery 100.

[0060] Furthermore, sensors 300-1 to 300-M may also include multiple characteristic sensors. Moreover, these multiple characteristic sensors can be positioned at different locations around the construction site surrounding the construction machinery 100, and their respective sensing ranges can overlap with at least one other characteristic sensor. Thus, for example, even if occlusion prevents the acquisition of measurement data representing the characteristics of a portion of the object within the sensing range from being obtained from the measurement data of one characteristic sensor, measurement data representing the characteristics of the object within that range can still be obtained from other shape sensors. Therefore, sensor group 300 can more reliably acquire measurement data representing the characteristics of objects at the construction site of the construction machinery 100.

[0061] Furthermore, sensors 300-1 to 300-M may also include sensors that have the functions of both shape sensors and characteristic sensors (hereinafter referred to as "integrated sensors"). In this case, sensors 300-1 to 300-M may include multiple integrated sensors. Moreover, multiple characteristic sensors may be set at different locations on the construction site surrounding the construction machinery 100, and each of them may be configured such that its sensing range overlaps with at least one other characteristic sensor.

[0062] Sensors 300-1 to 300-M can be fixed to the construction site or mounted on a mobile device capable of moving within the construction site using the construction machinery 100. The mobile device includes, for example, construction machinery or work vehicles that move within the construction site. Furthermore, the mobile device capable of moving within the construction site can include, for example, a drone or other flying object that flies above the work objects at the construction site.

[0063] The outputs (measurement data) of sensors 300-1 to 300-M are input to the information processing device 200 via communication line NW. The outputs of sensors 300-1 to 300-M may be directly input to the information processing device 200 via communication line NW, for example. Alternatively, the outputs of sensors 300-1 to 300-M may be temporarily input to construction machinery 100 via communication line NW, and then input to the information processing device 200 via construction machinery 100. Furthermore, when sensors 300-1 to 300-M are mounted on a predetermined device such as the aforementioned mobile body, the outputs of sensors 300-1 to 300-M may be temporarily input into the predetermined device and then input to the information processing device 200 from that device.

[0064] Alternatively, sensor group 300 can simply include only a shape sensor or a characteristic sensor. Furthermore, the operation support system SYS can simply replace sensor group 300 with only a sensor capable of acquiring measurement data indicating the state of objects at the construction site surrounding the construction machinery 100. In this case, sensor group 300 can be omitted altogether. The measurement data indicating the state of objects at the construction site can be acquired by the peripheral monitoring sensor 140 mounted on the construction machinery 100 and sent to the information processing device 200.

[0065] [Hardware structure of the operation support system] Next, besides reference Figure 1 , Figure 2 In addition, also refer to Figure 3 The hardware structure of the SYS operating support system will be described.

[0066] Furthermore, the hardware structure of the remote operation support device 400 can be the same as that of the information processing device 200. Therefore, illustrations and descriptions related to the hardware structure of the remote operation support device 400 are omitted.

[0067] <Hardware Structure of Construction Machinery> like Figure 1 As shown, the construction machinery 100 includes a control device 110, an actuator 120, a working device 125, an operating device 130, a peripheral monitoring sensor 140, a motion monitoring sensor 150, an output device 160, an input device 170, and a communication device 180.

[0068] The control device 110 performs controls related to the construction machinery 100.

[0069] The functions of the control device 110 are implemented through any hardware or any combination of hardware and software. The control device 110 includes, for example, an auxiliary storage device 110A, a memory device 110B, a CPU (Central Processing Unit) 110C, and an interface device 110D connected via a bus BS1.

[0070] Auxiliary storage device 110A is a non-volatile storage unit that stores the program to be installed and the necessary files or data. Auxiliary storage device 110A may be, for example, EEPROM (Electrically Erasable Programmable Read-Only Memory) or flash memory.

[0071] For example, when a program start instruction is present, memory device 110B loads the program from auxiliary storage device 110A so that CPU 110C can read the program. Memory device 110B is, for example, SRAM (Static Random Access Memory).

[0072] CPU 110C, for example, executes a program loaded into memory device 110B and implements various functions of control device 110 according to the program's commands.

[0073] The interface device 110D functions, for example, as a communication interface for connecting to communication lines inside the construction machinery 100. The interface device 110D may also include multiple different types of communication interfaces depending on the type of communication line to be connected.

[0074] Furthermore, the interface device 110D functions as an external interface for reading data from or writing data to the storage medium. The storage medium can be, for example, a special tool connected to a connector located inside the cab 10 via a detachable cable. The storage medium can also be a common storage medium such as an SD memory card or a USB (Universal Serial Bus) memory. Thus, programs implementing various functions of the control device 110 can be provided, for example, via a portable storage medium and installed in the auxiliary storage device 110A of the control device 110. Furthermore, the programs can also be downloaded from another computer (e.g., information processing device 200) outside the construction machinery 100 via the communication device 180 and installed in the auxiliary storage device 110A.

[0075] Furthermore, some of the functions of the control device 110 can also be implemented by other control devices. That is, the functions of the control device 110 can also be implemented in a distributed manner by multiple control devices mounted on the construction machinery 100.

[0076] Actuator 120 drives the driven component of construction machinery 100. Actuator 120 can be a hydraulic actuator that drives the driven component by hydraulic drive, an electric actuator that drives the driven component by electric drive, or it can drive the driven component by other drive methods.

[0077] The working device 125 acts directly on the work object to perform a predetermined operation. The working device 125 includes one or more driven components and is driven by an actuator 120. The working device 125 is, for example, the attachment AT of an excavator SLV (described later) or the bucket elevator 59 of a continuous ship unloader ULD (described later). Furthermore, the working device 125 includes a working section at the front end of the working device 125, corresponding to the application of changes to the work object. The working section is, for example, the bucket 6 of an excavator SVL (described later) or the scraper 61 of a continuous ship unloader ULD (described later).

[0078] An operating device 130 is disposed in the driver's seat of the construction machinery 100, for the operator sitting in the driver's seat to operate various driven components of the construction machinery 100. Specifically, the operating device 130 is used by the operator to operate the actuators 120 that drive each driven component, thereby enabling the operator to operate the driven components that are driven by the actuators 120. The operating device 130 may include, for example, a lever device operated by the operator's hand or a pedal device operated by the operator's foot.

[0079] The operating device 130 is, for example, electrically operated. In this case, the operating device 130 outputs an electrical signal (hereinafter referred to as the "operation signal") corresponding to the operation content, and the operation signal is input to the control device 110. Then, the control device 110 controls the actuator 120 to perform an action corresponding to the content of the operation signal. For example, if the actuator 120 is a hydraulic actuator such as a hydraulic cylinder or hydraulic motor, the control device 110 outputs a control signal corresponding to the content of the operation signal to a predetermined hydraulic control valve (hereinafter referred to as the "operation control valve") (e.g., a proportional valve). As a result, a pilot pressure corresponding to the operation content of the operating device 130 is input from the operation control valve to the hydraulic control device (e.g., a control valve). Therefore, the hydraulic control device can drive the actuator 120 according to the content of the operation signal, and as a result, the actuator 120 can perform an action corresponding to the operation content of the operating device 130. Furthermore, for example, if the actuator 120 is an electric actuator such as an electric motor, the control device 110 outputs a control signal corresponding to the content of the operation signal to the drive device (e.g., a motor driver) that drives the actuator 120. Thus, the drive device can drive the actuator 120 according to the content of the operation signal, and as a result, the actuator 120 can perform an action corresponding to the operation content of the operation device 130. Furthermore, if the actuator 120 is an electric actuator, the operation signal can also be directly input to the drive device.

[0080] Furthermore, the operating device 130 can also be hydraulically piloted. In this case, the operating device 130 uses a predetermined hydraulic source (e.g., a pilot pump) to output a pilot pressure (hereinafter referred to as "operating pilot pressure") corresponding to the operation content. For example, when the actuator 120 is a hydraulic actuator, the operating pilot pressure is input to a hydraulic control device. Thus, the hydraulic control device can drive the actuator 120 according to the operating pilot pressure, resulting in the actuator 120 performing an action corresponding to the operation content of the operating device 130. Furthermore, for example, when the actuator 120 is an electric actuator, the detected value of the operating pilot pressure is input to a control device 110, and the control device 110 outputs a control signal corresponding to the detected value of the operating pilot pressure to a drive device. Thus, the drive device can drive the actuator 120 according to the operating pilot pressure, resulting in the actuator 120 performing an action corresponding to the operation content of the operating device 130.

[0081] The peripheral monitoring sensor 140 acquires measurement data representing the state of the work object of the construction machinery 100 or objects in its vicinity.

[0082] The peripheral monitoring sensor 140 may include, for example, a shape sensor capable of acquiring measurement data representing the shape of the work object or objects in its vicinity. The shape sensor may include, for example, a camera device (e.g., a monocular camera) capable of acquiring two-dimensional image data, a range sensor, a 3D camera, etc. The output of the peripheral monitoring sensor 140 is input to the control device 110. Thus, the control device 110 can monitor the state of the work object of the construction machinery 100 or objects in its vicinity. Furthermore, the peripheral monitoring sensor 140 may also include a characteristic sensor capable of acquiring measurement data representing the characteristics of the work object or objects in its vicinity.

[0083] Motion monitoring sensor 150 acquires measurement data representing the operational state of construction machinery 100. For example, motion monitoring sensor 150 acquires measurement data representing the operational state of working device 125. The output of motion monitoring sensor 150 is input to control device 110. Thus, control device 110 can monitor the operational state of construction machinery 100. Therefore, control device 110 can, for example, perform control related to automatic operation functions while monitoring the operational state of construction machinery 100.

[0084] The output device 160 outputs various information to users of the construction machinery 100 (e.g., the operator in the driver's seat) or people around the construction machinery 100 (e.g., workers or drivers of work vehicles).

[0085] For example, output device 160 includes lighting equipment or display device 162 that outputs various information visually.

[0086] Lighting equipment may include, for example, warning lights (indicator lights). Display device 162 may include, for example, a liquid crystal display (LCD) or an organic EL (Electroluminescence) display. For example, the lighting equipment or display device 162 may be installed inside the driver's seat and output various information to the operator inside the driver's seat in a visual manner.

[0087] Furthermore, the lighting equipment or display device 162 may be configured to be exposed outside the construction machinery 100, and to output various information to workers and others around the construction machinery 100 in a visual manner.

[0088] Furthermore, the display device 162 may include, for example, a portable display device that can be carried by a user such as an operator. The portable display device is provided to the operator or other user by being brought into the cab 10 of the excavator SVL. The portable display device may include, for example, a wearable display device capable of displaying image information on the transmissive surface of glasses or goggles that the user can wear. The wearable display device can overlay image information onto the user's field of vision while allowing the user to visually perceive the surrounding situation from the transmissive surface. Data related to the position and posture of the wearable display device is input to the control device 110, for example, via a short-range communication line based on wireless communication standards such as Bluetooth or WiFi. Thus, the control device 110 can determine the viewpoint or direction of the user (operator) wearing the wearable display device and display content matching that field of vision on the transmissive surface. The wearable display device may be, for example, smart glasses or AR (Augmented Reality) goggles. Furthermore, the portable display device may include, for example, a display device mounted on a mobile terminal carried by the user.

[0089] Furthermore, the output device 160 may also include a sound output device 164 that outputs various information in an auditory manner. The sound output device 164 may include, for example, a buzzer or a loudspeaker. The sound output device 164 may be installed in at least one of the interior and exterior of the driver's seat, and output various information audibly to the operator inside the driver's seat or people (workers, etc.) around the construction machinery 100.

[0090] Furthermore, the output device 160 may also include a device that outputs various information in a tactile manner, such as by vibration of the operator's seat.

[0091] The input device 170 accepts various inputs from the user of the construction machinery 100, and the signal corresponding to the accepted input is input to the control device 110. For example, the input device 170 is installed inside the driver's seat and accepts inputs from the operator inside the driver's seat. Alternatively, the input device 170 may be installed outside the construction machinery 100 and accept inputs from workers around the construction machinery 100.

[0092] For example, input device 170 includes an operation input device that accepts mechanically operated input from a user. The operation input device may include a touch panel mounted on display device 162, a touchpad disposed around display device 162, a push-button switch, a lever, a toggle switch, a rotary switch disposed on operation device 130 (lever device), etc.

[0093] Furthermore, the input device 170 may also include a voice input device that accepts voice input from the user. The voice input device may include, for example, a microphone.

[0094] Furthermore, the input device 170 may also include a gesture input device that accepts gesture input from the user. The gesture input device may include, for example, a camera device that captures the state of the gestures performed by the user.

[0095] Furthermore, the input device 170 may also include a biometric input device for accepting biometric input from a user. Biometric input may include, for example, the input of biometric information such as the user's fingerprint or iris scan.

[0096] The communication device 180 is connected to an external communication line NW and communicates with a device separately installed from the construction machinery 100. The device separately installed from the construction machinery 100 may include, in addition to a device located outside the construction machinery 100, a portable terminal device (mobile terminal) brought into the driver's seat by the user of the construction machinery 100. The communication device 180 may, for example, include devices compliant with 4G (4G... th Generation: Fourth Generation Mobile Communication) or 5G (5G) th Generation: 5G mobile communication) and other standard mobile communication modules. Furthermore, the communication device 180 may include, for example, a satellite communication module. The communication device 180 may also include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module. Furthermore, in the presence of multiple connectable communication lines NW, the communication device 180 may include multiple communication devices depending on the type of communication line NW.

[0097] For example, communication device 180 communicates with external devices such as information processing device 200 or remote operation support device 400 within the construction site via local communication lines constructed at the construction site. These local communication lines may be, for example, mobile communication lines based on local 5G (so-called local 5G) constructed at the construction site or local area networks based on WiFi 6.

[0098] Furthermore, the communication device 180 can also communicate with the information processing device 200 or the remote operation support device 400 located outside the construction site via a wide area network, including the wide area communication line at the construction site.

[0099] <Hardware Structure of Information Processing Device> Figure 3 This is a block diagram illustrating an example of the hardware structure of the information processing device 200.

[0100] The functions of the information processing device 200 are implemented through any hardware or any combination of hardware and software. For example, such as Figure 5As shown, the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed computing device 205, a communication interface 206, an input device 207, a display device 208, and a sound output device 209. They are connected via a bus BS2.

[0101] External interface 201 functions as an interface for reading data from or writing data to storage medium 201A. Storage medium 201A may include, for example, floppy disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), BDs (Blu-ray Discs), SD cards, and USB storage devices. Thus, information processing device 200 can read various types of data used in processing through storage medium 201A and store them in auxiliary storage device 202, or install programs that implement various functions.

[0102] In addition, the information processing device 200 can also obtain various data or programs used in the processing from external devices via the communication interface 206.

[0103] Auxiliary storage device 202 stores various installed programs and files or data required for various processes. Auxiliary storage device 202 may include, for example, HDD (Hard Disc Drive), SSD (Solid State Disk), flash memory, etc.

[0104] When a program start instruction is present, memory device 203 reads the program from auxiliary storage device 202 and stores it. Memory device 203 may include, for example, DRAM (Dynamic Random Access Memory) or SRAM.

[0105] CPU 204 executes various programs loaded from auxiliary storage device 202 into memory device 203, and performs various functions related to information processing device 200 according to the programs.

[0106] The high-speed computing device 205 works in conjunction with the CPU 204 to perform computational processing at a relatively high speed. The high-speed computing device 205 may include, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or a FPGA (Field-Programmable Gate Array).

[0107] Alternatively, depending on the required processing speed, the high-speed computing unit 205 may be omitted.

[0108] The communication interface 206 serves as an interface for communication connection with external devices. Thus, the information processing device 200 can communicate with external devices, such as construction machinery 100, via the communication interface 206. Furthermore, the communication interface 206 can have multiple types of communication interfaces depending on the communication method with the device to be connected.

[0109] The input device 207 accepts various inputs from the user. The input device 207 includes a remote operation device for remotely operating the construction machinery 100.

[0110] Input device 207 may include, for example, an input device that accepts mechanical operation input from a user (hereinafter referred to as "operation input device"). Remote operation devices may be operation input devices. Operation input devices may include, for example, buttons, toggle switches, levers, keyboards, mice, touch panels mounted on display device 208, touchpads disposed separately from display device 208, etc.

[0111] Furthermore, the input device 207 may also include a voice input device capable of accepting voice input from a user. The voice input device may include, for example, a microphone capable of collecting the user's voice.

[0112] Furthermore, the input device 207 may also include a gesture input device capable of accepting gesture input from the user. The gesture input device may include, for example, a camera capable of capturing the state of the user's gestures.

[0113] Furthermore, the input device 207 may also include a biometric input device capable of accepting biometric input from a user. For example, a biometric input device may include a camera capable of acquiring image data containing information representing a user's fingerprint or iris.

[0114] Display device 208 displays information screens or operation screens to the user of information processing device 200. Display device 208 is, for example, a liquid crystal display or an organic EL display.

[0115] The sound output device 209 uses sound to transmit various information to the user of the information processing device 200. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, etc.

[0116] [Example of the functional structure of an operational support system] Next, besides reference Figures 1-3 In addition, also refer to Figure 4 , Figure 5 The first example of the functional structure of the SYS (System for Operation Support) will be explained.

[0117] Figure 4 This is a functional block diagram illustrating the first example of the functional structure of the SYS (System for Operation Support). Figure 5 This is a diagram illustrating an example of how security levels are calculated.

[0118] Hereinafter, "track of the working part of construction machinery 100" will be used to mean both the path (i.e., trajectory) that the working part of construction machinery 100 has already moved and the path that it may move in the future.

[0119] In this example, the Operation Support System SYS provides support to the user operating the construction machinery 100 and performing the work.

[0120] like Figure 4 As shown, the control device 110 of the construction machinery 100 includes an action log providing unit 1101 and an operation support unit 1102 as functional units.

[0121] Furthermore, when the operation support system SYS includes multiple construction machines 100, there may be construction machines 100 whose control device 110 only includes the former of the action log providing unit 1101 and the operation support unit 1102, and construction machines 100 that only include the latter. In this case, the former construction machine 100 only has the function of acquiring the action log of the construction machine 100 and providing it to the information processing device 200, and this function is used for the operation support function of the latter construction machine 100. The same applies to the second example of the operation support system SYS described later.

[0122] The information processing device 200 includes a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a training data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007 as functional units.

[0123] The action log providing unit 1101 is a functional unit for acquiring the action log of the construction machinery 100 when it performs a predetermined action and providing it to the information processing device 200.

[0124] The predetermined actions of the construction machinery 100 include, for example, digging actions, boom lifting and slewing actions, boom lowering and slewing actions, soil discharge actions, and sweeping actions used during excavation operations of the excavator SVL. Furthermore, the predetermined actions of the construction machinery 100 may also include digging actions, soil discharge actions, sweeping actions, horizontal traction actions, compaction actions, and sweeping actions used during land preparation operations of the excavator SVL. Furthermore, the predetermined actions of the construction machinery 100 may also include cutting actions and compaction actions used during slope operations of the excavator SVL. Additionally, the predetermined actions of the construction machinery 100 may include, for example, the scraping action of bulk cargo M used during unloading operations of the continuous unloader ULD.

[0125] Additionally, the sweeping action is, for example, an action in which the bucket 6 is pushed forward along the ground surface by operating the attachment AT, and sand is swept forward by the back of the bucket 6. During the sweeping action, for example, the attachment AT lowers the boom 4 and opens the stick 5. The horizontal traction action is, for example, an action in which the bucket teeth of the bucket 6 are moved forward in a roughly horizontal manner along the ground by operating the attachment AT, leveling the unevenness of the ground (terrain surface). During the horizontal traction action, for example, the attachment AT raises the boom 4 and closes the stick 5. The compaction action is, for example, an action in which the back of the bucket 6 is pressed against the ground by operating the attachment AT. Furthermore, the compaction action can also be an action in which the bucket 6 is moved up and down while the back of the bucket 6 strikes the ground to compact it. Furthermore, the compaction action can also be an action in which the bucket 6 is pushed forward along the ground surface, and sand is swept to a predetermined position in front by the back of the bucket 6, and then the back of the bucket 6 is pressed against the ground at that predetermined position. During the compaction operation, for example, the attachment AT lowers the boom 4 while pressing the ground. The sweeping operation, for example, involves the upper slewing body 3 moving to rotate the bucket 6 left and right along the ground. Furthermore, the sweeping operation can also involve the attachment AT and the upper slewing body 3 moving to alternately rotate the bucket 6 left and right along the ground while simultaneously pushing the bucket 6 forward. During the sweeping operation, for example, the upper slewing body 3 alternately rotates left and right repeatedly. Furthermore, during the sweeping operation, in addition to alternating left and right rotation of the upper slewing body 3, the attachment AT, similar to the sweeping operation, can also lower the boom 4 and open the stick 5.

[0126] The operation log of the construction machinery 100 is timing data representing the operational status of the construction machinery 100. For example, the operation log of the construction machinery 100 includes timing data representing the operator's actions. The timing data representing the operator's actions may be, for example, timing measurement data of the pilot pressure output from the hydraulic pilot-operated device 130 or timing output data of the electric pilot-operated device 130 (i.e., timing data of the operation signal). Furthermore, the operation log of the construction machinery 100 may also be timing output data of the motion monitoring sensor 150.

[0127] For example, the action log providing unit 1101 acquires the action logs of construction machinery 100 operated by an operator with a long and relatively experienced driving history (hereinafter, for convenience, referred to as a "skilled operator"), and provides them to the information processing device 200. Thus, as described later, a learned model (e.g., the learned model LM3 described later) capable of reproducing the actions of construction machinery 100 operated by a skilled operator can be generated through machine learning based on the action logs of construction machinery 100. Furthermore, the action logs of construction machinery 100 provided to the information processing device 200 may include action logs of operators other than skilled operators operating construction machinery 100. Also, the action logs of construction machinery 100 can be acquired separately depending on the type of learned model generated by the information processing device 200.

[0128] The action log providing unit 1101 includes an action log recording unit 1101A, an action log storage unit 1101B, and an action log sending unit 1101C.

[0129] The action log recording unit 1101A acquires the action log of the construction machinery 100 when it performs a predetermined action and records it in the action log storage unit 1101B. For example, the action log recording unit 1101A records the action log of the construction machinery 100 in the action log storage unit 1101B each time the construction machinery 100 performs a predetermined action.

[0130] The operation log of the construction machinery 100 is stored in the operation log storage unit 1101B. For example, for each predetermined action performed by the construction machinery 100, the operation log is associated with the time (e.g., year, month, day) of the predetermined action and stored in the operation log storage unit 1101B. The time of execution of the predetermined action includes data for the start and end of the predetermined action of the construction machinery 100. Furthermore, when multiple predetermined actions are specified, for each predetermined action performed by the construction machinery 100, the operation log, the time of execution of the predetermined action, and the identification information of the predetermined action performed are associated and stored in the operation log storage unit 1101B. Hereinafter, for convenience, the data associated with the operation log of the construction machinery 100 is sometimes referred to as "ancillary data". For example, in the operation log storage unit 1101B, for each predetermined action performed by the construction machinery 100, record data representing the correspondence between the operation log and the ancillary data is accumulated, thereby constructing a database of the operation log of the construction machinery 100 when performing the predetermined action.

[0131] Alternatively, the action logs that have been sent to the information processing device 200 by the action log sending unit 1101C (described later) can be deleted afterward.

[0132] The action log sending unit 1101C transmits the action logs stored in the action log storage unit 1101B, which record the actions performed by the construction machinery 100 during predetermined actions, along with associated data, to the information processing unit 200 via the communication device 180. Furthermore, the action log sending unit 1101C may also transmit recorded data showing the correspondence between the action logs and associated data of the construction machinery 100 for each predetermined action performed by the construction machinery 100, to the information processing unit 200.

[0133] For example, the operation log sending unit 1101C, upon receiving a request from the information processing device 200 to send the operation log of the construction machinery 100 that has not yet been sent, along with related data, stored in the operation log storage unit 1101B, to the information processing device 200. Furthermore, the operation log sending unit 1101C can also automatically send the operation log of the construction machinery 100 that has not yet been sent, stored in the operation log storage unit 1101B, along with related data, to the information processing device 200 at a predetermined time. The predetermined time could be, for example, when the construction machinery 100 stops operating (key switch is in the "off" position) or starts operating (key switch is in the "on" position).

[0134] The log acquisition unit 2001 acquires the logs of construction machinery 100 when it performs its predetermined actions.

[0135] The log for the construction machinery 100 when performing a predetermined action includes an action log for the construction machinery 100 performing the predetermined action and a status log for the work object. The status log for the work object includes data indicating the status of the work object before and after the construction machinery 100 performs the predetermined action. The status of the work object includes its shape or characteristics. The action log for the construction machinery 100 performing the predetermined action is uploaded from the construction machinery 100 based on the output of the action monitoring sensor 150. The status log for the work object when the construction machinery 100 performs the predetermined action is obtained based on measurement data uploaded from the sensor group 300 and supplementary data (data on the time of performing the predetermined action) uploaded from the construction machinery 100. Furthermore, the status log for the work object when the construction machinery 100 performs the predetermined action can also be uploaded from the construction machinery 100 based on the output of the surrounding monitoring sensor 140.

[0136] The simulator 2002 uses a virtual model of the construction machinery 100 and the work object to perform computer simulations related to the predetermined actions of the construction machinery 100.

[0137] For example, the Discrete Element Method (DEM) is used to model the work object, sand or bulk material M, as a collection of tiny particles. Thus, the simulator unit 2002 can virtually reproduce the overall behavior of the work object, sand, as a collection, or the reaction forces from the sand by having the virtual model of the construction machinery 100 perform predetermined actions such as digging or scraping, and by analyzing the individual movements of the tiny particles.

[0138] The simulator unit 2002 acquires data representing the operational status of the construction machinery 100 and data representing the status of the work object before and after performing the predetermined action, and uses this data as a log when the construction machinery 100 performs the predetermined action through computer simulation. The former data is equivalent to the operational log when the construction machinery 100 performs the predetermined action through computer simulation, and the latter data is equivalent to the status log of the work object when the construction machinery 100 performs the predetermined action through computer simulation.

[0139] The simulator unit 2002 uses the states of various work objects (sand) and the tracks of the working parts of various construction machines 100 to perform computer simulations of multiple modes related to the predetermined actions of the construction machines 100. As a result, the simulator unit 2002 can accumulate logs of the construction machines 100 performing predetermined actions through computer simulation under different conditions in the log storage unit 2003.

[0140] The log storage unit 2003 stores, in an accumulation manner, the logs acquired by the log acquisition unit 2001 and the simulator unit 2002 during the execution of predetermined actions by the construction machinery 100. For example, the log storage unit 2003 stores the logs in a manner that associates them with the action logs, work object status logs, and accompanying data of each predetermined action actually performed by the construction machinery 100 or performed through computer simulation. In the log storage unit 2003, the logs acquired by the log acquisition unit 2001 and the logs acquired by the simulator unit 2002 can be stored in a identifiable manner or in a mixed manner that is not identifiable.

[0141] The training data generation unit 2004 generates training data for machine learning based on logs stored in the log storage unit 2003 of the construction machinery 100 performing predetermined actions, and outputs a collection of multiple training data sets, i.e., a training dataset. The training data generation unit 2004 can automatically generate training data through batch processing, or it can generate training data based on input from a user in the information processing device 200. The training data generation unit 2004 includes training data generation units 2004A and 2004B.

[0142] The training data generation unit 2004A generates the training dataset TRD1 for generating the fully trained model LM1.

[0143] The learned model LM1 uses predetermined data as input to infer the trajectory of the work area predicted by the operator's actions. The predetermined data input to the learned model LM1 includes, for example, data representing the current operating state of the construction machinery 100 performed by the operator. Therefore, the learned model LM1 can predict the trajectory of the work area immediately following the operator's actions based on the current operating state of the construction machinery 100. The data representing the operating state of the construction machinery 100 is, for example, data representing the operating state of the construction machinery 100 itself, that is, data representing the operation input from the operator. Furthermore, the data representing the operating state of the construction machinery 100 can also be data reflecting the operating state of the construction machinery 100, representing the movement state of the construction machinery 100. Moreover, the predetermined data input to the learned model LM1 can also include historical data representing the current operating state of the construction machinery 100 performed by the operator, representing historical data of the operating state of the construction machinery 100. Therefore, the learned model LM1 can predict the trajectory of the work area immediately following the operator's actions with higher accuracy based on the most recent historical record of the operating state of the construction machinery 100 performed by the operator. Furthermore, the predetermined data input to the learned model LM1 may also include data representing the shape of the work object. Thus, the learned model LM1 can predict the trajectory of the subsequent work section to be performed by the operator, starting from the current shape of the work object. The predetermined data input to the learned model LM1 may include only either data representing the operating status of the construction machinery 100 operated by the operator or data representing the shape of the work object, or both.

[0144] The training dataset TRD1 contains a combination of the aforementioned predetermined data as input data and data representing the trajectory of the work area executed based on the input data as correct output data.

[0145] Furthermore, the training dataset TRD1 used to generate the learned model LM1 can be generated solely from the logs acquired by the log acquisition unit 2001 and the logs output from the simulator unit 2002. In this case, the simulator unit 2002 can be omitted. Similarly, the training dataset TRD1 used to generate the learned model LM1 can also be generated solely from the logs acquired by the log acquisition unit 2001 and the logs output from the simulator unit 2002. In this case, the motion log providing unit 1101 of the sensor group 300 and the construction machinery 100 can be omitted. Moreover, the training dataset TRD1 used to generate the learned model LM1 can include a basic training dataset and a final adjustment (micro-call) training dataset. In this case, since the basic training dataset requires multiple datasets, it can be generated based on the logs output from the simulator unit 2002, and the final adjustment training dataset can be generated based on the logs acquired by the log acquisition unit 2001. Hereinafter, the same applies to the training dataset TRD2 used to generate the learned model LM2 (described later) or the training dataset TRD3 used to generate the learned model LM3.

[0146] The training data generation department 2004B generates the training dataset TRD2 for generating the fully learned model LM2.

[0147] After learning, the model LM2 uses predetermined data as input to infer the shape of the work object after the construction machinery 100 performs predetermined actions. "After the construction machinery 100 performs predetermined actions" means, for example, in the case of repeated predetermined actions such as the digging action of an SVL excavator, where the predetermined action is performed once and then repeated at intervals. Furthermore, "after the construction machinery 100 performs predetermined actions" means, for example, in the case of continuous predetermined actions such as the scraping action of a ULD ship unloader, where the predetermined actions are performed continuously, meaning after a predetermined action of a predetermined time has elapsed since the start of the action. The predetermined data input to the model LM2 includes, for example, data representing the shape of the work object or the work object and surrounding objects (hereinafter referred to as "work object, etc.") and data on the trajectory (i.e., path) of the working part of the construction machinery 100 when performing the predetermined actions. Therefore, the model LM2 can predict the changed shape of the work object, etc., based on the trajectory of the working part of the construction machinery 100, using the shape of the work object, etc., as a premise. When the construction machinery 100 performs a predetermined action, if the predetermined action is repeated at intervals, it means that the predetermined action is performed once. Furthermore, when the construction machinery 100 performs a predetermined action continuously, it means that the predetermined action is performed for a predetermined time starting from a start time.

[0148] Furthermore, the input data for the learned model LM2 can also include data representing the characteristics of the work object of the construction machinery 100. For example, the data representing the characteristics of the work object includes the angle of repose of the sand or bulk cargo M. Thus, the learned model LM2 can consider the angle of repose of the sand or bulk cargo M to infer a more accurate shape of the work object. In this case, the input data in the training data includes data representing the characteristics of the work object (the angle of repose data).

[0149] Furthermore, the predetermined data input into the learned model LM2 can also include data representing the reaction force from the work object to the work site when the construction machinery 100 performs a predetermined action. Thus, for example, even if the bucket 6 of the excavator SVL comes into contact with underground rocks, and as a result, sand excavation is not possible, the learned model LM2 can still estimate the accurate shape of the sand by taking into account the reaction force of the work site (bucket 6).

[0150] Furthermore, the predetermined data input to the learned model LM2 may also include occlusion data corresponding to the data representing the shape of a pre-existing work object, etc. Occlusion refers to a state where observation is impossible due to obstructions such as those present between the surrounding monitoring sensor 140 or sensor 300-X and the work object. The occlusion data is, for example, a set of data representing whether each small region obtained by dividing the observation area into regions can be observed. Thus, for example, if occlusion is included in the data representing the shape of a pre-existing work object, etc., the learned model LM2 can predict the shape of the subsequent work object, etc., by interpolating the occluded parts from the data representing the surrounding shape. In this case, the learned model LM2 performs machine learning using training data that intentionally includes occlusion.

[0151] The training dataset TRD2 contains a combination of the aforementioned predetermined data as input data and the shape of the work object after the construction machinery 100 performs the predetermined action as correct output data.

[0152] The Machine Learning Division 2005 generated fully learned models LM1 and LM2 by performing machine learning on the basic learning model based on the training dataset generated by the Training Data Generation Division 2004. The fully learned models (basic learning models) include neural networks such as DNNs (Deep Neural Networks).

[0153] The Machine Learning Division 2005 includes Machine Learning Division 2005A and 2005B.

[0154] The machine learning unit 2005A performs machine learning on the basic learning model M1 based on the training dataset TRD1 output from the training data generation unit 2004A. As a result, the machine learning unit 2005A can generate a learned model LM1, which can take data related to the operating state of the construction machinery 100 or data representing the shape of the work object as input and output (infer) data representing the predicted trajectory of the working part of the construction machinery 100. Furthermore, the machine learning unit 2005A can correct (additionally learn) the learned model LM1 to reduce the error between the inference result based on the learned model LM1 and the measurement result of the actual motion monitoring sensor 150. In this case, the data based on the inference result of the learned model LM1 and the measurement result of the actual motion monitoring sensor 150 are uploaded from the construction machinery 100 to the information processing device 200.

[0155] The machine learning unit 2005B performs machine learning on the basic learning model M2 based on the training dataset TRD2 output from the training data generation unit 2004B. As a result, the machine learning unit 2005B can generate a learned model LM2, which takes as input data representing the shape of a pre-existing work object and data representing the track of the work area when the construction machinery 100 performs a predetermined action, and outputs (infers) data representing the shape of the work object after the construction machinery 100 performs the predetermined action. Furthermore, the machine learning unit 2005B can also correct (additionally learn) the learned model LM2 to reduce the error between the inference result based on the learned model LM2 and the actual measurement results from the surrounding monitoring sensor 140 or the sensor group 300. In this case, the data based on the inference result of the learned model LM2 and the actual measurement results from the surrounding monitoring sensor 140 are uploaded from the construction machinery 100 to the information processing device 200. And the data based on the actual measurement results from the sensor group 300 are uploaded from the sensor group 300 to the information processing device 200.

[0156] The learned models LM1 and LM2, output by the machine learning unit 2005, are stored in the learned model storage unit 2006. Furthermore, when the learned model LM1 is relearned or supplemented through the machine learning unit 2005A, the learned model LM1 in the learned model storage unit 2006 is updated. The same applies when the learned model LM2 is relearned or supplemented through the machine learning unit 2005B.

[0157] The distribution department 2007 distributed the completed learning data of models LM1 and LM2 to construction machinery 100.

[0158] For example, if the machine learning unit 2005A generates or updates the learned model LM1, the distribution unit 2007 distributes the most recently generated or updated learned model LM1 to the construction machinery 100. Furthermore, the distribution unit 2007 can also distribute the latest learned model LM1 from the learned model storage unit 2006 to the construction machinery 100 based on a signal received from the construction machinery 100 requesting the distribution of the learned model LM1. The same applies to the learned model LM2.

[0159] The Operation Support Unit 1102 is a functional unit used to provide support to users who operate the construction machinery 100 and perform or monitor the operation of the construction machinery 100.

[0160] The operation support unit 1102 includes a learning completed model storage unit 1102A, an object perception unit 1102B, a safety control unit 1102C, a trajectory prediction unit 1102D, a work object shape prediction unit 1102E, a safety calculation unit 1102F, and a safety control unit 1102G.

[0161] In addition, some or all of the functions of the object sensing unit 1102B, safety control unit 1102C, track prediction unit 1102D, work object shape prediction unit 1102E, safety calculation unit 1102F, and safety control unit 1102G can be transferred to the outside of the construction machinery 100 (e.g., information processing device 200).

[0162] The learned models LM1 and LM2, which are distributed from the information processing device 200 and received through the communication device 180, are stored in the learned model storage unit 1102A.

[0163] The object sensing unit 1102B senses objects around the construction machinery 100 based on the output of the peripheral monitoring sensor 140. Furthermore, the object sensing unit 1102B can also sense objects around the construction machinery 100 based on the output of the peripheral monitoring sensor 140 and the output of the sensor group 300 disposed at the construction site. The output of the sensor group 300 can be received directly from the sensor group 300 via the communication device 180, or indirectly from the sensor group 300 via the information processing device 200.

[0164] The periphery of the construction machinery 100 includes the periphery of the main body of the construction machinery 100 or the periphery of the working device 125. The main body of the construction machinery 100 is, for example, the part of an excavator SVL that includes the lower traveling body 1 and the upper rotating body 3. Furthermore, the periphery of the construction machinery 100 may include not only the area of ​​the work object of the construction machinery 100, but also the area around the work object.

[0165] The objects being monitored may include, for example, workers. Furthermore, the objects being monitored may also include the work object or other obstacles located around it. Other obstacles may include, for example, other moving objects specific to the excavator SVL construction site, such as other construction machinery or work vehicles. Other obstacles may also include, for example, fixed objects specific to the excavator SVL construction site, such as utility poles, fences, and traffic cones. Furthermore, other obstacles may include specific terrain shapes specific to the excavator SVL construction site, such as ditches and holes. And, for example, other obstacles may include the walls corresponding to the outer edge of a ship's hull (HD).

[0166] The object sensing unit 1102B processes objects that are monitored around the construction machinery 100 according to a predetermined processing cycle, and outputs the processing results. The processing results include, for example, data indicating the position of the object if the monitored object is detected or not, and if the monitored object is detected.

[0167] The safety control unit 1102C performs functional safety-related controls based on the output of the object sensing unit 1102B.

[0168] For example, when the object being monitored is detected within a predetermined range around the construction machinery 100 by the object sensing unit 1102B, the safety control unit 1102C activates the safety function.

[0169] Safety functions may include, for example, alarm outputs to at least one of the following: the inside of the operator's seat of the construction machinery 100, the outside of the operator's seat, and the remote operator of the construction machinery 100, and notification functions that notify objects that sense the monitored object. Thus, it is possible to alert the operator inside the operator's seat, workers around the construction machinery 100, and the operator remotely operating the construction machinery 100 to the presence of objects within the monitored area surrounding the construction machinery 100. Hereinafter, the notification function to the operator inside the operator's seat, etc., will sometimes be referred to as an "internal notification function," the notification function to workers outside the construction machinery 100, etc., will be referred to as an "external notification function," and the notification function to the operator remotely operating the construction machinery 100 or the monitor performing remote monitoring, etc., will be referred to as a "remote notification function."

[0170] Furthermore, safety functions may include, for example, a motion restriction function that limits the actions of the construction machinery 100 in response to the operation or remote operation of the operating device 130. This forcibly restricts the actions of the construction machinery 100, thereby reducing the possibility of the construction machinery 100 approaching or coming into contact with the monitored object. The motion restriction function may include, for example, a motion deceleration function that slows down the actions of the construction machinery 100 in response to the operation or remote operation of the operating device 130. The motion restriction function may also include a motion stop function that stops the construction machinery 100 from operating and maintains it in a stopped state regardless of the presence or absence of operation or remote operation of the operating device 130. As a motion restriction function, the safety control unit 1102C may execute only one of the motion deceleration function and the motion stop function, or both.

[0171] For example, when an object of surveillance is detected by the object sensing unit 1102B within a predetermined range (hereinafter referred to as the "notification range") around the construction machinery 100, the safety control unit 1102C activates the notification function. The notification range is, for example, a range where the distance D from a predetermined location on the construction machinery 100 is less than or equal to a threshold Dth1. The predetermined location on the construction machinery 100 is, for example, the main body of the construction machinery 100 (e.g., the fuselage including the lower traveling body 1 and upper rotating body 3 of an excavator SVL). Furthermore, the predetermined location on the construction machinery 100 may also be the working part at the front end of the working device 125 (e.g., the bucket 6 of an excavator SVL or the scraper 61 of a continuous unloader ULD). The threshold Dth1 may remain constant regardless of the direction of view from the predetermined location on the construction machinery 100, or it may vary depending on the direction of view from the predetermined location on the construction machinery 100.

[0172] Safety control unit 1102C, for example, activates an internal notification function or an external notification function based on sound (i.e., auditory mode) to at least one of the driver's seat's interior and exterior by controlling sound output device 164. At this time, safety control unit 1102C can vary the pitch, sound pressure, timbre, resonant period when the sound is periodically audible, and content of the voice output according to various conditions.

[0173] Furthermore, the safety control unit 1102C enables, for example, a visual-based internal notification function. Specifically, the safety control unit 1102C can control the display device 162 inside the driver's seat to display an image representing a detected monitored object along with the captured or processed images from the camera devices included in the peripheral monitoring sensor 140. The safety control unit 1102C can also emphasize the position of the monitored object reflected in the captured or processed images displayed on the display device 162 inside the driver's seat, or the position on the image corresponding to the detected monitored object. For example, the safety control unit 1102C can overlay a frame surrounding the detected monitored object onto the image displayed on the display device 162 inside the driver's seat, or overlay a marker at the position on the image corresponding to the detected monitored object. Thus, the control device 110 can implement a visual notification function for the operator. Furthermore, the safety control unit 1102C can also use warning lights or lighting devices inside the driver's seat to notify the operator inside the driver's seat of the detected monitored object.

[0174] Furthermore, the safety control unit 1102C can, for example, activate the vision-based external notification function by controlling an output device 160 (e.g., a lighting device such as a headlight or a display device 162) located on the side of the construction machinery 100. Additionally, the safety control unit 1102C can also activate the external notification function by sending a command signal indicating the operation of the notification function to a terminal device (mobile terminal) carried by personnel, supervisors, managers, or other individuals near the construction machinery 100 at the construction site. The terminal device carried by these personnel, supervisors, managers, etc., can be a common mobile terminal such as a smartphone or tablet. Furthermore, the terminal device carried by these personnel, supervisors, managers, etc., can also be a wearable terminal, such as smart glasses. Additionally, the safety control unit 1102C can, for example, activate the internal notification function via tactile means by controlling a vibration generating device that vibrates the operator's seat. Therefore, the control device 110 enables operators, workers around the construction machinery 100, and supervisors to identify objects (e.g., workers) that are in a relatively close location around the construction machinery 100. Thus, the control device 110 can prompt the operator to confirm the safety conditions around the construction machinery 100, or prompt workers within the monitored area to retreat from the monitored area.

[0175] Furthermore, the safety control unit 1102C can, for example, send a command signal indicating the operation of the notification function to the remote operation support device 400 via the communication device 180 to activate the remote notification function. In this case, if the remote operation support device 400 receives the command signal from the construction machinery 100, it can output an alarm based on visual or auditory means. Thus, the operator performing remote operation of the construction machinery 100 can be notified of the entry of a monitored object into the vicinity of the construction machinery 100 via the remote notification function of the remote operation support device 400.

[0176] Additionally, the remote notification function of the safety control unit 1102C can also be transferred to the remote operation support device 400. In this case, the remote operation support device 400 receives information from the construction machinery 100 related to the processing results based on the object sensing unit 1102B. Then, the remote operation support device 400 determines whether the object being monitored has entered the notification range based on the received information, and if the object being monitored is within the notification range, activates the remote notification function.

[0177] Furthermore, the safety control unit 1102C can also make the notification method (i.e., the notification method) different based on the positional relationship between the object being monitored and the predetermined part of the construction machinery 100 that serves as the reference for the notification range.

[0178] For example, if the object of the monitored target, sensed by the object sensing unit 1102B within the notification range, is located at a position relatively far from a predetermined location on the construction machinery 100, the safety control unit 1102C can output an alarm indicating a relatively low level of urgency (hereinafter referred to as an "attention level alarm") to remind the user to pay attention to the object of the monitored target. For convenience, the range within the notification range that is relatively far from the predetermined location on the construction machinery 100, i.e., the range corresponding to the attention level alarm, is sometimes referred to as the "attention notification range." On the other hand, if the object of the monitored target, sensed by the object sensing unit 1102B within the notification range, is located at a position relatively close to a predetermined location on the construction machinery 100, the safety control unit 1102C can output an alarm indicating a relatively high level of urgency (hereinafter referred to as an "alert level alarm") to remind the user that the object of the monitored target is approaching a predetermined location on the construction machinery 100 and the danger level is increasing. The range within the notification range that is relatively close to the predetermined location on the construction machinery 100, i.e., the range corresponding to the alert level alarm, is sometimes referred to as the "alert notification range."

[0179] In this situation, the safety control unit 1102C can differentiate the pitch, sound pressure, timbre, and ringing period of the sound output from the sound output device 164 or the remote operation support device 400 between the attention-level alarm and the alert-level alarm. Furthermore, the safety control unit 1102C can also differentiate the color, shape, size, flashing status, and flashing period of the image of the monitored object displayed on the display device 162 or the remote operation support device 400 between the attention-level alarm and the alert-level alarm. Similarly, the safety control unit 1102C can also differentiate the color, shape, size, flashing status, and flashing period of the image (e.g., a frame or marker) of the monitored object or its position displayed on the display device 162, emphasizing the image captured by the camera or the processed image. Thus, the control device 110 can, through the differences in the notification sound output from the sound output device 164 or the notification image displayed on the display device 162, enable the operator to grasp the level of urgency, in other words, to grasp the proximity of the monitored object to a predetermined location on the construction machinery 100. Furthermore, the safety control unit 1102C can execute alert-level alarms via a display on the display device 162 or the remote support device, and can execute warning-level alarms via voice output from the voice output device 164 or the remote operation support device 400 instead of displaying them, or execute warning-level alarms via both voice output from the voice output device 164 or the remote operation support device 400 and displaying them. Thus, the control device 110 can enable operators to grasp the level of urgency by switching notification units.

[0180] Furthermore, the safety control unit 1102C can change the notification method in multiple stages of three or more stages, or continuously change the notification method, based on the distance between the monitored object perceived within the notification range and the predetermined part of the construction machinery 100.

[0181] For example, if, after the notification function has started, the object of the monitored target sensed by the object sensing unit 1102B is not sensed within the notification range, the security control unit 1102C stops the notification function. Furthermore, if, after the notification function has started, a predetermined input to deactivate the notification function is received via the input device 170, the security control unit 1102C can stop the notification function.

[0182] Furthermore, for example, when the object sensing unit 1102B senses an object being monitored within a predetermined range (hereinafter referred to as the "motion restriction range") around the construction machinery 100, the safety control unit 1102C activates the motion restriction function. The motion restriction range can be set to be the same as or different from the notification range described above. For example, the motion restriction range can be set to a range whose outer edge is closer to a predetermined part of the construction machinery 100 than the notification range. Thus, the safety control unit 1102C can, for example, activate the notification function first when the object being monitored enters the notification range from the outside, and then activate the motion restriction function further when the object being monitored enters the inner motion restriction range. Therefore, the control device 110 can activate the notification function and the motion restriction function in stages according to the object being monitored moving inward within the monitoring area.

[0183] Specifically, if an object of the monitored target is sensed within the motion restriction range at a distance D from a predetermined location on the construction machinery 100 that is within a threshold value Dth2 (≤Dth1), the safety control unit 1102C can activate the motion restriction function. The threshold value Dth2 may remain constant regardless of the direction of observation from the predetermined location on the construction machinery 100, or it may change according to the direction of observation from the predetermined location on the construction machinery 100.

[0184] The motion restriction range includes a deceleration range that causes the construction machinery 100 to operate slower than normally in response to the operation or remote operation of the operating device 130. Furthermore, the motion restriction range may also include a stopping range that stops the construction machinery 100 and maintains it in a stopped state regardless of the presence or absence of operation or remote operation of the operating device 130. The motion restriction range may include only one of the deceleration range and the stopping range, or it may include both. For example, if the motion restriction range includes both the deceleration range and the stopping range, the stopping range is the area within the motion restriction range closest to a predetermined location on the construction machinery 100. Moreover, the deceleration range is the area within the motion restriction range located outside the stopping range.

[0185] When the actuator 120 is a hydraulic actuator, the safety control unit 1102C activates the motion limiting function, for example, by controlling the operating control valve. In this case, the safety control unit 1102C can limit the movement of all driven components (i.e., the corresponding hydraulic actuators), or it can limit the movement of a portion of the driven components (hydraulic actuators). Thus, if there is a monitored object around the construction machinery 100, the control device 110 can slow down or stop the movement of the construction machinery 100. Therefore, the control device 110 can prevent the monitored object around the construction machinery 100 from contacting the construction machinery 100. Furthermore, the safety control unit 1102C can also control an unshown solenoid switching valve at the upstream end of the pilot line between the hydraulic source (e.g., a pilot pump) and the operating control valve to cut off the pilot line and activate the motion stopping function. Furthermore, when the actuator 120 is an electric actuator, the safety control unit 1102C, for example, controls the drive device to slow down or stop the actuator 120 to activate the motion limiting function.

[0186] Furthermore, if, after the motion restriction function is activated, the object of the monitored target sensed by the object sensing unit 1102B is not sensed within the motion restriction range, the safety control unit 1102C can deactivate the motion restriction function. Also, if, after the motion restriction function is activated, the safety control unit 1102C can deactivate the motion restriction function upon receiving a predetermined input for deactivating the motion restriction function via the input device 170. The input content for deactivating the notification function and the input content for deactivating the motion restriction function via the input device 170 can be the same or different.

[0187] Furthermore, the safety control unit 1102C can also switch the function between ON (enabled) and OFF (disabled) based on the predetermined inputs from the operator to the input device 170 or the remote operation support device 400.

[0188] Alternatively, the safety control unit 1102C can be omitted.

[0189] The track prediction unit 1102D predicts the track of the working part of the work device 125 in the near future based on the operator's operation, according to the operating status of the construction machinery 100 or the shape of the current work object. The near future corresponds to the period of the predicted track of the working part; specifically, it refers to a relatively short period in the future based on the present. The near future, for example, is the period from the completion of a predetermined action assuming the excavator SVL is currently performing or about to begin a predetermined action. Furthermore, the near future can also be a period on the order of several seconds to tens of seconds from the present. Specifically, the track prediction unit 1102D obtains data representing the predicted track of the working part of the work device 125 based on data representing the operating status of the construction machinery 100 or data representing the shape of the current work object. Data representing the operating status of the construction machinery 100 is, for example, data of the operating signal corresponding to the output of the electric operating device 130 or data of the detected value of the pilot pressure for operation of the hydraulic pilot operating device 130. Furthermore, data representing the shape of the current work object is obtained, for example, based on the output data of the peripheral monitoring sensor 140. Furthermore, data representing the shape of the current work object can be obtained from the output data of the sensor group 300 received via the communication device 180, instead of the output data of the peripheral monitoring sensor 140. Alternatively, it can be obtained from the output data of the sensor group 300 received via the communication device 180, in addition to the output data of the peripheral monitoring sensor 140. Furthermore, if there are parts in the output data of the peripheral monitoring sensor 140 or the sensor group 300 where the shape of the work object cannot be observed due to obstruction, the data representing the shape of that part can be supplemented using the same method as the work object shape prediction unit 1102E described later. Specifically, the control device 110 can estimate the shape of the current work object based on the shape of the most recently observed work object and the trajectory of the work area when the construction machinery 100 performed a predetermined action, thereby interpolating the shape of the work object in the unobservable parts.

[0190] For example, the track prediction unit 1102D uses a learned model LM1 to predict the track of the work area in the near future based on the operator's actions, based on data related to the operating state of the construction machinery 100 or data representing the shape of the work object. Furthermore, the track prediction unit 1102D can also predict the track of the work area in the near future based on the operator's actions using MPC (Model Predictive Control), based on data representing the operating state of the construction machinery 100 or data representing the shape of the work object. In this case, the machine learning unit 2005A can be omitted.

[0191] Furthermore, when the construction machinery 100 is a continuous unloader (ULD), the multiple buckets 77 continue to move on the predetermined track of the chain bucket 79 without relying on the operator's operation. Therefore, the object shape prediction unit 1102E predicts the shape of the work object after a predetermined action of the construction machinery 100 in the near future, based on the movement on the predetermined track of the chain bucket, the prediction result (predicted track) of the track prediction unit 1102D, and the current shape of the work object. Specifically, the object shape prediction unit 1102E obtains data indicating the predicted shape of the work object after a predetermined action of the construction machinery 100 in the near future, based on the data representing the predicted track obtained by the track prediction unit 1102D and the data representing the current shape of the work object. Furthermore, the object shape prediction unit 1102E can also predict the shape of the work object and its surrounding objects after a predetermined action of the construction machinery 100 in the near future, based on the prediction result of the track prediction unit 1102D and the shape of the current work object and its surrounding objects. Specifically, the object shape prediction unit 1102E can acquire data representing the predicted shape of the work object and its surrounding objects after changes caused by a predetermined action of the construction machinery 100 in the near future, based on data representing the predicted trajectory and data representing the shape of the current work object and its surrounding objects. This is because sometimes the predetermined action of the construction machinery 100 on the work object can also affect the objects surrounding the work object. The shape of the objects surrounding the work object is, for example, the shape of the sand surrounding the work area that is continuous with the work area of ​​the excavator SVL. The data representing the shape of the objects surrounding the work object is acquired, for example, based on the output data of the sensor group 300 received via the communication device 180. Furthermore, the data representing the shape of the objects surrounding the work object can also be acquired based on the output data of the surrounding monitoring sensor 140 instead of the output data of the sensor group 300, or it can be acquired based on the output data of the surrounding monitoring sensor 140 in addition to the output data of the sensor group 300.

[0192] For example, the object shape prediction unit 1102E can use the learned model LM2 to predict the shape of the work object after changes caused by a predetermined action of the construction machinery 100 in the near future, based on data representing the prediction track and data representing the shape of the current work object. Similarly, the object shape prediction unit 1102E can also use the learned model LM2 to predict the shape of the work object and its surrounding objects after changes in the near future, based on data representing the prediction track and data representing the shape of the current work object and its surrounding objects.

[0193] The safety calculation unit 1102F evaluates the degree of safety (hereinafter referred to as "safety") based on changes in the shape of the work object or its surrounding objects within a predetermined range (hereinafter, for convenience, referred to as the "evaluation range"). Specifically, the safety calculation unit 1102F predicts changes in the shape of the work object, etc., within the evaluation range based on the prediction results of the work object shape prediction unit 1102E. Then, the safety calculation unit 1102F calculates the safety of the position of the monitored object perceived by the object sensing unit 1102B within the evaluation range based on the prediction results of the changes in the shape of the work object, etc. Furthermore, the safety calculation unit 1102F may calculate the safety of the possible location of the monitored object instead of the object sensing unit 1102B perceiving the object's position, or it may calculate the safety of the possible location of the monitored object in addition to the object sensing unit 1102B perceiving the object's position. The possible location of the monitored object refers, for example, to a location in the latest work data where, although within the observation range of the surrounding monitoring sensor 140 or sensor 300-X, obstruction has occurred. Hereinafter, the location of the monitored object sensed by the object sensing unit 1102B, or the possible location of the monitored object, will be referred to simply as the "location of the monitored object".

[0194] For example, the safety calculation unit 1102F calculates the safety level of a location by calculating that the safety level decreases as the change in shape of the work object or surrounding objects at the location of the monitored object increases. This is because if the shape of the work object or surrounding objects changes, the monitored object may be involved in that shape change. For example, as... Figure 5 As shown, the security calculation unit 1102F calculates the security level at the location of the object being monitored based on the case where the change is zero, and in a manner where the security level decreases as the change increases and the rate of decrease in security level increases.

[0195] Furthermore, the safety calculation unit 1102F can also calculate the safety level of a location based on the amount of change in the shape of the work object or objects surrounding it within a relatively narrow predetermined range, using the location of the monitored object as a reference. This is because even if only the shape of the work object or objects surrounding the location of the monitored object changes, the safety of the monitored object is considered to have decreased. This is because the monitored object may move. For example, the safety calculation unit 1102F calculates the safety level using the average value (hereinafter referred to as "average change") of the amount of change in the shape of the work object or objects surrounding it within the predetermined range. The safety calculation unit 1102F can also calculate the safety level using the maximum value (hereinafter referred to as "maximum change") of the amount of change in the shape of the work object or objects surrounding it within the predetermined range. Specifically, the safety calculation unit 1102F calculates the safety level at the location where the monitored object is perceived, in a manner that the safety level decreases as the average change or maximum change increases within the predetermined range. For example, with... Figure 5 Similarly, the safety calculation unit 1102F calculates the safety level of the monitored object's position based on the case where the average change or maximum change is zero, and in a manner that the safety level decreases as the average change or maximum change increases and the rate of decrease in safety level increases.

[0196] The safety calculation unit 1102F can, for example, use conversion formulas, conversion maps, or conversion tables pre-registered in the auxiliary storage device 110A to calculate the safety level based on the change in shape of the work object, etc.

[0197] The safety control unit 1102G performs functional safety-related controls based on the safety level calculated by the safety level calculation unit 1102F.

[0198] For example, when the safety level is relatively low compared to a predetermined benchmark, the safety control unit 1102G activates the safety function. A relatively low safety level compared to the predetermined benchmark can mean either the safety level is below the predetermined benchmark or the safety level is lower than the predetermined benchmark.

[0199] The safety control unit 1102G enables it to perform the same safety functions as the safety control unit 1102C.

[0200] For example, when the safety level is relatively low compared to the predetermined benchmark (hereinafter referred to as the "notification benchmark") for the notification function, the safety control unit 1102G activates the internal notification function or the remote notification function. As a result, the control device 110 can notify the operator that the shape of the work object or its surroundings may change due to the movement of the construction machinery 100, thereby reducing the safety level. Therefore, the operator can ensure the safety of the work object or its surroundings by stopping the operation of the construction machinery 100. Furthermore, in addition to activating the internal or remote notification function, the safety control unit 1102G can also activate the external notification function. Thus, the control device 110 can prompt workers or supervisors around the construction machinery 100 to pay attention. In this case, the notification benchmark for the internal or remote notification function and the notification benchmark for the external notification function may be the same or different.

[0201] Furthermore, when the security level is relatively low compared to the notification benchmark, the security control unit 1102G can adjust the notification method (i.e., the notification approach) according to the level of security.

[0202] For example, similar to the case of safety control unit 1102C, safety control unit 1102G distinguishes between alert-level alarms and warning-level alarms. Specifically, safety control unit 1102G can output an alert-level alarm when the security level is relatively high, and an alarm-level alarm when the security level is relatively low, within a range where the security level is relatively low relative to the notification reference. The relatively high and relatively low security levels are distinguished based on a predetermined reference (hereinafter referred to as the "warning-level notification reference") that is lower than the usual notification reference (hereinafter referred to as the "attention notification reference") specified for warning-level alarms. That is, safety control unit 1102G can output an alert-level alarm when the security level is relatively low relative to the attention notification reference but not relatively low relative to the warning-level notification reference, and output an alarm-level alarm when the security level is relatively low relative to the warning-level notification reference.

[0203] Furthermore, the security control unit 1102G can also change the notification method in multiple stages, either in three or more stages, or continuously, depending on the level of security within a range where the security level is relatively low compared to the notification benchmark. Specifically, the security control unit 1102G can issue a notification in a manner where the level of attention or alert increases as the security level decreases within a range where the security level is relatively low compared to the notification benchmark.

[0204] For example, if the operator's operation is stopped after the notification function has started working, the safety control unit 1102G stops the notification function.

[0205] Furthermore, when the safety level is relatively low compared to the predetermined benchmark for the motion restriction function (hereinafter referred to as the "motion restriction benchmark"), the safety control unit 1102G can activate the motion restriction function. Thus, when the shape of the work object or its surroundings changes due to the movement of the construction machinery 100, potentially reducing the safety level, the control device 110 can restrict the movement of the construction machinery 100. Therefore, the control device 110 can ensure the safety of the work object or its surroundings. When the safety level is relatively low compared to the motion restriction benchmark, the control device 110 can activate either the motion deceleration function or the motion stop function. Furthermore, a motion restriction benchmark (hereinafter referred to as the "motion deceleration benchmark") for the motion deceleration function and a motion restriction benchmark (hereinafter referred to as the "motion stop benchmark") set below the motion deceleration benchmark for the motion stop function can be provided. In this case, the safety control unit 1102G activates the motion deceleration function when the safety level is relatively low compared to the motion deceleration benchmark but not relatively low compared to the motion stop benchmark, and activates the motion stop function when the safety level is relatively low compared to the motion stop benchmark.

[0206] Furthermore, when the safety level is relatively low compared to the motion restriction reference, the safety control unit 1103G can change the degree of motion restriction in three or more stages, or continuously change the degree of motion restriction, depending on the level of safety within that range. Specifically, the safety control unit 1103G can activate the motion restriction function in a way that the degree of motion restriction (i.e., the degree to which the movement of the construction machinery 100 is slowed down) increases as the safety level decreases within a range where the safety level is relatively low compared to the motion restriction reference.

[0207] Furthermore, the safety control unit 1102G can take into account the reliability of the data representing the predicted shape of the work object, etc. (hereinafter referred to as "prediction reliability") to enable the safety function to operate.

[0208] The prediction reliability is set, for example, to be lower when the operator is not operating the operating device 130 than when the operator is operating it. This is because reflecting the operator's actual operation improves the accuracy of the data representing the predicted shape of the work object, etc. Similarly, the prediction reliability can also be set to be lower when the work part is not moving than when it is moving. This is because reflecting the actual movement of the work part improves the accuracy of the data representing the predicted shape of the work object. Furthermore, the prediction reliability can also be set to decrease as the occlusion range in the shape representing the current work object increases. This is because if the occlusion range in the shape representing the current work object increases relatively, the accuracy of the predicted shape based on that shape is relatively low.

[0209] For example, the safety control unit 1102G is configured such that the higher the predicted reliability, the higher the predetermined benchmark for enabling the safety function to operate, and vice versa. Therefore, when the predicted reliability is low, the control device 110 can prevent the safety function from operating. Thus, the control device 110 can suppress the reduction in work efficiency caused by inaccurate operation of the safety function under conditions of low predicted reliability. Therefore, the control device 110 can balance the safety of the work object or its surroundings with work efficiency.

[0210] Furthermore, when the level of security is relatively low compared to the notification benchmark, the security control unit 1102G can adjust the notification method according to the level of predicted reliability.

[0211] For example, the safety control unit 1102G can change the pitch, sound pressure, timbre, and ringing period of the notification tone output from the sound output device 164 or the remote operation support device 400 to increase the level of attention alert as the prediction reliability increases. Furthermore, the safety control unit 1102G can also change the color, shape, size, blinking status, and blinking period of the notification image on the display device 162 or the remote operation support device to increase the level of attention alert as the prediction reliability increases. Moreover, when the prediction reliability is relatively low, the safety control unit 1102G activates the visual-based notification function; conversely, when the prediction reliability is relatively high, it can activate the auditory-based notification function instead of the visual method, or activate both the visual and auditory-based notification functions. Thus, the control device 110 can suppress the decrease in work efficiency and ensure the safety of the work object or its surroundings.

[0212] For example, if the operator's operation is stopped after the motion restriction function has been activated, the safety control unit 1102G will deactivate the motion restriction function. This allows the operator to resume operation of the construction machinery 100.

[0213] Furthermore, the safety control unit 1102G can switch its function between ON (enabled) and OFF (disabled) based on the predetermined inputs from the operator to the input device 170 or the remote operation support device 400.

[0214] Alternatively, the safety control unit 1102G can also activate the safety function only for objects monitored outside the working scope of the safety function of the safety control unit 1102C. In this case, the safety calculation unit 1102F calculates the safety level of the location of the monitored object outside the working scope (i.e., the notification range or action restriction range) of the monitored object perceived by the object sensing unit 1102B.

[0215] [Example of handling the operational support system] Next, refer to Figure 6 , Figure 7 This section describes the first example of handling the SYS (System for Operation Support) system. Specifically, it describes the first case of handling the SYS system. Figure 4 The specific example of processing based on the first case of the SYS operation support system will be explained.

[0216] Figure 6 This is a flowchart illustrating the first example of the processing of the SYS (System for Operations) support system. Figure 7 This is a diagram showing an example of the area being observed.

[0217] This flowchart, for example, is repeated for each predetermined processing cycle during the period from the start to the end of the predetermined operation of construction machinery 100.

[0218] like Figure 6 As shown, in step S102, the control device 110 acquires historical data related to the operating status of the construction machinery 100, including data related to the latest operating status of the construction machinery 100 performed by the operator.

[0219] For example, control device 110 acquires the most recent historical data of the trajectory (i.e., track) of the work area of ​​construction machinery 100, reflecting the operation status of the construction machinery 100 performed by the operator. 1：k Orbit x 1：k The status x of the work area using the most recent processing cycle k times. n (n=1, ..., k) is represented by the following equation (1).

[0220] [Formula 1] The state x of the work area at time n n Use m parameters x1 of the specified work location. n ~x m n (m≥2) is expressed as shown in equation (2).

[0221] [Equation 2] Parameter x1 n ~x m nThe parameters for defining the position of the working part are obtained based on the output data of the motion monitoring sensor 150. For example, in the case of the construction machinery 100 being an excavator SVL, the parameters for defining the position of the bucket 6 are parameters for defining the position of the bucket 6. The parameters for defining the position of the bucket 6 are, for example, the posture angles of the boom 4, stick 5, bucket 6, and upper slewing body 3 of the excavator SVL. Furthermore, the parameters for defining the position of the working part are, for example, in the case of the construction machinery 100 being a continuous unloader ULD (described later), the posture angles of the slewing body 55, boom 57, and scraper 61 of the continuous unloader ULD, and the inter-axis distance of the driven rollers 81b and 81c of the scraper 61.

[0222] If step S102 is completed, the control device 110 proceeds to step S104.

[0223] In step S104, the control device 110 acquires data representing the shape of the latest work object, etc.

[0224] For example, such as Figure 7 As shown, the observation area TA around the construction machinery 100, which is the work object, is divided into a predetermined number of grids N. The observation area TA is the range of observed objects in the region corresponding to the work object and its surrounding objects, based on the shape of the surrounding monitoring sensor 140 or the surrounding monitoring sensor 140 and sensor group 300. Furthermore, the observation area TA is also the region of predicted objects in the region corresponding to the work object and its surrounding objects, based on the shape of the work object shape prediction unit 1102E.

[0225] For example, control device 110 acquires the latest shape h of the work object, etc. The shape h of the work object, etc., uses the height h of each cell in the observation object area TA. i (i=1, ..., N) are represented as shown in equation (3).

[0226] [Formula 3] Furthermore, in addition to acquiring the shape h of the work object, the control device 110 can also acquire an occlusion map o representing whether each cell in the observation area TA is occluded. The occlusion map o uses observation availability information representing whether each cell is occluded. j (j=1, ..., N) is represented as shown in equation (4).

[0227] [Formula 4] For example, is the observation information available? j Take the value "1" when the state is observable, and take the value "0" when the state is unobservable.

[0228] Furthermore, the shape h of the task object and the occlusion mapping o can be used as input, and the interpolated shape h of the task object can be obtained by using the function q, which is equivalent to the learned model, through the following equation (5). ip .

[0229] [Formula 5] If step S104 is completed, the control device 110 proceeds to step S106.

[0230] In step S106, the track prediction unit 1102D of the control device 110 predicts the nearest future track of the working part of the construction machinery 100 based on the data obtained in steps S102 and S104, and obtains data representing the predicted track.

[0231] For example, the trajectory prediction unit 1102D uses a function f, which is equivalent to the learned model LM1, to obtain the predicted trajectory y of the work site by the following equation (6).

[0232] [Formula 6] Furthermore, variables γ representing the characteristics of the work object, such as sand or bulk cargo M, can be imported, and the predicted trajectory y of the work location can be obtained through the following formula (7).

[0233] [Formula 7] In addition, the predicted orbit y can be compared with the above orbit x. 1：k The same form (refer to equation (1) above) is used.

[0234] If step S106 is completed, the control device 110 proceeds to step S108.

[0235] In step S108, the object shape prediction unit 1102E of the control device 110 predicts the shape of the object, etc., based on the data obtained in steps S104 and S106, and obtains data representing the predicted shape.

[0236] For example, the object shape prediction unit 1102E uses a function r equivalent to the learned model LM2 to obtain the predicted shape h′ of the object, etc., by the following equation (8).

[0237] [Formula 8] Similar to the shape h of the work object, the predicted shape h′ of the work object uses the predicted height h′ of each cell in the observed object region TA. s(s=1, ..., N) is represented as shown in equation (9).

[0238] [Formula 9] Furthermore, similar to the case of equation (7) above, the object shape prediction unit 1102E can import γ representing the characteristics of the object and obtain the predicted shape h′ of the object, etc., by the following equation (10).

[0239] [Formula 10] Furthermore, the function r, which is equivalent to having learned the model LM2, can take the shape h of the current task object and the predicted trajectory y as inputs, as well as the occlusion map o as inputs, and obtain the predicted shape h of the task object through the following equation (11) or equation (12). ip ′.

[0240] [Equation 11] If step S108 is completed, the control device 110 proceeds to step S110.

[0241] In step S110, the control device 110 acquires data indicating the position of the object being monitored. As described above, the position of the object being monitored includes at least one of the position of the object being monitored as sensed by the object sensing unit 1102B and the possible location of the object being monitored (e.g., the location where the shape of the work object cannot be observed due to obstruction).

[0242] If step S110 is completed, the control device 110 proceeds to step S112.

[0243] In step S112, the safety calculation unit 1102F of the control device 110 calculates the safety level of each position of the monitored object based on the data obtained in step S110.

[0244] For example, the safety calculation unit 1102F calculates the change in shape of the work object, etc., based on the difference between the shape h of the work object, etc., and the predicted shape h′. Furthermore, when calculating the safety of the position where the shape of the work object, etc., cannot be observed due to occlusion, the safety calculation unit 1102F can calculate the change in shape based on the shape h of the work object, etc. ip With predicted shape h ip The difference between the values ​​is used to calculate the change in shape of the work object, etc. Then, the safety calculation unit 1102F calculates the safety level of each position of the monitored object based on the calculated change.

[0245] If step S112 is completed, the control device 110 proceeds to step S114.

[0246] In step S114, the safety control unit 1102G of the control device 110 determines whether there is a position of a monitored object whose calculation result (safety level) in step S112 is relatively low compared to a predetermined reference. If there is a position of a monitored object whose safety level is relatively low compared to the predetermined reference, the safety control unit 1102G proceeds to step S116; otherwise, the processing of this flowchart ends.

[0247] In addition, when both the notification function and the motion restriction function, which are safety features, are working, the safety level is judged to be relatively low based on the notification benchmark and the motion restriction benchmark, respectively.

[0248] In step S116, the safety control unit 1102G activates the safety function (i.e., at least one of the notification function and the action restriction function).

[0249] If step S116 is completed, the control device 110 ends the current flowchart processing.

[0250] [Example 2 of the functional structure of an operational support system] Next, besides reference Figures 1-3 In addition, also refer to Figure 8 The second example of the functional structure of the SYS operation support system will be explained.

[0251] Hereinafter, the same reference numerals are used for structures that are the same as or correspond to those in the first example above, and the parts that are different from those in the first example above are explained in detail. Sometimes, the descriptions that are the same as or correspond to those in the first example above are omitted or simplified.

[0252] Figure 8 This is the second example of a functional block diagram illustrating the functional structure of the SYS operating support system.

[0253] In this example, unlike the first example above, the operation support system SYS provides support related to the operation of the construction machinery 100 that operates through the autonomous operation function.

[0254] like Figure 8 As shown, similar to the first example above, the control device 110 of the construction machinery 100 includes an action log providing unit 1101 and an operation support unit 1102 as functional units.

[0255] Similar to the first example above, the information processing device 200 includes a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a training data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007 as functional units.

[0256] The training data generation unit 2004 includes training data generation units 2004B and 2004C.

[0257] The training data generation unit 2004C generates the training dataset TRD3 for generating the fully trained model LM3.

[0258] After learning, the model LM3 uses the data of the state of the work object of the construction machinery 100 (e.g., the shape or characteristics of the work object) as input to infer the target trajectory of the work part under the predetermined action of the construction machinery 100.

[0259] The training dataset TRD3 contains training data that are combinations of the state of the pre-existing work object as input data and the trajectory of the work area when the construction machinery 100 performs a predetermined action through the operation of a skilled operator, serving as correct output data. That is, the training data generation unit 2004C generates the training dataset based on logs obtained by the log acquisition unit 2001 showing the construction machinery 100 performing predetermined actions through the operation of a skilled operator. Furthermore, when multiple types of predetermined actions are specified, a fully learned model LM3 can be generated for each type of predetermined action. In this case, the training data generation unit 2004C generates the training dataset TRD3 for each type of predetermined action.

[0260] The Machine Learning Division 2005 includes Machine Learning Division 2005B and 2005C.

[0261] The machine learning unit 2005C performs machine learning on the basic learning model M3 based on the training dataset output from the training data generation unit 2004C. As a result, the machine learning unit 2005C can generate a learned model LM3, which can take data representing the state of the work object of the construction machinery 100 as input and output (infer) the target track of the work part under the predetermined action of the construction machinery 100.

[0262] The learned models LM2 and LM3, output by the machine learning unit 2005, are stored in the learned model storage unit 2006. Furthermore, when the learned model LM3 is relearned or supplemented through the machine learning unit 2005C, the learned model LM3 in the learned model storage unit 2006 is updated.

[0263] The distribution department 2007 distributed the data from the learned models LM2 and LM3 to construction machinery 100.

[0264] For example, if the machine learning unit 2005C generates or updates the learned model LM3, the distribution unit 2007 distributes the most recently generated or updated learned model LM3 to the construction machinery 100. Furthermore, the distribution unit 2007 can also distribute the latest learned model LM3 from the learned model storage unit 2006 to the construction machinery 100 based on a signal received from the construction machinery 100 requesting the distribution of the learned model LM3.

[0265] Unlike the first example above, the operation support unit 1102 is a functional unit used to provide operation support for the construction machinery 100 that operates through the autonomous operation function.

[0266] The operation support unit 1102 includes a learned model storage unit 1102A, an object perception unit 1102B, a safety control unit 1102C, a work object shape prediction unit 1102E, a safety calculation unit 1102F, and a safety control unit 1102G. Furthermore, unlike the first example described above, the operation support unit 1102 includes a target trajectory generation unit 1102H and a motion control unit 1102I.

[0267] In addition, some or all of the functions of the object sensing unit 1102B, safety control unit 1102C, work object shape prediction unit 1102E, safety calculation unit 1102F, safety control unit 1102G, target track generation unit 1102H, and motion control unit 1102I can also be transferred to the outside of the construction machinery 100 (e.g., information processing device 200).

[0268] The learned models LM2 and LM3, which are distributed from the information processing device 200 and received through the communication device 180, are stored in the learned model storage unit 1102A.

[0269] The target track generation unit 1102H generates a target track for the work area in the predetermined operation of the construction machinery 100 based on data indicating the current state of the work object, and outputs data indicating the target track of the work area. The data indicating the current state of the work object is acquired, for example, from the surrounding monitoring sensor 140. Furthermore, the data indicating the current state of the work object can also be acquired from the sensor group 300 via the communication device 180.

[0270] For example, the target track generation unit 1102H generates the target track for the working part in the predetermined action of the construction machinery 100 using the learned model LM3 based on data representing the shape of the work object. Furthermore, in addition to data representing the shape of the work object, the target track generation unit 1102H can also generate the target track for the working part in the predetermined action of the construction machinery 100 using the learned model LM3 based on data representing the state of the work object.

[0271] Alternatively, the target track generation unit 1102H can also generate a target track for the working part of the construction machinery 100 that matches the state of the work object by applying any known method instead of the learned model LM3. In this case, the training data generation unit 2004C and the machine learning unit 2005C can be omitted. For example, the target track generation unit 1102H can generate data representing the target track of the working part of the construction machinery 100 in a predetermined action by using MPC (Model Predictive Control) based on data representing the state of the current work object. Furthermore, the target track generation unit 1102H can also generate data representing the target track of the working part of the construction machinery 100 in a predetermined action by optimizing a predefined reference track of the working part of the construction machinery 100 based on data representing the state of the work object.

[0272] The motion control unit 1102I causes the construction machinery 100 to perform a predetermined action, so that a predetermined part of the construction machinery 100 moves along a target track generated by the target track generation unit 1102H. Specifically, the motion control unit 1102I controls the actuator 120 while knowing the position of the working part based on the output of the motion monitoring sensor 150, thereby causing the construction machinery 100 to perform a predetermined action, so that the working part of the construction machinery 100 moves along the target track. Thus, the construction machinery 100 can autonomously perform work while executing predetermined actions according to the state of the work object, such as its shape or characteristics.

[0273] The object shape prediction unit 1102E predicts the shape of the work object after a predetermined action of the construction machinery 100 in the near future, based on the target track of the work area generated by the target track generation unit 1102H and the current shape of the work object. Specifically, the object shape prediction unit 1102E obtains data representing the predicted shape of the work object after a predetermined action of the construction machinery 100 in the near future, based on data representing the target track of the work area and data representing the current shape of the work object. Furthermore, the object shape prediction unit 1102E can also predict the shape of the work object and its surrounding objects after a predetermined action of the construction machinery 100 in the near future, based on the target track of the work area generated by the target track generation unit 1102H and the shape of the current work object and its surrounding objects. Specifically, the object shape prediction unit 1102E obtains data representing the predicted shape of the work object and its surrounding objects after a predetermined action of the construction machinery 100 in the near future, based on data representing the target track of the work area and data representing the shape of the current work object and its surrounding objects.

[0274] For example, the object shape prediction unit 1102E uses the learned model LM2 to predict the shape of the work object after changes caused by a predetermined action of the construction machinery 100 in the near future, based on data representing the target track of the work location and data representing the shape of the current work object. Similarly, the object shape prediction unit 1102E can use the learned model LM2 to predict the shape of the work object and its surrounding objects after changes in the near future, based on data representing the target track of the work location and data representing the shape of the current work object and its surrounding objects.

[0275] Similar to the first example above, the safety control unit 1102G performs functional safety-related controls based on the safety level calculated by the safety level calculation unit 1102F.

[0276] For example, when the safety level is relatively low compared to the notification benchmark, the safety control unit 1102G activates the remote notification function. Thus, the control device 110 can, for example, notify a remote monitor that the shape of the work object or its surroundings has changed due to the movement of the construction machinery 100, potentially reducing the safety level. Therefore, the monitor can intervene in the autonomous operation function of the construction machinery 100 to slow down or stop its movement, thereby ensuring the safety of the work object or its surroundings. Furthermore, in addition to activating the remote notification function, the safety control unit 1102G can also activate the external notification function.

[0277] For example, if the autonomous operation function of the construction machinery 100 is deactivated by the intervention of the monitor through remote monitoring after the notification function has started working, or if the construction machinery 100 stops after the deactivation, the safety control unit 1102G stops the notification function.

[0278] Furthermore, when the safety level is relatively low compared to the motion restriction benchmark, the safety control unit 1102G can activate the motion restriction function. Therefore, if the shape of the work object or its surroundings changes due to the autonomous operation of the construction machinery 100, potentially reducing the safety level, the control device 110 can restrict the actions of the construction machinery 100 in response to operating commands based on the autonomous operation function. Thus, the control device 110 can ensure the safety of the work object or its surroundings.

[0279] For example, if the autonomous operation function is deactivated after the motion restriction function has been activated, the safety control unit 1102G will stop the motion restriction function. Thus, the remote monitor can, for example, restart the scheduled operation of the construction machinery 100 based on the autonomous operation function after confirming the safety conditions of the work object and its surroundings. Furthermore, the remote monitor can, for example, restart the scheduled operation of the construction machinery 100 through its own operation after confirming the safety conditions of the work object and its surroundings.

[0280] [Example 2 of handling operational support systems] Next, refer to Figure 9 This section describes the second example of handling the SYS (System for Operation Support) system. Specifically, it describes the handling of... Figure 8 The specific example of processing based on the second example of the operation support system SYS will be explained.

[0281] The following focuses on the first case mentioned above ( Figure 6 Different parts are explained, and sometimes the explanation of the same or corresponding parts as in the first example above is omitted or simplified.

[0282] like Figure 9 As shown, the processing in step S202 is similar to... Figure 6 The process of step S104 is the same, so the explanation is omitted.

[0283] If step S202 is completed, the control device 110 proceeds to step S204.

[0284] In step S204, the target track generation unit 1102H of the control device 110 generates the target track of the working part of the construction machinery 100 and acquires data representing the target track of the working part.

[0285] For example, the data representing the target track of the work site is compared with the track x in equation (1) above. 1：k Or the predicted orbit y can be represented in the same form.

[0286] If step S204 is completed, the control device 110 proceeds to step S206.

[0287] In step S206, the object shape prediction unit 1102E of the control device 110 predicts the shape of the object, etc., based on the data obtained in steps S202 and S204, and obtains data representing the predicted shape.

[0288] If step S206 is completed, the control device 110 proceeds to step S208.

[0289] The processing in steps S208, S210, S212, and S214 is the same as described above. Figure 6 The processes in steps S110, S112, S114, and S116 are the same, so the explanation is omitted.

[0290] [Example of Operation Support System Application in Excavators] Next, refer to Figures 10-15 This paper describes an application example of the Operation Support System (SYS) in the SVL of an excavator.

[0291] <Structure of an excavator> Figure 10 , Figure 11 These are side and top views of an example of an excavator SVL using the SYS operational support system.

[0292] "summary" like Figure 10 , Figure 11 As shown, the excavator SVL, which is a construction machine 100, includes a lower traveling body 1, an upper slewing body 3, an attachment AT including a boom 4, a stick 5, and a bucket 6, and a cab 10. Hereinafter, the front of the excavator SVL corresponds to the direction in which the attachment extends relative to the upper slewing body 3 when viewed from directly above (hereinafter referred to as "top view") along the axis of rotation of the upper slewing body 3. Furthermore, the left and right sides of the excavator SVL correspond to the left and right sides as viewed from the operator's perspective inside the cab 10, respectively.

[0293] The lower traveling body 1 includes, for example, a pair of tracks 1C, left and right. Specifically, the pair of tracks 1C includes a left track 1CL and a right track 1CR. The lower traveling body 1 is hydraulically driven by travel hydraulic motors 1M via each track 1CL and 1CR, enabling the excavator SVL to move. Specifically, the travel hydraulic motors 1M include a travel hydraulic motor 1ML that drives the left track 1CL and a travel hydraulic motor 1MR that drives the right track 1CR.

[0294] The upper rotating body 3 is mounted on the lower traveling body 1 via the rotating mechanism 2. For example, the upper rotating body 3 rotates relative to the lower traveling body 1 by being hydraulically driven by the rotating hydraulic motor 2A through the rotating mechanism 2.

[0295] The AT accessory is equivalent to the working device 125.

[0296] The boom 4 is mounted in the center of the front part of the upper slewing body 3 in a tilting manner. The stick 5 is mounted on the front end of the boom 4 in a rotatable manner, and the bucket 6 is mounted on the front end of the stick 5 in a rotatable manner.

[0297] The bucket 6 is an example of an end attachment, equivalent to the working part of an excavator SVL. The bucket 6 is used, for example, for excavation operations. Furthermore, other end attachments can be installed at the front end of the boom 5 to replace the bucket 6, depending on the work being performed. These other end attachments can be, for example, slope buckets, dredging buckets, or other types of buckets. Moreover, these other end attachments can be types other than buckets such as mixer buckets or hydraulic breakers.

[0298] The boom 4, stick 5, and bucket 6 are hydraulically driven by the boom cylinder 7, stick cylinder 8, and bucket cylinder 9, which are respectively hydraulic actuators.

[0299] The driver's cab 10 is the operator's cab, for example, mounted on the front left side of the upper rotating body 3.

[0300] The excavator SVL includes components such as a hydraulic drive system related to the hydraulic drive of the driven component, an operating system related to the operation of the driven component, a user interface system related to information exchange with the user, a communication system related to communication with the outside world, and a control system related to various controls.

[0301] Hydraulic Drive System like Figure 10 , Figure 11As shown, the hydraulic drive system of the excavator SVL includes a hydraulic actuator 120, which serves as the driven component. The driven components of the excavator SVL include, for example, the tracks 1CL and 1CR of the lower traveling body 1, the upper slewing body 3, the boom 4, the stick 5, and the bucket 6. The hydraulic actuator 120 includes, for example, the traveling hydraulic motors 1ML and 1MR, the slewing hydraulic motor 2A, the boom cylinder 7, the stick cylinder 8, and the bucket cylinder 9. Furthermore, the hydraulic drive system of the excavator SVL includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0302] Engine 11 is the main power source in the hydraulic drive system. Engine 11 is, for example, a diesel engine that uses diesel fuel. Engine 11 is, for example, mounted at the rear of the upper rotating body 3. Engine 11 rotates at a constant speed at a preset target speed under the direct or indirect control of control device 110 to drive the main pump 14 and pilot pump 15.

[0303] The regulator 13 adjusts the discharge volume of the main pump 14 under the control of the control device 110. For example, the regulator 13 adjusts the angle of the swashplate of the main pump 14 (hereinafter referred to as the "deflection angle") according to the control command from the control device 110.

[0304] The main pump 14 supplies working oil to the control valve 17 via a high-pressure hydraulic line. Similar to the engine 11, the main pump 14 is, for example, mounted at the rear of the upper rotating body 3. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable-capacity hydraulic pump, and as described above, under the control of the control device 110, the piston stroke length is adjusted by regulating the deflection angle of the swashplate via the adjuster 13, thereby controlling the discharge flow rate or discharge pressure.

[0305] Control valve 17 controls the hydraulic actuators based on operator input to operating device 130, remote operation commands, or operation instructions related to the automatic operation function. Operation instructions related to the automatic operation function can be output from control device 110 or other control devices. Control valve 17 is, for example, mounted in the center of the upper rotating body 3. Control valve 17 selectively supplies working oil from main pump 14 to multiple hydraulic actuators based on the operation of operating device 130 or operation instructions corresponding to the automatic operation function.

[0306] "operating system" like Figure 10 , Figure 11 As shown, the operating system of the excavator SVL involved in this embodiment includes a pilot pump 15 and an operating device 130.

[0307] Pilot pump 15 supplies pilot pressure to various hydraulic devices via pilot lines. Similar to engine 11, pilot pump 15 is, for example, mounted at the rear of upper rotating body 3. Pilot pump 15 is, for example, a fixed-capacity hydraulic pump, as described above, driven by engine 11.

[0308] Alternatively, the pilot pump 15 can be omitted. In this case, for example, the pilot pressure is supplied to various hydraulic devices via pilot lines by depressurizing the working oil supplied from the main pump 14.

[0309] The operating device 130 is located near the driver's seat in the cab 10 and is used by the operator to operate various driven components. In other words, the operating device 130 is used by the operator to operate the hydraulic actuators that drive each driven component. The operating device 130 includes, for example, lever devices for operating the boom 4 (boom cylinder 7), stick 5 (stick cylinder 8), bucket 6 (bucket cylinder 9), and upper slewing body 3 (slewing hydraulic motor 2A). Furthermore, the operating device 130 includes, for example, pedal devices or lever devices for operating the left and right tracks 1CL and 1CR (travel hydraulic motors 1ML and 1MR) of the lower traveling body 1.

[0310] User Interface Systems like Figure 10 , Figure 11 As shown, the user interface system of the excavator SVL includes an operating device 130, an output device 160, and an input device 170.

[0311] As described above, the output device 160 includes a display device 162 and a sound output device 164.

[0312] Communication Systems like Figure 10 , Figure 11 As shown, the communication system of the excavator SVL involved in this embodiment includes a communication device 180.

[0313] Control Systems like Figure 10 , Figure 11 As shown, the control system of the excavator SVL includes a control device 110, a peripheral monitoring sensor 140, and a motion monitoring sensor 150.

[0314] The control unit 110 performs various controls related to the excavator's SVL.

[0315] The control device 110, for example, uses the aforementioned operating control valve as the controlled object to perform control related to the operation of the hydraulic actuator (driven element) of the excavator SVL.

[0316] For example, the control device 110 can control the operation of the hydraulic actuator (driven element) of the excavator SVL based on the operation of the operating device 130 by using the operating control valve as the controlled object.

[0317] Furthermore, the control device 110 can also control the remote operation of the hydraulic actuator (driven element) of the excavator SVL by using the operating control valve as the control object.

[0318] Furthermore, the control device 110 can also control the automatic operation function of the excavator SVL by using the operating control valve as the control object.

[0319] Furthermore, as described above, the control device 110 performs control related to the operation support system SYS.

[0320] The perimeter monitoring sensor 140 acquires and outputs data representing the status of the area surrounding the excavator SVL. The perimeter monitoring sensor 140 includes a camera device 141 and a distance measuring sensor 142.

[0321] Alternatively, either the camera device 141 or the range sensor 142 can be omitted.

[0322] The camera device 141 acquires video images representing the state of the area surrounding the excavator SVL. The camera device 141 is, for example, a monocular camera. Furthermore, the camera device 141 can be a 3D camera capable of acquiring data representing distance (depth) in addition to acquiring two-dimensional images. When the camera device 141 is a 3D camera, the range sensor 142 can be omitted. The camera device 141 includes cameras 141B, 141F, 141L, and 141R.

[0323] Cameras 141B, 141F, 141L, and 141R are respectively mounted on the upper rear, upper front, upper left, and upper right sides of the upper rotating body 3, capturing images of the rear, front, left, and right sides of the upper rotating body 3. For example, cameras 141B, 141L, and 141R are mounted on the upper surface of the machine room section of the upper rotating body 3. Furthermore, for example, camera 141F is mounted on the upper surface of the cab 10 of the upper rotating body 3. Moreover, cameras 141B, 141F, 141L, and 141R are mounted on the upper part of the upper rotating body 3 with their optical axes pointing diagonally downwards, capturing images of a vertical range extending from the ground near the excavator SVL to the distance beyond the excavator SVL. Furthermore, cameras 141B, 141F, 141L, and 141R can also be configured to have a variable vertical orientation, i.e., a variable rotation angle, depending on the operation of the operator or other user.

[0324] Cameras 141B, 141F, 141L, and 141R output video images at predetermined intervals (e.g., 1 / 30 of a second) during the period from the start of the excavator SVL (i.e., the key switch is "on") to its stop (i.e., the key switch is "off"). The video images output from cameras 141B, 141F, 141L, and 141R are input to control device 110.

[0325] Alternatively, some of cameras 141B, 141F, 141L, and 141R may be omitted. For example, if the operator is seated in the cab 10, at least one of cameras 141F and 141L may be omitted. This is because the operator can relatively easily assess the front and left-side position of the upper rotating body 3 from the cab 10 by looking through the rearview mirror or window.

[0326] The ranging sensor 142 acquires data representing the distance between itself and objects surrounding the excavator SVL. The ranging sensor 142 is, for example, a LiDAR. Furthermore, the ranging sensor 142 can also be a millimeter-wave radar, an ultrasonic sensor, an infrared sensor, etc. The following description focuses on the case where the ranging sensor 142 is a LiDAR. The ranging sensor 142 includes ranging sensors 142BL, 142BR, 142L, and 142R.

[0327] Ranging sensors 142BL, 142BR, 142L, and 142R are respectively mounted on the upper left rear end, upper right rear end, upper left end, and upper right end of the upper rotating body 3. Furthermore, these sensors acquire distance data to objects to the left rear, right rear, left side, and right side of the upper rotating body 3. Additionally, ranging sensors 142BL, 142BR, 142L, and 142R can also be mounted on the upper part of the upper rotating body 3 with the infrared illumination direction reference axis pointing diagonally downwards, and have an infrared illumination range in the vertical direction centered on the ground portion relatively close to the excavator SVL.

[0328] The ranging sensors 142BL, 142BR, 142L, and 142R are, for example, scanning LIDARs, which are three-dimensional laser scanners capable of scanning the direction of infrared laser illumination in both vertical and horizontal directions. Furthermore, the ranging sensors 142BL, 142BR, 142L, and 142R can also be, for example, flash LIDARs that illuminate a large area of ​​three dimensions with infrared light from a light-emitting module and capture the reflected light (infrared light) using a three-dimensional distance imaging element.

[0329] Distance sensors 142BL, 142BR, 142L, and 142R output data or reflections indicating the distance to objects surrounding the excavator SVL at predetermined intervals during the period from the start to the stop of the excavator SVL. The data output from distance sensors 142BL, 142BR, 142L, and 142R is input to control device 110.

[0330] Motion monitoring sensor 150 acquires data representing the motion status of the excavator SVL. Motion monitoring sensor 150 includes sensors 151 to 155.

[0331] Sensor 151 is mounted on boom 4 and measures the posture state of boom 4. Sensor 151 outputs measurement data representing the posture state of boom 4. The posture state of boom 4 is, for example, the posture angle of the base end of the connection between boom 4 and upper rotating body 3 about the rotation axis. Sensor 151 may include, for example, a rotary potentiometer, a rotary encoder, an accelerometer, an angular accelerometer, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. The same applies to sensors 152 to 154. Furthermore, sensor 151 may also include a cylinder sensor that detects the extension and retraction position of boom cylinder 7. The same applies to sensors 152 and 153. The output of sensor 151 (measurement data representing the posture state of boom 4) is input to control device 110. Thus, control device 110 can grasp the posture state of boom 4.

[0332] Sensor 152 is mounted on the boom 5 and measures the attitude state of the boom 5. Sensor 152 outputs measurement data representing the attitude state of the boom 5. The attitude state of the boom 5 is, for example, the attitude angle of the base end of the boom 5 connected to the boom 4 about the axis of rotation. The output of sensor 152 (the measurement data representing the attitude state of the boom 5) is input to control device 110. Thus, control device 110 can grasp the attitude state of the boom 5.

[0333] Sensor 153 is mounted on bucket 6 and measures the posture of bucket 6. Sensor 153 outputs measurement data representing the posture of bucket 6. The posture of bucket 6 is, for example, the posture angle of the base end of the connection between bucket 6 and stick 5 about the axis of rotation. The output of sensor 153 (the measurement data representing the posture of bucket 6) is input to control device 110. Thus, control device 110 can grasp the posture of bucket 6.

[0334] Sensor 154 measures the posture of the excavator SVL's body (e.g., the upper rotating body 3). Sensor 154 outputs measurement data representing the posture of the excavator SVL's body. The posture of the excavator SVL's body is, for example, the tilt state of the body relative to a predetermined reference plane (e.g., a horizontal plane). For example, sensor 154 is mounted on the upper rotating body 3 and measures the tilt angle of the excavator SVL about two axes in the forward and backward directions and the left and right directions. The output of sensor 154 (the measurement data representing the posture of the excavator SVL's body) is input to control device 110. Thus, control device 110 can grasp the posture state (tilt state) of the body (upper rotating body 3).

[0335] Sensor 155 is mounted on the upper rotating body 3 and measures the rotation state of the upper rotating body 3. Sensor 155 outputs measurement data indicating the rotation state of the upper rotating body 3. Sensor 155 measures, for example, the rotational angular velocity or rotational angle of the upper rotating body 3. Sensor 155 may include, for example, a gyroscope sensor, a resolver, and a rotary encoder. The output of sensor 155 (measurement data indicating the rotation state of the upper rotating body 3) is input to control device 110. Thus, control device 110 can grasp the rotation state of the upper rotating body 3, such as the rotational angle.

[0336] The outputs of sensors 151 to 155 are input to the control device 110. As a result, the control device 110 can estimate the position of the bucket 6 at the working part of the front end of the accessory AT based on the outputs of sensors 151 to 155.

[0337] Alternatively, if sensor 154 includes a gyroscope sensor, a six-axis sensor, an IMU, or the like capable of detecting angular velocities around three axes, the rotational state (e.g., rotational angular velocity) of the upper rotating body 3 can be detected based on the detection signal from sensor 154. In this case, sensor 155 can be omitted.

[0338] <Specific examples of the operation of the support system> Figures 12-15 These are the first to fourth examples showing the changes in the shape of the sand around the excavator caused by the excavator's digging action.

[0339] "Case 1" like Figure 12 As shown, in this example, the excavator SVL and the worker W1 around the excavator SVL are working in the area corresponding to the work object of the excavator SVL.

[0340] The operator W1 is positioned in front of the excavator SVL. The object sensing unit 1102B of the control device 110 can detect the operator W1 as the object of monitoring based on the image captured by the camera 141F. However, since the position of the operator W1 is a certain distance away from the upper rotating body 3 of the excavator SVL, the safety control unit 1102C of the control device 110 will not activate the notification function or the action restriction function, and the excavator SVL can continue to operate.

[0341] Here, the SVL excavator moves the bucket 6 along the track F120 according to the operator's operation or through the automatic operation function to excavate the sand and soil of the target object.

[0342] If the excavator SVL performs digging along track F120, the sand around track F120 will be removed, potentially allowing surrounding sand to flow into the area of ​​track F120. Therefore, in this example, the current shape of the sand at the work site changes from F121 to F122. As a result, the shape of the sand at the location of worker W1 changes significantly, potentially causing worker W1 to be washed away by the sand (refer to the blackened arrow in the diagram).

[0343] In this example, the control device 110 predicts the shape F122 of the work object based on the track F120 of the bucket 6, and calculates the safety level of the operator W1's position based on the change from shape F121 to shape F122. Furthermore, since the shape change of the sand at the operator W1's position is relatively large, the safety level is relatively low compared to a predetermined benchmark. Therefore, the safety control unit 1102G of the control device 110 can activate safety functions such as notification functions or motion restriction functions. Thus, the control device 110 can prompt the operator to stop operation based on the activation of the internal notification function or the remote notification function, or slow down or stop the excavator SVL's movement based on the motion restriction function. Therefore, the control device 110 can suppress the change from shape F121 to shape F122 and ensure the safety of the operator W1.

[0344] "Case 2" like Figure 13 As shown, in this example, the excavator SVL is working in the work area at the foot of the slope, and the worker W2 is working on the upper surface of the extension line of the apex of the slope.

[0345] In this example, worker W2 is located in the blind spot of the perimeter monitoring sensor 140 of the excavator SVL. Therefore, even using the output of the perimeter monitoring sensor 140, the object sensing unit 1102B of the control device 110 cannot detect worker W2. On the other hand, the object sensing unit 1102B of the control device 110 can detect worker W2 based on the output of sensor 300-X acquired through the communication device 180. However, since worker W2 is located far away from the excavator SVL, the safety control unit 1102C of the control device 110 will not activate the notification function or the action restriction function, and the excavator SVL can continue to operate.

[0346] Here, the SVL excavator, based on operator instructions or through automatic operation, intends to excavate the sand and soil at the foot of the slope.

[0347] If the SVL excavator digs the sand at the foot of the slope, the sand above the excavated area may collapse and flow downwards. Therefore, in this example, the shape F131 of the work object and the surrounding sand changes to shape F132. As a result, the shape of the sand at the location of worker W2 changes significantly, and worker W2 may be washed away by the sand (refer to the black arrow in the figure).

[0348] In this example, the control device 110 predicts the shape F132 of the work object after the excavator SVL's digging action changes, and calculates the safety level of the operator W2's position based on the change from shape F131 to shape F132. Furthermore, since the change in the shape of the sand at the operator W2's position is relatively large, the safety level is relatively low compared to a predetermined benchmark. Therefore, the safety control unit 1102G of the control device 110 can activate safety functions such as notification functions or motion restriction functions. Thus, the control device 110 can prompt the operator to stop operating based on the activation of the internal notification function or the remote notification function, or slow down or stop the excavator SVL's movement based on the motion restriction function. Therefore, the control device 110 can suppress the change from shape F131 to shape F132 and ensure the safety of the operator W2.

[0349] "Case 3" like Figure 14 As shown, in this example, the excavator SVL operates in the work area on the upper surface of the extension line at the top of the slope, while the operator W4 operates on the lower surface of the extension line at the foot of the slope.

[0350] In this example, worker W4 is located in the blind spot of the perimeter monitoring sensor 140 of the excavator SVL. Therefore, even using the output of the perimeter monitoring sensor 140, the object sensing unit 1102B of the control device 110 cannot detect worker W4. On the other hand, the object sensing unit 1102B of the control device 110 can detect worker W4 based on the output of sensor 300-X acquired through the communication device 180. However, since worker W4 is located far away from the excavator SVL, the safety control unit 1102C of the control device 110 will not activate the notification function or the action restriction function, and the excavator SVL can continue to operate.

[0351] Here, the SVL excavator, based on operator instructions or through automatic operation, intends to excavate the sand at the top of the slope.

[0352] If the SVL excavator digs the sand at the top of the slope, the sand near the top of the slope may collapse and flow down the slope (refer to the black arrow in the diagram). Therefore, in this example, the shape F141 of the work object and the surrounding sand changes to shape F142, and as a result, the sand flowing towards the location of worker W3 may reach that location.

[0353] In this example, the control device 110 predicts the shape F142 of the work object after the excavator SVL's digging action changes, and calculates the safety level of the operator W3's position based on the change from shape F141 to shape F142. Furthermore, since the change in the shape of the sand at the operator W3's position is relatively large, the safety level is relatively low compared to a predetermined benchmark. Therefore, the safety control unit 1102G of the control device 110 can activate safety functions such as notification functions or motion restriction functions. Thus, the control device 110 can prompt the operator to stop operating based on the activation of the internal notification function or the remote notification function, or slow down or stop the excavator SVL's movement based on the motion restriction function. Therefore, the control device 110 can suppress the change from shape F141 to shape F142 and ensure the safety of the operator W3.

[0354] "Case 4" like Figure 15 As shown, in this example, the excavator SVL is working in one work area separated from the obstacle OB buried underground below it, while the operator W4 is working in another work area.

[0355] In this example, the worker W4, the object being monitored, is located in a blind spot of the perimeter monitoring sensor 140 of the excavator SVL due to the influence of the obstacle OB. Therefore, even if the object sensing unit 1102B of the control device 110 uses the output of the perimeter monitoring sensor 140, it cannot detect the worker W4. Consequently, the safety control unit 1102C of the control device 110 will not activate the notification function or the motion restriction function, and the excavator SVL can continue to operate.

[0356] Furthermore, when the output of the sensor 300-X at the construction site can be obtained through the communication device 180, the object sensing unit 1102B of the control device 110 can detect the worker W4 based on the output of the sensor 300-X. However, since the worker W4 is located far away from the excavator SVL, the safety control unit 1102C of the control device 110 will not activate the notification function or the action restriction function, and the excavator SVL can continue to operate.

[0357] Furthermore, in this example, the control device 110 can detect the obstacle OB, which is the object being monitored, based on the image captured by the camera 141F included in the peripheral monitoring sensor 140. However, since the position of the operator W4 is a certain distance away from the upper rotating body 3 of the excavator SVL, the safety control unit 1102C of the control device 110 will not activate the notification function or the motion restriction function, and the excavator SVL can continue to operate.

[0358] Here, the SVL excavator begins digging sand near the base of the obstacle OB, either according to the operator's instructions or through the automatic operation function.

[0359] If the excavator SVL digs sand near the base of obstacle OB, the sand at the base of obstacle OB may flow into the track of the bucket 6 performing the digging action. Therefore, in this example, the current shape F151 of the sand on the work object changes to shape F152. As a result, the fixed state of obstacle OB becomes unstable, and obstacle OB may tip over toward the worker W4 (refer to the black arrow in the figure).

[0360] In this example, the control device 110 predicts the shape F152 of the work object after its shape changes due to the digging action of the excavator SVL, and calculates the safety level of the obstacle OB's location based on the change from shape F151 to shape F152. Furthermore, since the shape change of the sand at the obstacle OB's location is relatively large, the safety level is relatively low compared to a predetermined benchmark. Therefore, the safety control unit 1102G of the control device 110 can activate safety functions such as notification or motion restriction functions. Thus, the control device 110 can prompt the operator to stop operation based on the activation of the internal or remote notification function, or slow down or stop the excavator SVL's movement based on the motion restriction function. Therefore, the control device 110 can suppress the change from shape F151 to shape F152 and ensure the safety of the operator W4.

[0361] [Example of Operational Support System Application in Continuous Ship Unloaders] Next, refer to Figures 16-20 This paper describes an application example of the Operation Support System (SYS) in a continuous unloader (ULD).

[0362] <Structure of a Continuous Ship Unloader> Figures 16-19 This diagram illustrates an example of the application of the SYS (System for Continuous Unloading) system in a ULD (Unmanned Dock). Specifically, Figure 16 This is an overall diagram showing an example of a continuous unloader (ULD). Figures 17-19 These are the front view, side view, and transverse sectional view of the bucket elevator 59.

[0363] like Figure 16 As shown, the continuous unloader ULD (an example of construction machinery) is a so-called bucket elevator type, installed at the quay QY, which continuously unloads (unloads) bulk cargo M from the hold HD of the ship SP docked at the quay QY onto the shore. Bulk cargo M includes, for example, coal, coke, iron ore, etc.

[0364] The quay QY is constructed of reinforced concrete, for example, and two guide rails 53 are arranged on the quay QY parallel to its extension direction, i.e., the length direction of the berthed vessel SP. The continuous unloader ULD is configured to move on the two guide rails 53 and unload cargo from the vessel SP when stopped at a predetermined position.

[0365] The continuous unloader ULD includes a traveling section 52, a slewing body 55, a boom 57, a bucket elevator 59, and an operator's cab 66. The structural assembly including the boom 57 and the bucket elevator 59 corresponds to the aforementioned working device 125.

[0366] The traveling part 52 is configured to move on the two guide rails 53 of the dock QY.

[0367] The rotating body 55 is rotatably mounted on the walking unit 52.

[0368] The boom 57 is configured to extend forward from the slewing body 55 to the sea section from the dock QY to the berthing of the vessel SP, and is configured to be able to undulate relative to the slewing body 55. Specifically, the boom 57 can undulate up and down according to the extension and retraction of the cylinder 65 installed between it and the slewing body 55.

[0369] The bucket elevator 59 is configured to extend downwards from the front end of the boom 57, i.e., towards the ship's SP (hull HD). For example... Figures 16-19 As shown, the bucket elevator 59 has a scraping section 61 at its front end, which scrapes the bulk cargo M by the scraping section 61 and conveys it upward through the bucket 77, and unloads it onto the shore. The scraping section 61 is equivalent to the working part.

[0370] like Figure 18 As shown, the scraping unit 61 can change its tilt angle in the forward and backward direction relative to the hoist body 64. For example, when scraping bulk cargo M directly below the opening OP of the hold HD, the scraping unit 61 is used in a state where it is not tilted in the forward and backward direction relative to the hoist body 64 (the state shown by the solid line in the figure). On the other hand, when scraping bulk cargo M that exists on the inner wall side of the hold HD, the scraping unit 61 is used in a state where it is tilted forward relative to the hoist body 64 (the state shown by the dashed line in the figure).

[0371] A parallel connecting rod 58 is provided between the rotating body 55 and the bucket elevator 59. Through the action of the parallel connecting rod 58, the bucket elevator 59 is configured to maintain a vertical position regardless of the undulation angle of the boom 57. Furthermore, the bucket elevator 59 can move vertically according to the up-and-down movement of the boom 57. A counterweight 63 is supported on the rotating body 55 via a connecting rod extending rearward to the side opposite to the boom 57, and a balance bar 62 is provided to connect the counterweight 63 to the boom 57. Thus, load balance can be maintained between the bucket elevator 59 and the counterweight 63.

[0372] The operator's cab 66 is located at the front of the slewing body 55 (i.e., in the direction in which the boom 57 extends) for the operator to ride in and operate the continuous unloader ULD.

[0373] An operating device 130 is provided in the operator's cab 66, for example. Thus, the operator can use the operating device 130 to operate driven components such as the slewing body 55, the boom 57, and the bucket elevator 59.

[0374] Furthermore, a control device 110 is installed in the control room 66, for example. Also, an output device 160 and an input device 170 are provided in the control room 66, for example.

[0375] When unloading bulk cargo M using the continuous unloader ULD, in addition to the operator in the control room 66, a staff member is also stationed inside the hold HD to communicate the status of the bulk cargo to the operator. Then, through the cooperation of the operator in the control room 66 and the staff member in the hold HD, the continuous unloader ULD performs the unloading operation of bulk cargo M.

[0376] The operator in the control room 66 can operate the continuous unloader ULD while confirming the image information displayed on the display device 162, which shows the status of the opening OP of the hold HD or its interior, and the communication content from the operators regarding the status of the interior of the hold HD.

[0377] like Figure 17 , Figure 18 As shown, the bucket elevator 59 includes an elevator body 64 that extends in the vertical direction and chain buckets 79 that rotate between the upper part 59a and the lower part (scraping part 61) of the bucket elevator 59.

[0378] The chain bucket 79 includes a pair of roller chains 75 and a plurality of buckets 77 supported and suspended on the pair of roller chains 75.

[0379] The roller chain 75 passes through the interior of the elevator body 64 and is connected in a ring shape in a circulating manner between the upper part 59a and the lower part (scraping part 61) of the bucket elevator 59.

[0380] Furthermore, the bucket elevator 59 includes a drive roller 81a that supports the roller chain 75 and driven rollers 81b, 81c and a guide roller 83 that guide the roller chain 75.

[0381] The drive roller 81a is disposed on the upper part 59a of the bucket elevator 59, and the driven rollers 81b and 81c are disposed in the scraping part 61 at predetermined intervals in the front-to-back direction.

[0382] The steering roller 83 is located on the upper part 59a of the bucket elevator 59 below the drive roller 81a and is configured to change the travel direction of the roller chain 75.

[0383] A cylinder 85 is connected between the driven rollers 81b and 81c. The distance between the shafts of the driven rollers 81b and 81c can be changed according to the extension and retraction of the cylinder 85. As a result, the moving and rotating trajectory of the chain bucket 79 can be changed.

[0384] The roller chain 75 is driven by the drive roller 81a and rotates relative to the elevator body 64 in the direction of arrow W. The chain bucket 79 rotates and circulates between the upper part 59a of the bucket elevator 59 and the scraping part 61.

[0385] In the scraping section 61, as the roller chain 75 moves from the driven roller 81b toward the driven roller 81c in a generally horizontal direction, the bucket 77 scrapes the bulk material M from its opening into its interior. The bucket 77, having scraped and contained the bulk material M, rises with its opening facing upwards in response to the rising of the roller chain 75 from the driven roller 81c toward the drive roller 81a. Upon reaching the upper part 59a of the bucket elevator 59, the bucket 77 turns downwards, its opening facing downwards, as the roller chain 75 passes the drive roller 81a. Thus, the bulk material M inside the bucket 77 is conveyed from the discharge chute to the rotary feeder 87 located on the outer periphery of the bucket elevator 59.

[0386] The rotary feeder 87 conveys the bulk material M, which is transported from the bucket 77 via a discharge chute, to the boom conveyor 89 located on the boom 57.

[0387] A boom conveyor 89 is located inside the boom 57. The bulk material M from the rotary feeder 87 of the bucket elevator 59 is transferred to the boom conveyor 89 and conveyed toward the rotating body 55. On the rotating body 55 side, a hopper is provided at the end of the boom conveyor 89, through which the bulk material M conveyed by the boom conveyor 89 is fed to the belt conveyor 93.

[0388] A belt conveyor 93 is installed in the traveling section 52. The belt conveyor 93 transports the bulk cargo M to the ground belt conveyor 95. Thus, the bulk cargo M is moved out to the ground equipment 99 via the ground belt conveyor 95.

[0389] And, as Figures 16-19 As shown, the continuous unloader ULD includes a peripheral monitoring sensor 140 and a motion monitoring sensor 150.

[0390] The perimeter monitoring sensor 140 acquires and outputs data indicating the state of the interior of the hold HD of the bulk cargo M, which is loaded with work objects. The perimeter monitoring sensor 140 includes camera devices 143 and 144.

[0391] A camera device 143 is mounted on the upper part 59a of the bucket elevator 59 and configured to capture images of the interior of the cabin HD from above through the opening OP at the top of the cabin HD of the ship SP. The camera device 143 includes cameras 143A to 143D. Cameras 143A to 143D are evenly spaced in the circumferential direction on the outer peripheral surface of the upper part 59a of the bucket elevator 59 when viewed from above.

[0392] A camera device 144 is mounted on the scraping section 61 at the lower part of the bucket elevator 59 and is configured to capture images of the bulk cargo M inside the hold HD from a relatively close distance. The camera device 144 includes cameras 144A and 144B. Cameras 144A and 144B are located on the left and right outer sides of the fixing part of the scraping section 61. Cameras 144A and 144B are configured with their optical axes pointing downwards, enabling them to capture images of the bulk cargo M from above.

[0393] Cameras 143A-143D and cameras 144A and 144B output video images (image data) at predetermined intervals (e.g., 1 / 30 second) during the period from the start-up to the stop of the continuous unloader ULD. The video images output from cameras 143A-143D and cameras 144A and 144B are input to control device 110.

[0394] Furthermore, in addition to the camera devices 143 and 144, other types of peripheral monitoring sensors 140 (e.g., range sensors) can also be installed in the continuous unloader ULD (bucket elevator 59). For example, one or more range sensors can be installed as peripheral monitoring sensors 140 capable of acquiring point cloud data of the range including the camera range of camera device 143. Similarly, one or more range sensors can be installed as peripheral monitoring sensors 140 capable of acquiring point cloud data of the range including the camera range of camera device 144.

[0395] Motion monitoring sensor 150 acquires data representing the operational status of the continuous unloader (ULD). Motion monitoring sensor 150 includes sensors 156 to 159.

[0396] Sensor 156 is mounted on the rotating body 55 and measures the rotational state of the rotating body 55. Sensor 156 outputs data indicating the rotational state of the rotating body 55. Sensor 156 may include, for example, a rotary potentiometer, a rotary encoder, an accelerometer, an angular accelerometer, a six-axis sensor, etc. The same applies to sensors 157 to 159 below.

[0397] Sensor 157 is mounted on boom 57 and measures the posture of boom 57. Sensor 157 outputs data representing the posture of boom 57.

[0398] Sensor 158 is mounted on the fixing part of scraping part 61 and measures the posture state of scraping part 61. Sensor 158 outputs data indicating the posture state of scraping part 61.

[0399] Sensor 159 outputs data representing the inter-axial distance between the driven rollers 81b and 81c in the scraping section 61. Sensor 158 is, for example, a cylinder sensor that measures the extension and retraction state of the cylinder block 85.

[0400] The outputs of sensors 156 to 159 are input to control device 110. As a result, control device 110 can estimate the position of the front end of scraping section 61, which is the working part, opposite to the bulk material M, based on the outputs of sensors 156 to 159.

[0401] <Specific examples of the operation of the support system> Figure 20 This is a diagram illustrating an example of the change in shape of the bulk cargo M around the scraping section 61 caused by the scraping action of the bulk cargo M in the continuous unloader ULD.

[0402] like Figure 20 As shown, in this example, inside the ship's hold HD, the scraping section 61 of the continuous unloader ULD performs a scraping action on the bulk cargo M, and while the status of the bulk cargo M is confirmed in its vicinity, the operator W5 sends instructions to the operator via wireless communication or the like.

[0403] In this example, the object sensing unit 1102B of the control device 110 can sense the operator W5, which is the object being monitored, based on the image captured by the camera device 143. However, since the position of the operator W5 is a certain distance away from the scraping unit 61, the safety control unit 1102C of the control device 110 will not activate the notification function or the action restriction function, and the continuous unloader ULD can continue to carry out the unloading operation.

[0404] Here, if the scraping part 61 performs a scraping action, the surrounding loose material M may flow into the area scraped by the scraping part 61. Therefore, in this example, the current shape F201 of the loose material M of the work object changes to shape F202. As a result, the shape of the loose material M at the location of the worker W5 changes significantly, which may cause the worker W5 to be swept away by the loose material M (refer to the black arrow in the figure).

[0405] In this example, the control device 110 predicts the shape F202 of the work object after the scraping action of the scraping part 61 changes, and calculates the safety level of the operator W5's position based on the change from shape F201 to shape F202. Furthermore, since the change in shape of the bulk material M at the operator W5's position is relatively large, the safety level is relatively low compared to a predetermined benchmark. Therefore, the safety control unit 1102G of the control device 110 can activate safety functions such as notification or motion restriction functions. Thus, the control device 110 can prompt the operator to stop operating based on the operation of the internal notification function or the remote notification function, or slow down or stop the scraping action of the scraping part 61 based on the motion restriction function. Therefore, the control device 110 can suppress the change from shape F201 to shape F202 and ensure the safety of the operator W5.

[0406] [effect] Next, the functions of the construction machinery, information processing device, and program involved in this embodiment will be explained.

[0407] In the first embodiment of this invention, the construction machinery includes a control device. The construction machinery is, for example, an excavator (SVL) or a continuous unloader (ULD) as described in the construction machinery 100. The control device is, for example, the control device 110 described above. Specifically, the control device acquires information representing the shape of the work object or information representing the shape of the work object and surrounding objects, predicts changes in the shape of the work object or surrounding objects corresponding to the movement of the construction machinery based on this information, and notifies the operator or an external source based on the predicted changes, or restricts the movement of the construction machinery.

[0408] Furthermore, in the first embodiment of this invention, the information processing device acquires information representing the shape of the work object of the construction machinery or information representing the shape of the work object and surrounding objects. Based on this information, it predicts changes in the shape of the work object or surrounding objects corresponding to the movement of the construction machinery. Based on the predicted changes, it can notify the operator or the outside of the construction machinery, or it can restrict the movement of the construction machinery. The information processing device is, for example, the control device 110 or the information processing device 200 described above.

[0409] Furthermore, in the first embodiment of this invention, the program instructs the information processing device to perform the following processing: acquire information representing the shape of the work object of the construction machinery or information representing the shape of the work object and surrounding objects; predict, based on this information, changes in the shape of the work object or surrounding objects corresponding to the movement of the construction machinery; and, based on the predicted change, notify the operator or the outside of the construction machinery, or restrict the movement of the construction machinery. For example, the program is stored in the auxiliary storage device 110A of the control device 110 or the auxiliary storage device 202 of the information processing device 200.

[0410] Therefore, construction machinery or information processing devices (hereinafter referred to as "construction machinery, etc.") can predict changes in the shape of the work object or the like corresponding to the movement of the construction machinery. For example, if a decrease in safety is predictable due to the change in shape, the operator is notified to stop the operation. Similarly, construction machinery, etc., can predict changes in the work object or the like corresponding to the movement of the construction machinery. For example, if a decrease in safety is predictable due to the change in shape, the operator is notified to move the object being monitored from a position where safety is reduced. Similarly, construction machinery, etc., can predict changes in the work object or the like corresponding to the movement of the construction machinery. For example, if a decrease in safety is predictable due to the change in shape, the operator restricts the movement of the construction machinery and suppresses changes in the shape of the work object or the like. Therefore, construction machinery, etc., can further improve the safety around the construction machinery.

[0411] Furthermore, in the second aspect of this embodiment, based on the first aspect described above, the control device or the information processing device (hereinafter referred to as "the control device, etc.") can evaluate the safety of the location based on the prediction result of the change in the location of the object being monitored or the location where the object being monitored may exist within the range of the predicted change, and issue the notification or restrict the operation of the construction machinery based on the evaluation result.

[0412] Therefore, even when the safety of the monitored object may be reduced due to changes in the shape of the work object corresponding to the movement of the construction machinery, the safety of the monitored object can be ensured.

[0413] Furthermore, in the third aspect of this embodiment, based on the second aspect described above, the control device or the like can evaluate the security of the location based on the prediction results of the changes within a predetermined range of the location of the object being monitored or the location where the object being monitored may exist, within the range of the predicted object containing the changes.

[0414] Therefore, even when the shape of construction machinery or other work objects within a predetermined range of the location of the object being monitored may change relatively significantly, the safety of the monitored object can be ensured.

[0415] Furthermore, in the fourth embodiment of this invention, based on any of the first to third embodiments described above, the control device or the like can predict the operation of the construction machinery based on information representing the shape of the work object, and predict the change based on the prediction result of the operation of the construction machinery and information representing the shape of the work object.

[0416] Therefore, construction machinery can predict the movement of the construction machinery based on the shape of the current work object, and predict changes in the shape of the work object based on the prediction results.

[0417] Furthermore, in the fifth embodiment of this invention, based on any of the first to third embodiments described above, the control device can acquire information indicating the operator's operating status of the construction machinery, predict the movement of the construction machinery based on this information and information indicating the shape of the work object, and predict the change based on the prediction result of the movement of the construction machinery and information indicating the shape of the work object.

[0418] Therefore, construction machinery can predict the movement of the construction machinery based on the shape of the work object and the operating status of the construction machinery, and predict the changes in the shape of the work object based on the prediction results.

[0419] Furthermore, in the sixth embodiment of this invention, based on any of the first to third embodiments described above, the construction machinery can have an automatic operation function. Moreover, the control device or the like can acquire information related to the operation plan of the construction machinery based on the automatic operation function, and predict the changes based on this information and information indicating the shape of the work object or information indicating the shape of the work object and surrounding objects. The information related to the operation plan of the construction machinery is, for example, data indicating the target track of the aforementioned work area.

[0420] Therefore, construction machinery and the like can predict changes in the shape of the work object based on the action plan of the construction machinery based on the automatic operation function.

[0421] Furthermore, in the seventh embodiment of this invention, based on any of the first to sixth embodiments described above, the construction machinery may be equipped with a sensor that outputs information representing the shape of the work object or the shape of the work object and its surroundings. The sensor, for example, is the aforementioned perimeter monitoring sensor 140.

[0422] As a result, construction machinery and other equipment can use sensors mounted on the machinery to acquire information representing the shape of the work object or other objects.

[0423] Furthermore, in the eighth embodiment of this invention, based on any of the first to seventh embodiments described above, the construction machinery, etc., can be equipped with a communication device for communicating with the outside. This communication device is, for example, the communication device 180 described above. Moreover, the control device, etc., can obtain information representing the shape of at least one of the work object and its surrounding objects from outside the construction machinery via the communication device. The information representing the shape of at least one of the work object and its surrounding objects obtained from the outside is, for example, the output data of the sensor group 300 described above.

[0424] Therefore, construction machinery and the like can use external sensors to acquire information representing the shape of the work object. Thus, construction machinery and the like can, for example, predict changes in the shape of the work object over a wider range of applications.

[0425] The embodiments have been described in detail above, but the present invention is not limited to this specific embodiment. Various modifications and alterations can be made within the scope of the spirit described in the technical solution.

[0426] Finally, this application claims priority based on Japanese Patent Application No. 2023-204120, filed on December 1, 2023, and the entire contents of the Japanese Patent Application are incorporated herein by reference.

[0427] Symbol Explanation 1-Lower traveling body, 3-Upper slewing body, 4-Boom, 5-Stick, 6-Bucket, 52-Traveling unit, 55-Slewing body, 57-Boom, 59-Bucket elevator, 61-Scraping unit, 100-Construction machinery, 110-Control device, 120-Actuator, 125-Working device, 130-Operating device, 140-Surround monitoring sensor, 141-Camera device, 141B, 141F, 141L, 141R-Camera, 142-Range sensor, 142BL, 142BR, 142L, 142R-Range sensor 143-Camera device, 143A~143D-Camera, 144-Camera device, 144A, 144B-Camera, 150-Motion monitoring sensor, 151~155-Sensor, 156~159-Sensor, 160-Output device, 162-Display device, 164-Sound output device, 170-Input device, 180-Communication device, 200-Information processing device, 300-Sensor group, 300-1~300-M-Sensor, 400-Remote operation support device, 1101-Motion log providing unit, 1 101A - Action Log Recording Department, 1101B - Action Log Storage Department, 1101C - Action Log Sending Department, 1102 - Operation Support Department, 1102A - Model Storage Department, 1102B - Object Perception Department, 1102C - Safety Control Department, 1102D - Track Prediction Department, 1102E - Work Object Shape Prediction Department, 1102F - Safety Calculation Department, 1102G - Safety Control Department, 1102H - Target Track Generation Department, 1102I - Action Control Department, 1103G - Safety Control Department, 2001 - Log Acquisition Department, 200 2-Simulator Department, 2003-Log Storage Department, 2004-Training Data Generation Department, 2004A-Training Data Generation Department, 2004B-Training Data Generation Department, 2004C-Training Data Generation Department, 2005-Machine Learning Department, 2005A-Machine Learning Department, 2005B-Machine Learning Department, 2005C-Machine Learning Department, 2006-Model Storage Department, 2007-Distribution Department, HD-Ship Hold, M-Bulk Cargo, SLV-Excavator, SP-Ships, SVL-Excavator, SYS-Operations Support System, ULD-Continuous Unloader.

Claims

1. A construction machine, wherein, The construction machinery is equipped with a control device that acquires information representing the shape of the work object or the shape of the work object and surrounding objects. Based on this information, the control device predicts changes in the shape of the work object or surrounding objects corresponding to the movement of the construction machinery, and notifies the operator or external parties based on the predicted changes, or restricts the movement of the construction machinery.

2. The construction machinery according to claim 1, wherein, The control device evaluates the safety of a location based on the predicted changes in the location of the monitored object within the predicted range of the change, or the predicted location where the monitored object may exist, and issues the notification or restricts the operation of construction machinery based on the evaluation results.

3. The construction machinery according to claim 2, wherein, The control device evaluates the security of a location based on the prediction results of the changes within a predetermined range of the location of the object being monitored, which includes the predicted object of the change, or the location where the object being monitored may exist.

4. The construction machinery according to any one of claims 1 to 3, wherein, The control device predicts the movement of the construction machinery based on information representing the shape of the work object, and predicts the change based on the prediction result of the movement of the construction machinery and information representing the shape of the work object.

5. The construction machinery according to any one of claims 1 to 3, wherein, The control device acquires information indicating the operator's operating status of the construction machinery, predicts the movement of the construction machinery based on this information and information indicating the shape of the work object, and predicts the change based on the predicted movement of the construction machinery and information indicating the shape of the work object.

6. The construction machinery according to any one of claims 1 to 3, wherein, The construction machinery has an automatic operation function. The control device acquires information related to the action plan of the construction machinery based on the automatic operation function, and predicts the changes based on the information and information representing the shape of the work object or the shape of the work object and its surrounding objects.

7. The construction machinery according to any one of claims 1 to 3, wherein, The construction machinery is equipped with sensors that output information representing the shape of the work object or the shape of the work object and its surrounding objects.

8. The construction machinery according to any one of claims 1 to 3, wherein, The construction machinery is equipped with a communication device for communicating with the outside world. The control device obtains information from the outside, representing the shape of at least one of the work object and its surrounding objects, through the communication device.

9. An information processing apparatus, wherein, The information processing device acquires information representing the shape of the work object of the construction machinery or information representing the shape of the work object and the objects surrounding the work object. Based on this information, it predicts changes in the shape of the work object or the objects surrounding the work object corresponding to the actions of the construction machinery. Based on the predicted changes, it notifies the operator or the outside of the construction machinery, or restricts the actions of the construction machinery.

10. A program in which, The program causes the information processing device to perform the following processing: The system acquires information representing the shape of the work object of the construction machinery or the shape of the work object and surrounding objects. Based on this information, it predicts changes in the shape of the work object or surrounding objects corresponding to the actions of the construction machinery. Based on the predicted changes, it notifies the operator or the outside of the construction machinery, or restricts the actions of the construction machinery.

Images