Mobile working machine

Abstract

The present disclosure relates to mobile work machines. The mobile work machine includes: a frame; a material loading system having a material receiving area and an actuator, the material receiving area being configured to receive material, the actuator being configured to control the material loading system to move the material receiving area relative to the frame; and a control system configured to: receive an indication of a detected object; determine the position of the object relative to the material loading system; and generate a control signal for controlling the mobile work machine based on the determined position.

Description

Technical Field

[0001] This specification generally relates to object detection and control systems for mobile work machines. More specifically, but not by way of limitation, this specification relates to dump body control of mobile work machines using object detection. Background Art

[0002] There are many different types of work machines. These work machines can include construction machines, turf management machines, forestry machines, agricultural machines, and others. Many of these mobile devices have controllable subsystems that include mechanisms controlled by an operator during operation.

[0003] For example, a construction machine can have multiple different mechanical, electrical, hydraulic, pneumatic, and electro - mechanical subsystems, among others, all of which can be operated by an operator. Generally, the tasks of a construction machine often involve transporting materials on a construction site or transporting materials into or out of a construction site according to construction operations at the site. Different construction operations can include moving materials from one location to another or leveling the construction site, etc. During construction operations, various construction machines can be used, including articulated dump trucks, wheel loaders, excavators, boom cranes, and others.

[0004] The above discussion is provided only for general background information and is not intended to assist in determining the scope of the claimed subject matter. Summary of the Invention

[0005] A mobile work machine includes: a frame; a material loading system having a material receiving area and an actuator, the material receiving area being configured to receive materials, the actuator being configured to control the material loading system to move the material receiving area relative to the frame; and a control system configured to: receive an indication of a detected object; determine the position of the object relative to the material loading system; and generate a control signal for controlling the mobile work machine based on the determined position.

[0006] This summary of the invention is provided to introduce some concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to assist in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in the background. Brief Description of the Drawings

[0007] Figure 1 is a block diagram showing an example of a work machine architecture including a mobile work machine.

[0008] Figure 2An illustration showing an example of a mobile working machine.

[0009] Figure 3 An illustration showing an example of a mobile working machine.

[0010] Figure 4 An illustration showing an example of a mobile working machine.

[0011] Figure 5 A block diagram illustrating an example of an object detection system.

[0012] Figure 6 A flowchart illustrating an example operation of an object detection system.

[0013] Figure 7 Shows an example of a mobile device that can be used in the architecture shown in the previous figures.

[0014] Figure 8 A block diagram showing an example of a computing environment that can be used in the architecture shown in the previous figures. Detailed Description

[0015] Mobile working machines operate at job sites that are often complex, with many other working machines and workers performing many different operations at any given time. Job site operations can involve a large number of steps or phases and can be quite complex. Additionally, job site operations often require precise mechanical control by an operator, and this mechanical control can be highly repetitive. To make matters worse, a typical job site contains a variety of different hazards or obstacles, which are generally referred to herein as objects.

[0016] The more complex a job site becomes, the more difficult it is for the operator of a working machine to complete their tasks while avoiding collisions with objects such as another working machine, hazards above or below the ground, people in the job site, or any other object. For example, the material loading system of a mobile working machine includes a material receiving area (e.g., a dump body such as a bucket, container, etc. on a construction machine), which can be moved by a dump actuator to unload the material receiving area. For such a dumping operation, overhead hazards can include overhead facilities, aircraft, drones, building overhangs, etc. In many cases, an operator cannot easily identify such objects, especially when they are above and / or behind the operator.

[0017] This specification generally relates to an object detection and control system for mobile working machines. More specifically, but not by way of limitation, this specification relates to dump body control of a mobile working machine using object detection.

[0018] Figure 1 FIG. Figure 1 is a block diagram showing an example of a work machine architecture 100 including a mobile work machine 102 (such as a construction machine, examples of which will be discussed below). The work machine 102 includes a control system 104 configured to control a set of controllable subsystems 106 that perform operations at a work site. For example, an operator 108 can interact with and control the work machine 102 through an operator interface mechanism 110. The operator interface mechanism 110 can include things such as a steering wheel, pedals, joysticks, control levers, buttons, dials, linkages, etc. Additionally, they can include a display device that displays user-actuable elements such as icons, links, buttons, etc. In the case where the display device is a touch-sensitive display, those user-actuable items can be actuated by touch gestures. Similarly, in the case where the mechanism 110 includes a voice processing mechanism, the operator 108 can provide input and receive output through a microphone and a speaker, respectively. The operator interface mechanism 110 can include any of a variety of other auditory, visual, or tactile mechanisms.

[0019] The work machine 102 includes a communication system 112 configured to communicate with other systems or machines in the architecture 100. For example, the communication system 112 can communicate with other local machines, such as other machines operating at the same work site as the work machine 102. In the illustrated example, the communication system 112 is configured to communicate with one or more remote systems 114 through a network 116. The network 116 can be any of a variety of different types of networks. For example, it can be a wide area network, a local area network, a near field communication network, a cellular communication network, or any of a variety of other networks, or a combination of networks.

[0020] A remote user 118 is illustrated as interacting with the remote system 114, such as to receive information from or send information to the work machine 102 through the communication system 112. For example but not limited to, the remote user 118 can receive information from the work machine 102, such as notifications, help requests, etc., using a mobile device.

[0021] Figure 1 Also shown is that the work machine 102 includes one or more processors 122, one or more sensors 124, an object detection system 126, a data memory 128, and may also include other items 130. Depending on the type of work machine 102, the sensors 124 can include any of a variety of sensors. For example, the sensors 124 can include object detection sensors 132, material sensors 134, position / route sensors 136, speed sensors 138, work site imaging sensors 140, and can also include other sensors 142.

[0022] The material sensor 134 is configured to sense the material being moved, processed, or otherwise handled by the work machine 102. The position / route sensor 136 is configured to confirm the position of the work machine 102 and the corresponding route (e.g., forward direction) of the work machine 102 as the work machine 102 traverses the work site. The sensor 136 includes sensors configured to generate signals indicative of the angle or turning radius of the machine 102. This may include, but is not limited to, steering angle sensors, articulation angle sensors, wheel speed sensors, differential drive signals, gyroscopes, to name a few.

[0023] The speed sensor 138 is configured to output a signal indicative of the speed of the work machine 102. The work site imaging sensor 140 is configured to obtain an image of the work site, and the image can be processed to identify objects or conditions at the work site. Examples of the imaging sensor 140 include, but are not limited to, one or more cameras (e.g., monocular cameras, stereo cameras, etc.) that obtain a still image of an area of the work site, a time series of images, and / or a video feed. For example, the field of view (FOV) of the camera can include any area of interest at the work site. This can include areas above and / or behind the machine 102 that would otherwise be invisible to the operator 108 when in the operator's compartment or cab of the machine 102.

[0024] The camera can include any suitable image acquisition system that includes, but is not limited to, area array devices such as charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) image devices. Additionally, the camera can be coupled to any suitable optical system to increase or decrease the field of view under the control of the control system 104. Further still, the camera can be provided with additional lighting such as reverse lights or dedicated illuminators so that an image can be easily acquired when operating the excavator under low light conditions. Further still, in one example, multiple cameras are used to provide stereo vision. In this way, using stereo vision techniques, three-dimensional images and visual odometry can be employed.

[0025] The object detection sensor 132 can include an electromagnetic radiation (EMR) sensor (e.g., transmitter, receiver, transceiver) 144. Examples of the EMR sensor include the imaging sensor 140 (discussed above), radio frequency (RF) devices 146 (such as radar), lidar devices 148, and may also include other devices 150. The object detection sensor 132 can also include a sonar device 152 and may also include other devices 154.

[0026] The control system 104 interacts with a data memory 128 (e.g., that stores or retrieves data). The data memory 128 can store various information about the work site and the objects / terrain of the work site. As illustratively shown, the data memory 128 includes object data 156 and can also include other data, as indicated by block 158.

[0027] The object data 156 includes data related to various objects in the work site (e.g., work machine 102, other work machines, operator 108, hazards, etc.). For example, the object data 156 can include position data identifying the location of the object in the work site. For example, this can include the GPS coordinates or local coordinates of the object and / or the vertical portion of the object relative to the ground. The control system 104 can use the position data of the objects to avoid collisions between the objects.

[0028] In addition, the object data 156 can include the dimensions of the objects in the work site. For example, the physical dimensions of the objects. For example, in one example, the work machine 102 is thirteen feet long, eight feet wide, and nine feet high. The control system 104 can use the dimensions of the objects in the work site to prevent collisions between the objects.

[0029] In addition, the object data 156 can include pose data of the objects in the work site. For example, the pose data of the objects includes the positions and dimensions of their components. For example, the pose data of the work machine 102 can include the positions of various components (e.g., dump body, bucket, etc.) relative to the frame and / or other components. The control system 104 can use the pose data of the objects in the work site to prevent collisions between the objects with greater precision than just standard dimensions. For example, a work machine 102 in one pose at a given location will not cause a collision, but a work machine 102 in a different pose at the same location will cause a collision.

[0030] As illustratively shown, the control system 104 includes setup control logic 160, route control logic 162, position logic 163, mechanical geometry logic 164, collision logic 166, proximity logic 168, display generator logic 170, and it can include other items 172. The controllable subsystem 106 can include a propulsion system 174, a steering system 176, a material handling (or manipulation) system 178, one or more different actuators 180 (e.g., that can be used to change machine settings, machine configurations, etc.), and it can include various other systems 182.

[0031] As illustrated, the material handling system 178 includes a material receiving area 184 and one or more actuators 186 configured to move (e.g., load / unload) the material receiving area. The material receiving area 184 is configured to receive material (e.g., soil, building material, etc.) and can take a variety of different forms depending on the type of machine 102. For example, in the case of a dump truck (e.g., rear dump or side dump truck), the material receiving area 184 includes a pivotable (or otherwise movable) dump body. In the case of an excavator, the material receiving area 184 includes a bucket supported on an articulated arm.

[0032] In one example, the control subsystem 106 may further include an operator interface mechanism 110, such as a display device, an audio output device, a haptic feedback mechanism, and an input mechanism. Examples are discussed in more detail below.

[0033] The setting control logic 160 can control one or more subsystems 106 based on the objects, conditions, and / or characteristics of the work site to change the machine settings. For example, the setting control logic 160 can actuate the actuators 180 and / or 186, which change the operation of the propulsion system 174, the steering system 176, and / or the material handling system 178.

[0034] The route control logic 162 can control the steering system 176. For example, but not limited to, if the object detection system 126 detects an object, the route control logic 162 can control the propulsion system 174 and / or the steering system 176 to avoid the detected object. The position logic 163 determines the position of the machine 102 on the work site.

[0035] The machine geometry logic 164 obtains the physical dimensions of the work machine 102 and its controllable subsystems 106 and the sensor signals from the sensors 124. The machine geometry logic 164 uses this data to determine the pose of the work machine 102 (e.g., the orientation and position of the work machine 102 and all its controllable subsystems 106). The pose of the work machine 102 can be useful in determining whether a part of the work implement 102 will collide with another object.

[0036] The collision logic 166 is operatively or communicatively coupled to the setup control logic 160, the path control logic 162, the position logic 163, the mechanical geometry logic 164, and / or a source of external object positions to collect information for determining whether the work machine 102 will collide with another object. Before the setup control logic 160 and / or the path control logic 162 send a control signal, the collision logic 166 can determine whether a potential action will cause a collision with an external object, and if so, prevent the actuation signal from being sent. For example, the collision logic 166 can simulate the requested actuation and determine whether the actuation will cause an intersection of objects. If so, there will be a collision and the actuation will be prevented.

[0037] Similar to the collision logic 166, the proximity logic 168 is coupled to the setup control logic 160, the path control logic 162, the position logic 163, the mechanical geometry logic 164, and / or a source of external object positions to collect information for determining whether the work machine 102 will collide with another object. However, the proximity logic 168 determines that the work machine 102 is within a threshold distance of an external object and prevents the work machine 102 from moving within the threshold distance of the object. Based on the type of object, the threshold distance can be variable. For example, the threshold distance from an overhead power line can be twenty feet, while the threshold distance to a tree can be five feet.

[0038] The display generator logic 170 illustratively generates a control signal for controlling a display device to generate a user interface display for the operator 108. The display can be an interactive display having a user input mechanism for interacting with the operator 108.

[0039] The object detection system 126 is configured to receive signals from the object detection sensors 132 and, based on those signals, detect objects on the worksite (such as in the path of the machine 12) that are in proximity to the work machine 102. Thus, the object detection system 126 can assist the operator 108 in avoiding objects while moving the work machine 102 along the worksite (e.g., backing up) and / or while moving the receiving area 184 (e.g., while transporting / unloading / dumping materials). Before discussing the object detection system 126 in further detail, an example of a mobile work machine will be referenced Figures 2 to 4 to discuss an example of a mobile work machine.

[0040] As described above, the mobile work machine can take many different forms. Figure 2An example of a mobile work machine 202 in the form of an off-road construction vehicle (exemplarily, a rear dump truck or a rear dump lorry) is illustrated. The machine 202 includes a power head portion 204 and a load carrying portion 206. The power head portion 204 includes a vehicle engine or motor 208, a cab 210, and axles and wheels 212, all of which are coupled to a front frame 214. The load carrying portion 206 includes a dump body 216, a first rear axle and wheels 218, and a second rear axle and wheels 220, all of which are coupled to a rear frame 222. The front frame 214 of the power head portion 204 is coupled to the rear frame 222 of the load carrying portion 206 by an articulation and oscillation joint 224. The articulation joint allows the power head portion 204 and the load carrying portion 206 to pivot relative to each other about a vertical axis to steer the machine 202, and the oscillation joint allows the power head portion 204 and the load carrying portion 206 to rotate relative to each other about a longitudinal axis extending along the length of the machine 202. When in the dump position, the dump body 216 is illustrated in dashed lines as the dump body 216'.

[0041] The machine 202 includes an object detection system 230 (e.g., system 126) and a control system 232 (e.g., system 104). The object detection system 230 detects objects located within the range of the machine 202. In the illustrated example, the system 230 receives signals from an object detection sensor 234 (e.g., sensor 132), which is mounted to detect objects behind and / or above the dump body 216. In one example, components of the system 230 and / or the system 232 communicate via the CAN network of the machine 202.

[0042] The object detection sensor 234 is configured to generate signals indicative of the detected object (generally represented in box 236) and the position of the detected object relative to the dump body 216 (or other parts of the machine 202). For example, the sensor 234 may include an imaging sensor or a camera. Alternatively, or additionally, the sensor 234 may be configured to transmit a detection signal towards the rear of the machine 202 and receive the reflection of the detection signal to detect an object 236 behind the machine 202. In one example, the detection signal includes electromagnetic radiation transmitted to the rear of the machine 200. For example, this may include radio frequency (RF) signals. Some specific examples include radar and long range navigation (LORAN), to name just a few. In other examples, the object detection sensor 234 uses sonar, ultrasound, and also light (e.g., lidar) to image the object. Example lidar systems image objects using ultraviolet light, visible light, and / or near infrared light.

[0043] Of course, other types of object detectors can be utilized. In any case, the object detection system 230 generates outputs indicative of the objects, which the control system 232 can utilize to control the operation of the machinery 202.

[0044] Figure 3 Another example of a mobile work machine 302 in the form of an off-road construction vehicle (exemplarily, a front or wheel loader) is illustrated. The machine 302 includes a cab 304 having a display device 306, ground engaging elements 308 (e.g., wheels), motors 310, speed sensors 312, a frame 314, and a boom assembly 316. The boom assembly 316 includes a boom 318, a boom cylinder 320, a bucket 322, and a bucket cylinder 324. The boom 318 is pivotally coupled to the frame 314 and can be raised and lowered by extending or retracting the boom cylinder 320. Figure 3 The bucket 322 in the unload or dump position 322’ is illustrated.

[0045] The bucket 322 is pivotally coupled to the boom 318 and can be moved by extending or retracting the bucket cylinder 324. During operation, the mobile machine 302 can be controlled by an operator within the cab 304, and the mobile machine 302 can traverse a work site. In one example, each of the motors 310 is exemplarily coupled to the wheels 308 of the mobile machine 302 and is configured to drive the wheels 308. The speed sensors 312 are exemplarily coupled to each of the motors 310 to detect the operating speed of the motors.

[0046] In the illustrated example, the machine 302 includes an articulated body, wherein a front portion 326 is pivotally connected to a rear portion 328 at a pivot joint 330. The articulation angle at the pivot joint 330 can be determined using an articulation sensor, and the articulation angle can be used to determine the path of the machine 302. In another example where the body of the machine 302 is non-articulated, the angle of the front wheels 308 and / or the rear wheels 308 can be rotated relative to the frame.

[0047] The machine 302 includes an object detection system 332 and a control system 334. In one example, systems 332 and 334 are similar to Figure 2 the systems 230 and 232 illustrated in. The object detection system 332 detects objects located within the range of the machine 302. In the illustrated example, the system 332 receives signals from an object detection sensor 336 (e.g., sensor 132), which is mounted to detect an object 338 above (or at least potentially in the path of) the bucket 322.

[0048] Figure 4Another example of a mobile working machine 402 in the form of an off-road construction vehicle (exemplarily, a hydraulic excavator) is illustrated. The machine 402 includes a cabling chamber 404, and the cabling chamber 404 has a cab 406 rotatably provided above a crawler section 408. The cabling chamber 404 can rotate 360 degrees around the crawler section 408 via a rotatable coupling 410. An arm 412 extends from the cabling chamber 404 and can be raised or lowered in the direction indicated by arrow 414 based on the actuation of a hydraulic cylinder 416. A boom 418 is pivotally connected to the arm 412 via a joint 420 and can be moved in the direction of arrow 422 based on the actuation of a hydraulic cylinder 424. A bucket 426 is pivotally connected to the boom 418 at a joint 428 and can rotate around the joint 428 in the direction of arrow 430 based on the actuation of a hydraulic cylinder 432.

[0049] When an operator in the cab 406 needs to move the arm 412, he or she engages appropriate controls, and in one example, a camera that provides a camera image on a display in the cab 406 can be automatically activated. The camera image corresponds to the field of view of an area in which the arm 412 is moving or will move for the operator-commanded movement.

[0050] The machine 302 includes an object detection system 434 and a control system 436. In one example, the systems 434 and 436 are similar to Figure 2 the systems 230 and 232 illustrated in. The object detection system 434 detects objects located within the range of the machine 402. In the illustrated example, the system 434 receives signals from an object detection sensor 438 (e.g., sensor 132), and the object detection sensor 438 is installed to detect an object 440 above (or at least potentially in the path of) the arm 412.

[0051] Figure 5 An example of an object detection system 126 is illustrated. For purposes of illustration, but not by way of limitation, the object detection system 126 will be described in the context of the mobile working machine 102 illustrated in Figure 1 The object detection system 126 includes activation logic 502 configured to activate and control the object detection performed by the system 126. For example, this can be in response to a mode selector 504 determining that the machine 102 has entered a specific mode, and the system 126 will be activated for that specific mode. For example, this can be in response to determining, by sensing operator input and / or machine settings, that the material loading system 178 is being moved or otherwise actuated (or is ready to be moved or actuated).

[0052]

[0053] ​The sensor control logic 506 is configured to control the object detection sensor 132 to detect any objects in the predicted path of the material handling system 178. This may include controlling the imaging sensor (e.g., camera) 140 to acquire an image. In another example, the sensor control logic 506 controls the sensor 144 to send a detection signal and receive the corresponding reflection of the detection signal. The signals from the sensor 132 are used by the object detection logic 508 to detect the presence of objects (e.g., objects 236, 338, 440) at the work site. The object position determination logic 510 is configured to determine the positions of the objects detected by the object detection logic 508.

[0054] In one example, the visual recognition system 512 is configured to perform visual recognition on the acquired image to evaluate the objects detected by the object detection logic 508. Illustratively, the system 512 includes image processing logic 514 and object evaluation logic 516. The image processing logic 514 is configured to perform image processing on the image, and the object evaluation logic 516 is configured to evaluate the objects based on the image processing performed by the object detection logic 514. This may include, but is not limited to, object size detection 518, object shape detection 520, object classification performed by the object classifier 522, and may also include other items 524.

[0055] The path determination logic 526 is configured to determine the path of the material handling system 178 (e.g., the movement of the receiving area 184) and / or the paths of other components of the machine 102, and the control signal generator logic 528 is configured to generate control signals either autonomously or in conjunction with the control system 104. The system 126 is illustrated as having one or more processors 530 and may also include other items 532.

[0056] Figure 6 A flowchart 600 illustrating an example operation of the object detection system 126. For purposes of illustration, but not limitation, it will be described in the context of the mobile work machine 102 shown in Figure 1 below. Figure 6

[0057] In block 602, the startup logic 502 starts object detection. This may be in response to a manual input from the operator 108, such as the operator 108 actuating an input mechanism. This is represented by block 604. Alternatively, or additionally, object detection may be automatically started, such as in response to the logic 502 detecting that the machine 102 has entered a particular operating mode (such as changing to the opposite direction, starting / actuating the material handling system 178 (e.g., the lift dump body 216, the bucket 322, the boom 412, etc.)). This is represented by block 606. Of course, object detection may also be started in other ways. This is represented by block 608.

[0058] ​In block 610, the sensor control logic 506 controls the object detection sensor 132 to detect objects on the construction site. This can include objects at or near the ground plane and objects above the ground plane, such as trees, power lines, building overhangs, building ceilings, etc. This can be done in any of a variety of ways. For example, an image can be obtained from the imaging sensor 140. This is represented by block 612. In another example, a radar transmitter can transmit radar signals, which is represented by block 614. Alternatively, or additionally, the detection signal can include a lidar device. This is represented by block 616. Of course, other types of detections can also be performed. This is represented by block 618.

[0059] The object position determination logic 510 receives the sensor signal (or a representation of the sensor signal) indicating the detected object and determines the position of each object relative to the machine 102. This is represented by block 620.

[0060] In one example of block 620, the logic 510 determines the distance of the object from the sensor mounting position on the machine 102. This is represented by block 622. For example, in Figure 2 the example of, the system 230 determines the distance between the object 236 and the sensor 234.

[0061] The logic 510 can also determine the angle of the detected object relative to the sensor mounting orientation. This is represented by block 624. Using the received information, the logic 510 determines the position of the object relative to the machine 102 and / or the construction site itself. Of course, the position of each object can also be determined in other ways. This is represented by block 626.

[0062] Using this information, the position of each detected object can be associated with the current position and / or the expected movement path of the material loading system 178. This is represented by block 628. For example, the path determination logic 526 can utilize the machine geometry logic 164 to determine the path of the system 178 for a commanded movement. This is represented by block 630. In Figure 2 the example of, this includes the logic 526 determining the path of the dump body 216 as it moves from the loading position to the unloading or dumping position 216'.

[0063] In one example, the association can be determined based on the object location (box 632), object size (box 634), and / or object shape (box 636). Additionally, in one example, an image classifier is applied to detect the type of the object. This is represented by box 638. In one example, the visual recognition system 512 can be configured to apply the image object classifier 522 to determine whether the detected object is a person, a utility line (such as a high-voltage power line), a tree, a building, etc. In one example, the type of the object is used to select a threshold distance for collision avoidance. As mentioned above, if the object is a high-voltage power line, the threshold can be set to twenty feet, and if the object is a tree, the threshold can be set to five feet. Of course, these are for illustrative purposes only. Of course, the position of each object can also be associated with the mechanical position in other ways. This is represented by box 640.

[0064] In box 642, each object is evaluated based on the probability of entering the threshold distance. Based on this determination, a control signal for controlling the machine is generated in box 644. The machine 102 can be controlled in any of a variety of ways. In one example, one or more of the subsystems 106 can be controlled by the control signal generator logic 528 and / or the control system 104. This is represented by box 646.

[0065] For example, the control system 104 can automatically move the material loading system 178 to avoid collision with the object. In another example, the threshold position of the material loading system 178 can be set to avoid collision with the object. For illustrative purposes, but not limited to, in Figure 2 the example, box 646 can set the threshold dump position to seventy-five percent (or other), such that the dump body 216 cannot be raised above the height that will result in contact with the object 236 (or entering the threshold distance from the object 236). Of course, this is for illustrative purposes only. In another example, the propulsion system 174 and / or the steering system 176 can be automatically controlled to avoid contact with the object (e.g., automatically braking or stopping the wheels, stopping the engine, shifting gears, etc.).

[0066] Alternatively, or in addition, the operator interface mechanism can be controlled to present visual, auditory, tactile, or other types of output to the operator 108 indicating the detected object. This is represented by block 648. For example, a visual warning can be generated for the operator 108 on a display device to indicate that the system 178 is approaching the detected object. In one example, the visual indication can include an enhanced video feed from a camera that shows the object in the camera's field of view. The display can be enhanced in any of a variety of ways, such as but not limited to highlighting the object on the display. Alternatively, or in addition, an audible alarm can also be generated and the volume (or other) can be varied based on the detected distance to the object (i.e., the audible alarm gets louder as the machine 102 approaches the object). In another example, tactile feedback in the form of seat and / or steering wheel vibration can be provided to the operator 108.

[0067] Of course, control signals for controlling the machine 102 (and / or other machines / systems) can also be generated in other ways. This is represented by block 650. Block 652 determines whether to continue object detection. This can be done automatically, manually, or otherwise.

[0068] This discussion has referred to processors and servers. In one embodiment, the processors and servers include computer processors having associated memory and timing circuitry not shown separately. They are functional parts of the systems or devices to which they belong and are initiated by those systems or devices, and contribute to the functionality of other components or items in those systems.

[0069] It will be noted that the above discussion has described various different systems, components, and / or logic. It should be understood that such systems, components, and / or logic can include hardware items (such as processors and associated memory or other processing components described below in some cases) that perform the functions associated with those systems, components, and / or logic. Additionally, the systems, components, and / or logic can include software that is loaded into memory and subsequently executed by the processor or server or other computing components as described below. The systems, components, and / or logic can also include different combinations of hardware, software, firmware, etc., some examples of which are described below. These are only some examples of the different structures that can be used to form the above systems, components, and / or logic. Other structures can also be used.

[0070] In addition, a variety of user interface displays have been discussed. They can take various different forms and can be provided with a variety of different user-actuable input mechanisms thereon. For example, the user-actuable input mechanisms can be text boxes, check boxes, icons, links, drop-down menus, search boxes, etc. They can also be actuated in a variety of different ways. For example, a pointing device (such as a trackball or mouse) can be used to actuate them. Hardware buttons, switches, joysticks, or keyboards, thumb switches, or thumb pads can be used to actuate them. Virtual keyboards or other virtual actuators can also be used to actuate them. Additionally, in the case where the screen on which they are displayed is a touch-sensitive screen, touch gestures can be used to actuate them. Additionally, in the case where the device on which they are displayed has a speech recognition component, voice commands can be used to actuate them.

[0071] A variety of data memories have also been discussed. It should be noted that each of them can be divided into multiple data memories. All of them can be local to the system accessing them, all can be remote, or some can be local while others are remote. All of these configurations are contemplated herein.

[0072] In addition, these figures show multiple boxes, and the functions of these multiple boxes are attributed to each box. It will be noted that fewer boxes can be used, so the functions are performed by fewer components. Additionally, more boxes can be used with functions distributed among more components.

[0073] It should also be noted that Figure 1 the components of or some parts of can be provided on a variety of different devices. Some of these devices include servers, desktop computers, laptop computers, tablet computers, or other mobile devices such as handheld computers, mobile phones, smartphones, multimedia players, personal digital assistants, etc.

[0074] Figure 7 is a simplified block diagram of an exemplary example of a handheld or mobile computing device that can be used as a handheld device 16 for a user or customer, in which the present system (or a part thereof) can be deployed. For example, the mobile device can be deployed in the operator's cab of a work machine 102 or deployed as a remote system 114.

[0075] Figure 7 provides an overall block diagram of the components of device 16, which can run Figure 1Some of the components shown, interact with these components, or do both. In device 16, a communication link 13 is provided that allows the handheld device to communicate with other computing devices and, in some embodiments, provides a channel for automatically receiving information (such as by scanning). Examples of communication link 13 include allowing communication via one or more communication protocols such as wireless services that provide cellular access to a network and protocols that provide local wireless connectivity to a network.

[0076] In other examples, applications may be received on a removable Secure Digital (SD) card connected to interface 15. Interface 15 and communication link 13 communicate along bus 19 with processor 17 (which may also implement the processor or server of the previous figures), and bus 19 is also connected to memory 21 and input / output (I / O) components 23 as well as clock 25 and positioning system 27.

[0077] In one example, I / O components 23 are provided to facilitate input and output operations. For various embodiments of device 16, I / O components 23 may include input components such as buttons, touch sensors, optical sensors, microphones, touchscreens, proximity sensors, accelerometers, orientation sensors and output components such as display devices, speakers, and / or printer ports. Other I / O components 23 may also be used.

[0078] Clock 25 illustratively includes a real-time clock component that outputs time and date. Illustratively, it may also provide timing functions for processor 17.

[0079] Illustratively, positioning system 27 includes components that output the current geographical location of device 16. For example, this may include a Global Positioning System (GPS) receiver, LORAN system, dead reckoning system, cellular triangulation system, or other positioning system. For example, it may also include mapping software or navigation software that generates desired maps, navigation routes, and other geographical functions.

[0080] Memory 21 stores operating system 29, network settings 31, applications 33, application configuration settings 35, data storage 37, communication driver 39, and communication configuration settings 41. Memory 21 may include all types of tangible volatile and non-volatile computer-readable storage devices. It may also include (as described below) computer storage media. Memory 21 stores computer-readable instructions that, when executed by processor 17, cause the processor to perform computer-implemented steps or functions in accordance with the instructions. Processor 17 may also be initiated by other components to facilitate its functions.

[0081] Examples of apparatus 16 include, but are not limited to, a smart phone or a tablet computer having a user interface display screen, such as a touch screen or a pen-enabled interface that receives input from a pen or stylus. It may also use a virtual keyboard on the screen. Of course, it may also be attached to a keyboard or other user input device through a suitable attachment mechanism (e.g., a wireless link or a USB port, for example). Additionally, the computer may illustratively receive voice input. Of course, various forms of apparatus 16 are possible.

[0082] Figure 8 is Figure 1 an example of a computing environment in which an element or a portion thereof (e.g.) may be deployed. Referring to Figure 8 , an example system for implementing some embodiments includes a computing device in the form of a computer 810. Components of the computer 810 may include, but are not limited to, a processing unit 820 (which may include a processor or a server from a previous figure), a system memory 830, and a system bus 821 that couples various system components including the system memory to the processing unit 820. The system bus 821 may be any of several types of bus structures using any of a variety of bus architectures, including a memory bus or memory controller, a peripheral bus, and a local bus. With respect to Figure 1 the memory and programs described may be deployed in Figure 8 the corresponding portions of

[0083] The computer 810 generally includes a variety of computer-readable media. Computer-readable media can be any available media that is accessible to the computer 810 and includes both volatile and non-volatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media is different from (and does not include) a modulated data signal or a carrier wave. It includes hardware storage media that includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that is accessible to the computer 810. Communication media may embody computer-readable instructions, data structures, program modules, or other data in a transmission mechanism and includes any information delivery media. The term "modulated data signal" means a signal in which one or more characteristics are set or changed in order to encode information in the signal.

[0084] System memory 830 includes computer storage media in the form of volatile and / or non-volatile memory such as read only memory (ROM) 831 and random access memory (RAM) 832. A basic input / output system 833 (BIOS) is typically stored in ROM 831, and the basic input / output system 833 (BIOS) contains basic routines that help transfer information between elements within computer 810 during startup. RAM 832 typically contains data and / or program modules that are immediately accessible to and / or currently being operated on by processing unit 820. By way of example, and not limitation, Figure 8 operating system 834, application programs 835, other program modules 836, and program data 837 are illustrated.

[0085] Computer 810 may also include other removable / non-removable volatile / non-volatile computer storage media. By way of example only, Figure 8 hard disk drive 841 that reads from and writes to non-removable, non-volatile magnetic media, optical disk drive 855, and non-volatile optical disk 856 are illustrated. Hard disk drive 841 is typically connected to system bus 821 through a non-removable memory interface such as interface 840, and optical disk drive 855 is typically connected to system bus 821 through a removable memory interface such as interface 850.

[0086] Alternatively, or in addition, the functions described herein may be performed, at least in part, by one or more hardware logic components. By way of example, and not limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGA), application specific integrated circuits (e.g., ASIC), application specific standard products (e.g., ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), etc.

[0087] The drives discussed above and illustrated in Figure 8 store computer readable instructions, data structures, program modules, and other data for computer 810. In Figure 8 for example, hard disk drive 841 is illustrated as storing operating system 844, application programs 845, other program modules 846, and program data 847. Note that these components may be the same as or different from operating system 834, application programs 835, other program modules 836, and program data 837.

[0088] A user may input commands and information into computer 810 through input devices such as keyboard 862, microphone 863, and pointing device 861 (such as a mouse, trackball, or touchpad). Other input devices (not shown) may include a joystick, gamepad, satellite dish, scanner, etc. These and other input devices are often connected to processing unit 820 through user input interface 860 coupled to the system bus, but may be connected through other interfaces and bus structures. Visual display 891 or other types of display devices are also connected to system bus 821 through an interface such as video interface 890. In addition to the monitor, the computer may also include other peripheral output devices such as speaker 897 and printer 896, and these peripheral output devices may be connected through output peripheral interface 895.

[0089] Computer 810 operates in a networked environment using a logical connection (such as a local area network - LAN, wide area network - WAN, or controller area network - CAN) with one or more remote computers (such as remote computer 880). When used in a LAN networked environment, computer 810 is connected to LAN 871 through network interface or adapter 870. When used in a WAN networked environment, computer 810 typically includes modem 872 or other devices for establishing communication through a WAN 873 such as the Internet. In a networked environment, program modules may be stored in a remotely stored storage device. For example, Figure 8 illustrates that remote application 885 may reside on remote computer 880.

[0090] It should also be noted that the different examples described herein may be combined in different ways. That is, parts of one or more examples may be combined with parts of one or more other examples. All of these are contemplated herein.

[0091] Example 1 is a mobile working machine, which includes:

[0092] A frame;

[0093] A material loading system having a material receiving area and an actuator, the material receiving area being configured to receive material, the actuator being configured to control the material loading system to move the material receiving area relative to the frame; and

[0094] A control system configured to:

[0095] Receive an indication of a detected object;

[0096] Determine the position of the object relative to the material loading system; and

[0097] Generate a control signal for controlling the mobile working machine based on the determined position.

[0098] Example 2 is a mobile work machine of any or all of the previous examples, wherein the mobile work machine includes a construction machine.

[0099] Example 3 is a mobile work machine of any or all of the previous examples, wherein the construction machine includes a dump truck.

[0100] Example 4 is a mobile work machine of any or all of the previous examples, wherein the material receiving area includes a bucket.

[0101] Example 5 is a mobile work machine of any or all of the previous examples, wherein the construction vehicle includes one of an excavator and a loader.

[0102] Example 6 is a mobile work machine of any or all of the previous examples, wherein the material receiving area includes a tipping body.

[0103] Example 7 is a mobile work machine of any or all of the previous examples, wherein the actuator includes a tipping actuator configured to move the tipping body, and the control signal controls the tipping actuator.

[0104] Example 8 is a mobile work machine of any or all of the previous examples, wherein the control system includes:

[0105] Collision logic configured to receive a commanded movement from an operator and determine whether the commanded movement will result in contact between the material loading system and the detected object; and

[0106] Control logic configured to generate a control signal to prevent contact between the material loading system and the detected object.

[0107] Example 9 is a mobile work machine of any or all of the previous examples, wherein the control signal controls the tipping actuator to limit movement of the material loading system.

[0108] Example 10 is a mobile work machine of any or all of the previous examples, the mobile work machine further including a set of ground engaging elements supported for movement relative to the frame, wherein the control signal controls at least one of a propulsion system and a steering system for controlling the set of ground engaging elements.

[0109] Example 11 is a mobile work machine of any or all of the previous examples, wherein the control signal controls a user interface mechanism to provide an output to the user that indicates the position of the object relative to the material loading system.

[0110] Example 12 is a mobile work machine of any or all of the previous examples, wherein the output includes at least one of the following:

[0111] Display output;

[0112] Auditory output; and

[0113] Tactile output.

[0114] Example 13 is a mobile working machine of any or all of the previous examples, the mobile working machine further comprising:

[0115] An object detection system configured to receive a signal from an object detection sensor and detect an object based on the signal.

[0116] Example 14 is a mobile working machine of any or all of the previous examples, wherein the object detection sensor includes an imaging sensor.

[0117] Example 15 is a mobile working machine of any or all of the previous examples, wherein the signal includes a radio frequency (RF) signal.

[0118] Example 16 is a computer-implemented method for controlling a mobile working machine, the method comprising:

[0119] Receiving an indication of a detected object on a construction site;

[0120] Determining the position of the object relative to a material loading system of the mobile working machine, the material loading system having a material receiving area and an actuator, the material receiving area being configured to receive material, the actuator being configured to control the material loading system to move the material receiving area relative to the frame; and

[0121] Generating a control signal for controlling the mobile working machine based on the determined position.

[0122] Example 17 is a computer-implemented method of any or all of the previous examples, wherein the mobile working machine includes a construction machine, and the method further comprises:

[0123] Receiving a commanded movement from an operator;

[0124] Determining whether the commanded movement will result in contact between the material loading system and the detected object; and

[0125] Generating the control signal to prevent contact between the material loading system and the detected object.

[0126] Example 18 is a control system for a mobile working machine, the control system comprising:

[0127] An object detection system configured to:

[0128] Receiving a signal from an object detection sensor and detecting an object based on the signal; and

[0129] Determining a position of the object relative to a material loading system of the mobile work machine, the material loading system having a material receiving area configured to receive material and an actuator configured to control the material loading system to move the material receiving area relative to the frame;

[0130] Collision logic configured to receive a commanded movement from an operator and determine whether the commanded movement will result in contact between the material loading system and the detected object; and

[0131] Control logic configured to generate a control signal to prevent contact between the material loading system and the detected object.

[0132] Example 19 is a control system of any or all of the previous examples, wherein the mobile work machine includes a construction machine.

[0133] Example 20 is a control system of any or all of the previous examples, wherein the material receiving area includes a dump body and the actuator includes a dump actuator configured to move the dump body, and the control signal controls the dump actuator.

[0134] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the above specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A mobile working machine (102; 202; 302; 402), the mobile working machine comprising: Frames (214, 222; 314); A material loading system (178) having a material receiving area (184; 216; 322; 426) and an actuator (186; 320; 424; 432), the material receiving area being configured to receive material, the actuator being configured to control the material loading system to move the material receiving area relative to the frame; and A control system (104; 232; 334; 436) configured to: Receive an indication of a detected object; Determine the position of the object relative to the material loading system; Define a threshold distance based on the object type of the object; Evaluate the object based on the probability of entering the threshold distance; and Generate a control signal for controlling the mobile working machine based on the probability such that the material receiving area cannot be raised above a height that would result in entering the threshold distance.

2. The mobile working machine according to claim 1, wherein, The mobile working machine includes a construction machine.

3. The mobile working machine according to claim 2, wherein, The construction machine includes a dump truck.

4. The mobile working machine according to claim 2, wherein, The material receiving area includes a bucket.

5. The mobile working machine according to claim 4, wherein, The construction machine includes one of an excavator and a loader.

6. The mobile working machine according to claim 3, wherein, The material receiving area includes a tipping body.

7. The mobile working machine according to claim 6, wherein, The actuator includes a tipping actuator configured to move the tipping body, and the control signal controls the tipping actuator.

8. The mobile working machine according to claim 7, wherein, The control system includes: Collision logic configured to receive a commanded movement from an operator and determine whether the commanded movement will result in contact between the material loading system and the detected object; and Control logic configured to generate the control signal to prevent contact between the material loading system and the detected object.

9. The mobile working machine according to claim 8, wherein, The control signal controls the tipping actuator to limit the movement of the material loading system.

10. The mobile work machine according to claim 1, wherein the mobile work machine further includes a set of ground engaging elements supported to be movable relative to the frame, wherein, The control signal controls at least one of a propulsion system and a steering system for controlling a set of ground engaging elements.

Images