WORK VEHICLE WITH DECISION SUPPORT AND INTERVENTION ZONES BASED ON SPEED AND / OR DISTANCE

Abstract

A computer-implemented system and a method for dynamically adapting warnings based on detected objects outside a work vehicle are provided. Object sensors mounted on the vehicle generate object signals representing detected objects in their respective fields of view, and cameras mounted on the vehicle generate images corresponding to their respective image areas. The images are displayed selectively, along with initial symbols that define, or at least partially define, concentric zones, which are dynamically configured based on vehicle operating characteristics (e.g., driving speed).Distances are determined from respective object sensors to any detected objects, with second characters relating to the selectively generated images and corresponding to a respective intervention state for the objects in the images being displayed, the respective intervention state being determined based on at least the determined distance and the operating characteristics of the work vehicle.

Description

AREA OF REVELATION

[0001] The present disclosure relates to self-driving vehicles, such as work vehicles, having on-board obstacle detection systems, and more particularly to systems and methods for augmenting conventional obstacle detection with speed and / or distance-based decision support and intervention. BACKGROUND

[0002] Work vehicles discussed herein may generally refer to self-propelled vehicles or machines, including, but not limited to, four-wheel drive wheel loaders, excavators, and other equipment that modify the terrain or an equivalent work environment in any way. These self-propelled vehicles or machines may have ground-engaging units with tracks or wheels that support the undercarriage from the ground surface and may further include one or more work tools used to modify the terrain in coordination with the movement of the machine. Various situations arise with such self-propelled vehicles or machines where the human operator must control the movements of the work vehicle position and / or the relative positions of the work tools based on detected obstacles in the work area, which may or may not be visible to the operator.

[0003] It is conventionally known that object detection systems provide the operator with feedback regarding the detected objects in the workspace. When designing an object detection system, it is advantageous to have a system that can distinguish between different types of objects (e.g., moving or transient obstacles such as people or vehicles, or fixed obstacles such as structures or material), as the system may want to respond differently to different types of objects in the environment.

[0004] Conventional object detection systems are also known that use distance-based warnings, which are capable, for example, of alerting an operator if an object is detected to be within a threshold distance from the work vehicle or a specified portion of the work vehicle. SUMMARY OF REVELATION

[0005] The present disclosure provides an improvement to conventional systems, at least in part, by providing dynamic zones for decision support and potential intervention regarding detected objects. For example, a plurality of zones may be created, with warnings in a farthest zone masked if the work vehicle is traveling slowly. One advantage of dynamic (e.g., speed-adapted) zones is that the zones are smaller when traveling slowly, resulting in fewer misdetections or false positives, particularly on congested construction sites, which may often have shorter stopping distances, requiring smaller warning zones.

[0006] Since braking deceleration can be assumed to be constant, all other operating characteristics of a work vehicle, such as stopping distance, stopping time, and time to engage an object (e.g., collision or crossing), are proportional to one another and linear with the speed of the work vehicle. Accordingly, warnings may be permitted to be relayed to the operator based on any one or more of such operating characteristics, further wherein the farthest edge of the farthest zone may be an adjustable parameter scaled with the vehicle speed or related operating characteristics or parameters. For example, a surrounding area relative to the work vehicle may be divided into several concentric zones, which are also scaled linearly or nonlinearly.

[0007] According to a first embodiment, a method is provided for dynamically adapting alerts based on detected objects external to a work vehicle. Object signals are generated by one or more of a plurality of vehicle-mounted object sensors, the object signals representing detected objects within a respective field of view. Images are selectively generated on a display unit, the generated images corresponding to a respective image area for each of a plurality of vehicle-mounted cameras, the respective field of view for each of the plurality of object sensors overlapping with the image area for at least one of the plurality of cameras. First indicia are displayed with respect to the selectively generated images, defining or at least partially based on one or more concentric zones that are dynamically configured based on at least one operating characteristic of the work vehicle.For each object sensor, a distance from the respective object sensor to any one or more of the detected objects in the respective field of view is determined, wherein second indicia are displayed with respect to the selectively generated images, and the displayed second indicia correspond to a respective intervention state for the any one or more of the detected objects in the respective field of view and further within the selectively generated images, wherein the respective intervention state is determined at least based on the determined distance and the at least one operating characteristic of the work vehicle.

[0008] In an exemplary aspect according to the above-mentioned embodiment, the at least one operating characteristic of the work vehicle may include a travel speed of the work vehicle.

[0009] In another exemplary aspect according to the above embodiment, the at least one operating characteristic of the work vehicle may include movement of a work implement that is independently movable with respect to a frame of the work vehicle.

[0010] In another exemplary aspect according to the above embodiment, the at least one operating characteristic of the work vehicle may include an estimated braking distance and / or an estimated braking time.

[0011] In another exemplary aspect according to the above-mentioned embodiment, the respective intervention state may correspond to an estimated time until the work vehicle engages the respective detected object based on at least the determined distance and the at least one operating characteristic of the work vehicle.

[0012] For example, the respective intervention state may correspond to an estimated time until the work vehicle engages the respective detected object, further based on a potential movement state of the respective detected object. If the detected object is, for example, animate or otherwise not necessarily static, where there is uncertainty regarding the potential movement, the respective intervention state may correspond to an estimated time until the work vehicle engages the respective detected object, further based on a detected movement and / or a predicted future position or predicted possible future position of the respective detected object. The displayed second indicia may further comprise, for example, a demarcated area around the respective detected object based on its potential movement state.

[0013] As another example, the respective intervention state may further correspond to a predicted response in the at least one operating characteristic of the work vehicle.

[0014] In another exemplary aspect according to the above-mentioned embodiment, the method may include generating control signals to one or more actuators for adjusting the at least one operating characteristic of the work vehicle based on at least the respective intervention state.

[0015] In another exemplary aspect according to the above-mentioned embodiment, the displayed first indicia may include a plurality of concentric zones, each independently configured based on the at least one operating characteristic. For example, each of the plurality of concentric zones may be non-linearly scaled relative to the other plurality of concentric zones based on changes in the travel speed of the work vehicle over time.

[0016] In another embodiment, a work vehicle disclosed herein includes a plurality of vehicle-mounted object sensors, each configured to generate object signals representing detected objects within a respective field of view, a plurality of vehicle-mounted cameras, each configured to generate image data corresponding to a respective image area, wherein the respective field of view for each of the plurality of object sensors overlaps with the image area for at least one of the plurality of cameras, and a controller coupled to receive the generated object signals and the generated image data. The controller is configured to control the performance of steps in a method according to the above-mentioned embodiment, and optionally any of the described aspects thereof.

[0017] In another embodiment, a system disclosed herein includes one or more processors in communication with a plurality of vehicle-mounted object sensors, each configured to generate object signals representing detected objects within a respective field of view, and a plurality of vehicle-mounted cameras, each configured to generate image data corresponding to a respective image region, wherein the respective field of view for each of the plurality of object sensors overlaps with the image region for at least one of the plurality of cameras. The one or more processors may be configured to control the performance of steps in a method according to the above-referenced embodiment and optionally any of the described aspects thereof.

[0018] Numerous objects, features, and advantages of the embodiments set forth herein will become readily apparent to those skilled in the art upon reading the following disclosure in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a perspective view illustrating an exemplary work vehicle according to the present disclosure. Fig. 2 is a block diagram illustrating an example control system and inputs thereto for the example work vehicle according to the present disclosure. Fig. 3-9 are graphical diagrams illustrating embodiments of a generated visual display for an operator of a work vehicle, including speed and / or distance-adjusted signs, according to the present disclosure. Fig. 10 is a flowchart illustrating one embodiment of a method according to the present disclosure. DETAILED DESCRIPTION

[0019] The implementations disclosed in the above drawings and the following detailed description are not intended to be exhaustive or to limit the present disclosure to those implementations.

[0020] Fig. 1, in a particular embodiment disclosed herein, shows a representative work vehicle 100 in the form of, for example, a four-wheel drive loader with a front-mounted work implement 102 that includes and controllably adjusts the position of a work tool 104, such as a bucket, generally facing away from an operator of the work vehicle 100 for forwardly excavating, carrying, and dumping earth and other materials into a loading area, such as a stationary bin or hopper, or a container integrated with an articulated dump truck.It is within the scope of the present disclosure that the work vehicle 100 may be in the form of various alternative devices or vehicles, typically but not exclusively self-propelled, and which may or may not include a work implement for modifying the nearby terrain and / or carrying material from the terrain to a loading area for loading. Other examples of suitable work tools 104 within the scope of the present disclosure may include, without limitation, knives, pallet forks, bale lifters, augers, harvesters, tillers, mowers, grapples, and the like.

[0021] As in Fig. 1, the work vehicle 100 includes an operator's cab 106, a rear chassis section 108 with rear ground-engaging units 110 (e.g., wheels), and a front chassis section 112 with front ground-engaging units 114 (e.g., wheels). The work implement 102 includes a linkage for adjusting a position of the work tool 104 and a hydraulic cylinder 118 for driving the linkage. The work tool 104 is rotatably coupled to boom arms 120 at respective pivot points 142 on opposite sides of the work tool 104. The hydraulic cylinder 118, which drives the linkage, and the boom arm 120 provide a boom actuator that raises and lowers the work implement 102 relative to the frame (i.e., at least including the rear frame section 108 and the front frame section 112 of the work vehicle 100).

[0022] An articulated joint in one embodiment may allow angular adjustment of the rear frame section 108 relative to the front frame section 112. Hydraulic cylinders 122 allow angular changes between the rear and front frame sections 108 and 112 under hydraulic power derived from conventional hydraulic pumps (not shown).

[0023] A user interface 116 (in Fig. 2) may be located within the operator's cab 106 for use by an operator of the work vehicle 100. The user interface 116 may include, or otherwise be functionally linked to, one or more corresponding user interface tools 208 for input and / or output related to a controller 220, as discussed below. Such user interface tools 208 may include, for example, a plurality of user-selectable touch buttons (e.g., soft buttons) for selecting from a variety of commands or menus, each of which may be selectable via a touchscreen having a display unit 210. Touch buttons respond to a touch and do not include a mechanical component that requires a sufficiently large force to actuate mechanical features.The touchscreen may be a graphical user interface configured to display icons and content from applications of the work vehicle. The display unit 210 may be configured to display still images, moving images, and video content on the touchscreen via one or more different types of displays. The display unit 210 may include, among others, cathode ray tube (CRT) displays, light-emitting diode (LED) displays, and liquid crystal displays (LCD).

[0024] Such an in-vehicle user interface 116 may be provided as part of a vehicle control system 200 or otherwise functionally linked thereto, for example, via a CAN bus arrangement or other equivalent forms of electrical and / or electromechanical signal transmission. Another form of user interface (not shown) may take the form of a display unit generated on a remote (i.e., non-in-vehicle) computing device that can display outputs such as status indications and / or otherwise enable user interaction such as providing inputs to the system. In the context of a remote user interface, data transmission between, for example, the vehicle control system 200 and the user interface may take the form of a wireless communication system and associated components well known in the art.

[0025] The user interface 116 may further include an accelerator pedal, or as separately defined with respect to operator-accessible interface tools, that allows the operator to adjust the speed of the vehicle. In other embodiments, a hand lever provides this function. Other example tools located within or otherwise accessible from the operator cab 106 may include a steering wheel, a plurality of operator-selectable touch buttons configured to allow the operator to control the operation and function of the work vehicle 100, and any accessories or work implements powered by the work vehicle's powertrain, including, for example, the work implement 104.

[0026] As used herein, directions may be referred to with respect to the work vehicle 100 from the perspective of an operator seated in the operator cab 106; the left side of the work vehicle 100 is to the left of such operator, the right side of the work vehicle is to the right of such operator, a front portion (or front) 112 of the work vehicle 100 is the direction such operator is facing, a rear portion (or rear) 108 of the work vehicle 100 is behind such operator, a top of the work vehicle 100 is above such operator, and a bottom of the work vehicle 100 is below such operator.

[0027] As in Fig. 1, the work implement 102 is in a partially lowered, or intermediate, position relative to the ground surface. The work tool 104 may be adjustable relative to the boom arm 120 by activating the cylinder 118, one end of which is coupled to a portion of the work vehicle 100, as understood by one of ordinary skill in the art, and another end of which is operatively connected to an implement link 134. The implement link 134 is rotatably coupled to one end of the cylinder 118 at a pivot point 136. Another end of the implement link 134 is rotatably coupled to a portion of the work implement 104 at a pivot point (not shown). An intermediate portion 138 of the implement link 134 is rotatably coupled to a flange 140 that is fixedly connected to the arm 120. The extension and retraction of the hydraulic cylinder 118 rotates the working tool 104 around the pivot point 142.The hydraulic cylinder 118 and the implement link 134 provide an implement actuator for moving the work tool 104 relative to the boom arm 120.

[0028] One or more position sensors 204 (as in Fig. 2, but not in Fig. 1) may include a kinematic sensor located at or near one or more of the pivot points referenced above, for example, to determine an angle of rotation of the arm 120 relative to the frame 112. In one embodiment, a position sensor 204 may be operatively connected to the arm 120 by a four-bar linkage, as will be understood by one of ordinary skill in the art. In another embodiment, a position sensor 204 may be located at the pivot point 150. As the hydraulic cylinder 118 is retracted and extended, the arm 120 is raised and lowered relative to the ground surface. Another position sensor 204 may be located at or near a rotation axis of the implement link 134 relative to the flange 140. As the hydraulic cylinder 118 is retracted and extended, the work tool 104 rotates about the axis of the pivot point 142.For example, an output of a first position sensor 204 may be used to determine a height of the work tool 104 relative to the ground surface, and an output of another position sensor 204 may be used to determine the inclination of the work tool 104 relative to the arm 120.

[0029] One or more imaging devices 202, such as stereo cameras, may be provided. As in Fig. 1, a first imaging device 202a is provided at a first (e.g., upper) position on the work vehicle 100, and a second imaging device 202b is provided at a second (e.g., lower) position on the work vehicle 100 and relative to the first imaging device 202a. In various embodiments, only one imaging device 202 is provided, or, for example, more than two imaging devices (not shown) may be provided for the work vehicle 100 and implemented alongside or substituted for one or more of the first imaging device 202a and the second imaging device 202b within the scope of the present disclosure.

[0030] In the illustrated embodiment, the first imaging device 202a is located on or near the hydraulic cylinder 118, for example, at or near the pivot point 136, and moves relative to the frame 112 of the work vehicle 100 along with the movement of the work implement 102. Furthermore, in the illustrated embodiment, the second imaging device 202b is located on or near an axle of the work vehicle 100 and remains at a fixed position relative to the frame 112 of the work vehicle 100 regardless of the relative movement of the work implement 102.

[0031] The first imaging device 202a and the second imaging device 202b may be cameras arranged to capture images corresponding to at least respective fields of view 203a, 203b. Positioning the upper imaging device 202a on the work implement 102 may generally be preferred over positioning the upper imaging device 202a on an elevated and static portion of the work vehicle 100, such as above the operator's cab 106, at least because the respective field of view from above the operator's cab 106 is only moderately enhanced over the field of view from within the operator's cab 106 itself, and is significantly more obscured by the work implement 102 at most, if not all, stages of the available motion trajectory.However, it can be appreciated that locations for the first imaging device 202a and the second imaging device 202b are not limited to those illustrated in the figures, and various alternatives may be used for the purposes described in more detail below and within the scope of the present disclosure. In one embodiment, for example, the lower imaging device 202b may be attached to a lower portion of the work tool 102 (e.g., a bucket link mount) such that raising the work tool 102 presents a field of view 203b, or a third imaging device (not shown) may be attached to a lower portion of the work tool 102 (e.g., a bucket link mount) such that raising the work tool 102 presents an additional field of view that can be merged with the field of view 203b when the work tool 102 is sufficiently raised.A fourth imaging device (not shown) may, in one embodiment, be provided on the operator cab 106, for example, to present a different field of view that can be merged with the field of view 203a when the work implement 102 is sufficiently lowered, or for use if the imaging device 202a becomes, for example, no longer functional or otherwise unavailable.

[0032] The imaging devices 202 may include video cameras configured to record an original image stream and transmit corresponding data to the controller 220. Alternatively or additionally, the imaging devices 202 may include one or more of a digital (CCD / CMOS) camera, an infrared camera, a stereo camera, a PMD camera, a time-of-flight / depth sensing camera, high-resolution LiDAR (Light Detection and Ranging) scanners, radar detectors, laser scanners, and the like within the scope of the present disclosure. An orientation and / or location of the one or more imaging devices 202 may vary according to the type of work vehicle 100 and relevant applications.

[0033] Object detection devices 206, in various embodiments, may include the imaging devices 202, supplemental imaging devices, alternative devices including, for example, radar, lidar, etc. The object detection devices 206, which may also include object sensors or perception sensors or may otherwise be identified herein as such, may be generally configured to detect and / or classify the environment of the work vehicle 100, and various examples thereof, in addition to or alternatively to cameras, may include ultrasonic sensors, laser scanners, radar wave transmitters and receivers, thermal sensors, structured light sensors, other optical sensors, and the like.The types and combinations of imaging devices in these contexts may vary depending on the type of work vehicle, work area, and / or application, but may generally be provided and configured to optimize the detection and classification of a material being loaded and unloaded, as well as the working conditions corresponding to at least these working conditions, at least in association with a particular work area (loading, unloading, and associated travel paths) of the work vehicle 100 for a particular application. In some embodiments, the object sensors 206 may include the capability of determining an object position as well as an object distance.In other embodiments, output signals from one or more object sensors 206 are used in combination with output signals from one or more other object sensors 206 and / or output signals from one or more imaging devices 202 to determine the position of an object and a corresponding distance between the object and the work vehicle.

[0034] In various embodiments, a respective field of view for each object sensor 206 may overlap with the respective image area 203 for at least one imaging device 202.

[0035] In one embodiment, an imaging device 202 may include an ultra-wide-angle lens (e.g., a "fisheye" lens) with a sufficiently wide field of view to capture a region of interest at any position along an available motion trajectory (if any) of a component to which the imaging device 202 is attached and to provide image data comprising the region of interest projected onto a plane for image processing functions, as further described elsewhere herein. The imaging devices 202 may be equipped with a zoom lens so that the field of view 203, and accordingly the respective output image data, compensates for, for example, the movement of the position of the imaging device 202 relative to an object or region of interest.Such an embodiment may eliminate or at least reduce the need for data processing after the imaging device 202 to change the size of the field of view, for example, when the scale of the resulting image may otherwise vary depending on the relative heights of the imaging devices as they transition between them during operation, as additionally described below.

[0036] In one embodiment, it may be contemplated that an imaging device 202 is equipped with a movable / rotatable mount such that the corresponding field of view 203 is dynamic to correspond as closely as possible to a detected object or area of ​​interest throughout the movement of the work vehicle 100 or a component to which the imaging device 202 is attached relative to the frame 112.

[0037] Those skilled in the art will appreciate that the image processing functions may be performed discretely within a given imaging device 202 if properly configured, but most, if not all, of the image processing may generally be performed by the controller 220 or another downstream data processor. For example, image data from an imaging device 202 may be provided for three-dimensional point cloud generation, image segmentation, object delineation and classification, and the like, using image processing tools known in the art in combination with the disclosed objectives.

[0038] Further with reference to Fig. 2, the controller 220 may be configured to receive inputs from some or all of various data sources, such as imaging devices 202, one or more position sensors 204, object detection sensors 206, and optionally other data sources (not shown) such as on-board payload sensors, on-board work condition sensors, and external data sensors such as from a remote computing device, the user interface, and / or a machine control system for the work vehicle 100, if separately defined with respect to the controller 220. The hydraulic pressure sensors may provide a pressure input indicative of, for example, the hydraulic pressure in one side of, for example, a hydraulic cylinder 118 of the work implement assembly 102.The other sensors that provide input to the controller 220 may include, for example, sensors coupled to piston-cylinder units to detect their relative hydraulically actuated displacements, imaging devices, and / or any known alternatives known to those skilled in the art to provide position or motion inputs indicative of a position (e.g., angular position or displacement) or motion (e.g., angular velocity or acceleration) of one or more components of the work device 102 and / or the work tool 104.

[0039] In one embodiment, the position sensors 204, as noted above, may include kinematic sensors for tracking a position of the upper imaging device 202a relative to a predetermined region of interest, a position of the implement 102 relative to the frame 112, a component in the implement 102 relative to another component therein, and / or the like. Kinematic sensors may be provided in the form of inertial measurement units (each an IMU), including, for example, a number of sensors, including, but not limited to, accelerometers that measure (among other things) velocity and acceleration, gyroscopes that measure (among other things) angular velocity and angular acceleration, and magnetometers that measure (among other things) the strength and direction of a magnetic field.In general, an accelerometer provides measurements related to (among other things) the force caused by gravity, while a gyroscope provides measurements related to (among other things) the motion of a rigid body. The magnetometer provides measurements of the strength and direction of the magnetic field, related to (among other things) known internal constants or to a known, precisely measured magnetic field. The magnetometer provides measurements of a magnetic field to obtain information about the position or angular orientation of the IMU; similarly to the magnetometer, the gyroscope also provides information about the position or angular orientation of the IMU. Accordingly, the magnetometer can be used instead of the gyroscope or in combination with the gyroscope and complementary to the accelerometer to obtain local information and coordinates about the position, motion, and orientation of the IMU.

[0040] In another embodiment, non-kinematic sensors may be implemented for position detection, such as markers or other machine-readable components affixed or printed on the work vehicle 100 and within the field of view 203 of the imaging device 202. In one example, April tags or an equivalent may be provided so that, depending on how the marker appears in the field of view 203 of the imaging device 202, data processing elements can calculate a distance to the marker and / or an orientation of the marker relative to the imaging device 202 to spatially determine the position of the imaging device 202.As another example, machine learning techniques may be implemented based on inputs for two or more known components of the work vehicle 100, such as a front cab mount and a rear fender, such that the computing units may spatially determine a position of the imaging device 202 based on a distance between the two or more components and their respective positions in the field of view 203 of the imaging device 202.

[0041] A controller 220 for the purposes indicated above may be embodied by or include a processor 212, such as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed and programmed to perform or cause the functions described herein to be performed. A general-purpose processor may be a microprocessor, but alternatively, the processor may be a microcontroller or a state machine, combinations thereof, or the like. A processor may also be implemented as a combination of computing devices, e.g.,as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0042] The controller 220 may further generate control signals for controlling the operation of the respective actuators or signals for indirect control via intermediate control units associated with a work vehicle steering control system 224, an implement control system 226, and / or a work vehicle propulsion control system 228. For example, the controller 220 may generate control signals for controlling the operation of various actuators, such as hydraulic motors or hydraulic piston-cylinder units, and electronic control signals from the controller 220 may actually be received by electro-hydraulic control valves associated with the actuators, such that the electro-hydraulic control valves regulate the flow of hydraulic fluid to and from the respective hydraulic actuators to control their actuation in response to the control signal from the controller 220.

[0043] The controller 220, which is further communicatively coupled to a hydraulic system as a work implement control system 226, may accordingly be configured to operate the work vehicle 100 and to operate a work implement 102 coupled thereto, including, without limitation, a lift mechanism, tilt mechanism, roll mechanism, tilt mechanism, and / or auxiliary mechanisms, for example, as relevant to a given type of work implement or work vehicle application.

[0044] The controller 220, which is further communicatively coupled to a hydraulic system as a steering control system 224 and / or a propulsion control system 228, may be configured to move the work vehicle 100 forward and backward, move the work vehicle left and right, control the travel speed of the work vehicle, etc. The propulsion control system 228 may be embodied as or otherwise include any device or collection of devices (e.g., one or more motors, engines, or the like) capable of providing rotational power to a drivetrain and, if applicable, other components to drive the operation of those components. The drivetrain may be part of the propulsion control system 228 or may, for example, be embodied as any device or collection of devices (e.g.,one or more transmissions, differentials, axles, or the like) capable of transmitting the rotational force provided by the drive control system 228 to the ground engaging units 110, 114 to drive the movement of the work vehicle 100.

[0045] The controller 220 may, in addition to the processor 212, be or include a computer-readable medium 214, a communication unit 216, a data storage 218, such as a database network, and the aforementioned user interface 116 or a control panel with a display 210. It should be understood that the controller 220 described herein may be a single controller having all of the described functionality or may include multiple controllers with the described functionality distributed among the multiple controllers.

[0046] Various operations, steps, or algorithms described in connection with the controller 220 may be embodied directly in hardware, in a computer program product such as a software module executed by the processor 212, or in a combination of both. The computer program product may be located in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, on a hard disk, a removable disk, or any other form of computer-readable medium 220 known in the art. An exemplary computer-readable medium 214 may be coupled to the processor 212 so that the processor 212 can read and write information to the memory / storage medium 218. Alternatively, the medium 218 may be integrated with the processor 212.

[0047] The communication unit 216 may support or provide communication between the controller 220 and external systems or devices, for example, via a wireless communication network, and / or may support or provide a communication interface to internal components of the work vehicle 100. The communication unit may include wireless communication system components (e.g., via a cellular modem, Wi-Fi, Bluetooth, or the like) and / or may include one or more wired communication ports, such as Universal Serial Bus ports.

[0048] The data storage 218 discussed herein, unless otherwise stated, may generally include hardware such as volatile or non-volatile storage devices, drives, memories, or other storage media, as well as one or more databases located thereon. For example, the data storage 218 may store image data from each imaging device 202, position data from at least the position sensors 204, object data from at least the object detection sensors 206, and one or more models developed to correlate any of the relevant data regarding, for example, operating states, object states, intervention states, regions of interest, occlusions in the fields of view for respective imaging devices relative to a region of interest, etc.The data storage 218, in some embodiments, may include a data store physically located at a different location than the work vehicle 100 and the controller 220, and may include any cache memory in a processing device, as well as any storage capacity used as virtual memory, e.g., as stored on a mass storage device or other computer coupled to the controller 220. The mass storage device may include a cache or other data space, which may include databases. The data storage, in other embodiments, is located in the "cloud," for example, with storage located at a remote location that wirelessly provides the stored information to the controller 220.

[0049] To illustrate this, we will now use the Fig. 3-10 illustrates an embodiment of a method 400 according to the present disclosure relating to dynamically adapting alerts based on detected objects external to a work vehicle, for example, with respect to a work vehicle 100 described above. Although an embodiment as described herein and with respect to the illustrated work vehicle 100 in Fig. 1 may typically represent objects and associated alerts in displayed areas of interest in front of the work vehicle, in other embodiments, for example, utilizing alternative work vehicles, the area of ​​interest may be in other directions.

[0050] For example, the method 400 may be performed by the above-mentioned controller 220 alone or in at least partial combination with separate components of a vehicle or machine control unit, mobile user devices, and / or distributed computing elements, such as in a cloud computing environment.

[0051] The illustrated method 400 may begin with a step 410 of inputting or otherwise receiving input signals from various perception sensors or other sensors related to the position, orientation, and / or environment of the work vehicle, such as the sensors 202, 204, 206 described above.

[0052] In another step 420, method 400 includes selecting one or more images for display to an operator or other user associated with the work vehicle. In one embodiment, the displayed image may be associated with a camera directed toward a predetermined area of ​​interest, e.g., toward an area in front of the work vehicle, at least while the work vehicle is moving in a forward direction.

[0053] Alternatively, the displayed image may be associated with a camera determined based on user input, for example, by receiving input signals from the user interface that identify a camera, an area around the work vehicle, a direction relative to the work vehicle, a particular detected object relative to the work vehicle, etc.

[0054] In one embodiment, the displayed image can be dynamically determined based on the detection of an object within a specified area of ​​the work vehicle and within the field of view of an associated camera. For example, a first image can be displayed during "normal" operation, but the display unit can be dynamically switched to display a second image when an object is detected within the field of view of a camera associated with the second image.

[0055] In one embodiment, the displayed image may be determined at least in part based on a particular work condition for the work vehicle. For example, a first image may be displayed during a work condition associated with forward movement of the work vehicle, a second image may be displayed during a work condition associated with reverse movement of the work vehicle, and a third image may be further displayed during a work condition associated with particular movements of a work implement for the work vehicle. The relevant work condition may be determined at least in part based on operating parameters of the work vehicle, such as ground speed or engine speed, user-initiated commands, and / or detected positions and / or orientations of the work vehicle frame, the work implement, the work tool, and / or the like.As one example, it may be determined that a current or predicted work condition corresponds to loading material from the ground surface into a bucket as the work tool or picking up a pallet on the ground with forks as the work tool, where the region of interest may be aligned with a corresponding area at ground level and in front of the work vehicle 100. As another example, it may be determined that a current or predicted work condition corresponds to unloading material from the bucket into a container with a raised edge or placing a pallet on an elevated surface of a flatbed transport vehicle, where the region of interest may be aligned with a corresponding area at a specified height above ground level and in front of the work vehicle 100.Other examples may, of course, be relevant based on the type of work vehicle, the type of work equipment or work tool attached to it, etc.

[0056] In various embodiments described above, selecting an image for display may involve selectively repositioning a single camera to obtain a different selected image for display relative to an original image for display. In other words, instead of switching from a first image provided via a first camera to a second image provided via a second camera, the first camera itself may be repositioned or otherwise modified to switch from a first image output to a second image output based on, for example, a detected object within the field of view of the first camera, or image processing may be performed to zoom in on or otherwise switch from a first display to a second display within the same field of view associated with the same first camera.

[0057] In another step 430, the method 400 may include generating first indicators 310 relating to one or more zones that extend concentrically from a point associated with the work vehicle 100 and are adjusted in size, orientation, coloring, and the like according to a current or predicted speed of the work vehicle, a distance from the work vehicle, a trajectory of the work vehicle, etc. In an exemplary embodiment, as in Fig. 3, the first display characters 310 include lines 310 superimposed on the underlying display image 300 and extending away from the work vehicle according to a current trajectory. The lines 310 may, for example, be generated to approximately represent a path of each respective tire or track of the work vehicle. Additional markings may be provided, for example, as shown in Fig. 3, as tabs or short lines extending inwardly from the lines 310 and substantially perpendicular thereto to define a plurality of zones 312, 314, 316.

[0058] One or more of the concentric zones 312, 314, 316 can be generated with sizes relative to the travel speed of the work vehicle, so that the zones 312, 314, 316 visually decrease or increase in size with respect to corresponding changes in the travel speed of the work vehicle. For example, at least the closest zone 316 is Fig. 3 in response to a slower driving speed of the work vehicle is relatively short, whereas at least the nearest zone 316 in Fig. 4 is relatively long in response to a faster travel speed of the work vehicle. In some embodiments, all of the zones may decrease or increase in size together, for example, linearly with respect to changes in the travel speed of the work vehicle. In other embodiments, one or more of the zones may be dynamically adjusted nonlinearly and / or substantially independently of other zones in response to changes in the travel speed of the work vehicle.

[0059] In some embodiments, one or more of the zones 312, 314, 316 may have sizes determined according to other parameters and depending on a current type of work vehicle, work tool, work area, work condition, and the like. For example, conditions associated with lower visibility or uncertainty regarding the identification of objects in the field of view or lower control of the work vehicle, such as slides or work vehicles with particularly dangerous work tool extensions, may result in longer zones.

[0060] In the Fig. In the exemplary embodiment illustrated in Figure 3, the display area may include, in addition to the display image 300 from at least one of the cameras and the overlaid first display indicia 310, a number of additional user interface tools. A pan tool 318 and / or a zoom tool 320 may enable user-selective transition of the display view, whether within a field of view of a camera, transitioning from one camera to another, or a combination thereof. A status indicator 322 and / or notification bar 324 may dynamically indicate changes in operating conditions, the status of communications with sensors, the presence of objects within a threshold number of zones from the work vehicle, and / or the like. A view reset tab 326 may enable user-selective resetting of the current display view to a previous or default view.

[0061] User-selectable operating conditions may also be specified, and the system may be configured to define the sizes of one or more zones, the thresholds for alerts, and the like, at least in part, based thereon. For example, user selection of a confined space mode tab 328, or various settings in a selectable menu tab 332, as illustrated. Such settings may include user-selectable options for the shape and characteristics of the first indicator symbols 310 and / or the second indicator symbols 340, whether or not to shade zones upon detection of an object therein, whether or not to demarcate potentially moving objects (as further described below), and the like. A mute tab 330 or button may be provided for muting or activating audio alerts.

[0062] The method 400 may further include a step 440 of determining relative distances of detected objects in the display image 300 from the work vehicle. The detection and classification of objects in a field of view and the calculation of distances to the respective objects may be performed using object detection systems known in the art. Preferably, the object detection sensors 206 and the controller 220 are capable not only of accurately detecting the presence or absence of an object in the (actual or projected) work path of the work vehicle, but also of distinguishing between different types of objects (e.g., people, other vehicles, static elements) and providing appropriate feedback.

[0063] In another step 450, the method 400 may include generating second indicators, for example, superimposed on or otherwise displayed alongside the first indicators 310, that indicate or are based on an intervention status for detected objects in the display image 300. The intervention status may be determined based on the detected distance and the travel speed of the work vehicle. The intervention status may further be determined based on a working condition of the work vehicle.

[0064] For example, although the current travel speed and trajectory of the work vehicle may otherwise indicate a potential for collision with the detected object in a period of time, which would result in an intervention status corresponding to an operator warning, the controller 220 may determine based on the work condition that the current travel speed and / or trajectory will not be maintained and therefore at least provisionally calculate a different intervention status.

[0065] In various embodiments, the controller 220 may make this determination based on predetermined rules, such that, for example, the thresholds are fixed with respect to other work conditions, or the controller 220 may learn correlations between the determined work condition and driving patterns for the work vehicle over time, resulting in confidence in the predicted movements of the work vehicle and associated work equipment. This may, for example, avoid false positives regarding generated visual and audio warnings.

[0066] As in Fig. 4, an object 340 was detected and identified as being located in the furthest zone 312, and second indicator characters (in this case coloring or shading, which may depend on the associated intervention status) 342 were displayed over the underlying image in the zone 312. In contrast, as in Fig. 5, an object 340 was detected and identified as being in a second zone 314, or an intermediate zone for that matter, and second indicator characters 342 were displayed over the underlying image in the zone 314.

[0067] In both Fig. 4 and 5, the detected object 340 is appropriately classified as being of a first type, which is static, so that a degree of uncertainty regarding the potential movement of the object 340 is inferred. As in Fig. 6-8, the detected "object" 340 may alternatively be classified as being of a second type with the potential to move independently of the work vehicle, in this case, a pedestrian standing in the furthest generated zone 312. In addition to generating second indicator indicia 342 in association with the detected object 340, determined, for example, based on an intervention status that considers the potential movement and any associated uncertainty, the system may further generate visual indicia 350 indicating, for example, that the detected object 340 is a pedestrian. The visual indicia 350 may correspond to a demarcated area surrounding the detected object 340 and may further consider priority levels based, in part, on proximity and / or previously observed movements of the detected object, in addition to the detection type.

[0068] The method 400 may further include generating alerts based on the determined intervention condition (step 460). Generally speaking, a detected object 340 at a first distance from the work vehicle may result in a first type of alert if the work vehicle is moving slowly, whereas the same object and distance may result in a second type of alert if the work vehicle is moving faster and / or, for example, in relatively unstable work conditions. Similarly, at a first travel speed, a detected object 340 at a first distance from the work vehicle may result in a first type of alert, whereas the same object and distance may result in a second type of alert if the detected object is at a second distance.

[0069] In various embodiments, different warning levels may be generated based on different levels or types of intervention conditions. For example, different levels of audio warnings and / or visual warnings may correspond to the proximity of the relevant zone for a detected object. Different levels of audio warnings and / or visual warnings may be generated based on different detection types (e.g., static versus passerby). Certain warning levels may be muted or otherwise deactivated according to other warning levels, which may, for example, be predetermined, automated for certain work states or conditions, or otherwise manually selectable by a user associated with the work vehicle.In effect, whether or not a warning (or a particular warning level) is generated may vary according to vehicle speed, time to collision or other engagement with the detected object, stopping distance for the work vehicle, stopping time for the work vehicle, predicted movement of the detected object relative to a trajectory of the work vehicle, and the like, as demonstrated at least in part by the generated visual zones and indicators.

[0070] In an example of a visual warning, which may or may not be provided along with an audio warning, Fig. 8 a change in the color, shade, or other visual characteristic of the second indicator symbols 342 for a detected object 340 in the nearest zone 316, as opposed to the second indicator symbols 342 for a similar detected object 340 in the intermediate zone 314 (as in Fig. 7) or in the furthest zone 312 (as in Fig. 6). As further shown in Fig. 9, different second indicator symbols 342a, 342b may be generated to account for one or more detected objects 340 located in or across different zones, with the different second indicator symbols 342a, 342b representing warnings to the viewing user according to, for example, proximity with the work vehicle.

[0071] In one embodiment, alerts (in addition to or instead of audio and / or visual alerts) may include output signals to an automated control feature associated with the work vehicle. For example, in association with a particular intervention condition, controller 220 may generate control signals to steering control unit 224, implement control unit 226, and / or propulsion control unit 228 for automatically steering away from a detected object, automatically braking to avoid collision with the detected object, etc.In one embodiment, automated control features may be implemented after a predetermined window after an initial warning is generated, wherein the operator or other user associated with the work vehicle has an opportunity to respond to the warning but has not yet acted, but before a threshold that may correspond to a necessary amount of time to respond to the detected object (e.g., a calculated braking time, a calculated braking distance, a predicted movement of the detected object, etc., which may further take into account certain working conditions).

[0072] In one embodiment, method 400 may continuously determine whether a different image should be displayed (step 470), for example, based on objects detected in fields of view for respective cameras other than a currently in-use camera, or, for example, a detected or predicted change in the trajectory of the work vehicle. The user may be further authorized, for example, via the user interface, to select individual cameras, deselect individual cameras, toggle between cameras, identify a particular camera associated with a current display image 300, etc. After selecting a new image and returning to step 420, whether automatically or manually initiated, method 400 may continue with the following steps.

[0073] It will be apparent to one of ordinary skill in the art that terms such as "above," "below," "upward," "downward," "upper," "lower," etc., are used descriptively for the figures and do not represent limitations on the scope of the present disclosure, which is defined by the appended claims.

[0074] Terms of degree, such as "generally," "substantially," or "approximately," refer to reasonable ranges outside of a stated value or orientation, as understood by one of ordinary skill in the art, such as general tolerances or positional relationships associated with the manufacture, assembly, and use of the described implementations.

[0075] As used herein, "e.g.," is used to list examples non-exhaustibly and has the same meaning as alternative illustrative phrases such as "including," "including, but not limited to," and "including without limitation." As used herein, lists containing items separated by connectives (e.g., "and") and further preceded by the phrase "one or more of" or "at least one of," unless otherwise limited or modified, indicate configurations or arrangements that may include individual items of the list or any combination thereof. For example, "at least one of A, B, and C" or "one or more of A, B, and C" each indicate the possibilities of only A, only B, only C, or any combination of two or more of A, B, and C (e.g.,A and B; B and C; A and C; or A, B and C).

[0076] It will thus be appreciated that systems, work vehicles, and / or methods in accordance with the present disclosure readily achieve the stated objectives and advantages, as well as the benefits inherent therein. While certain preferred embodiments of the disclosure have been illustrated and described for purposes of illustration, numerous changes in the arrangement and construction of the parts and steps may be made by those skilled in the art, which changes will fall within the scope and spirit of the present disclosure as defined by the appended claims. Any disclosed feature or embodiment may be combined with any of the other disclosed features or embodiments, unless expressly stated otherwise.

Claims

[1] A method (400) for dynamically adapting warnings based on detected objects (340) outside a work vehicle (100), the method comprising: Generating (410) object signals from one or more of a plurality of object sensors (206) mounted on the vehicle, the object signals representing detected objects in a respective field of view; selectively generating (420) images on a display unit (210), the generated images corresponding to a respective image area for each of a plurality of cameras (202) mounted on the vehicle, the respective field of view for each of the plurality of object sensors overlapping with the image area (203) for at least one of the plurality of cameras; Displaying (430) first indicia (310) with respect to the selectively generated images that define or are based at least in part on one or more concentric zones (312, 314, 316) that are dynamically configured based on at least one operating characteristic of the work vehicle; for each generated image, determining (440) a distance to one or more arbitrary detected objects in the respective field of view; and Displaying (450) second indicia (342, 350) with respect to the selectively generated images, wherein the displayed second indicia correspond to a respective intervention state for the one or more of the detected objects in the respective field of view and further within the selectively generated images, wherein the respective intervention state is determined based at least on the determined distance and the at least one operating characteristic of the work vehicle. [2] The method of claim 1, wherein the at least one operating characteristic of the work vehicle comprises a travel speed of the work vehicle. [3] The method of claim 1 or 2, wherein the at least one operating characteristic of the work vehicle comprises a movement of a work implement that is independently movable with respect to a frame of the work vehicle. [4] Method according to one of the preceding claims, wherein the at least one operating characteristic of the work vehicle comprises an estimated braking distance and / or an estimated braking time. [5] Method according to one of the preceding claims, wherein the respective intervention state corresponds to an estimated time until the work vehicle engages with the respective detected object based on the at least determined distance and the at least one operating characteristic of the work vehicle. [6] The method of claim 5, wherein the respective intervention state corresponds to an estimated time until the work vehicle engages the respective detected object further based on a potential motion state of the respective detected object. [7] The method of claim 5 or 6, wherein the respective intervention state corresponds to an estimated time until the work vehicle engages the respective detected object further based on a detected movement and / or a predicted future position of the respective detected object. [8] The method of claim 6, wherein the displayed second indicia comprise a demarcated area around the respective detected object based on its potential motion state. [9] The method of any one of claims 5 to 8, wherein the respective intervention state further corresponds to a predicted response in the at least one operating characteristic of the work vehicle. [10] Method according to one of the preceding claims, further comprising generating control signals to one or more actuators (224, 226, 228) for adjusting the at least one operating characteristic of the work vehicle based on at least the respective intervention state. [11] A method according to any one of the preceding claims, wherein the displayed first indicia comprise a plurality of concentric zones (312, 314, 316) each independently configured based on the at least one operating characteristic. [12] The method of claim 11, wherein each of the plurality of concentric zones is non-linearly scaled with respect to other ones of the plurality of concentric zones based on changes in the travel speed of the work vehicle over time. [13] Work vehicle (100), comprising: a plurality of object sensors (206) mounted on the vehicle, each configured to generate object signals representing detected objects (340) in a respective field of view; a plurality of cameras (202) mounted on the vehicle, each configured to generate image data corresponding to a respective image area, wherein the respective field of view for each of the plurality of object sensors overlaps with the image area (203) for at least one of the plurality of cameras; and a controller (220) coupled to receive the generated object signals and the generated image data and configured to control the performance of steps in a method according to any one of claims 1 to 12. [14] A system (200) for dynamically adjusting warnings based on detected objects (340) outside a work vehicle (100), the system comprising: one or more processors (212) communicatively linked to a plurality of vehicle-mounted object sensors (206), each configured to generate object signals representing detected objects in a respective field of view, and a plurality of vehicle-mounted cameras (202), each configured to generate image data corresponding to a respective image area, wherein the respective field of view for each of the plurality of object sensors overlaps with the image area (203) for at least one of the plurality of cameras; wherein the one or more processors are further configured to control the performance of steps in a method according to any one of claims 1 to 12.

Images