System and method for customized visualization of the surroundings of a self-propelled work vehicle
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DEERE & CO
- Filing Date
- 2021-10-11
- Publication Date
- 2026-06-19
Smart Images
Figure CN114537280B_ABST
Abstract
Description
Technical Field
[0001] This disclosure generally relates to self-propelled work vehicles such as construction machinery and forestry machinery, and more particularly to systems and methods for customized visualization of the surrounding environment of such self-propelled work vehicles. Background Technology
[0002] The self-propelled work vehicles discussed in this article may include, for example, excavators, loaders, forestry machinery, and other equipment that modifies terrain or equivalent working environments in some way. These work vehicles may have tracked or wheeled ground engagement units with underframes supported by the ground, and may also include one or more implementations for modifying the terrain in coordination with the movement of the work vehicle.
[0003] In the field of such work vehicles, there remains a continuous need for solutions that provide operators with better operational awareness. One problem for operators is that the surrounding environment of the work vehicle can only be observed to a limited extent directly from the operator's cab. This difficulty is compounded in the case of front-loading and rear-loading machinery, where the operator's view can be essentially limited to the front or rear, depending on the orientation and position of the work implement. Therefore, the operator may not be able to adequately identify external objects hidden from the work implement in their field of vision from a typical working position. This may be particularly important regarding objects behind the operator's field of vision and within the pivoting range of the machine frame and / or the work implement.
[0004] Some conventional systems include cameras that record images of the area behind the work vehicle and display those images on a display unit located in the operator's cab. Other known systems include cameras mounted to provide a "bird's-eye view" of the machinery when their respective images are stitched together. These systems can help the operator see what's around the vehicle, but they require individual selection of a given image and are limited in scope to specific images captured by cameras at predetermined locations on the work vehicle.
[0005] This can be problematic for situations where an operator wants to view a smaller segment of the entire surrounding (i.e., 360-degree) field of view or manipulate the shape and size of the image against a background. For example, when performing certain mechanical functions, or when an obstacle detection system detects an obstacle, a more detailed or less distorted individual image may be highly desirable. Summary of the Invention
[0006] This disclosure provides enhancements to conventional systems by introducing, in at least part, novel systems and methods that enable simple and intuitive manipulation of displayed images using gestures on a touchscreen interface, and / or automatically change the area of interest of a surround view camera unit based on identified operational conditions.
[0007] In the context of manual operation as described above, certain implementations of the control system disclosed herein may be provided such that a set of natural touch gestures are used to manipulate images generated by the peripheral vision camera unit, including but not limited to: "pinch" movements to change the size and distortion of the peripheral vision image; tapping an area over the entire 360-degree peripheral vision to enter a smaller sub-view corresponding to the position of the "tapping"; and / or sliding a finger (or an equivalent instrument at the junction) left or right to rotate the peripheral vision image around a mechanical pivot.
[0008] In the context of the aforementioned automated operation, certain embodiments of the control systems disclosed herein may also be provided such that if a work vehicle is detected as performing a function, the surrounding view image can be automatically changed to a smaller sub-view of view, which provides a more focused visibility suitable for that function. Some exemplary but non-limiting examples may include: where the work vehicle is reversing straight, the surrounding view is changed to a sub-view of view behind the work vehicle; where, on an excavator, the boom and bucket controls are engaged in digging or trenching mode, and the surrounding view is changed to a sub-view of view in front of the machine, which is focused on the boom and bucket; and / or on an excavator, when the machine is commanded to swing, the surrounding view is changed to a sub-view of view showing the area where the counterweight is swinging.
[0009] In addition to altering the area of interest, in some implementations, the distortion and simulated field of view of the surrounding view image can be automatically manipulated based on detected functionality. Some illustrative but non-limiting examples may include: the simulated field of view becoming larger when the machinery travels faster; the simulated field of view being automatically adjusted based on the instantaneous movement or functionality of attachments or implements, such as increasing the field of view when the excavator boom extends further; and / or the surrounding view image dynamically changing to indicate the identification of specific attachments attached to or otherwise equipped to the work vehicle. Further illustrative but non-limiting examples of the latter may include: when a fork attachment is equipped to a loader, the surrounding view is altered to a sub-view with a simulated field of view and distortion to optimize the view of the fork tips; and / or when a cold planer is mounted on a skid-steer loader, the surrounding view is altered to a sub-view of the rear of the work vehicle that is expected to reverse.
[0010] In some implementations, the control system may also dynamically modify the surrounding view image based on the detected work status or work vehicle cycle. Some exemplary but non-limiting examples may include: if the excavator is detected to be operating in a specific work status such as a 180-degree truck loading with a twelve-second cycle, the surrounding view system may anticipate when the operator will turn the work vehicle and preemptively propose a sub-view of the area the operator will turn to.
[0011] In some implementations, the control system can also dynamically change the region of interest in the surrounding field of view system based on the output from the obstacle detection system. For example, if an obstacle is detected, the surrounding field of view image can be automatically changed to focus on a sub-field of view of the area where the object was detected. If the obstacle detection system outputs the height of the object, the surrounding field of view system can automatically manipulate the image distortion and simulated field of view to give the detected object a less distorted appearance in the image.
[0012] In some implementations, the control system can also selectively and dynamically switch or even lock a sub-field of view of the surrounding view image to an identified or designated object of interest, such as a truck or ditch or a specific individual. For example, the controller can be programmed to identify the object of interest based on image recognition algorithms and specified conditions, or even based on specified input from RFID devices located in the surrounding environment of the work vehicle.
[0013] In one particular and exemplary embodiment, this document provides a method for displaying the surrounding environment of a self-propelled work vehicle including a main frame. One or more images corresponding to the surrounding environment of the work vehicle are received, said one or more images being recorded via corresponding one or more cameras supported by the main frame. The recorded images are processed to map the images to a shape corresponding to a first defined display radius and a first defined display depth. In response to at least one triggering action, one or more display images corresponding to at least one selected area of interest in the surrounding environment of the work vehicle are generated on a display unit.
[0014] In one exemplary aspect of the implementation described above, the generated one or more display images may include both overhead downward facing display images and outward facing display images, or selectively switch between overhead and outward facing display images.
[0015] In another exemplary aspect of the implementation described above, the generated display image may include a top-view display image corresponding to a first defined display radius and a first defined display depth, the method comprising the steps of: generating a top-view display image corresponding to a second defined display radius and a second defined display depth in response to at least a first type of triggering action.
[0016] In addition to the embodiments referred to above and another exemplary aspect of the exemplary aspects, at least the first type of triggering action may include one or more automatically determined operating conditions.
[0017] In another exemplary aspect, also according to the embodiments referenced above and the exemplary aspects, the triggering action of at least the first type may include a detected speed change of the self-propelled work vehicle, wherein the top-view display image is reconfigured to increase the second defined display radius and the second defined display depth in association with an increase in the speed of the self-propelled work vehicle, and the top-view display image is reconfigured to decrease the second defined display radius and the second defined display depth in association with a decrease in the speed of the self-propelled work vehicle.
[0018] In accordance with the embodiments referenced above and another exemplary aspect of the exemplary aspects, the at least one triggering action may include selectively manually engaging a top-down display image via a user interface.
[0019] In accordance with the embodiments and exemplary aspects referenced above, in response to at least a second type of triggering action, the method may further include the step of manipulating a selected area of interest for the generated external view display image.
[0020] For example, at least the second type of triggering action may include a detected configuration change of the self-propelled work vehicle.
[0021] Self-propelled work vehicles may include work implements connected to the main frame, wherein detected configuration changes of the self-propelled work vehicle include detected configuration changes of the work implements.
[0022] The working implement can be controllably movable relative to the main frame, wherein at least the second type of triggering action includes detected movement of the working implement and / or predicted movement of the working implement.
[0023] The movement of the work implements can be predicted based on the determined work cycle of the self-propelled work vehicle.
[0024] As another or alternative example, at least the second type of triggering action may include detected obstacles in the environment surrounding the work vehicle.
[0025] In addition to the embodiments and exemplary aspects referenced above, the method may further include the step of: laterally pivoting an external view display image in response to at least a third type of triggering action.
[0026] In another exemplary aspect, which is also based on the embodiments referred to above and the exemplary aspects, the external view display image may be generated in response to at least a second type of triggering action, instead of the top view display image; and / or the external view display image and the top view display image are displayed independently on the same display unit and modified independently in response to different types of triggering actions.
[0027] In another embodiment, a self-propelled work vehicle may be provided, comprising: a main frame; one or more cameras supported by the main frame and configured to record images of the corresponding surrounding environment of the work vehicle; at least one work implement connected to the main frame and configured to work on the terrain surrounding the work vehicle; and a controller communicatively linked to the one or more cameras and a display unit. The controller may also be configured to direct the execution of method steps according to any of the embodiments referred to above and the associated exemplary aspects.
[0028] Many objects, features, and advantages of the embodiments set forth herein will readily become apparent to those skilled in the art when the following disclosure is read in conjunction with the accompanying drawings. Attached Figure Description
[0029] Figure 1 This is a side view of an exemplary self-propelled work vehicle that includes embodiments of the control systems and methods disclosed herein.
[0030] Figure 2 yes Figure 1 A top view of a self-propelled work vehicle, with multiple cameras associated with the self-propelled work vehicle and represented in an exploded perspective view.
[0031] Figure 3 This is a block diagram illustrating an exemplary implementation of the control system disclosed herein.
[0032] Figure 4 This is a flowchart illustrating an exemplary implementation of the method disclosed herein.
[0033] Figure 5 This is a graphical representation of the simplified surrounding field of view image mapping shape disclosed in this paper.
[0034] Figure 6 It is a graphic representation of a rectangular 360-degree surrounding field of view in the form of rectangular blocks in a top-down view of the surrounding field of view image of the mapping disclosed herein.
[0035] Figures 7A to 7B It is a graphical representation of the horizontal viewing area corresponding to the manual 'tapping' operation, representing the surrounding field of vision.
[0036] Figures 8A to 8C It is a graphical representation of the horizontal viewing area corresponding to the manual 'swipe' operation, representing the surrounding field of vision image. Detailed Implementation
[0037] Now, referring to Figures 1 to 8C This can describe various implementations of systems and methods for custom visualization of the surrounding environment of self-propelled work vehicles.
[0038] In certain embodiments as disclosed herein, Figure 1 A representative self-propelled work vehicle 100, for example in the form of a loader, is shown, which has a forward-mounted work implement 120 for modifying the nearby terrain. Within the scope of this disclosure, the work vehicle 100 may take the form of any other self-propelled vehicle that uses attachment implements to modify the nearby terrain, particularly work vehicles designed for off-highway environments, such as construction or forestry vehicles.
[0039] The illustrated work vehicle 100 includes a main frame 132 supported by a left-side tracked ground engaging unit 122 and a right-side tracked ground engaging unit 124, and at least one travel motor (not shown) for driving the respective ground engaging units. Each of the tracked ground engaging units 122, 124 typically includes a front idler, a drive sprocket, and a track chain extending around the front idler and drive sprocket. The travel motor of each tracked ground engaging unit 122, 124 drives its respective drive sprocket. Each tracked ground engaging unit 122, 124 has a forward direction defined from the drive sprocket toward the front idler. The forward direction of the tracked ground engaging units 122, 124 also defines the forward direction of the underframe, thereby defining the forward direction of the self-propelled work vehicle 100.
[0040] The work implement 120 for the illustrated self-propelled work vehicle 100 includes a front-mounted loader bucket 120 coupled to a boom assembly 102. The loader bucket 120 generally faces away from the operator of the loader 100 and is movably coupled to the main frame 132 via the boom assembly 102 for forward shoveling, transporting, and dumping of soil and other materials. In an alternative embodiment of the self-propelled work vehicle, such as a tracked excavator, the boom assembly 102 may be defined as including at least a boom and a boom 144 pivotally connected to the boom. In this example, the boom is pivotally attached to the main frame 132 to pivot about a generally horizontal axis relative to the main frame 132. A coupling mechanism may be located at the end of the boom assembly 102 and configured to couple to the work implement 120; this coupling mechanism may also be characterized as a work tool, and in various embodiments, the boom assembly 102 may be configured to engage and secure various types and / or sizes of attachment implements 120.
[0041] In other embodiments, depending on the type of self-propelled work vehicle 100, the work implement 120 may be a sweeper, hay bale fork, hay bale hugger, grapple, scraper, debris blower, debris blower, shovel, snowplow, etc., for performing specific tasks.
[0042] The operator's cab can be located on the main frame 132. Both the operator's cab and the boom assembly 102 (or, depending on the type of work vehicle 100, the work implement 120) can be mounted on the main frame 132 such that the operator's cab faces the working direction of the work implement 120. A control console, including a user interface 116, can be located within the operator's cab.
[0043] The user interface 116 described herein can be configured as part of a display unit that graphically displays symbols, data, and other information, and in some embodiments, can also provide other outputs from the system, such as indicator lights, audible alarms, etc. The user interface may also, or alternatively, include various controls or user inputs (e.g., steering wheel, joystick, lever, buttons) 208 for operating the work vehicle 100 (including the operation of the engine, hydraulic cylinders, etc.). Such an in-vehicle user interface can be coupled to the vehicle control system, for example, via a CAN bus or other equivalent form of electrical and / or electromechanical signal transmission. Another form of user interface (not shown) can take the form of a display unit generated on a remote (i.e., non-in-vehicle) computing device, which can display outputs such as status indications and / or otherwise enable user interactions such as providing input to the system. In the context of a remote user interface, data transmission between, for example, the vehicle control system and the user interface can take the form of wireless communication systems and associated components conventionally known in the art.
[0044] like Figure 3 As illustrated schematically, the work vehicle 100 includes a control system 200, which includes a controller 112. The controller 112 may be part of the work vehicle's mechanical control system, or it may be a separate control module. The controller 112 may include a user interface 116 and may optionally be mounted at a control panel in the operator's cab.
[0045] The controller 112 is configured to receive input from some or all of a variety of sources, such as the camera system 202, the machine position detection system 204, and the obstacle detection system 206.
[0046] Reference Figure 2The camera system 202 may include one or more imaging devices, such as a camera 202 mounted on the self-propelled work vehicle 100 and configured to capture images corresponding to the surrounding environment of the self-propelled work vehicle 100. In the illustrated embodiment, from the perspective of the work vehicle 100's operating direction, four cameras 202 are respectively positioned on the front, left, rear, and right sides to record individual image areas of the surrounding environment of the work vehicle 100 from different image recording positions. However, this number and orientation of cameras are merely exemplary and are not limiting to the scope of this disclosure unless specifically indicated herein. The cameras 202 may, in various cases, record specific image areas of the ground surface 138, or alternatively, may be controlled to different positions associated with other image areas. The position and size of the image areas recorded by the respective cameras 202 may depend on the arrangement and orientation of the cameras and camera lens system, particularly the focal length of the camera lenses.
[0047] An exemplary implement position detection system 204 may include: an inertial measurement unit (IMU) mounted to a corresponding component of the implement 120 and / or boom assembly 102 and / or main frame 132, a sensor coupled to a piston-cylinder unit to detect its relative hydraulically actuated extension, or any other known alternative that may be known to those skilled in the art.
[0048] In various implementations, additional sensors can be provided to detect mechanical operating conditions or positioning, such as orientation sensors, global positioning system (GPS) sensors, vehicle speed sensors, vehicle and implement position sensors, etc., and one or more of these sensors may be discrete in nature, and the sensor system may also reference signals provided from the mechanical control system.
[0049] Other sensors may collectively define the obstacle detection system 206, and various examples may include: ultrasonic sensors, laser scanners, radar wave transmitters and receivers, thermal sensors, imaging devices, structured light sensors, and other optical sensors. The type and combination of sensors used for obstacle detection may vary depending on the type of work vehicle, the work area, and / or the application, but they are generally set and configured to optimize the identification of objects near or associated with the work area of the determined vehicle.
[0050] The controller 112 can typically work in conjunction with the user interface 116 referenced above to display various symbols to a human operator. The controller can also generate control signals for controlling the operation of the corresponding actuators, or signals for indirect control via intermediate control units associated with the mechanical steering control system 224, the machine tool control system 226, and / or the mechanical drive control system 228.
[0051] Controller 112 includes or can be associated with processor 212, computer-readable medium 214, communication unit 216, data storage device 218 (e.g., database network), and the aforementioned user interface 116 or control panel with display 210. Input / output devices 208, such as keyboards, joysticks, or other user interface tools, are provided to allow a human operator to input instructions to controller 112. It should be understood that controller 112 described herein may be a single controller having all the described functions, or it may comprise multiple controllers, wherein the described functions are distributed among the multiple controllers.
[0052] The various operations, steps, or algorithms described in conjunction with controller 112 can be implemented directly in hardware, as a computer program product such as a software module executed by processor 212, or a combination of both. The computer program product can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, or any other form of computer-readable medium 214 known in the art. An exemplary computer-readable medium 214 can be coupled to processor 212, allowing processor 212 to read information from and write information to the memory / storage medium 214. In this alternative embodiment, medium 214 can be integrated with processor 212. Processor 212 and medium 214 can reside in an application-specific integrated circuit (ASIC). The ASIC can reside in a user terminal. In this alternative embodiment, processor 212 and medium 214 can reside as discrete components in a user terminal.
[0053] As used herein, the term "processor" 212 may refer to at least general-purpose or special-purpose processing devices and / or logic that can be understood by those skilled in the art, including but not limited to microprocessors, microcontrollers, state machines, etc. The processor 212 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other such configuration.
[0054] Communication unit 216 may support or provide communication between controller 112 and external systems or devices, and / or support or provide communication interfaces regarding internal components of the self-propelled work vehicle 100. The communication unit may include wireless communication system components (e.g., via cellular modem, WiFi, Bluetooth, etc.), and / or may include one or more wired communication terminals, such as Universal Serial Bus ports.
[0055] Unless otherwise stated, the data storage device 218 discussed herein may generally encompass hardware (such as volatile or non-volatile storage devices, drives, memory or other storage media) and one or more databases residing on that hardware.
[0056] Overall reference Figures 4 to 8C The exemplary operating mode can be further described with reference to the aforementioned operating vehicle.
[0057] Now, further reference Figure 4 This can describe an implementation of a method 400 for custom visualization of the surrounding environment of a self-propelled work vehicle 100.
[0058] The illustrated method 400 begins at step 410, which captures an image of the environment surrounding the propulsion work vehicle 100. As previously mentioned, this image can be provided by a single camera 202 or by multiple imaging devices 202 positioned at different locations around the work vehicle 100.
[0059] In step 420, method 400 continues by processing the captured image to generate at least a first image (e.g., a top-down image) of the surrounding environment, which may typically include the self-propelled work vehicle 100 itself.
[0060] Reference Figure 5 and Figure 6 Exemplary surrounding view images, as disclosed herein, can be created by mapping individual camera images into, for example, a circular "bowl" shape 240 and blending them together. The 360-degree surrounding view image 242 is a top-down view of the mapped "bowl," formatted to fit the shape of the display 210 (e.g., rectangular, square). The depth d and radius r of the bowl affect the simulated field of view of the surrounding view image 242 and the distortion of objects in the image. The center of the bottom of the bowl 240 can be correlated with the center of the self-propelled work vehicle 100. Where multiple images can be overlaid on the bowl 240, these images can be blended together, for example, by combining corresponding pixel values from the respective cameras 202 using conventional alpha blending techniques.
[0061] In step 430, method 400 continues by monitoring any one of one or more possible trigger actions, the detection of which results in the generation of a second and modified image on display 210 (step 440). Various exemplary types of trigger actions (431 to 435) and corresponding actions (441 to 443) in response to such trigger actions are described below, but are not intended to be limiting unless otherwise indicated herein. Various alternative examples of trigger actions and / or corresponding actions in response to such trigger actions may be considered within the scope of this disclosure.
[0062] In various implementations, the triggering action detected in step 430 may take the form of a manual engagement trigger 431 relative to the display 210 or an equivalent user interface tool.
[0063] For example, a detected "pinch-expand" action can cause a change in both the depth d and radius r of the bowl 240. A first direction of the pinch-expand action (e.g., the first and second initial engagement points pointing towards each other) can cause an increase in the depth d and radius r of the bowl 240 corresponding to a "shrink" characteristic, and a second direction of the pinch-expand action (e.g., the first and second initial engagement points facing away from each other) can cause a decrease in the depth d and radius r of the bowl 240 corresponding to a "magnify" characteristic. In various cases, the control system 200 responds to the detected trigger action 431 by manipulating a first image on the display 210 into a second image on the display 210 in a magnification / shrink orientation (step 441), which, for example, corresponds to a revised depth d and radius r of the bowl 240, which may also correspond to the magnitude of the pinch-expand action.
[0064] Alternatively, or in another embodiment, the triggering action detected in step 430 may be a “tapping” action detected relative to one of several areas that are a smaller portion of the bowl 240 (e.g., at the rear, either side, front, or even a corner of the work vehicle), wherein a region-specific square / rectangular sub-viewpoint 244 is generated from the external (i.e., lateral) stereoscopic view. The region corresponding to the tapping action may be one of several predetermined region locations around the perimeter of the bowl 240. In various embodiments, alternatively, the regions may be dynamically defined, wherein the tapping point defines the center of the region, and the perimeter of the sub-viewpoint 244 is generated accordingly around the user-defined center, such that the number of regions is limited only by the number of potential tapping points relative to the bowl 240. In sub-viewpoint mode 244, only a portion of the entire 360-degree surrounding view image is shown, and this portion is scaled up to give the operator a more focused view of that particular region.
[0065] In one implementation, a subsequent tapping gesture detected anywhere on subview 244 can prompt the control system to exit subview mode 244 and return to overhead surround view 242. However, in other implementations, the overhead surround view 242 and the selected subview 244 can be displayed simultaneously, or the selected subview 244 can be generated next to a correspondingly reduced overhead surround view 242, and then, when a tapping gesture is subsequently detected anywhere on subview 244 or overhead surround view 242, the selected subview 244 is removed to facilitate the correspondingly enlarged overhead surround view 242.
[0066] Reference Figure 7A If a trigger action 431 in the form of a first tap is detected on the display 210 or an alternative interface tool, the control system 200 responds to the detected trigger action 431 (step 442) by manipulating a first (e.g., top-down) image on the display 210 into a second image on the display 210 located in, for example, an external stereoscopic field of view 244a corresponding to a selected area.
[0067] Reference Figure 7B If a different trigger action 431 in the form of a second tap is detected on the display 210 or an alternative interface tool, the control system 200 responds to the detected trigger action 431 by manipulating a first (e.g., top-down) image on the display 210 into a different second image on the display 210 in an external stereoscopic field of view 244b corresponding to a selected area.
[0068] Alternatively, or in another alternative embodiment, the triggering action detected in step 430 may be a “slide” action 431 detected relative to the display 210 or an equivalent interface tool when currently in the external view sub-field of view 244, wherein the current external view sub-field of view pivots to the left or right around the periphery of the top-view bowl field of view 240 based on the corresponding left or right direction of the slide trigger 431.
[0069] Reference Figures 8A to 8CIf a trigger action 431 in the form of a right swipe gesture is detected via display 210, then control system 200 responds to the detected trigger action 431 by shifting the corresponding area to the right (i.e., clockwise) along the circumference of the bowl view 240 based on the corresponding region around the central axis. The current (at the moment before the trigger action) external view image 244a on display 210 is then manipulated into a modified external view image 244b on display 210 (step 443). A subsequent trigger action 431 in the form of another right swipe gesture can also be detected via display 210. In this case, control system 200 responds to the subsequently detected trigger action 431 by manipulating the external view image 244b on display 210 into an external view image 244c on display 210. It is understood that, according to the described implementation, the trigger action 431 in the form of a leftward swipe gesture can result in a corresponding leftward (i.e., counterclockwise) shift of the corresponding area around the central axis and along the periphery of the bowl view 240, manipulating the external view image 244a on the display 210 into a modified external view image 244b on the display 210. However, these directions are intended as illustrative, and theoretically it is possible that alternative swipe gestures can be alternatively programmed by the control system 200 to generate corresponding manipulations of the displayed image.
[0070] Alternatively, the control system 200 may manipulate the initial external view image 244a into an external view image 244c using only a single slide trigger action, which extends further to the right than the slide trigger action that generates the external view image 244b. Furthermore, the control system 200 may manipulate the initial external view image 244a into modified external view images 244b, 244c, ..., 244x, depending on the orientation, such as the direction, length, and speed of the swipe gesture (which can also be collectively described as detection characteristics of the swipe gesture).
[0071] In previous reference Figures 5 to 8C In the discussion, each of the detected trigger actions 431 corresponds to a manual gesture on display 210 or an equivalent interface tool. However, the various display operations in the display operations of step 440 (and sub-steps 441 to 443) can also be automatically implemented using a variety of alternative triggers, examples of which are now provided in a non-limiting manner.
[0072] In various implementations, the surrounding view image 242 of 440 can be automatically manipulated based on a trigger 432 associated with a detected configuration change of the self-propelled work vehicle 100, such as identifying the installation of a specific type of work implement 120. In the context of, for example, equipping a fork attachment 120 to a loader, the surrounding view image 242 can be automatically manipulated to an appropriate sub-view 244 with a simulated field of view and determined deformations to provide the operator with optimal visibility of the fork tips. As another example, when the cold milling machine 120 is detected as being mounted on a skid steer loader (SSL), the surrounding view image 242 can be automatically manipulated to a sub-view 244 of the rear side of the self-propelled work vehicle 100, with the expectation that the work vehicle 100 will reverse.
[0073] In some implementations, the surrounding view image 242 can be automatically manipulated based on a trigger 433 associated with the detection of certain operating conditions or predetermined work states. In other words, when the control system 200 determines that the self-propelled work vehicle 100 is performing a function, the surrounding view image 242 can be automatically changed to a smaller sub-view 244 that provides a more focused visibility suitable for that function. For example, if the work vehicle 200 is determined to be reversing in a straight line (or based on a detected steering command or a detected work state consistent with such movement), the surrounding view 242 is changed to a sub-view 244 behind the work vehicle 100. As another example, on an excavator, the boom and bucket controls can be engaged in digging or trenching modes, wherein the control system 200 detects the relevant work state and changes the surrounding view image 242 to a sub-view 244 in front of the work vehicle 100, which focuses on the boom and bucket 120. On the same excavator, when the work vehicle 100 is commanded to turn, the control system 200 can also detect the change in the working state and change the surrounding view image 242 to a sub-view 244 showing the area where the counterweight is turning.
[0074] In addition to transforming the surrounding field of view image 242 into a sub-field of view 244 based on a determined specific area of interest, the deformation and simulated field of view of the surrounding field of view image 242 can also be manipulated based on detected functional or operational conditions. For example, when it is detected that the work vehicle 100 is moving faster, the control system 200 can manipulate the simulated field of view to become larger.
[0075] In some implementations, the surrounding field of view image 242 of 440 can be automatically manipulated based on a trigger 434 associated with detected or predicted movement of the work implement 120. For example, when it is detected that the work implement 120, such as an excavator boom, is extending further, the simulated field of view can automatically adjust to increase the field of view. The surrounding field of view image 242 can also be automatically manipulated by the control system 200 according to the detected work status or cycle. For example, if it is detected that the excavator 100 is operating with a 180-degree truck loading at a 12-second cycle, the control system 200 can anticipate when the operator will turn the main frame 132 of the work vehicle 100 and preemptively propose a sub-field of view 244 of the area where the operator will turn.
[0076] In some implementations, the surrounding view image 242 of 440 can be automatically manipulated, such as changing the region of interest, based on a trigger 435 associated with obstacle detection. For example, if the obstacle detection system 206 detects an obstacle alone or in coordination with the controller 112, the surrounding view image can be automatically changed to focus on a sub-view 244 of the area where the object was detected. If the obstacle detection system 206 outputs the height of the object, the controller 112 can be configured to automatically manipulate the distortion and simulated field of view of images 242, 244 to give the detected object a less distorted appearance in images 242, 244.
[0077] In various implementations, the control system 200 may also lock the sub-view of view 244 onto an identified object of interest, such as a truck or ditch. For example, the object of interest may be defined based on detected known obstacles or targets, estimated operational status, type of work equipment, operator selection / commands, etc.
[0078] In various implementations, the user interface 116 can be configured to selectively apply manual triggers 431 and / or one or more automatic triggers 432 to 435 as disclosed herein. For example, the operator can select an automatic mode, wherein any of the aforementioned automatic triggers is allowed to result in manipulation of the displayed image, but further overrides the automatically generated display screen and the option to revert the display screen to a standard top-down image or any other optional image. The operator can, for example, selectively differentiate between multiple available automatic trigger options in other ways, wherein, for example, the operator can specify that automatic manipulation of the displayed image is appropriate for certain automatic triggers 432 to 435, while other (e.g., unspecified) triggers are ignored.
[0079] The control system 200 also enables the supplementation of the automatically generated display image based on detected manual gestures 431, rather than simply overwriting the automatic result. For example, the radius and / or depth of the bowl 240 can be automatically modified according to changes in speed, wherein the top-view image 242 is modified accordingly, and the operator can make a manual gesture that generates a further modified radius and / or depth of the bowl 240, rather than reverting the image to the initial image, which can otherwise be achieved using separate manual selections.
[0080] In various implementations, the control system 200 may be configured to prioritize detected or predicted triggering actions to determine which display image(s)(s) to present or manipulate, or to show the order of multiple display images. For example, a first display portion may typically include a display image that moves or alternates between positions corresponding to excavator operating states (e.g., digging, swinging to dumping, dumping, swinging to digging), but the first display portion may instead switch to a display image corresponding to a detected obstacle or other safety issue that occurs during excavator operation, or alternatively, a second display portion may be generated and highlighted to draw attention to a recently detected obstacle.
[0081] As used herein, when used with a list of items, the phrase “one or more of…” means that different combinations of one or more of these items may be used, and only one of the individual items in the list may be required. For example, “one or more of items A, B, and C” may include, but is not limited to, item A, or items A and B. The example may also include items A, B, and C, or items B and C.
[0082] Therefore, it can be seen that the apparatus and methods of this disclosure readily achieve the mentioned and inherent purposes and advantages. While certain preferred embodiments of this disclosure have been illustrated and described for these purposes, many changes can be made by those skilled in the art to the arrangement and construction of components and steps, and these changes are covered within the scope and spirit of this disclosure as defined by the appended claims. Features or embodiments of each disclosure can be combined with any of the features or embodiments of other disclosures.
Claims
1. A computer-implemented method (400) for displaying the surrounding environment of a self-propelled work vehicle (100) including a main frame (132), the method comprising the following steps: Receive one or more images (410) corresponding to the surrounding environment of the work vehicle, the images being recorded via one or more corresponding cameras (202) supported by the main frame; The recorded image is processed (420) to map the image to a shape corresponding to a first defined display radius and a first defined display depth (240). In response to at least one trigger action (430), one or more display images (440) corresponding to at least one selected area of interest in the surrounding environment of the work vehicle are generated on the display unit. In response to at least a second type of triggering action, the selected region of interest is manipulated in relation to the generated external view display image. The triggering action of at least the second type includes the detected configuration change (432) of the self-propelled working vehicle. The self-propelled work vehicle includes a work implement (120) connected to the main frame, and the detected configuration change of the self-propelled work vehicle includes the detected configuration change of the work implement.
2. The method according to claim 1, wherein, The generated one or more display images include both top-view display images and outside-view display images, or selectively switch between the top-view display images and the outside-view display images.
3. The method according to claim 1 or 2, wherein, The generated display image includes a top-view display image corresponding to the first defined display radius and the first defined display depth. The method includes the following steps: in response to at least a first type of triggering action, generating a top-view display image corresponding to a second defined display radius and a second defined display depth.
4. The method according to claim 1 or 2, wherein, At least the first type of triggering action includes one or more automatically determined operational conditions (433).
5. The method according to claim 2, wherein: At least the first type of triggering action includes the detected speed change of the self-propelled working vehicle. The top-down display image is reconfigured to increase the second-defined display radius and the second-defined display depth in association with the increase in the speed of the self-propelled working vehicle, and The top-down display image is reconfigured to reduce the second-defined display radius and the second-defined display depth in association with the reduction in the speed of the self-propelled work vehicle.
6. The method according to claim 1 or 2, wherein, The at least one triggering action includes selectively manually engaging the top-down display image via a user interface (116).
7. The method according to claim 1 or 2, wherein, The self-propelled work vehicle includes a work implement that is controllably movable relative to the main frame, and the at least second type of triggering action includes one or more of the following: detected movement (434) of the work implement; and predicted movement (434) of the work implement.
8. The method according to claim 7, wherein, The movement of the work implement is predicted based on the determined work cycle of the self-propelled work vehicle.
9. The method according to claim 1 or 2, wherein, The at least second type of triggering action includes the detection of obstacles (435) in the surrounding environment of the work vehicle.
10. The method according to claim 1 or 2, further comprising the following step: In response to at least a third type of triggering action, the external view display image is pivoted laterally.
11. The method according to claim 1 or 2, wherein: The external view display image is generated in response to at least a second type of triggering action, instead of the top view display image; and / or The external view image and the top view image are displayed independently on the same display unit (210) and are modified independently in response to different types of triggering actions.
12. A self-propelled work vehicle (100), the self-propelled work vehicle (100) comprising: Main frame (132); One or more cameras (202), which are supported by the main frame and configured to record images (410) of the corresponding surrounding environment of the work vehicle. At least one working implement (120) is connected to the main frame and configured to work on the terrain around the working vehicle; A controller (112) communicatively linked to the one or more cameras and the display unit, and the controller is configured to perform the steps of the method according to any one of claims 1 to 11.