Method and user interface for selective assistance automation of loading operation phases of a work vehicle

Abstract

Method and user interface for selective assistance automation relating to a loading operation phase of a work vehicle. In particular, a system and method for selective input confirmation of automated loading of a work vehicle comprising a main frame and a work attachment movable relative to the main frame for loading / unloading material in a loading area of the work vehicle during a loading process having a plurality of loading phases. A position input of the loading area is detected for the main frame and / or the work attachment, respectively. A first user input corresponds to a selected automation of a respective loading phase for which a detection routine is performed with respect to parameters of the loading area based on the detected position input. If certain parameters with respect to the loading area determine that a second user input is required, the second user input is received and the movement of the main frame and / or the work attachment is controlled for automated loading of the respective loading phase based at least in part on the second user input.

Description

Technical Field

[0001] This disclosure generally relates to work vehicles, and more particularly to systems and methods for selective automation of vehicle movement and / or work attachment movement during a specified portion of a loading operation. Background Technology

[0002] For illustrative purposes, the term "operating vehicle" as used herein may specifically refer to a wheelloader, but may also include, for example, excavator machines, forestry machines, and other equipment that modifies terrain or equivalent working environments in some way. The operating vehicles discussed herein are often self-propelled via tracked or wheeled ground engagement units supported by a ground-based underframe, and may also include one or more working attachments for transporting material from a location to unload it into a loading area, for example, associated with a truck or bucket. However, some operating vehicles within the scope of this disclosure are not necessarily self-propelled, such as, for example, knuckle boom loaders.

[0003] Those skilled in the art will recognize that finding experienced operators for certain conventional work vehicles is an ongoing challenge. With regard to wheel loaders, which serve as such exemplary work vehicles, a particularly challenging part of the operating cycle for novice operators is approaching and loading loading areas, such as those associated with trucks or hoppers. Novice operators can typically learn the “digging” part of the operating cycle relatively quickly, but often hesitate for some time when approaching a truck or hopper.

[0004] As an example, an operation for unloading bulk material from an attachment (e.g., a bucket) of a work vehicle may include pivotal movement of the attachment relative to the main frame and loading area of ​​the work vehicle, and also includes movement of the work vehicle itself relative to the ground and loading area. Therefore, it is important to ensure that the attachments and / or other parts of the work vehicle do not collide with the loading area during the unloading operation; this includes not only the attachments approaching the loading area but also their retraction after the unloading of the bulk material is complete. Summary of the Invention

[0005] This disclosure provides enhancements to conventional systems, at least in part, by introducing novel systems and methods for selectively loading auxiliary features.

[0006] An exemplary purpose of such loading assistance features could be to add value to customers by automating multiple aspects of truck loading operations related to the movement of control attachments (e.g., boom) and / or the distance the work vehicle stops relative to the loading area. The loading area can be, for example, the bed of a dump truck, hopper, trailer, etc., and thus the term can encompass both moving and stationary containers and the associated loading area, as understood by those skilled in the art.

[0007] A detection system, including, for example, a stereo camera, can be used to measure the distance to a loading area, the height of the top edge of the loading area from the ground, and the horizontal extent of the loading area. One challenge in operating such a system is that point cloud data obtained using stereo parallax is not always clear. There may be many defects, especially in the presence of visual phenomena such as reflections, glare, and partial fog-like blurring. In such cases, the system can automatically identify the distance to the side of the loading area, but cannot identify attributes related to the edges of the loading area.

[0008] In this scenario, the system can mark areas requiring operator attention and obtain from the operator several types of visual backgrounds from the camera system that will trigger additional detection subroutines. As disclosed herein, such a collaborative approach may include an automated loading feature that uses visual sensors to identify loading areas (e.g., trucks or hoppers) and enables the work vehicle to automatically load material into the loading area once the bucket on the work vehicle is full.

[0009] In some implementations, the methods disclosed herein can be implemented while an operator is in the cab of a work vehicle monitoring the operation of the automated system. In other implementations, the methods disclosed herein can be implemented remotely, or in situations where an operator is monitoring a fleet of autonomous vehicles and must be able to quickly and effectively assist semi-autonomous vehicles in troubleshooting while remaining operational.

[0010] Therefore, the systems and methods disclosed herein can not only provide site owners with increased confidence that the loader bucket will not come into contact with the truck floor or hopper when loading materials, but the methods disclosed herein can also serve as an autonomous middle ground—the work vehicle system can utilize simpler / faster algorithms that do not necessarily rely on machine learning, and the operator can quickly and remotely add the visual background required by the machine learning classification algorithm when the operation of the semi-autonomous system requires it.

[0011] In one embodiment, a computer-implemented method for selective input verification of automated loading of a work vehicle, as disclosed herein, is provided. The work vehicle includes a plurality of ground engagement units supporting a main frame and at least one work attachment movable relative to the main frame and configured to load and unload materials in a loading area outside the work vehicle during a loading process having one or more loading stages. The method includes the steps of: detecting one or more location inputs in the loading area, respectively, for the main frame and / or the at least one work attachment, via at least one detector associated with the work vehicle. The method further includes the steps of: receiving a first user input corresponding to a selected automation of any one or more of the one or more loading stages; and for one or more selectively automated loading stages, executing a detection routine in relation to parameters of the loading area based on the detected one or more location inputs, and determining whether one or more second user inputs are required in relation to one or more of the parameters of the loading area. The method further includes the steps of: receiving the one or more second user inputs and, at least in part based on the one or more second user inputs, automatically controlling the movement of the main frame and / or the at least one job attachment for automating the corresponding loading phase.

[0012] In one exemplary aspect of the implementation described above, where one or more second user inputs are required for any one or more of the parameters of the loading area, the user may be prompted to input the one or more second user inputs via a user interface, and thereby the one or more second user inputs may be received.

[0013] In another exemplary aspect of the implementation according to the above reference, the step of detecting one or more location inputs may include: capturing an image via an imaging device.

[0014] In another exemplary aspect of the embodiments referred to above, the loading area parameters may include one or more of the following: the distance between the loading area and the main frame; the distance between the loading area and the at least one working attachment; the orientation of the loading area relative to the main frame and / or at least one working attachment; the height of the loading area; the polygonal profile of the container associated with the loading area; and the circular or elliptical profile of the wheels supporting the loading area.

[0015] In another exemplary aspect of the implementation according to the above reference, the method may further include the step of generating an image of a loading area on a user interface associated with at least one or more required second user inputs, wherein the one or more second user inputs include one or more engagement points of the user interface and / or continuous movement of the engagement points along the user interface.

[0016] Further according to this exemplary aspect, the method may include the step of automatically estimating one or more contours of a container associated with the loading area, at least based on an image of the generated loading area and one or more second user inputs from the user interface, including swiped engagements.

[0017] Further according to this exemplary aspect, the method may include the step of: automatically estimating one or more contours of a container associated with the loading area, at least based on an image of the generated loading area and the one or more second user inputs of the user interface, including a plurality of sweep engagements for defining closed areas in the user interface.

[0018] Further according to this exemplary aspect, the method may include the step of automatically estimating the profile of a wheel associated with the loading area, at least based on an image of the generated loading area and one or more second user inputs from the user interface, including a circular or elliptical sweep engagement for defining a closed area in the user interface.

[0019] Further according to this exemplary aspect, the method may include the step of automatically estimating the profile of a wheel associated with the loading area, at least based on an image of the generated loading area and the one or more second user inputs via a user interface, including one or more tapped engagements.

[0020] Another embodiment disclosed herein may provide a work vehicle comprising a plurality of ground engagement units supporting a main frame, and at least one work attachment movable relative to the main frame and configured to load and unload material in a loading area outside the work vehicle during a loading process having at least one or more loading stages. At least one detector is configured to detect one or more location inputs in the loading area, respectively, for the main frame and / or the at least one work attachment. A user interface is configured to enable at least a first user input corresponding to a selected automation of any one or more of the one or more loading stages. A controller is provided and configured to direct the execution of methods according to the embodiments referenced above and optional exemplary aspects.

[0021] In another embodiment disclosed herein, a system for automated assistance of multiple work vehicles, substantially according to the embodiments referenced above, may be provided, wherein a user computing device is remotely configured relative to each of the plurality of work vehicles and includes a user interface configured to enable at least a first user input for a corresponding work vehicle corresponding to a selected automation of any one or more of the one or more loading stages. Each of the plurality of work vehicles also includes a controller, which is accordingly configured to direct the execution of methods according to the embodiments referenced above and optional exemplary aspects.

[0022] Many objects, features, and advantages of the embodiments set forth herein will readily become apparent to those skilled in the art when the following disclosure is read in conjunction with the accompanying drawings. Attached Figure Description

[0023] Figure 1 This is a side view of an exemplary embodiment of the work vehicle according to the present disclosure.

[0024] Figure 2 This is a block diagram illustrating a control system according to an embodiment of the present disclosure.

[0025] Figure 3 It is a graphical representation of the user interface that displays the loading area and the user-selectable stages of the loading process.

[0026] Figure 4 It means Figure 3 A graphical representation of the user interface shows the loading area and some of its highlighted outlines.

[0027] Figure 5 It means Figure 3 The graphical representation of the user interface shows the loading area and the overlay trajectory used to approach the loading area.

[0028] Figure 6 It means Figure 5 The graphical representation of the user interface shows the loading area and the superimposed trajectory of the approaching loading area after it has already approached a horizontal distance relative to the loading area.

[0029] Figure 7 It means Figure 3 The graphical representation of the user interface shows the loading area and highlights contours that cannot be adequately detected by automated methods.

[0030] Figure 8A , Figure 8B as well as Figures 9 to 12 It means Figure 7The graphical representation of the user interface shows the loading area and user inputs used to help define the specific outline of the loading area.

[0031] Figure 13 It means Figure 7 The graphical representation of the user interface shows the loading area and highlights the contours that are adequately detectable for automated methods.

[0032] Figure 14 It means Figure 3 The graphical representation of the user interface shows the loading area and highlights that the contours are not adequately detected by automated methods even with user assistance.

[0033] Figure 15 This is a flowchart illustrating an exemplary method according to an embodiment of the present disclosure. Detailed Implementation

[0034] Now, referring to Figures 1 to 15 This can describe various implementations of the system and method of the invention.

[0035] In certain embodiments as disclosed herein, Figure 1 A representative work vehicle 100, for example, in the form of a loader, is shown, which has a pre-installed work attachment 120 for modifying proximate terrain. Within the scope of this disclosure, the work vehicle 100 may take the form of any other self-propelled vehicle using one or more work implements 120 for modifying proximate terrain and transporting materials from that terrain for loading into a loading area, and the vehicle is generally designed for use, for example, in off-highway environments, such as construction or forestry vehicles. In the remaining examples provided herein, the work implement 120 is referred to as a work attachment; however, the scope of this disclosure contemplates the use of various implements and is not limited to the examples specifically provided herein, unless otherwise stated.

[0036] In the various embodiments further described herein, the loading area is associated with a truck and typically includes a loading surface surrounded by multiple walls and an open area opposite the loading surface to accommodate the material unloaded therein.

[0037] The illustrated work vehicle 100 includes: a main frame 132 supported by a first pair of wheels serving as a left ground engagement unit 122 and a second pair of wheels serving as a right ground engagement unit 124; and at least one travel motor (not shown) for driving the ground engagement units. Although wheels are used in the illustrated embodiment, it is conceivable within the scope of this disclosure that the ground engagement units are tracked.

[0038] The working attachment 120 for the illustrated work vehicle 100 includes a front-mounted loader bucket 120 coupled to a boom assembly 102. The loader bucket 120 generally faces away from the operator of the loader 100 and is movably coupled to a main frame 132 via the boom assembly 102 for shoveling, transporting, and dumping soil and other materials, for example, forward shoveling, transporting, and dumping into a loading area 302, such as that associated with an articulated dump truck. In an alternative embodiment where the work vehicle is, for example, a tracked excavator, the boom assembly 102 may be defined as including at least a boom and a boom pivotally connected to the boom. In this example, the boom is pivotally attached to the main frame 132 to pivot about a generally horizontal axis relative to the main frame 132. A coupling mechanism may be located at the end of the boom assembly 102 and configured to couple to the working attachment 120; this coupling mechanism may also be characterized as a working tool, and in various embodiments, the boom assembly 102 may be configured to engage and secure attachments 120 of various types and / or sizes.

[0039] In other embodiments, for example, depending on the type of work vehicle 100, the work attachment 120 may take other appropriate forms as understood by those skilled in the art, but for the purposes of this disclosure, a work attachment 120 will be included that is used to transport materials from a first location for unloading or otherwise unloading into a second location which is a loading area (e.g., a truck or hopper) 300.

[0040] The operator's cab can be located on the main frame 132. Both the operator's cab and the boom assembly 102 (or, depending on the type of work vehicle 100, the work attachment 120) can be mounted on the main frame 132 such that the operator's cab faces the working direction of the work attachment 120. A console, including a user interface 116, can be located in the operator's cab. As used herein, the orientation of the work vehicle 100 can be referenced from the perspective of the operator seated in the operator's cab; the left side of the work vehicle is the operator's left side, the right side of the work vehicle is the operator's right side, the front (or front) portion 112 of the work vehicle is the direction the operator is facing, the rear (or rear) portion of the work vehicle is behind the operator, the top of the work vehicle is above the operator, and the bottom of the work vehicle is below the operator.

[0041] An example of the user interface 116 described herein can be set as part of a display unit configured to graphically display symbols, data, and other information, and in some embodiments, may also provide other outputs from the system, such as indicator lights, audible alarms, etc. The user interface may also or alternatively include various controls or user inputs (e.g., steering wheel, joystick, lever, buttons) 208 for operating the work vehicle 100 (including operation of the engine, hydraulic cylinders, etc.). Such an onboard user interface can be coupled to the vehicle control system, for example, via a CAN bus or other equivalent form of electrical and / or electromechanical signal transmission. Another form of user interface disclosed herein can take the form of a display unit generated on a remote (i.e., off-vehicle) computing device, which can display outputs such as status indications and / or otherwise enable user interactions such as providing input to the system. In the context of a remote user interface, data transmission between, for example, the vehicle control system and the user interface can take the form of wireless communication systems and associated components conventionally known in the art. In some implementations, the remote user interface and vehicle control system for the corresponding work vehicle may also coordinate with or otherwise interact with a remote server or other computing device to perform operations within a system as disclosed herein.

[0042] For example Figure 2 As illustrated schematically, the work vehicle 100 includes a control system 200, which includes a controller 112. The controller 112 may be part of the work vehicle's mechanical control system, or it may be a separate control module. The controller 112 may include a user interface 116 and may optionally be mounted at a control panel in an operator's cab.

[0043] The controller 112 is configured to receive inputs from some or all of a variety of sources, including image data sources such as the camera system 202 and the work vehicle motion sensor 204, as well as machine parameters 206, such as those from the user interface 116 and / or the machine control system for the work vehicle (if defined separately relative to the controller).

[0044] Image data sources such as camera system 202 are suitable. Implementations may include one or more detectors, such as imaging devices like camera 202 mounted on the work vehicle 100, configured to capture images or otherwise generate image data corresponding to the surrounding environment of the work vehicle 100. Camera system 202 may include a video camera configured to record a raw image stream and send corresponding data to controller 112. In alternatives or additionally, camera system 202 may include one or more cameras such as infrared cameras, stereo cameras, PMD cameras, etc. Those skilled in the art will recognize that high-resolution light detection and ranging (LiDAR) scanners, radar detectors, laser scanners, etc., can be implemented as image data sources within the scope of this disclosure. The number and orientation of image data sources may vary depending on the type of work vehicle and the associated application, but they may be positioned at least relative to an area along the direction of travel of the work vehicle 100 and configured to capture image data associated with the loading area 302 that the work vehicle 100 is traveling towards. The location and size of the image area recorded by the corresponding camera 202 may depend on the arrangement and orientation of the camera and camera lens system, particularly the focal length of the camera lens, but can be desirously configured to capture essentially the entire loading area 302 during the approach and retraction of the work vehicle 100 and associated attachment 120. Those skilled in the art will also appreciate that, with proper configuration, image data processing functions can be performed separately at a given image data source, but typically, at least some image data processing can also be included, or otherwise, by a controller or other downstream data processor. For example, image data from any one or more image data sources can be provided for use with image data processing tools known in the art in conjunction with the objectives disclosed herein for 3D point cloud generation, image segmentation, object depiction, and classification, etc.

[0045] An exemplary work vehicle motion sensing system 204 may include: an inertial measurement unit (IMU) mounted to a corresponding component of the work attachment 120 and / or boom assembly 102 and / or main frame 132, a sensor coupled to a piston-cylinder unit to detect its associated hydraulically actuated extension, or any other known alternative that may be known to those skilled in the art.

[0046] In various implementations, additional sensors can be provided to detect mechanical operating conditions or positioning, such as orientation sensors, global positioning system (GPS) sensors, vehicle speed sensors, vehicle and implement position sensors, etc., and one or more of these sensors may be discrete in nature, and the sensor system may also reference signals provided from the mechanical control system.

[0047] In an implementation, any of the aforementioned sensors may be supplemented by radio frequency identification (RFID) devices or equivalent wireless transceivers located on one or more attachments 120, loading areas 302, etc. Such devices may be implemented, for example, to determine and / or confirm the distance and / or orientation between them.

[0048] Other sensors may be used in conjunction with or combined with one or more of the aforementioned sensors to improve data collection in order to collectively define the obstacle detection system 206. Various examples of such sensors may include: ultrasonic sensors, laser scanners, radar wave transmitters and receivers, thermal sensors, imaging devices, structured light sensors, and other optical sensors. The type and combination of sensors used for obstacle detection may vary depending on the type of vehicle, the work area, and / or the application, but they are generally set and configured to optimize the identification of objects associated with the work area 302 of the determined vehicle and / or the associated loading area 302 of the given application, or objects associated with the work area 302 of the determined vehicle and / or the associated loading area 302 of the given application.

[0049] Controller 112 can typically cooperate with the user interface 116 referenced above to display various symbols to a human operator. Controller 112 can also generate control signals for controlling the operation of corresponding actuators, or signals for indirect control via intermediate control units associated with the mechanical steering control system 224, mechanical accessory control system 226, and / or mechanical drive control system 228. Controller 112 can, for example, generate control signals for controlling the operation of various actuators such as hydraulic motors or hydraulic piston-cylinder units, and the electronic control signals from controller 112 can actually be received by an electro-hydraulic control valve associated with the actuator, such that the electro-hydraulic control valve, in response to the control signals from controller 112, controls the flow of hydraulic fluid to and from the corresponding hydraulic actuator to control the actuation of that hydraulic actuator. The controller 112, which is also communicatively connected to the hydraulic system as a mechanical accessory control system 226, can be configured to operate the work vehicle 100 and the accessories 120 connected to the work vehicle, including but not limited to, lifting mechanisms, tilting mechanisms, roll mechanisms, pitch mechanisms, and / or auxiliary mechanisms of the accessories, as appropriate for a given type of accessory or work vehicle application. The controller 112, which is also communicatively connected to the hydraulic system as a mechanical steering control system 224 and / or a mechanical drive control system 228, can be configured to move the work vehicle in the forward and reverse directions, move the work vehicle left and right, control the travel speed of the work vehicle, etc.

[0050] The controller 112 includes or may be associated with a processor 212, a computer-readable medium 214, a communication unit 216, a data storage device 218 (e.g., a database network), and the aforementioned user interface 116 or control panel with a display 210. Input / output devices 208, such as a keyboard, joystick, or other user interface tools, are provided to allow a human operator to input commands to the controller 112. It should be understood that the controller 112 described herein may be a single controller having all the functions described, or it may include multiple controllers, with the described functions distributed among the multiple controllers.

[0051] The various operations, steps, or algorithms described in conjunction with controller 112 can be implemented directly in hardware, as a computer program product such as a software module executed by processor 212, or a combination of both. The computer program product can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, or any other form of computer-readable medium 214 known in the art. An exemplary computer-readable medium 214 can be coupled to processor 212, allowing processor 212 to read information from and write information to the memory / storage medium 214. In this alternative embodiment, medium 214 can be integrated with processor 212. Processor 212 and medium 214 can reside in an application-specific integrated circuit (ASIC). The ASIC can reside in a user terminal. In this alternative embodiment, processor 212 and medium 214 can reside as discrete components in a user terminal.

[0052] As used herein, the term "processor" 212 may refer to at least general-purpose or special-purpose processing devices and / or logic that can be understood by those skilled in the art, including but not limited to microprocessors, microcontrollers, state machines, etc. The processor 212 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other such configuration.

[0053] Communication unit 216 may support or provide communication between controller 112 and external systems or devices, and / or support or provide communication interfaces with respect to internal components of work vehicle 100. Communication unit may include wireless communication system components (e.g., via cellular modem, WiFi, Bluetooth, etc.), and / or may include one or more wired communication terminals, such as Universal Serial Bus ports.

[0054] Unless otherwise stated, the data storage device 218 discussed herein may generally encompass hardware (such as volatile or non-volatile storage devices, drives, memory or other storage media) and one or more databases residing on that hardware.

[0055] Next, refer to Figure 15 And further illustrative references Figures 3 to 14 The present invention describes an implementation of method 400, which is exemplary and not intended to limit the scope of the disclosure unless specifically indicated otherwise. Those skilled in the art will recognize that alternative implementations may include fewer or additional steps, and that some of the disclosed steps may be performed, for example, in a different time order or simultaneously.

[0056] In this implementation, as previously mentioned, a remote user can selectively assist in the automation of multiple work vehicles, wherein the first step 410 of method 400 may accordingly include selecting a communication channel and / or a corresponding work vehicle via a system-generated user interface 116. Upon selection of a specific channel, images generated by sensors / cameras 202 for the corresponding work vehicle can be generated. The user interface 116 may also be prompted via push notifications from the corresponding work vehicle or a central server, and generate images for the user to view and interact with.

[0057] Of course, in implementations where, for example, the user physically presents the operation of the corresponding work vehicle and a user interface is generated separately for each work vehicle, step 410 can be omitted.

[0058] In another step 420, method 400 may include: a user interface 116 that allows a user to select one or more portions of the loaded sequence for automation. Figure 3 As shown in the diagram, a touchscreen serving as user interface 116 can generate a background input screen 300, which includes one or more user-selectable buttons or equivalent interface tools associated with different parts of the loading sequence. For example, a first section 320 related to the approach of the work vehicle 100 and / or attachment (e.g., boom) 120 to the loading area 302 (e.g., "Approach & Boom"); a second section 330 related to loading material from the work vehicle attachment 120 to the loading area 302 (e.g., "Dump"); and a third section 340 related to a separate sequence of retraction of the work vehicle 100 and / or attachment 120 from the loading area 302 and / or transition to work vehicle operation (e.g., "Return to Excavation"). Also, Figure 3 As shown, the cursor, touchscreen interface 350, or equivalent can be implemented as a selection from various buttons.

[0059] In another step 430, method 400 may include generating a background input screen 300, which includes: a first image layer on user interface 116 corresponding to a captured image of loading area 302, for example, a captured image associated with a dump truck having multiple wheels 308a, 308b, 308c and associated with an axle supporting a loading container (e.g., truck floor), the loading container having, for example, a loading surface at the bottom of an interior area surrounded by sidewalls 306; and a top edge 304, at least a portion of which may generally be parallel to ground surface 138.

[0060] like Figures 4 to 6 As further indicated, the imaging routine can be executed by the system to process a defined portion of the first image layer and to programmatically determine or otherwise estimate parameters associated with the loading area 302. In step 440, method 400 may include generating a second image layer, for example, as an augmented reality image layer superimposed on the first image layer. This second image layer may, for example, provide visual markers to the user relative to the determined or otherwise estimated parameters, such as the distance between a portion of the loading area 302 and a portion of the work vehicle 100, the height of the portion of the loading area 302 relative to the ground surface 138, and the trajectory to be taken between the current position of the work vehicle 100 (and / or attachment 120) and the portion of the loading area 302.

[0061] In one embodiment, the imaging routine according to method 400 may include processing stereo camera parallax measurements and stored or otherwise developed models to segment the corresponding measurements into floor planes, such as those associated with the loading surface, the loading area sidewall 306, the top edge of the bin 304, etc., wherein the processing may take into account the position, orientation, and speed of movement of the camera 202. In some embodiments, the segmentation may be further improved via known markers (e.g., printed text, barcodes, etc.) associated with the loading area 302, attachment 120, or other objects within the image frame. In embodiments where multiple imaging devices 202 may be used, the known relative positions and orientations of the imaging devices may be further used, for example, triangulation techniques to enable object position determination. In short, the controller 112 and / or the discrete image processing unit (not shown) can, for example, utilize conventional image recognition and processing techniques, floor plan modeling, machine learning algorithms, stored loading area data, etc., to analyze the shape and size of objects, measure the distance from the stereo camera to the object, identify or predict the extent of objects in image frames, measure the orientation of objects in image frames, and convert the measurement results from image frames into work vehicle frames.

[0062] However, it should be noted that the above-described techniques are not necessary, and in fact, one of the potential advantages of the background input available in this disclosure is that it alleviates any problem in other aspects of complex image processing techniques or systems that fail to properly identify all elements in an image frame.

[0063] Now return to Figure 4 For the selected first portion 320 of the loading sequence, the system in this embodiment identifies parameters from the first image layer and generates a first marker 360 corresponding to the top edge 304 of the truck bed, a second marker 362 corresponding to the wheel 308b, and a third marker corresponding to the wheel 308c as the second image layer. Therefore, in this example, the system indicates that it has automatically positioned the top edge of the truck bed and the two prominent tires, thus eliminating the need for manual assistance.

[0064] In one implementation, the system allows the user to overlay automatically determined feature locations. For example, the user may recognize from the displayed information that the system has automatically, but not correctly, identified feature locations, but has color-coded or otherwise highlighted these features to indicate that manual assistance is not required. In such an implementation, the user can choose or otherwise trigger a manual assistance mode, wherein the system prompts the user to engage the first image layer and provides background information in a manner further described below, and this generally involves the user generating an augmented reality image on the first image layer for use in enhancing image processing algorithms.

[0065] Having identified the necessary elements (e.g., “approach & boom”) for the first part 320 of the automated loading feature, the system can proceed by generating signals associated with the desired material unloading to control at least the work vehicle 100 and attachments 120 to approach the loading area 302. This may include, for example: calculating and implementing a drivetrain trajectory that begins at the current work vehicle position and speed and ends at an appropriate location corresponding to the loading area 302 with zero ground speed; generating and implementing a steering trajectory using visual measurements of the position and orientation of the loading area 302 relative to the work vehicle 100, and dynamically adjusting the steering angle of the work vehicle 100 to follow the trajectory as the work vehicle 100 approaches the loading area 302; and further calculating and implementing trajectories for one or more attachments 120 (e.g., via boom cylinders) that begin at the current height and end at a loading height (which is substantially synchronized with the arrival of the work vehicle 100 relative to the loading area 302), and / or applying closed-loop control to ensure that the boom and drivetrain follow the calculated trajectory. The automated loading feature may also include: calculating a trajectory based on visual measurements of the height of the loading area (e.g., truck floor) 302 to automatically adjust the height of the attachment 120 (e.g., boom lifting height).

[0066] Further reference Figure 5 Furthermore, for the selected first portion 320 of the loading sequence, the system identifies additional parameters from the first image layer and generates a second image layer, which further includes, for example, a marker 382 indicating the current position of the closest portion of the work vehicle 100 and / or attachment 120 relative to the loading area 302, a marker 384 indicating the distance between the closest portion of the work vehicle / attachment and the desired stopping distance from the loading area 302, a marker 386 indicating the path along the ground surface 138 between the work vehicle 100 and the loading area 302, a marker 388 indicating the vertical height of a marker 360 corresponding to the top edge 304 of the truck bed, and a marker 390 indicating the distance between the desired stopping distance from the loading area and the marker 360 corresponding to the top edge 304 of the truck bed.

[0067] like Figure 6 Furthermore, as the work vehicle 100 travels along the path toward the loading area 302, the displayed image and corresponding generated markers are modified. In the example shown, the work vehicle has effectively traversed the loading area. Figure 5 The illustrated position and the stop point are horizontal paths, and the second image layer now includes: a marker 382 indicating the current position of the work vehicle and / or attachments relative to the closest part of the loading area, a marker 384 indicating a distance value, a marker 388 indicating a vertical path to be transverse to the marker 360 corresponding to the top edge 304 of the truck bed, and a marker 390 indicating the remaining horizontal path to be transverse to the marker 360 corresponding to the top edge 304 of the truck bed.

[0068] In some cases, for example, such as Figure 4 This indicates that image processing can produce results that are clear enough to enable automation without manual assistance, wherein the second marker can be highlighted in a specific way, such as in bold or in green.

[0069] In other cases, such as when image processing may be adversely affected by environmental conditions, the system may pause the automated sequence to identify the top edge 304 of the truck bed that it cannot correctly identify. Figure 7 An example is illustrated where the top edge has been automatically identified via the second image layer component 366, but the system recognizes that this component does not correspond to the actual top edge 304, or at least recognizes an insufficient level of certainty in such identification. Therefore, the system can prompt the user via user interface 116 to perform manual assistance in the image processing and automation sequence (i.e., in response to the query in step 450 being "yes"). Again, as Figure 7As shown, the first part of the loading sequence (“Approach & Boom”) is highlighted accordingly, for example, by using a flashing indicator, or by using a different color such as yellow to distinguish it from the aforementioned green.

[0070] In the implementations illustrated herein, various specific forms of user interaction and associated background input may be described with respect to step 460; however, such examples are illustrative and are not intended to limit the range of available user interaction and associated background input unless otherwise specifically stated herein.

[0071] Firstly, one such exemplary background input may include a swipe engagement 462 of a specific portion of the user interface displayed on the background input 300. Such a portion of the background input screen 300 may correspond to the upper edge 304 or side edge 306 of the truck bed on the truck 302. For example... Figure 8A and Figure 8B This indicates that after the cursor 350 engages with the touchscreen at the first point 368, it performs a sweeping movement to the second point 370. These inputs can trigger the vision system to perform an edge detection algorithm on the pixels near the sweep line and record the pixel positions of the detected edges. The pixel lines of the detected edges can be projected onto a detected surface plane representing a detectable portion of the sidewall of the truck bed, wherein the position of the projected lines can, for example, be used as the edge of the sidewall 306 of the truck bed.

[0072] Other such exemplary background inputs may include: a tap engagement 464 of the user on a specific portion of the background input screen 300, and / or a series of continuous engagements 466 of the user on a specific portion of the background input screen 300, such as defining a closed region associated with the outline of the loading area 302.

[0073] For example, a user can provide a tap 464 or draw a circle / ellipse 466 via cursor 350, and implement this tap or draw to indicate the position of one or more wheels associated with the first image layer. This can be done in conjunction with... Figure 9 The axle of wheel 308b represented by the indicated point 373 provides a tap 464, and / or can provide a drawn circle / ellipse 466 to define the point as shown. Figure 10 The closed region 374 corresponding to the outline of wheel 308b is indicated by cursor 350. Figure 11 The point corresponding to the axle of wheel 308c represents a subsequent tap 464, and / or a drawn circle / ellipse 466 can be provided to define the point as shown. Figure 12The outline of the wheel 308c represents the closed region 378. These inputs can accordingly trigger the vision system to perform a blob detection algorithm on pixels near the corresponding tapped or drawn circle, and run an ellipse fitting algorithm on the detected blobs. This algorithm can associate all pixels within the ellipse with the tire's sidewall and perform a best-fit analysis on the plane of the tire's sidewall. Stereo parallax measurements of the tire's sidewall can be used to resolve scaling ambiguity in the ellipse fitting algorithm. The tire's position and size can be calculated for the imaging device, where the estimated lower edge of the tire is further used to define ground contact for ground segmentation or sidewall height detection.

[0074] In other embodiments, a series of taps 464 may define other closed regions, such as a first tap identifying a first (e.g., left) side of the loading container on the first image layer, and a second tap identifying a second (e.g., right) side of the loading container contained on the first image layer. These taps may be provided based on prompts from the system to identify specific sides of the loading container or other visible features that may require further identification to benefit, for example, ellipse fitting algorithms.

[0075] The term "tap" as discussed herein can generally include directly engaging a touchscreen user interface using a manual tool or gesture (such as via a stylus, finger, etc.). Such a tap can also include indirectly engaging the user interface (e.g., "click") using an interface tool such as a mouse or trackpad associated with a cursor on the screen. In other words, the tap discussed herein generally refers to any tool, gesture, and / or backend function known to those skilled in the art for enabling selection points on a corresponding image layer displayed on the user interface, regardless of whether the user interface / display unit is associated with an in-vehicle user interface, mobile computing device, remote desktop computer, laptop computer, etc.

[0076] In another example, a closed region 466, comprising a multi-directional joining sequence of polygons (e.g., typically quadrilaterals), has been delineated to indicate a contour such as the surface of a truck bed sidewall. This type of user input can trigger the system to run an edge detection algorithm along the lines of the polygon, as in the edge indication process referenced above, and associate the pixels within the drawn polygon as a single surface. All available distance measurements within the polygon can be combined to describe the best-fit location and plane of the surface. The pixels of the detected edges of the drawn polygon can be projected onto the best-fit surface, where the actual location of the indicated edge can be determined on the detected surface, and the edges of the sidewall are associated with the necessary measurements of the control system.

[0077] Based at least in part on the background input from step 460, method 400 can continue by algorithmically predicting the contours and / or locations of one or more features of the ground surface 470, where such details cannot be sufficiently determined from the first image layer alone.

[0078] If the necessary outlines and / or other elements can be determined with sufficient clarity based on the processed image and user input for the background, then the method can proceed to automate the user-selected portion of the loading sequence (e.g., approaching loading area 302, loading material into loading area 302, and / or retrieving it from loading area 302). Figure 13 This indicates that the background input screen 300 indicates the system is now able to continue the first automated task (“Approach & Boom”).

[0079] In this alternative example, such as Figure 14 This indicates that the box corresponding to the first automated task can be highlighted (e.g., in red) 380 to indicate that manual takeover of the associated operation is required due to automation being unavailable or other considerations, such as problems with identifying image features using background input as described above.

[0080] Upon completion of the user-selected portion of the loading sequence, the method may further determine whether subsequent portions of the loading sequence are further expected to be automated, whether manual intervention is required, and / or whether user prompts are appropriate for clarification. An automated unloading (“dumping”) routine may, for example (in the context of using a loader bucket for illustrative purposes): shifting the work vehicle 100 to neutral, automatically dumping the bucket while raising the boom to prevent the bucket from contacting the loading area 302, and instructing the operator that unloading is complete and the work vehicle should be shifted to reverse.

[0081] If, for example, the user has specified an automated sequence including the return of work vehicle 100 from loading area 302 to digging area and the digging process, the method may further include such a process in step 490, and optionally prompt the user to clarify the captured images and the elements associated with them. The system may, for example, generate control signals associated with the retraction of work vehicle 100 and attachment 120 from loading area 302, which may be provided for one or more of the following: controlling the ground speed or steering of work vehicle 100 when work vehicle 100 reverses to prevent the bucket from contacting loading area 302; controlling the boom and bucket to prevent the bucket from contacting loading area (e.g., truck floor) 302 when the loader reverses from loading area 302; and returning the attachment to a predetermined position based on system settings. For example, during an illustrative and non-limiting retraction operation, the bucket may be guided to a digging or transporting position, and the boom may be guided to a transporting position.

[0082] As used herein, the phrase “one or more of…” when used with a list of items means that different combinations of one or more of these items may be used, and that only one of the individual items in the list may be required. For example, “one or more of items A, B, and C” may include, but is not limited to, item A, or items A and B. The example may also include items A, B, and C, or items B and C.

[0083] Those skilled in the art will recognize that when an element is referred to herein as “linked to” another element, the element may be directly connected to the other element, or there may be an inserting element.

[0084] Therefore, it can be seen that the apparatus and methods of this disclosure readily achieve the mentioned and inherent purposes and advantages. While certain preferred embodiments of this disclosure have been illustrated and described for these purposes, many changes can be made by those skilled in the art to the arrangement and construction of components and steps, and these changes are covered within the scope and spirit of this disclosure as defined by the appended claims. Features or embodiments of each disclosure can be combined with any of the features or embodiments of other disclosures.

Claims

1. A computer-implemented method (400) for selective input confirmation of automated loading of a work vehicle (100), the work vehicle (100) comprising a main frame (132) and at least one work attachment (120), the at least one work attachment (120) being movable relative to the main frame and configured to load and unload materials in a loading area (302) outside the work vehicle during a loading process having one or more loading stages, the method being characterized in that: One or more location inputs for the loading area are detected via at least one detector (202, 204, 206) associated with the work vehicle; Receive a first user input (420), the first user input (420) corresponding to the selected automation of any one or more of the one or more loading stages; For one or more selective automated loading phases, a detection routine (430) is executed in relation to the parameters of the loading region based on one or more detected location inputs, and it is determined in relation to one or more of the parameters of the loading region whether one or more second user inputs are required (460). as well as If any one or more parameters of the loading area require second user input, the system receives the one or more second user inputs and, at least in part based on the one or more second user inputs, automatically controls the movement of the main frame and / or the at least one work attachment to automate the corresponding loading stage. The image of the loading area is generated on a user interface (300) associated with at least one or more required second user inputs, wherein the one or more second user inputs include one or more engagement points of the user interface and / or continuous movement of the engagement points along the user interface. The method further includes at least one of the following steps: Based at least on the generated image of the loading area and the one or more second user inputs of the user interface, including sweep engagement (462), one or more contours of the container associated with the loading area are automatically estimated (470). Based at least on the generated image of the loading area and the one or more second user inputs of the user interface, including multiple sweeping engagements for defining a closed region (466) in the user interface, one or more contours of the container associated with the loading area are automatically estimated (470). Based at least on the generated image of the loading area and the one or more second user inputs of the user interface, including circular or elliptical sweeping engagements for defining a closed region (466) in the user interface, the contour of the wheel associated with the loading area is automatically estimated (470), and The contour of the wheel associated with the loading area is automatically estimated (470) based at least on the generated image of the loading area and the one or more second user inputs via the user interface, including one or more tap engagements (464).

2. The method according to claim 1, characterized in that, If any one or more of the parameters in the loading area require one or more second user inputs, the user is prompted to input (450) the one or more second user inputs via the user interface, and the one or more second user inputs are received therefrom.

3. The method according to claim 1 or 2, characterized in that, The steps of detecting one or more location inputs include: capturing an image via an imaging device (202).

4. The method according to claim 1 or 2, characterized in that, The parameters of the loading area include one or more of the following: the distance between the loading area and the main frame; the distance between the loading area and the at least one working attachment; the orientation of the loading area relative to the main frame and / or the at least one working attachment; the height of the loading area; the polygonal outline of the container associated with the loading area; and the circular or elliptical outline of the wheels supporting the loading area.

5. A work vehicle (100), said work vehicle (100) comprising: Main frame (132); At least one working attachment (120) is movable relative to the main frame and is configured to load and unload materials in a loading area (302) outside the work vehicle during a loading process having one or more loading stages. At least one detector (202, 204, 206) is configured to detect one or more location inputs for the loading area, respectively, for the main frame and / or the at least one working attachment; A user interface (116, 300) configured to enable at least a first user input (420) corresponding to a selected automation of any one or more of the one or more loading stages; and A controller (112) is configured to direct the execution of the method according to any one of claims 1 to 4.

6. A system comprising: Multiple work vehicles (100), each work vehicle comprising: a main frame (132); at least one work attachment (120) movable relative to the main frame and configured to load and unload materials in a loading area (302) outside the work vehicle during a loading process having one or more loading stages; and at least one detector (202, 204, 206) configured to detect one or more location inputs for the loading area, respectively, for the main frame and / or the at least one work attachment. A user computing device, which is remotely configured relative to each of the plurality of work vehicles, and includes a user interface (116, 300) configured to enable at least a first user input (420) for the corresponding work vehicle corresponding to a selected automation of any one or more of the loading stages. Each of the plurality of work vehicles further includes a controller (112), which is configured accordingly to direct the execution of the method according to any one of claims 1 to 4.

Images