Cargo handling controllers, methods, and computer program products

Abstract

To provide improved means for controlling vehicles equipped with cargo handling equipment, particularly for improving human-machine interaction. [Solution] A cargo handling controller, method, and computer program product are disclosed. Provided are a first virtual marker corresponding to a vehicle's cargo, and one or more commands for positioning the first virtual marker within a representation of the area surrounding the cargo handling vehicle. From the position of the first virtual marker in the representation, a desired position for unloading the cargo relative to the area surrounding the vehicle is determined. Based on the desired position for unloading the cargo relative to the area surrounding the vehicle, control commands for the cargo handling vehicle are provided.

Description

Technical Field

[0001] The present invention relates to controlling a vehicle equipped with load handling equipment, such as a vehicle equipped with a hook lift, a skip loader, etc.

Background Art

[0002] As industries increasingly rely on automation and advanced machinery, it has become essential to have improved means for controlling load handling vehicles. This is particularly true in scenarios involving autonomous operation where accuracy, safety, and efficiency are of utmost priority.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The object is to provide improved means for controlling a vehicle equipped with load handling equipment, particularly for improving human - machine interaction. This can be used for different levels of autonomous operation of the vehicle.

Means for Solving the Problems

[0004] According to a first aspect, a cargo handling controller (also referred to herein as the "Controller") is provided for a vehicle having cargo handling equipment. The cargo handling controller is configured to receive sensor inputs from a system of sensors in the vehicle for monitoring and mapping the area around the vehicle. The system of sensors may include at least an image sensor unit and / or a distance mapping sensor unit. The Controller is also configured to generate a representation of the area around the vehicle based on the sensor inputs. The representation may include one or more images and / or point cloud maps of the area around the vehicle, along with a set of virtual markers for one or more images and / or point cloud maps of the area around the vehicle to facilitate cargo handling. This enables human-machine interaction for a particular aspect of cargo handling.

[0005] The controller is further configured to provide a visual representation of the vehicle's surroundings to the operator of the cargo handling controller on one or more human-machine interface devices. The controller is also configured to collect user commands for cargo handling from one or more human-machine interface devices. User commands may belong to a set of commands available to the operator of the cargo handling controller.

[0006] The set of virtual markers includes a first virtual marker corresponding to the vehicle's load, particularly around the vehicle when it is unloaded. This allows the load to be visualized for the operator within the representation. The set of commands includes one or more commands for positioning the first virtual marker within the representation. This allows the first virtual marker to be moved within the representation, thereby allowing the operator to control the placement of the load within the representation.

[0007] The cargo handling controller is configured to determine a desired position for unloading the cargo relative to the vehicle's surroundings, based on the position of a first virtual marker in the above representation. Furthermore, the controller is configured to provide control commands for the vehicle for cargo handling based on the desired position for unloading the cargo relative to the vehicle's surroundings. This enables efficient human-machine interaction for unloading the cargo, as the first virtual marker can represent the cargo in the above representation, thereby allowing the operator to confirm the correct positioning of the cargo before unloading. This correct positioning may be subject to any conditions relating to the cargo and / or its surroundings. The operator may also visualize the cargo in the surrounding representation before handing control to the autonomous system to move the vehicle to the unloading position.

[0008] In one embodiment, the command set has one or more commands to indicate the desired orientation of the load relative to the representation around the vehicle. Thus, the load handling controller may be configured to provide control commands based on the desired orientation of the load relative to the representation around the vehicle, in addition to the desired position. This enables further improved control of load handling.

[0009] In a further embodiment, one or more commands for indicating the desired orientation of the load relative to a representation of the vehicle's surroundings include a command for rotating a first virtual marker in the representation. Alternatively or additionally, the load handling controller may be configured to determine the desired orientation of the load relative to the representation of the vehicle's surroundings from the orientation of the first virtual marker in the representation. Thus, the operator may have control over the orientation of the load for unloading, for example, in two dimensions, and even in three dimensions.

[0010] In one embodiment, the control command is based on the vehicle's trajectory for unloading a cargo at a target location around the vehicle. The target location around the vehicle may, in particular, correspond to a desired location relative to a representation of the vehicle's surroundings.

[0011] In a further embodiment, the set of virtual markers may have a second virtual marker corresponding to the travel path, which allows the travel path to be visualized for the operator. Alternatively or additionally, the set of commands may include confirmation of the travel path, for example, from the operator of the cargo handling controller. The cargo handling controller may be configured to initiate cargo handling based on the fact that the above confirmation has been provided.

[0012] In one embodiment, the controller is configured to receive one or more user commands as audible input. In particular, the controller may be configured to receive the above confirmation as audible input from the operator of the cargo handling controller. This enables voice control of the cargo handling controller, which can improve the speed and convenience of control under various cargo handling conditions.

[0013] In one embodiment, the representation of the vehicle's surroundings is generated in a view from above the vehicle. This allows for improved control of cargo handling because the operator is not limited to a cabin view, as in conventional systems. The controller may still be configured to allow the operator to switch between two or more views of the representation, including, for example, a view from above the vehicle and / or a view from inside the vehicle.

[0014] In one embodiment, the size of the first virtual marker in the representation, that is, relative to the vehicle in the representation, corresponds to the size of the cargo relative to the vehicle, i.e., its physical size. Since the virtual size of the cargo relative to its surroundings corresponds to the physical size of the cargo relative to its surroundings, cargo handling can be performed with improved accuracy even when the area around the vehicle is congested.

[0015] In one embodiment, the cargo handling controller is configured to determine whether a desired position for unloading a cargo relative to a representation of the vehicle's surroundings is effective for unloading the cargo. This may involve determining whether a first virtual marker in the representation satisfies a minimum threshold distance to an obstacle around the vehicle, and / or whether a travel trajectory can be effectively provided for the desired position and / or desired orientation.

[0016] In one embodiment, cargo handling includes or consists of moving a vehicle transporting the cargo in order to unload the cargo at a desired location.

[0017] In one embodiment, the cargo corresponds to a cargo-carrying object, such as an ISO container, flat rack, flatbed, or dumpster body.

[0018] In one embodiment, the control command includes a drive command to move the vehicle to unload a cargo at a target location around the vehicle. The target location around the vehicle may correspond to a desired location relative to a representation of the vehicle's surroundings.

[0019] According to a second aspect, a method for controlling a vehicle having cargo handling equipment is disclosed. The method comprises receiving sensor inputs from a system of sensors in the vehicle for monitoring and mapping the area around the vehicle. The system of sensors may include at least an image sensor unit and / or a distance mapping sensor unit. The method further comprises generating a representation of the area around the vehicle based on the sensor inputs. The representation may include one or more images and / or a point cloud map of the area around the vehicle, along with a set of virtual markers for one or more images and / or a point cloud map of the area around the vehicle to facilitate cargo handling.

[0020] The method further includes providing a representation of the vehicle's surroundings for the operator to see visually. The method also includes collecting user commands for cargo handling. User commands may belong to a set of commands available to the operator. A set of virtual markers has first virtual markers corresponding to the vehicle's cargo, particularly around the vehicle to be unloaded. A set of commands has one or more commands for positioning the first virtual markers within the representation.

[0021] The method comprises determining a desired position for unloading the cargo relative to the representation of the vehicle's surroundings, based on the position of a first virtual marker in the above representation. Furthermore, the method comprises providing control commands for the vehicle for cargo handling based on the desired position for unloading the cargo relative to the representation of the vehicle's surroundings.

[0022] According to a third aspect, a computer program product, when executed by a computer, has instructions that cause the computer to perform the methods described above.

[0023] It should be understood that the aspects and examples described above can be used in any combination with each other. Some of those aspects and examples can be combined together to form further examples of the present invention.

[0024] Included to provide further understanding and forming part of this specification, the accompanying drawings illustrate examples and, together with the description, help to explain the principles of the present disclosure.

Brief Description of the Drawings

[0025] [Figure 1] It is a diagram schematically showing a system according to an example. [Figure 2] It is a diagram showing a method according to an example. [Figure 3] It is a diagram showing a view for load handling according to an example.

Modes for Carrying Out the Invention

[0026] Similar reference numerals are used to designate equivalent or at least functionally equivalent parts in the accompanying drawings.

[0027] The detailed description provided below together with the accompanying drawings is intended as a description of examples and is not intended to represent the only form in which examples can be constructed or utilized. However, the same or equivalent functions and structures can be achieved by different examples.

[0028] The solutions of the present disclosure can be used for load handling. In particular, they can be utilized for any of user-assisted load handling, remotely operated load handling, or autonomous load handling.

[0029] Figure 1 shows a system 100 having at least a load handling controller 112 (also referred to herein as the “controller”) for a vehicle 120 having load handling equipment. System 100 may also have any or all of the additional components described herein, which may also be provided separately. In particular, the system may have a load handling system (LHS) 110 having a load handling controller 112, and the load handling controller 112 may also be provided as its own unit, i.e., completely separately from the LHS 110 or system 100. The controller may be configured to perform any of the functions described herein for the LHS or to cause them to be performed.

[0030] Controller 112 and LHS 110 may be configured to control load handling for vehicle 120. In particular, controller 112 may be configured to provide travel commands for the vehicle to move to the load and / or to a position for unloading the transported load. Furthermore, controller 112 may be configured to provide operation commands for load handling equipment to move the load, in particular for unloading the load from vehicle 120.

[0031] For any load-handling vehicle, such as the vehicles disclosed herein, the route to the load and / or to the location for unloading the transported load may depend on the loading / unloading process, and therefore, the driving commands may also depend on the loading / unloading process, for example, with respect to the orientation and / or position of the vehicle for loading / unloading. For load handling, the controller 112 may be configured to park the vehicle 120 near the loading and / or unloading location so that the load can be loaded and / or unloaded by the load-handling equipment. For example, in the case of a loader crane, the load may be lifted to or from the loading platform of the vehicle, and in the case of a hook lift, the load may be lifted onto or pushed off the loading platform of the vehicle by the hook lift arm. The controller 112 may also be configured to cause the vehicle's stabilizer legs to be activated for loading / unloading.

[0032] Vehicle 120 may be a freely moving vehicle, in which case it may face a variety of navigation challenges, from collision avoidance to target location. Vehicle 120 may be configured for free movement in two dimensions, for example, on paved or unpaved ground. The vehicle may be a wheeled vehicle that can provide freedom of movement on a work surface. The vehicle may be an autonomous or semi-autonomous vehicle. For example, the vehicle may be configured to move autonomously, but manual braking may still be easily accessible to allow a user monitoring the vehicle's movements to slow down and / or stop the vehicle. The vehicle may be a by-wired vehicle. In particular, the vehicle may be a truck. The vehicle may be configured for a variety of applications, such as transportation, waste disposal, or construction.

[0033] The vehicle may have the necessary safety compliance for autonomous or semi-autonomous cargo handling. In this case, an operator (in-vehicle or out-of-vehicle) may supervise the operation of the vehicle and cargo handling equipment. The vehicle may be equipped with a safety-compliant braking system (e.g., SAE L3, L4, or L5 autonomy) for operator use. Alternatively, the vehicle may be reliable but lack the necessary safety compliance for autonomous cargo handling. In such a case, the vehicle may be configured so that a supervisor initiates the operation of the cargo handling equipment, allowing the supervisor to guide and monitor the cargo handling. The supervisor may then be responsible for exceptional handling (e.g., SAE L2-L3 autonomy).

[0034] Vehicle 120 may be equipped with cargo handling equipment. The cargo handling equipment may be configured as an integral part of the vehicle. The cargo handling equipment may be configured to facilitate loading and / or unloading of the vehicle, i.e., loading and / or unloading cargo, where, in the case of unloading, the cargo is the transported cargo. For this purpose, the cargo handling equipment may have one or more movable cargo handling actuators. The cargo handling equipment may have, or consist of, any of the following, for example, a hook lift, a skip loader, a crane, a roll-off truck hoist, a tail lift, and any other movable arm for cargo handling. The crane may be a loader crane, a truck-mounted crane, a forestry crane, a recycling crane, or any vehicle-mounted crane. Generally, cargo handling equipment may be vehicle-mounted equipment, such as a vehicle-mounted crane, as opposed to a mobile crane having its own chassis. The cargo handling equipment may be detachable from the vehicle.

[0035] The cargo may correspond to a cargo transport object. The cargo transport object may be a container, such as an ISO container, intermodal container, or flat rack container. However, generally speaking, the cargo transport object may be any structured cargo, such as a flatbed, flat rack, bin, recycling container, building material package, dumpster body, or any other type of goods. The cargo may have a gripping interface for (one or more) working units, such as a hook handle or bar, for loading and / or unloading the cargo onto a vehicle.

[0036] System 100 may have a vehicle control unit (VCU) 130, which may be provided as part of LHS 110, as a controller 112 of LHS 110, or as a separate vehicle control system (VCS). For example, the VCU may be integrated into vehicle 120. The VCU may be configured to directly control the vehicle's operation, while the controller 112 may be configured for a higher level of planning for controlling the vehicle. The VCU 130 may be configured to provide actuator commands to vehicle 120 and / or to receive unique measurement data from vehicle 120. The VCU 130 may be configured to provide operation information to the controller 112, such as unique estimates or vehicle status, and / or to receive control settings from the controller 112, such as travel trajectory. Conversely, the controller 112 may be configured to receive operation information from the VCU 130 and / or to provide control settings to the VCU 130.

[0037] The solutions disclosed herein may be offered as vehicle-agnostic solutions, which allows them to be used, for example, as modifications to existing vehicles, without special requirements for the vehicle 120. This can be achieved by configuring the controller 112 to interact with different types of VCUs 130.

[0038] System 100 may also have one or more human-machine-interface (HMI) devices 140 (also referred to herein as “HMI devices”) for enabling the operator 10 of the vehicle 120, in particular a human operator, to interact with the LHS 110 or the controller 112. Thus, the operator of the LHS and / or the controller may be the operator of the vehicle. Thus, “operator” as used herein refers to the operator of the controller, but may additionally refer to the operator of the LHS and / or the vehicle. The HMI may have one or more user interfaces for enabling the operator to input information about the controller and / or receive information from the controller. The user interface may have a display, such as a touch screen, an augmented reality device, virtual reality goggles, or a regular display. The HMI devices may be configured in the vehicle and / or separate from the vehicle, in the latter case enabling the operator to remotely control the vehicle with the HMI devices, for example, with a remote control unit for a loader crane. The HMI device may have, for example, a cloud-connected device for communicating with the controller 112. The HMI device may be configured to provide a distributed user interface in which some components are incorporated into the vehicle and other components are provided for remote control of the vehicle. The system 100 may also have an emergency brake 122 for stopping the vehicle 120, which may be configured to bypass the LHS 110 or the controller 112, thereby reducing the delay in stopping the vehicle.

[0039] System 100 may have a sensor system 150 (also referred to herein as the “Sensor System”) which may be configured in a vehicle. The Sensor System may be configured to monitor and map the surroundings of the vehicle 120. For this purpose, the Sensor System may have at least an image sensor unit and / or a distance mapping sensor unit. The image sensor unit may have one or more cameras, such as RGB cameras, which may be configured for video imaging and / or color imaging. The distance mapping sensor unit may have one or more optical sensors, such as LIDAR and / or time-of-flight cameras. Any other distance mapping sensors may also be used, such as ultrasonic sensors or radar-based sensors.

[0040] System 100 may have an interface 114, which is configured as part of LHS 110 but may also be configured separately from LHS 110. The interface is a communication interface between the controller 112 and the HMI device 140, and may have a wired interface such as one or more data transmission cables, and / or a wireless interface such as a radio frequency communication interface. Similar or corresponding interfaces may also exist between other parts of System 100, for example, between VCU 130 and controller 112 and / or between sensor system 150 and controller. However, these do not need to be described in detail here.

[0041] Controller 112 is a load handling controller, such as an autonomous load handling system (ALHS). Controller 112 can be a centralized or distributed controller. Controller 112 may have one or more control units, which may be collectively positioned or separate. Controller 112 may be configured to be mounted on the vehicle 120 for operation, either completely or partially. For example, Controller 112 may be included as part of a work equipment assembly that has load handling equipment and is configured to be mounted on the vehicle for operation. Alternatively or additionally, Controller 112 may be configured to operate via a remote connection to the vehicle 120, for example, through a VCU 130. This enables remote control of the vehicle. The vehicle may also have a receiver and / or transmitter receiver for communicating with Controller. Generally, The vehicle may have a receiver and / or transmitter coupled to Controller (both options also include the possibility of a transmitter receiver) for providing commands to Controller and / or facilitating the transmission of information from Controller such as status information.

[0042] The controller 112 may have one or more processors (also referred to herein as “processors”) and one or more memories. The memories may have instructions that, when executed by the processors, cause any of the actions described herein with respect to the controller and / or LHS to be performed. In particular, these actions may include any or all of the actions described below, and the controller may be configured accordingly. Similarly, a computer program product may include any or all of the actions described herein, particularly with reference to Figure 2.

[0043] Figure 2 shows a method 200 that may be used to control a vehicle 120 having cargo handling equipment, as described herein. This method, or any part of this method in any combination, may be carried out by the controller 112 and / or LHS 110. Method 200 has several parts that can be carried out independently of each other. These parts may be carried out in the order shown, or in any other applicable order, including in parallel.

[0044] Method 200 comprises receiving sensor inputs from a system of sensors 150 in a vehicle 120 for monitoring and mapping the area around the vehicle 120. Sensor inputs may be received repeatedly over time to continuously monitor the area around the vehicle. Point clouds may be provided through the sensor inputs, in particular from distance mapping sensor inputs, as raw data point clouds and / or adjusted point clouds. Point clouds may be provided according to any method available to those skilled in the art, and point clouds should be in contrast to images, which are obviously constructed in a distinct form, for example, from pixels. Images may also be provided through sensor inputs, in particular from image sensor units. Point clouds may be produced by one or more scanners, in contrast to images that may be produced by cameras. Sensor inputs from image sensor units may be convertible into quantized indications, such as a pixelated view of the area around the vehicle. The system of sensors and / or controllers may be configured to provide machine-readable data matrices for automatic identification of targets such as cargo, and / or human-readable images for interaction with human operators inside / outside the vehicle.

[0045] Sensor inputs are used to generate a representation of the vehicle's surroundings 220. This representation may be generated based solely on the sensor inputs, or on the sensor inputs and additional data, such as additional surrounding data, which may be obtained earlier by the controller 112 and / or from one or more external sources. The representation may be repeatedly updated, particularly for continuous monitoring of the surroundings. This is done especially when the vehicle is moving, but also when the vehicle is stationary and the surroundings may change. The representation may be a three-dimensional representation of the surroundings. The representation may have one or more images, such as color images, and / or a point cloud map, of the vehicle's surroundings. The representation may be provided dynamically so that it can be modified as the vehicle moves and / or the vehicle's surroundings change. The representation may also be provided in multiple views, such as camera views and point cloud views, and the operator may be provided with options for switching between multiple views. Alternatively or additionally, two views may be provided simultaneously, such that one view, for example, a point cloud view, can be incorporated into another view, such as a camera view. Therefore, the solution enables the provision of a virtual user interface for controlling load handling with one or more virtual views of the vehicle's surroundings.

[0046] The virtual user interface also allows the representation of the vehicle's surroundings to be generated in a view from above the vehicle. The operator may still be provided with the possibility to switch between two or more views of the above representation, including, for example, a view from above the vehicle and / or a view from the vehicle itself.

[0047] The above representation may be provided with a set of virtual markers for one or more images and / or point cloud maps around the vehicle to facilitate cargo handling. The set of virtual markers may be provided as part of the above representation, for example, incorporated into the above representation. The virtual markers may be positioned around the vehicle to represent objects and / or commands around the vehicle. The set of virtual markers has a first virtual marker 320 corresponding to the vehicle's cargo. In particular, this may be used as a virtual cargo, which may be used to indicate the drop-off location for the cargo. Thus, the first virtual marker may represent cargo being unloaded around the vehicle.

[0048] The above representation may be generated by the controller 112 itself based on sensor input. The above representation may also be generated based on sensor input and a pre-generated map received by the controller. The controller 112 may be configured to store the above representation in one or more memories. The controller 112 may be configured to activate the pre-generated representation when the vehicle arrives at (e.g., re-arrives at) the surroundings described by the pre-generated representation.

[0049] The above representation describes the area around vehicle 120. The surroundings may have one or more landmarks, such as obstacles. These may include other vehicles, buildings, containers, etc. The surroundings may also have cargo, particularly as a first virtual marker. The above representation may be a two-dimensional or three-dimensional map. The map may have a fixed coordinate system. This makes it possible to provide a reference system that may be stationary. The controller 112 may be configured to facilitate the graphical representation of the above representation, for example, particularly through interface 114 and HMI device 140.

[0050] Method 200 may involve causing a representation of the vehicle's surroundings to be provided to the HMI device 140 to visually show the operator 10 the area around the vehicle. This representation may be provided by the controller 112, in particular by causing the representation to be transmitted to the HMI device 140 via the interface 114. The representation may be shown to the operator through a display, such as a touchscreen, an augmented reality device, virtual reality goggles, or a regular display.

[0051] Similarly, user commands are collected from the HMI device 140 for cargo handling. This can also be done through interface 114, in particular, so that controller 112 collects user commands from the HMI device 140 via interface 114. This may involve collecting at least some of the user commands through a touchscreen and / or through one or more physical actuators, such as buttons, a keyboard, or a joystick. User commands are collected through the same device of the HMI device, such as a touchscreen, used to show the above representation to the operator, but may also be collected through different devices. In addition or alternatively, one or more of the user commands may be received as audible input and / or visual input. Thus, the operator can control cargo handling by voice control and / or gesture control in addition to or as an alternative to physical contact with the HMI device.

[0052] The user commands described above may belong to a set of commands available to operator 10. The set of commands has one or more commands for positioning the first virtual marker within the representation. This allows the first virtual marker to be moved within the representation, thereby allowing the operator to control the placement of the load within the representation, but also to monitor the appearance of the load within the representation, such as the size and / or orientation of the load. This allows the operator to ensure that the load can be effectively unloaded at the desired location before the vehicle is moved to the unloading position. The size (and / or shape) of the first virtual marker within the representation may correspond to the size (and / or shape) of the load relative to the perimeter of the vehicle, which allows the first virtual marker to also represent the space required for the load in the representation relative to the perimeter of the vehicle. The first virtual marker may also have or consist of a bounding box for the load in the representation, such as a rectangular bounding box. This allows for avoidance of overlap of the first virtual marker within the representation, particularly with respect to obstacles, in a simple and straightforward manner. One or more commands for positioning the first virtual marker may consist of one or more translational commands for the first virtual marker, for example, as two pairs in opposite directions, or in four distinct directions.

[0053] The method comprises determining a desired position for unloading the cargo from the position of a first virtual marker in the above expression.250 The desired position may correspond to the position of the vehicle for unloading or the position of the cargo after unloading. Therefore, moving the vehicle to the desired position may correspond to either or both cases. The position of the vehicle for unloading the cargo, which may correspond to the position of the cargo after unloading and corresponds to the position of the first virtual marker in the above expression, may depend on the type of cargo handling equipment that the vehicle is equipped with for cargo handling. As exemplary examples, a hook lift can generally unload cargo from the rear of the vehicle, and a loader crane can unload cargo from the side of the vehicle.

[0054] The set of commands may also include one or more commands for indicating the desired orientation of the load relative to a representation of the vehicle's surroundings. The one or more commands may include or consist of one or more commands for rotating a first virtual marker in the representation. The desired orientation of the load relative to the representation of the vehicle's surroundings may be determined from the orientation of the first virtual marker in the representation. The desired orientation may correspond to the orientation of the load after unloading. The desired position and / or desired orientation may be determined relative to the representation of the vehicle's surroundings.

[0055] The cargo handling described herein includes, or may include, moving a vehicle transporting a cargo in order to unload the cargo at a desired location. Alternatively or additionally, the cargo handling may include, or may include, operating cargo handling equipment in particular to unload the cargo from vehicle 120 at a desired location and, optionally, to move the cargo to a desired orientation.

[0056] The command set may include commands for selecting a desired position and / or for confirming the desired position by, for example, confirming a travel trajectory to a target position that corresponds to the desired position. The command set may include commands for entering an operating state for unloading, i.e., loading / unloading mode, for which the controller may provide a first virtual marker for positioning. The command set may also include commands for stopping the vehicle 120 and / or commands for changing the vehicle's control mode from human operation to autonomous operation. Confirmation commands and / or stopping commands may be received as audible input from the operator, and the controller may be configured accordingly.

[0057] Method 200 may involve determining a travel path for moving the vehicle 120 from its current position to unload a cargo at a target location. The travel path may be determined over the entire distance to the target location. The travel path may be determined in a spatial coordinate system, such as xy or xyz coordinates, in which case the space may have or consist of a two-dimensional or three-dimensional coordinate system. Alternatively or additionally, the travel path may be determined in some other space for moving the vehicle, such as actuator space, i.e., space of control commands for actuators for moving the vehicle. In this case, the number of dimensions of the space may correspond to the number of degrees of freedom for the actuators. The travel path may be represented in vector or matrix form. The travel path is determined based on at least sensor inputs and a desired position. For example, the travel path may be determined based on a representation generated based on sensor inputs. The travel path may be determined as the shortest path from the vehicle's current position to unload a cargo at a target location under arbitrary conditions. Optional conditions may include requirements regarding the orientation of the vehicle and / or cargo handling equipment for unloading at the target location. Optional conditions may also include requirements for avoiding obstacles, such as objects or people, identified in the sensor input.

[0058] The vehicle may be a vehicle with steering, such as a truck with a hook lift, which has non-holonomic dynamics. In such dynamics, the vehicle cannot be commanded to produce lateral movement without steering and having longitudinal velocity. Therefore, the travel trajectory can generally be determined such that the trajectory starts from the vehicle's current position and ends at the unloading position, with a curve that allows the vehicle to smoothly unload the cargo by making free use of cargo handling equipment, such as a hook lift, to position the cargo in the desired orientation, for example, while avoiding possible obstacles.

[0059] A representation of the area around the vehicle 120, along with a representation of the travel path, may be provided to the HMI device 140. The travel path may be represented by lines or by a set of symbols defining lines. This allows for the provision of a graphical representation of the above representation to the operator 10, which may include a representation of the travel path within the representation of the area around the vehicle. The representation of the travel path may extend along the entire length or a portion of the length of the travel path.

[0060] It can be determined whether the desired position for unloading the load relative to the representation of the vehicle's surroundings is effective for unloading the load. This can be done automatically, in particular by the controller 112. Optical recognition means may be used for this purpose. The verification is performed based on the desired position of the load, but may also be performed based on the desired orientation of the load, and / or the properties of the load, such as the size and / or shape of the load. The verification may involve determining whether the vehicle is effectively positioned for unloading and / or whether the load is effectively positioned after unloading. The verification may involve determining whether a first virtual marker in the representation representing the load satisfies one or more conditions in the representation, such as the minimum threshold distance to obstacles around the vehicle, and / or whether a travel trajectory can be effectively provided for the desired position and / or desired orientation. Control commands may be provided to the vehicle for load handling if the desired position and / or desired orientation is effective, in particular if the travel trajectory is effective. The controller may be configured accordingly. For example, the travel path may be made visible to the operator, but may be accompanied by an indication that the travel path (or desired position and / or desired orientation) is invalid. In that case, the operator may be required to re-select the desired position and / or desired orientation.

[0061] Any or all of the commands may be requested in response to a prompt from the controller. The set of commands may have one or more user interface commands, such as commands for zooming in on the views of the above representation, including a view from above the vehicle and / or a view from the vehicle, and / or switching between different views of the above representation.

[0062] Furthermore, method 200 includes providing control commands 260 for a vehicle for cargo handling, particularly for unloading. These may be provided based on a desired position and / or orientation of the cargo relative to the vehicle's surroundings. Control commands may be generated by controller 112. The controller may be configured to cause control commands to be transmitted, in particular to VCU 130, for controlling the vehicle. Control commands may be transmitted for autonomous operation of the vehicle, particularly for autonomous unloading of the cargo, based on or in response to the determination of a desired position and / or orientation.

[0063] Control commands may include drive commands for moving the vehicle and / or actuation commands for activating cargo handling equipment. Control commands may also be commands for the autonomous operation of the vehicle. In particular, drive commands may be for moving the vehicle to unload cargo at a target location around the vehicle. This target location around the vehicle may correspond to a desired location relative to a representation of the vehicle's surroundings.

[0064] Control commands may be based on a travel trajectory for a vehicle to unload a cargo at a target location around the vehicle. The target location around the vehicle may correspond to a desired location relative to a representation of the vehicle's periphery. Thus, travel commands may be provided based on a travel trajectory, but they do not necessarily need to correspond to the entire length of the travel trajectory from the vehicle's current position to the unloading location, which may be represented by the desired location. A single set of travel commands may extend to the unloading location at the end of the travel trajectory, but providing travel commands in shorter increments allows for the saving of both computational and data transmission resources. Travel commands may be provided to extend beyond a threshold length that allows the vehicle to decelerate and stop within an expected trajectory in response to an abrupt stop command, where the expected trajectory represents the spatial trajectory corresponding to the travel command. The threshold length may correspond to the spatial and / or temporal length of the vehicle movement due to the travel command. For example, if a vehicle is instructed to travel at a speed that takes 4 seconds to come to a complete stop, the threshold length corresponds to the travel command for the next 4 seconds of operation. Therefore, if there are only 4 seconds of travel commands remaining and then another travel command is received, the vehicle will be instructed to stop rather than continue until the end of the previous travel command. This increases operator safety and comfort because the next operation is intuitive from the screen and predictable several seconds in advance.

[0065] A set of virtual markers may have a second virtual marker 350 corresponding to the travel path. Thus, the travel path can be visualized for the operator. The second virtual marker may have or consist of one or more path overlays of the travel path.

[0066] The second virtual marker may extend from the vehicle's current position in the above representation and / or to the desired position in the above representation. The second virtual marker may extend over the entire distance from the vehicle's current position to the desired position, but this is not required. The second virtual marker indicates the trajectory from the vehicle's current position to the desired position as a continuous indication, but alternatively, it may indicate it as a discontinuous indication. The second virtual marker may have or consist of one or more lines corresponding to the trajectory. In some embodiments, the second virtual marker is of a length that is at least half the distance between the vehicle's current position and the desired position.

[0067] A second virtual marker may be visualized in the above representation while the operator is selecting a desired position and / or desired orientation, which allows the operator to control the desired position and / or desired orientation based on the visualization of the obtained travel trajectory. Alternatively or additionally, a second virtual marker may be visualized in the above representation based on the confirmation of a desired position and / or desired orientation.

[0068] The set of commands may include, for example, confirmation of the travel trajectory from the operator and / or the supervisory system. The controller may be configured to initiate load handling based on the above confirmation being provided.

[0069] The disclosed solutions may be used for user interfaces for human-machine or human-robot interaction. These solutions may be used, for example, for driver-assisted automated and autonomous operation of vehicles and / or cargo handling equipment for pick-and-place tasks.

[0070] A first virtual marker may be provided as a virtual reality object, such as a virtual loading / unloading cargo. A second virtual marker may be provided as an augmented reality object, such as a route overlay. The first and / or second virtual markers may be overlaid on the above representation, in particular on an image that visualizes the above representation. The first and / or second virtual markers may be visualized on the HMI device 140, for example, on the windshield of a vehicle and / or on a remote device. The HMI device 140 may have virtual reality glasses and / or one or more glasses-free mixed reality devices.

[0071] The first virtual marker may be provided as a virtual load, in particular as a customizable virtual loading / unloading load. The set of commands may have one or more commands for positioning the virtual load within the above representation to indicate a desired position and / or desired orientation for unloading the load.

[0072] Virtual cargo, such as virtual containers, may have dimensions corresponding to the dimensions of cargo being transported by the vehicle. Information regarding these dimensions may be provided to the controller automatically and / or manually. For example, a sensor system may have one or more sensors, such as perceptual sensors, to provide measurements for determining the dimensions of the cargo. Alternatively or additionally, the operator may be provided with a set of alternative forms for cargo types, on which their dimensions can be determined, as they may be provided as predetermined information for each type. The vehicle may also be designed to handle unisized cargo, in which case the size of the first virtual marker may be preset to correspond to the size of unisized cargo. A set of commands may have one or more commands for changing the size of the first virtual marker, or its size may be fixed.

[0073] The orientation of the first virtual marker, i.e., the desired position and / or orientation, can be changed, for example, by dragging the first virtual marker via touchscreen input or mouse input, and / or by using a virtual control interface such as a real joystick and / or virtual joystick.

[0074] Any or all of the user commands may be received through voice recognition, for example, by configuring the controller for voice recognition. The set of commands may include start commands, stop commands, and confirmation commands. Thus, the vehicle may be subject to hands-free control. The controller may provide audio outputs, such as warnings, when a serious safety event occurs. This may include the identification of obstacles around the vehicle and / or issues with the vehicle's internal condition.

[0075] Figure 3 shows an example of a view 300 for load handling as described herein. The view may be provided as a graphical representation, as described above. A controller may be configured to cause the view to be displayed for the operator to select a desired position and / or orientation.

[0076] The view shows how obstacles, such as a person 310 and / or surrounding cargo 330, and the first virtual marker 320 can be visualized based on the representation. For those obstacles, the controller can automatically determine the corresponding virtual markers, such as designated objects 312, 332, and / or bounding boxes in the representation. The view also shows how the minimum threshold distance 322 can be determined and / or visualized. The conditions for satisfying the minimum threshold distance can be determined based on the distance between the first virtual marker and the bounding box of any of the surrounding obstacles 312, 332.

[0077] The desired position 340 may be indicated, for example, simply as the end of the track 342. The track may be indicated by a second virtual marker 350.

[0078] This example further illustrates how a travel command 360 may also be indicated based on its representation, for example, by a third virtual marker. Thus, in the example shown, the desired position 340 may already be confirmed, and therefore the travel command 360 may be provided.

[0079] The user interface may also have one or more controls for user commands, for example, for one or more commands for positioning a first virtual marker. An example of a virtual joystick having translation commands 370 and rotation commands 372 is shown.

[0080] An example of the operation of system 100 is as follows: Operator 10 may provide a loading / unloading location as a desired position through the HMI device 140. Controller 112 may apply optical recognition means to detect whether the intended location is suitable for loading / unloading the cargo in its current view. The operator may select an empty space in the representation around the vehicle, for example from a touch screen, by positioning a virtual cargo in a desired orientation. The controller may then visualize the travel trajectory, for example, as an overall path and / or as a passage from the vehicle to the empty space. Confirmation from the operator may be obtained (for example, by the operator shifting the vehicle's gear into reverse, pressing the throttle pedal once, and / or clicking the HMI device). This may initiate an automatic driving operation for the vehicle. The controller may take control of the vehicle, which allows the vehicle to automatically drive to an orientation suitable for loading / unloading the cargo. Lateral and longitudinal control can minimize discrepancies with the view of the desired loading and unloading space. After the vehicle has been driven to the desired position, the automated driving can end, and the vehicle can reach a stationary state. This can be indicated to the operator by the controller. The controller and / or operator may involve load handling equipment, such as hook lifts, to load and unload the cargo.

[0081] The controllers described above may be implemented in software, hardware, application logic, or a combination of software, hardware, and application logic. The application logic, software, or instruction set may be maintained on any one of various conventional computer-readable media. "Computer-readable media" can be any medium or means that can store, communicate, propagate, or transfer instructions for use by or in relation to instruction execution systems, apparatus, or devices, such as a computer. Computer-readable media may have computer-readable storage media, which can be any medium or means that can store or contain instructions for use by or in relation to instruction execution systems, apparatus, or devices, such as a computer. Examples of such media can store information relating to the various processes described herein. This information may be stored in one or more memories, such as hard disks, optical disks, magneto-optical disks, or RAM. One or more databases may store the information used to implement the embodiments. Databases may be organized using data structures (e.g., records, tables, arrays, fields, graphs, trees, lists, etc.) contained in one or more memory or storage devices listed herein. The database may reside on one or more devices, including local and / or remote devices such as servers. The processes described in relation to the embodiment may include appropriate data structures for storing the data collected and / or generated by the processes of the embodiment's devices and subsystems in one or more databases.

[0082] As will be understood by (one or more) parties in the computer and / or software field, all or part of the embodiments may be implemented using one or more general-purpose processors, microprocessors, digital signal processors, microcontrollers, etc., programmed in accordance with the teachings of the embodiments. As will be understood by parties in the software field, appropriate software may be readily prepared by a programmer skilled in the art based on the teachings of the embodiments. Furthermore, as will be understood by (one or more) parties skilled in the electrical field, the embodiments may be implemented by preparing application-specific integrated circuits or by interconnecting appropriate networks of conventional component circuits. Thus, the embodiments are not limited to any particular combination of hardware and / or software.

[0083] The different functions described herein may be performed in different orders and / or in parallel with one another.

[0084] Unless otherwise specified, any range or device value given herein may be extended or modified without loss of the desired effect. Furthermore, unless expressly denied, any example may be combined with another example.

[0085] While the subject matter has been described using terminology specific to structural features and / or actions, it should be understood that the subject matter as defined in the attached claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as examples of implementing the claims, and other equivalent features and actions are intended to fall within the scope of the claims.

[0086] It should be understood that the benefits and advantages described above may relate to one or more embodiments. The embodiments are not limited to those that solve any or all of the problems described, or that possess any or all of the benefits and advantages described. Furthermore, it should be understood that a reference to an item may refer to one or more of those items.

[0087] The term “comprising” is used herein to mean that the identified method, block, or element includes, but such block or element does not have an exclusive list, and that the method or apparatus may include additional blocks or elements.

[0088] Numeric descriptors such as "first," "second," etc., are used in this document merely as a way to distinguish between parts that may have similar names. Numeric descriptors should not be interpreted as indicating a specific order, such as the order of preferences, production, or occurrence in a particular structure.

[0089] While the present invention has been described in relation to a particular type of apparatus and / or method, it should be understood that the present invention is not limited to a particular type of apparatus and / or method. While several examples, embodiments, and implementations have been described, the present invention is not limited in this respect, but rather covers a variety of modifications and equivalent configurations that fall within the scope of the claims. Although various examples have been described above with some degree of specificity, or with respect to one or more individual embodiments, those skilled in the art can make numerous modifications to the disclosed examples without departing from the scope of this specification.

Claims

1. A cargo handling controller for a vehicle equipped with cargo handling equipment, wherein the cargo handling controller is - Receiving sensor input from a sensor system in the vehicle having at least an image sensor unit and a distance mapping sensor unit for monitoring and mapping the area around the vehicle, - To generate a representation of the vehicle's surroundings based on the sensor input, wherein the representation comprises one or more images and / or a point cloud map of the vehicle's surroundings, and a set of virtual markers for the one or more images and / or point cloud map of the vehicle's surroundings to facilitate cargo handling. - To provide the representation of the vehicle's surroundings for visual explanation to the operator of the cargo handling controller on one or more human-machine interface devices, - Collecting user commands for cargo handling from one or more of the aforementioned human-machine interface devices, wherein the user commands belong to a set of commands available to the operator of the cargo handling controller. It is configured to do the following: The set of virtual markers has a first virtual marker corresponding to the load of the vehicle, and the set of commands has one or more commands for positioning the first virtual marker within the representation, The cargo handling controller is configured to determine a desired position for unloading the cargo relative to the representation of the vehicle's surroundings, based on the position of the first virtual marker in the representation, and to provide control commands for the vehicle for cargo handling based on the desired position for unloading the cargo relative to the representation of the vehicle's surroundings.

2. The cargo handling controller according to claim 1, wherein the set of commands comprises one or more commands for indicating the desired orientation of the cargo to the representation of the vehicle's surroundings, and the cargo handling controller is configured to provide the control commands based on the desired orientation of the cargo to the representation of the vehicle's surroundings.

3. The cargo handling controller according to claim 2, wherein the one or more commands for indicating the desired orientation of the cargo with respect to the representation of the vehicle's surroundings include a command for rotating the first virtual marker in the representation, and the cargo handling controller is configured to determine the desired orientation of the cargo with respect to the representation of the vehicle's surroundings from the orientation of the first virtual marker in the representation.

4. A cargo handling controller according to any one of claims 1 to 3, wherein the control command is based on a travel path for the vehicle in order to unload the cargo at a target position in the vicinity of the vehicle, and the target position in the vicinity of the vehicle corresponds to the desired position relative to the representation of the vicinity of the vehicle.

5. The cargo handling controller according to claim 4, wherein the set of virtual markers has a second virtual marker corresponding to the travel trajectory, the set of commands has confirmation of the travel trajectory, and the cargo handling controller is configured to initiate cargo handling based on the provided confirmation.

6. The cargo handling controller according to any one of claims 1 to 5, wherein the controller is configured to receive one or more of the user commands as an audible input.

7. The cargo handling controller according to any one of claims 1 to 6, wherein the representation of the vehicle's surroundings is generated in a view from above the vehicle.

8. A cargo handling controller according to any one of claims 1 to 7, wherein the size of the first virtual marker in the representation corresponds to the size of the cargo relative to the surroundings of the vehicle.

9. A cargo handling controller according to any one of claims 1 to 8, wherein the cargo handling controller is configured to determine whether the desired position for unloading the cargo relative to the representation of the vehicle around the cargo is effective for unloading the cargo, and provides the vehicle with the control command for cargo handling if the desired position is effective.

10. The cargo handling controller according to any one of claims 1 to 9, wherein the cargo handling comprises moving the vehicle transporting the cargo in order to unload the cargo at the desired location.

11. The cargo handling controller according to any one of claims 1 to 10, wherein the cargo corresponds to a cargo transport object such as an ISO container, flat rack, flatbed, or dumpster body.

12. The cargo handling controller according to any one of claims 1 to 11, wherein the control command has a drive command for moving the vehicle to unload the cargo at a target position in the vicinity of the vehicle, and the target position in the vicinity of the vehicle corresponds to the desired position in relation to the representation of the vicinity of the vehicle.

13. A method for controlling a vehicle equipped with cargo handling equipment, wherein the method is - Receiving sensor input from a sensor system in the vehicle having at least an image sensor unit and a distance mapping sensor unit for monitoring and mapping the area around the vehicle, - To generate a representation of the vehicle's surroundings based on the sensor input, wherein the representation comprises one or more images and / or a point cloud map of the vehicle's surroundings, and a set of virtual markers for the one or more images and / or point cloud map of the vehicle's surroundings to facilitate cargo handling. - To provide a visual representation of the vehicle's surroundings for the operator, - To collect user commands for the aforementioned cargo handling, wherein the user commands belong to a set of commands available to the operator. Includes, The set of virtual markers has a first virtual marker corresponding to the load of the vehicle, and the set of commands has one or more commands for positioning the first virtual marker within the representation, The method comprises determining a desired position for unloading the cargo relative to the representation surrounding the vehicle, based on the position of the first virtual marker in the representation, and providing a control command for the vehicle for cargo handling based on the desired position for unloading the cargo relative to the representation surrounding the vehicle.

14. A computer program product having an instruction, when executed by a computer, that causes the computer to perform the method according to claim 13.

Images