Cargo handling systems, methods, and computer program products

Abstract

In particular, to provide improved means for controlling vehicles equipped with cargo handling equipment, for the purpose of improving human-machine interaction. [Solution] A cargo handling system, method, and computer program product are disclosed. In a target selection operation state, a target position is identified as the first selected target position in the representation around the cargo handling vehicle and is confirmed as a confirmed target position by obtaining confirmation of the first selected target position. In a vehicle driving operation state, a driving command provided based on the confirmed target position is transmitted to a vehicle control unit.

Description

Technical Field

[0001] The present invention relates to controlling a vehicle having load handling equipment, such as a vehicle having a hook-lift, a skip loader, etc.

Background Art

[0002] As industries increasingly rely on automation and advanced machinery, it has become essential to have improved means for controlling load handling vehicles. This is particularly true in scenarios involving autonomous operation where accuracy, safety, and efficiency are of utmost priority.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The object is to provide improved means for controlling a vehicle having load handling equipment, particularly for improving human-machine interaction. This can be used for different levels of autonomous operation of the vehicle.

Means for Solving the Problems

[0004] According to a first aspect, a cargo handling system for a vehicle having cargo handling equipment is disclosed. The cargo handling system comprises a controller configured to receive sensor inputs from a system of sensors on the vehicle for monitoring and mapping the area around the vehicle. The system of sensors may comprise at least an image sensor unit and / or a distance mapping sensor unit. The controller is also configured to identify a target location for the vehicle for cargo handling and to determine a travel trajectory for moving the vehicle from its current location to the target location, based on the sensor inputs and the target location. The controller is further configured to provide travel commands to a vehicle control unit that controls the movement of the vehicle, based on the travel trajectory, and to generate a representation of the area around the vehicle, based on the sensor inputs. The representation may include one or more images and / or a point cloud map of the area around the vehicle.

[0005] The cargo handling system may also include an interface between the controller and one or more human-machine interface devices. Therefore, the cargo handling system may be configured to provide one or more human-machine interface devices with a representation of the vehicle's surroundings to the operator of the cargo handling system through the interface. The cargo handling system may also be configured to collect user commands from one or more human-machine interface devices to the controller. These user commands may belong to a set of commands available to the operator of the cargo handling system.

[0006] The controller is further configured to operate in two distinct operating states, including a target selection operating state and a vehicle driving operating state. In the target selection operating state, the controller is configured to identify a target position in the representation of the vehicle's surroundings as the initially selected target position and to confirm the target position as a confirmed target position by obtaining confirmation of the initially selected target position. In the vehicle driving operating state, the controller is configured to cause a driving command provided based on the confirmed target position to be transmitted to the vehicle control unit. The controller is configured to transition from the target selection operating state to the vehicle driving operating state based on the confirmation of the target position.

[0007] The solution enables effective separation of human control over a vehicle from machine control, while providing appropriate security measures through verification. The target selection operation allows for both target identification / selection and verification for autonomous driving of the vehicle during vehicle operation.

[0008] In one embodiment, the set of commands available to the operator of the cargo handling system has a first subset of commands available to the operator of the cargo handling system when the controller is operating in a target selection operating state, and a second subset of commands available to the operator of the cargo handling system when the controller is operating in a vehicle driving operating state. Unlike the first subset of commands and the second subset of commands, this allows different operating states to be adapted for different degrees of human-machine interaction, in particular the vehicle driving operating state having an increased degree of autonomous control over the vehicle, for example, autonomous driving of the vehicle. In the vehicle driving operating state, human control over the movement of the vehicle may be limited to stop commands, in which case the controller may be configured to exit the vehicle driving operating state.

[0009] In one embodiment, the representation of the vehicle's surroundings is provided to one or more human-machine interface devices, along with the representation of the travel path, in order to provide the operator of the cargo handling system with a graphical representation of the vehicle's surroundings, including a representation of the travel path.

[0010] In one embodiment, the controller is configured to transition from a target selection operation state to a vehicle driving operation state in response to the confirmation of the target position.

[0011] In one embodiment, the controller is configured to obtain confirmation of the initially selected target position by obtaining confirmation of the travel trajectory for moving the vehicle from the vehicle's current position to the initially selected target position.

[0012] In one embodiment, the controller is further configured to obtain confirmation of the initially selected target position as confirmation of a graphical representation, the graphical representation including a travel trajectory to the initially selected target position. This allows the operator to visually observe how the travel trajectory behaves around the vehicle and provide the above confirmation accordingly.

[0013] In one embodiment, confirmation of the initially selected target position includes at least one of the following: a command from the operator of the cargo handling system collected from one or more human-machine interface devices via an interface to the controller; a command from the operator of the cargo handling system obtained from a confirmation device to the controller; and a response signal, such as an automatic response signal, from one or more human-machine interface devices confirming that a graphical representation has been provided for visual indication to the operator of the cargo handling system.

[0014] In one embodiment, the controller is configured to provide a zoomed-in view of the representation (or indication thereof) of the area around the vehicle around the initially selected target location in order to obtain confirmation of the initially selected target location during the target selection operation state.

[0015] In one embodiment, the target position is the position for loading or unloading cargo using cargo handling equipment.

[0016] In one embodiment, the target location is defined relative to an identified load for loading, or relative to an identified location relative to a transported load for unloading.

[0017] In one embodiment, the controller is configured to repeatedly update sensor inputs to provide updated sensor inputs during vehicle driving, and to repeatedly update driving commands to the vehicle control unit based on the updated sensor inputs. This allows, on the one hand, to reduce the impact of deviations on controlling the vehicle, and on the other hand, to provide driving commands to the vehicle in a reduced-size portion, thereby simplifying the requirements for data control in the vehicle and data transfer to the vehicle. In a further embodiment, the controller is configured to repeatedly update the driving trajectory based on updated sensor inputs and confirmed target positions in vehicle driving, in order to provide an updated driving trajectory. Then, driving commands to the vehicle control unit may be repeatedly updated based on the updated driving trajectory and, by extension, based on the updated sensor inputs.

[0018] According to a second aspect, a method for controlling a vehicle having cargo handling equipment may be provided. The method includes receiving sensor inputs from a system of sensors on the vehicle for monitoring and mapping the area around the vehicle. The system of sensors may include at least an image sensor unit and / or a distance mapping sensor unit. The method further includes identifying a target location for the vehicle for cargo handling and determining a travel trajectory for moving the vehicle from its current location to the target location based on the sensor inputs and the target location. The method includes providing travel commands to a vehicle control unit that controls the movement of the vehicle based on the travel trajectory and generating a representation of the area around the vehicle based on the sensor inputs. The representation may include one or more images and / or a point cloud map of the area around the vehicle.

[0019] The method may include causing a representation of the vehicle's surroundings to be provided to one or more human-machine interface devices to visually indicate the vehicle's surroundings to the operator of the cargo handling system. The method may also include collecting user commands from one or more human-machine interface devices. User commands may belong to a set of commands available to the operator of the cargo handling system.

[0020] This method may include operations in two separate operating states, including a target selection operating state and a vehicle driving operating state. This method includes a transition from the target selection operating state to the vehicle driving operating state based on the confirmation of the target position. In the target selection operating state, the target position is identified as the first selected target position in the representation of the vehicle's surroundings and is confirmed as a confirmed target position by obtaining confirmation of the first selected target position. A driving command provided based on the confirmed target position is triggered to be transmitted to the vehicle control unit in the vehicle driving operating state.

[0021] According to a third aspect, a computer program product is disclosed. When executed by a computer, this computer program product comprises instructions for causing the computer to perform the above-described method.

[0022] It should be understood that the aspects and embodiments described above can be used in any combination with each other. Some of those aspects and embodiments can be combined together to form further embodiments of the present invention.

[0023] Included to provide further understanding and forming part of this specification, the accompanying drawings illustrate examples and, together with the description, help to explain the principles of the present disclosure.

Brief Description of the Drawings

[0024] [Figure 1] A diagram schematically showing a system according to an example. [Figure 2] A diagram showing a method according to an example. [Figure 3] A diagram showing a view for load handling according to an example.

Modes for Carrying Out the Invention

[0025] Like reference numerals are used in the accompanying drawings to designate equivalent or at least functionally equivalent parts.

[0026] The detailed description provided below together with the accompanying drawings is intended as an explanation of examples and is not intended to represent the only form in which examples can be constructed or utilized. However, the same or equivalent functions and structures can be achieved by different examples.

[0027] <o The solutions of the present disclosure can be used for load handling. In particular, they can be utilized for any of user-assisted load handling, remote-controlled load handling, or autonomous load handling.

[0028] [[ID=4I]] Figure 1 shows a system 100 comprising at least a load handling system (LHS) 110 (also referred to herein as "LHS") for a vehicle 120 having load handling equipment. System 100 may also comprise any or all of the additional components described herein, which may also be provided separately.

[0029] LHS110 may be configured to control cargo handling for vehicle 120. For this purpose, LHS110 may include a controller 112. In particular, controller 112 may be configured to provide travel commands for the vehicle to move to the cargo and / or to a position for unloading the transported cargo. Furthermore, controller 112 may be configured to provide operational commands for cargo handling equipment to move the cargo. For any cargo handling vehicle, such as the vehicle disclosed herein, the route to the cargo and / or to the position for unloading the transported cargo may depend on the loading / unloading process, and therefore travel commands may also depend on the loading / unloading process, for example, with respect to the orientation and / or position of the vehicle for loading / unloading. With respect to cargo handling, controller 112 may be configured to park vehicle 120 near the loading and / or unloading position so that the cargo can be loaded and / or unloaded by the cargo handling equipment. For example, in the case of a loader crane, the load may be lifted to or from the vehicle's loading platform, and in the case of a hook lift, the load may be lifted onto or pushed off the vehicle's loading platform by the hook lift arm. The LHS may also be set to cause the vehicle's stabilizer legs to be activated for loading / unloading.

[0030] Vehicle 120 may be a freely moving vehicle, in which case it may face a variety of navigation challenges, from collision avoidance to target location. Vehicle 120 may be configured for free movement in two dimensions, for example, on paved or unpaved ground. The vehicle may be a wheeled vehicle that can provide freedom of movement on a work surface. The vehicle may be an autonomous or semi-autonomous vehicle. For example, the vehicle may be configured to move autonomously, but manual braking may still be easily accessible to allow a user monitoring the vehicle's movements to slow down and / or stop the vehicle. The vehicle may be a by-wired vehicle. In particular, the vehicle may be a truck. The vehicle may be configured for a variety of applications, such as transportation, waste disposal, or construction.

[0031] The vehicle may have the necessary safety compliance for autonomous or semi-autonomous cargo handling. In this case, an operator (in-vehicle or out-of-vehicle) may supervise the operation of the vehicle and cargo handling equipment. The vehicle may be equipped with a safety-compliant braking system (e.g., SAE L3, L4, or L5 autonomy) for operator use. Alternatively, the vehicle may be reliable but lack the necessary safety compliance for autonomous cargo handling. In such a case, the vehicle may be configured so that a supervisor initiates the operation of the cargo handling equipment, allowing the supervisor to guide and monitor the cargo handling. The supervisor may then be responsible for exceptional handling (e.g., SAE L2-L3 autonomy).

[0032] Vehicle 120 may be equipped with cargo handling equipment. The cargo handling equipment may be configured as an integral part of the vehicle. The cargo handling equipment may be configured to facilitate loading and / or unloading of the vehicle, i.e., loading and / or unloading cargo, where, in the case of unloading, the cargo is the transported cargo. For this purpose, the cargo handling equipment may comprise one or more movable cargo handling actuators. The cargo handling equipment may comprise, or consist of, for example, a hook lift, a skip loader, a crane, a roll-off truck hoist, a tail lift, and any other movable arm for cargo handling. The crane may be a loader crane, a truck-mounted crane, a forestry crane, a recycling crane, or any vehicle-mounted crane. Generally, cargo handling equipment may be vehicle-mounted equipment, such as a vehicle-mounted crane, as opposed to a mobile crane having its own chassis. The cargo handling equipment may be detachable from the vehicle.

[0033] The cargo 320 may correspond to a cargo-carrying object. The cargo-carrying object may be a container, such as an ISO container, intermodal container, or flat rack container. However, generally speaking, the cargo-carrying object may be any structured cargo, such as a flatbed, flat rack, bin, recycling container, building material package, dumpster body, or any other type of goods. The cargo may be equipped with a gripping interface for (one or more) working units, such as a hook handle or bar, for loading and / or unloading the cargo onto a vehicle.

[0034] System 100 may include a vehicle control unit (VCU) 130, which may be provided as part of LHS 110, as a controller 112 of LHS 110, or as a separate vehicle control system (VCS). For example, the VCU may be integrated into vehicle 120. The VCU may be configured to directly control the vehicle's operation, while the controller 112 may be configured for a higher level of planning for controlling the vehicle. The VCU 130 may be configured to provide actuator commands to vehicle 120 and / or to receive unique measurement data from vehicle 120. The VCU 130 may be configured to provide operation information to the controller 112, such as unique estimates or vehicle status, and / or to receive control settings from the controller 112, such as travel trajectory. Conversely, the controller 112 may be configured to receive operation information from the VCU 130 and / or to provide control settings to the VCU 130.

[0035] The solutions disclosed herein may be offered as vehicle-agnostic solutions, which allows them to be used, for example, as modifications to existing vehicles, without special requirements for the vehicle 120. This can be achieved by configuring the controller 112 to interact with different types of VCUs 130.

[0036] System 100 may also include one or more human-machine interface (HMI) devices 140 (also referred to herein as “HMI devices”) for enabling the operator 10 of the vehicle 120, in particular a human operator, to interact with the LHS 110 or its controller 112. Thus, the operator of the LHS may be the operator of the vehicle. The HMI may include one or more user interfaces for enabling the operator to input information about the controller and / or receive information from the controller. The user interface may include a display, such as a touch screen, an augmented reality device, virtual reality goggles, or a conventional display. The HMI device may be configured in the vehicle and / or separate from the vehicle, in the latter case enabling the operator to remotely control the vehicle with the HMI device, for example, with a remote control unit for a loader crane. The HMI device may include, for example, a cloud-connected device for communicating with the controller 112. The HMI device may be configured to provide a distributed user interface in which some components are incorporated into the vehicle and other components are provided for remote control of the vehicle. System 100 may also include an emergency brake 122 for stopping the vehicle 120, which may be configured to bypass the LHS 110 and thereby reduce the delay for stopping the vehicle.

[0037] System 100 may comprise a sensor system 150 (also referred to herein as the “Sensor System”) which may be configured in a vehicle. The Sensor System may be configured to monitor and map the surroundings of the vehicle 120. For this purpose, the Sensor System may comprise at least an image sensor unit and / or a distance mapping sensor unit. The image sensor unit may comprise one or more cameras, such as RGB cameras, which may be configured for video imaging and / or color imaging. The distance mapping sensor unit may comprise one or more optical sensors, such as LiDAR and / or time-of-flight cameras. Any other distance mapping sensors, such as ultrasonic sensors or radar-based sensors, may also be used.

[0038] System 100 may include an interface 114, which is configured as part of LHS 110 but may also be configured separately from LHS 110. The interface is a communication interface between the controller 112 and the HMI device 140, and may include a wired interface such as one or more data transmission cables, and / or a wireless interface such as a radio frequency communication interface. Similar or corresponding interfaces may also exist between other parts of System 100, for example, between VCU 130 and controller 112 and / or between sensor system 150 and controller. However, these do not need to be described in detail here.

[0039] Controller 112 may be a load handling controller, such as an autonomous load handling system (ALHS). Controller 112 may be a centralized or distributed controller. Controller 112 may comprise one or more control units, which may be collectively positioned or separate. Controller 112 may be configured to be mounted on the vehicle 120 for operation, either entirely or partially. For example, Controller 112 may be included as part of a work equipment assembly that includes load handling equipment and is configured to be mounted on the vehicle for operation. Alternatively or additionally, Controller 112 may be configured to operate via a remote connection to the vehicle 120, for example, through a VCU 130. This enables remote control of the vehicle. The vehicle may also comprise a receiver and / or transmitter receiver for communicating with Controller. Generally, The vehicle may comprise a receiver and / or transmitter coupled to Controller (both options also include the possibility of a transmitter receiver) for providing commands to Controller and / or facilitating the transmission of information from Controller such as status information.

[0040] The controller 112 may comprise one or more processors (also referred to herein as “processors”) and one or more memories. The memories may comprise instructions that, when executed by the processors, cause any of the actions described herein with respect to the controller and / or LHS to be performed. In particular, these actions may include any or all of the actions described herein with reference to Figure 2, and the controller may be configured accordingly. Similarly, a computer program product may comprise any or all of the actions described herein with reference to Figure 2.

[0041] Figure 2 shows a method 200 that may be used to control a vehicle 120 having cargo handling equipment, as described herein. This method, or any combination of parts thereof, may be implemented by the controller 112 and / or LHS 110. Method 200 comprises several parts that may be implemented independently of each other. These parts may be implemented in the order shown, or in any other applicable order, including in parallel.

[0042] Method 200 includes receiving sensor inputs from a system of sensors 150 in a vehicle 120 for monitoring and mapping the area around the vehicle 120. Sensor inputs may be received repeatedly over time to continuously monitor the area around the vehicle. Point clouds may be provided through the sensor inputs, in particular from distance mapping sensor inputs, as raw data point clouds and / or adjusted point clouds. Point clouds may be provided according to any method available to those skilled in the art, and point clouds should be in contrast to images, which are obviously constructed in a distinct form, for example, from pixels. Images may also be provided through sensor inputs, in particular from image sensor units. Point clouds may be produced by one or more scanners, in contrast to images that may be produced by cameras. Sensor inputs from image sensor units may be convertible into quantized indications, such as a pixelated view of the area around the vehicle. The system of sensors and / or controllers may be configured to provide machine-readable data matrices for automatic identification of targets such as cargo, and / or human-readable images for interaction with human operators inside / outside the vehicle.

[0043] Sensor inputs are used to generate a representation of the vehicle's surroundings in 220. This representation may be generated based solely on sensor inputs, or on sensor inputs and additional data, such as additional surrounding data, which may be obtained earlier by LHS110 and / or from one or more external sources. The representation may be repeatedly updated, particularly for continuous monitoring of the surroundings. This is done especially when the vehicle is moving, but may also be done when the vehicle is stationary and the surroundings may change. The representation may be a three-dimensional representation of the surroundings. The representation may include one or more images, such as color images, and / or a point cloud map, of the vehicle's surroundings. The representation may be provided dynamically so that it changes when the vehicle moves and / or the vehicle's surroundings change. The representation may also be provided in multiple views, such as camera views and point cloud views, and the operator may be provided with options for switching between multiple views. Alternatively or additionally, two views may be provided simultaneously, such that one view, for example, a point cloud view, can be incorporated into another view, such as a camera view.

[0044] The above representation may be generated by the controller 112 itself based on sensor input. The above representation may also be generated based on sensor input and a pre-generated map received by the controller. The controller 112 may be configured to store the above representation in one or more memories. The controller 112 may be configured to activate the pre-generated representation when the vehicle arrives at (e.g., re-arrives at) the surroundings described by the pre-generated representation.

[0045] The above representation describes the area around vehicle 120. The surroundings may include one or more landmarks, such as obstacles 310. These may include other vehicles, buildings, containers, etc. The surroundings may also include cargo. The above representation may be a two-dimensional or three-dimensional map. The map may have a fixed coordinate system. This makes it possible to provide a reference system that may be stationary. The controller 112 may be configured to facilitate the graphical representation of the above representation, for example, particularly through interface 114 and HMI device 140.

[0046] Method 200 may include causing an HMI device 140 to provide an HMI device 140 with a representation of the vehicle's surroundings to the operator 10 of the LHS 110 to visually show the vehicle's surroundings. This representation may be provided through interface 114, in particular by causing controller 112 to transmit the representation to HMI device 140 via interface 114. The above representation may be shown to the operator through a display, such as a touch screen, an augmented reality device, virtual reality goggles, or a regular display. Similarly, user commands may be collected from HMI device 140, and this may also be done through interface 114, in particular by controller 112 collecting user commands from HMI device 140 via interface 114. This may involve collecting at least some of the user commands through a touch screen and / or through one or more physical actuators, such as buttons, a keyboard, or a joystick. User commands may be collected through the same device of the HMI device, such as a touch screen, used to show the above representation to the operator, but may also be collected through different devices.

[0047] Method 200 includes identifying target locations 230 for a vehicle 120 for cargo handling. Target locations may be locations for loading and / or unloading cargo with cargo handling equipment. Target locations may be identified manually, for example by operator 10, and / or automatically, for example by controller 112. Target locations are identified by user commands, but may also be identified by automated pattern recognition, for example. One or more initial targets 330 for selection by operator 10 may also be identified automatically. For example, target locations may be identified from the front of cargo 320, such as a container. The operator may also be provided with an opportunity to change the default selection for target locations and / or to add target locations, even when it is not provided as an existing alternative. Target locations may be identified from the above expression as described above, or from something indicating the above expression. The target location is defined, for example, as the location of a vehicle for loading / unloading, relative to an identified load for loading, or as the location of an identified load for unloading a transported load.

[0048] Method 200 includes determining a travel path 240 for moving the vehicle 120 from its current position to a target position. The travel path may be determined over the entire distance to the target position. The travel path may be determined in space of spatial coordinates, such as xy or xyz coordinates, in which case the space may include or consist of a two-dimensional coordinate space or a three-dimensional coordinate space. Alternatively or additionally, the travel path may be determined in some other space for moving the vehicle, such as actuator space, i.e., space of control commands for actuators for moving the vehicle. In this case, the number of dimensions of the space may correspond to the number of degrees of freedom for the actuators. The travel path may be represented in vector or matrix form. The travel path is determined based on at least sensor inputs and the target position. For example, the travel path may be determined based on a representation generated based on sensor inputs. The travel path may, under optional conditions, be determined as the shortest path from the vehicle's current position to the target position. Optional conditions may include requirements for the attitude of the vehicle and / or cargo handling equipment at the target position. The optional conditions may also include requirements for avoiding obstacles 310, such as objects or people, identified in the sensor input.

[0049] The vehicle may be a vehicle with steering, such as a truck with a hook lift, which has non-holonomic dynamics. In such dynamics, the vehicle cannot be commanded to produce lateral movement without steering and having longitudinal velocity. Therefore, the travel trajectory can generally be determined such that the trajectory starts from the vehicle and ends at the target position with a curve that allows the vehicle to smoothly perform loading / unloading by latching loading handling equipment, such as a hook lift, onto the load.

[0050] A representation of the area around the vehicle 120, along with a representation of the travel path, may be provided to the HMI device 140. The travel path may be represented by lines or by a set of symbols defining lines. This allows for the provision of a graphical representation to the operator 10 of the LHS 110, which may include the travel path among the representations of the area around the vehicle. The travel path representation may extend along the entire length or a portion of the length of the travel path.

[0051] In this solution, operations can be performed in two separate operating states, including a target selection operating state and a vehicle driving operating state. In the target selection operating state, the target location is identified and confirmed, and in the vehicle driving operating state, the vehicle moves towards the target location. Therefore, the vehicle driving operating state can correspond to an autonomous operating state for the vehicle. The vehicle may be stationary when the target selection operating state is active (operational).

[0052] In the target selection operation state, the target position is identified as the initially selected target position in the representation around the vehicle, for example, by any means detailed above for identifying the target position 230. Method 200 then includes confirming the target position as a confirmed target position 250 by obtaining confirmation of the initially selected target position. The confirmation may be provided by the operator 10 and / or by the supervisory system, i.e., as an automatic confirmation, for example based on autonomous pattern recognition. To obtain confirmation of the initially selected target position, a zoomed-in view of the representation around the vehicle around the initially selected target position may be provided. This may be provided in response to the initially selected target position being selected by the operator or by automatic selection.

[0053] Confirmation of the initially selected target position can be obtained by obtaining confirmation of the travel trajectory for moving the vehicle from the vehicle's current position to the initially selected target position. Thus, the target position is initially selected, but the target position is confirmed as a confirmed target position only when it is confirmed that the obtained travel trajectory to that target position is acceptable. Confirmation of the travel trajectory can be obtained, for example, by the operator and / or by the supervisory system, by any means disclosed herein for obtaining confirmation of the target position / confirming the target position. In particular, confirmation of the initially selected target position can be obtained as confirmation of a graphical representation, which may include, as described above, a travel trajectory to the initially selected target position. To obtain the above confirmation, the graphical representation may be provided to the operator, and the above confirmation may be obtained from the operator, for example, by collecting user commands for confirmation.

[0054] Generally, the above confirmation can be obtained based on the display of a graphical representation, which includes at least an indication of the initially selected target location. The graphical representation may also include image data from an image sensor unit received through the sensor input, which enables a clear visualization of the vehicle's surroundings. The graphical representation may, of course, also include data from a distance mapping sensor unit, such as point cloud data, also received through the sensor input. Importantly, the graphical representation may include a representation of the vehicle's trajectory. Thus, this solution allows the vehicle's trajectory to be confirmed before it is sent out. Furthermore, the trajectory can be confirmed visually and / or with respect to the vehicle's surroundings. Thus, the vehicle can be sent out in response to the confirmation of the target location and, in particular, the trajectory. When the vehicle is sent out, it can transition to autonomous operation.

[0055] Generally, confirmation of the initially selected target position may include commands from the LHS 110 operator 10 to the controller 112, collected from the HMI device 140 via interface 114, and / or commands from the LHS 110 operator 10 to the controller 112, obtained from a confirmation device. Alternatively or additionally, the confirmation may include a response signal, which may be an automated response signal, from the HMI device 140 confirming that a graphical representation has been provided for visual indication to the LHS operator 10. Thus, the operator may be given the opportunity to withdraw the confirmation. In all cases, the confirmation may further include a separate sub-confirmation by the supervisory system.

[0056] Based on the confirmation of the target position, method 200 includes transitioning indirectly or directly from the target selection operation state to the vehicle driving operation state 260. The transition may be performed in response to the confirmation of the target position, in particular in response to the confirmation of the target position being obtained. Thus, when the target is confirmed, the target selection operation state may be terminated directly. Thus, when the target is confirmed, the vehicle driving operation state may be entered directly.

[0057] Method 200 includes providing one or more travel commands (also referred to herein as “travel commands”) to a vehicle control unit (VCU) that controls the movement of a vehicle. Travel commands may be provided based on a travel trajectory, but they do not necessarily need to correspond to the entire length of the travel trajectory. A single set of travel commands may extend to a target position at the end of the travel trajectory, but providing travel commands in shorter increments allows for the saving of both computational and data transmission resources. Travel commands may extend beyond a threshold length that allows the vehicle to decelerate and stop within an expected trajectory in response to an abrupt stop command, where the expected trajectory represents the spatial trajectory corresponding to the travel command. The threshold length may correspond to the spatial and / or temporal length of the vehicle movement due to the travel command. For example, if a vehicle is instructed to travel at a speed that takes 4 seconds to come to a complete stop, the threshold length corresponds to the travel command for the next 4 seconds of operation. Therefore, if there are only 4 seconds of travel commands remaining and then another travel command is received, the vehicle will be instructed to stop rather than continue until the end of the previous travel command. This increases operator safety and comfort because the next operation is intuitive from the screen and predictable several seconds in advance.

[0058] A driving command may be set to cause the vehicle to follow a driving trajectory where the target position has been confirmed. Therefore, the driving command corresponds to the driving trajectory to the confirmed target position. The driving command may be transmitted to the vehicle control unit when the vehicle is in a driving operation state, and therefore, in particular, the controller 112 causes the driving command to be transmitted to the VCU 130. The LHS 110 may be configured to prevent the driving command from being provided to and / or transmitted to the VCU 130 unless the target position is confirmed, for example, by preventing the transmission of a driving command unless the target position is confirmed and / or by generating only driving commands for confirmed target positions.

[0059] It is sufficient to confirm the target position once, after which the vehicle can enter a driving operation state, and the vehicle can autonomously move to the target position. While in the driving operation state, the above expression can still be repeatedly updated, which allows the LHS110 to react to deviations in the vehicle's movement and / or changes in the vehicle's surroundings, for example, by stopping the vehicle. The LHS can be configured to recognize exceptions and stop the vehicle in response to (recognized) exceptions. The LHS can also be configured to transition back to a target selection operation state for reconfirming the target position. In the driving operation state, sensor inputs can be repeatedly updated to provide updated sensor inputs. Similarly, driving commands can be repeatedly updated based on updated sensor inputs even when the confirmed target position remains constant. Therefore, there is no need to request commands from the user. Driving commands can be updated multiple times per second, for example, 5 to 10 times per second or more frequently. The driving command may be updated to account for deviations and / or changes in the surroundings along the driving trajectory, but the driving trajectory may be updated primarily to guide the vehicle toward and along the (original) driving trajectory to lead the vehicle to a confirmed target position. However, the driving trajectory may also be iteratively updated based on updated sensor inputs and the confirmed target position to provide an updated driving trajectory. The driving command to the vehicle control unit may then be iteratively updated based on the updated driving trajectory, which allows the driving trajectory itself to be changed. This allows for greater changes in the vehicle's path even when the confirmed target position does not change. This may be used for obstacle avoidance, but the vehicle may also stop in the case of an obstacle 310. Importantly, all of the above actions may be performed autonomously while the vehicle is in motion.

[0060] Entering (i.e., transitioning to) a vehicle driving state may involve sending the vehicle out. This may optionally involve releasing the vehicle's parking brake and setting the vehicle to move towards a target location. The controller may be configured to send one or more commands to the VCU to send the vehicle out in response to the target location being confirmed and / or the transition to the vehicle driving state. In the vehicle driving state, the vehicle may move to the target location and optionally perform loading and / or unloading. The controller may be configured to provide commands to cause this to happen automatically based on the confirmed target location. User input is not necessarily required in the vehicle driving state.

[0061] The user commands described above may belong to a set of commands available to the operator 10 of the LHS 110. The set of commands may include commands for selecting a target location and / or for confirming the target location, for example, by confirming the travel trajectory to the target location. The set of commands may also include commands for stopping the vehicle 120 and / or commands for changing the vehicle's control mode from autonomous to human operation. The set of commands available to the operator of the LHS may have a first subset of commands available to the operator of the LHS when the controller is operating in a target selection operation state, and a second subset of commands available to the operator of the LHS when the controller is operating in a vehicle travel operation state. The first subset of commands and the second subset of commands may be completely or partially different. The set of commands available to the operator in a target selection operation state may include commands for selecting a target location, commands for automatically identifying the target location, and commands for confirming the target location. All commands may be distinct. Any or all of the commands may be requested in response to a prompt from the controller. The target selection operation state may also include one or more user interface commands, such as commands for zooming in on the view of the above representation and / or switching between different views of the above representation. The set of commands available to the operator in the vehicle driving operation state may include, or consist of, a stop command for stopping the vehicle and / or a command for changing the vehicle's control mode from autonomous to human operation, as well as one or more user interface commands that, at will, do not affect the vehicle's movement, for example, to change the view of the vehicle's surroundings displayed to operator 10.

[0062] Figure 3 shows an example of a view 300 for cargo handling as described herein. The view may be provided as a graphical representation, as described above. It may be displayed in the target selection operation state and / or the vehicle driving operation state.

[0063] The view shows how obstacles 310 and loads 320 can be visualized based on their representations. The view also shows how the first target 330 can be highlighted to provide the initially selected target position 340 for selection for the user. In the example shown, the first target is already selected to obtain the initially selected target position 340, since the travel trajectory has already been determined and a representation of the travel trajectory 350 is shown.

[0064] This example further illustrates how a drive command 360 may be represented based on its representation. Thus, the initially selected target position 340 may have already been confirmed as the confirmed target position 342, and therefore the drive command 360 may be provided.

[0065] The controllers described above may be implemented in software, hardware, application logic, or a combination of software, hardware, and application logic. The application logic, software, or instruction sets may be maintained on any one of various conventional computer-readable media. “Computer-readable media” can be any medium or means that can store, communicate, propagate, or transfer instructions for use by, or relating to, an instruction execution system, apparatus, or device, such as a computer. Computer-readable media may include computer-readable storage media, which can contain or store instructions for use by, or relating to, an instruction execution system, apparatus, or device, such as a computer. Examples of such media can store information relating to the various processes described herein. This information may be stored in one or more memories, such as hard disks, optical disks, magneto-optical disks, or RAM. One or more databases may store information used to implement the embodiments. Databases may be organized using data structures (e.g., records, tables, arrays, fields, graphs, trees, lists, etc.) contained in one or more memory or storage devices listed herein. The database may reside on one or more devices, including local and / or remote devices such as servers. The processes described in relation to the embodiment may include appropriate data structures for storing the data collected and / or generated by the processes of the embodiment's devices and subsystems in one or more databases.

[0066] As will be understood by (one or more) parties in the computer and / or software field, all or part of the embodiments may be implemented using one or more general-purpose processors, microprocessors, digital signal processors, microcontrollers, etc., programmed in accordance with the teachings of the embodiments. As will be understood by parties in the software field, appropriate software may be readily prepared by a programmer skilled in the art based on the teachings of the embodiments. Furthermore, as will be understood by (one or more) parties skilled in the electrical field, the embodiments may be implemented by preparing application-specific integrated circuits or by interconnecting appropriate networks of conventional component circuits. Thus, the embodiments are not limited to any particular combination of hardware and / or software.

[0067] The different functions described herein may be performed in different orders and / or in parallel with one another.

[0068] Unless otherwise specified, any range or device value given herein may be extended or modified without loss of the desired effect. Furthermore, unless expressly denied, any example may be combined with another example.

[0069] While the subject matter has been described using terminology specific to structural features and / or actions, it should be understood that the subject matter as defined in the attached claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as examples of implementing the claims, and other equivalent features and actions are intended to fall within the scope of the claims.

[0070] It should be understood that the benefits and advantages described above may relate to one or more embodiments. The embodiments are not limited to those that solve any or all of the problems described, or that possess any or all of the benefits and advantages described. Furthermore, it should be understood that a reference to an item may refer to one or more of those items.

[0071] The term “comprising” is used herein to mean that the identified method, block, or element includes, but such block or element does not constitute an exclusive list, and that the method or apparatus may include additional blocks or elements.

[0072] Numeric descriptors such as "first," "second," etc., are used in this document merely as a way to distinguish between parts that may have similar names. Numeric descriptors should not be interpreted as indicating a specific order, such as the order of preferences, production, or occurrence in a particular structure.

[0073] While the present invention has been described in relation to a particular type of apparatus and / or method, it should be understood that the present invention is not limited to a particular type of apparatus and / or method. While several examples, embodiments, and implementations have been described, the present invention is not limited in this respect, but rather covers a variety of modifications and equivalent configurations that fall within the scope of the claims. Various examples have been described above, with some degree of specificity, or with respect to one or more individual embodiments, but those skilled in the art can make numerous modifications to the disclosed examples without departing from the scope of this specification.

Claims

1. A cargo handling system for a vehicle equipped with cargo handling equipment, wherein the cargo handling system is - It is a controller, 〇 Receiving sensor input from a sensor system in the vehicle, which includes at least an image sensor unit and a distance mapping sensor unit for monitoring and mapping the area around the vehicle, ○ Identifying the target location for the vehicle for handling cargo, Based on the sensor input and the target position, a driving trajectory is determined for moving the vehicle from its current position to the target position. ○ To provide a driving command to a vehicle control unit that controls the movement of the vehicle based on the aforementioned driving trajectory, ○ To generate a representation of the vehicle's surroundings based on the sensor input, wherein the representation includes one or more images and / or a point cloud map of the vehicle's surroundings. A controller configured to perform the following actions: - An interface between the controller and one or more human-machine interface devices, wherein the cargo handling system, through the interface, ○ To provide the one or more human-machine interface devices with a representation of the surroundings of the vehicle in order to visually show the surroundings of the vehicle to the operator of the cargo handling system, 〇 Collecting user commands from one or more human-machine interface devices to the controller, wherein the user commands belong to a set of commands available to the operator of the cargo handling system. An interface configured to perform the following: Equipped with, The aforementioned controller, A target selection operation state in which the controller is configured to identify the target position in the representation of the vehicle's surroundings as the initially selected target position, and to confirm the target position as a confirmed target position by obtaining confirmation of the initially selected target position, ○ A vehicle driving operation state in which the controller is configured to cause the driving command provided based on the confirmed target position to be transmitted to the vehicle control unit, and It is further configured to operate in two separate operating states, including A cargo handling system further configured such that the controller transitions from the target selection operation state to the vehicle driving operation state based on the confirmation of the target position.

2. The cargo handling system according to claim 1, wherein the set of commands available to the operator of the cargo handling system has a first subset of commands available to the operator of the cargo handling system when the controller is operating in the target selection operation state, and a second subset of commands available to the operator of the cargo handling system when the controller is operating in the vehicle driving operation state, wherein the first subset of commands and the second subset of commands are different.

3. The cargo handling system according to claim 1 or 2, wherein the representation of the vehicle's surroundings is provided to the one or more human-machine interface devices together with the representation of the travel trajectory, in order to provide the operator of the cargo handling system with a graphical representation including a representation of the travel trajectory among those representing the vehicle's surroundings.

4. The cargo handling system according to any one of claims 1 to 3, wherein the controller is configured to transition from the target selection operation state to the vehicle driving operation state in response to the confirmation of the target position.

5. The cargo handling system according to any one of claims 1 to 4, wherein the controller is configured to obtain confirmation of the first selected target position by obtaining confirmation of the travel trajectory for moving the vehicle from the vehicle's current position to the first selected target position.

6. The cargo handling system according to claim 5, as dependent on claim 3, wherein the controller is further configured to obtain confirmation of the first selected target position as confirmation of the graphical representation, and the graphical representation includes showing the travel trajectory to the first selected target position.

7. The confirmation regarding the initially selected target position is as follows: - Commands from the operator of the cargo handling system, collected from one or more human-machine interface devices via the interface to the controller, - Commands from the operator of the cargo handling system obtained from the confirmation device to the controller, - Confirmation that the graphical representation is provided for visual indication to the operator of the cargo handling system, and response signals from the one or more human-machine interface devices. A cargo handling system according to any one of claims 1 to 6, comprising at least one of the above.

8. The cargo handling system according to any one of claims 1 to 7, wherein the controller is configured to provide a zoomed-in view of the representation of the vehicle around the initially selected target position in order to obtain the confirmation of the initially selected target position in the target selection operation state.

9. The cargo handling system according to any one of claims 1 to 8, wherein the target position is a position for loading or unloading cargo with the cargo handling equipment.

10. The cargo handling system according to claim 9, wherein the target position is defined with respect to an identified cargo for loading, or with respect to an identified position with respect to a transported cargo for unloading.

11. The cargo handling system according to any one of claims 1 to 10, wherein the controller is configured to repeatedly update the sensor input to provide the updated sensor input in the vehicle driving operation state, and repeatedly update the driving command to the vehicle control unit based on the updated sensor input.

12. A method for controlling a vehicle equipped with cargo handling equipment, wherein the method is - Receiving sensor input from a sensor system in the vehicle, which includes at least an image sensor unit and a distance mapping sensor unit for monitoring and mapping the area around the vehicle, - Identifying the target location for the vehicle for cargo handling, - Based on the sensor input and the target position, determine the travel trajectory for moving the vehicle from its current position to the target position, - To provide a driving command to a vehicle control unit that controls the movement of the vehicle based on the aforementioned driving trajectory, - To generate a representation of the vehicle's surroundings based on the sensor input, wherein the representation includes one or more images and / or a point cloud map of the vehicle's surroundings. - The representation of the surroundings of the vehicle is provided to one or more human-machine interface devices to visually indicate the surroundings of the vehicle to the operator of the cargo handling system, - Collecting user commands from one or more human-machine interface devices, wherein the user commands belong to a set of commands available to the operator of the cargo handling system. - Based on the confirmation of the target position, the system transitions from the target selection operation state to the vehicle driving operation state. Includes, The aforementioned target position is identified as the first selected target position in the representation of the vehicle's surroundings during the target selection operation state, and is confirmed as a confirmed target position by obtaining confirmation of the first selected target position. A method to cause the driving command provided based on the confirmed target position to be transmitted to the vehicle control unit during the vehicle driving operation state.

13. A computer program product comprising, when executed by a computer, an instruction causing the computer to perform the method described in claim 12.

Images