Autonomous or assisted driving of an industrial truck using a camera and a lidar

Abstract

An industrial truck having an autonomous and / or assisted driving function for positioning the industrial truck with respect to a target object, comprising: means for handling a load, a video camera configured to generate image data relating to a visual field, a LIDAR device configured to generate point cloud data relating to the visual field, and control means connected with the video camera in order to receive image data generated by the video camera and connected with the LIDAR device to receive point cloud data generated by the LIDAR device. The control means is configured to: acquire image data from the video camera and point cloud data from the LIDAR device relating to the same visual field; recognise the target object in an image by image processing applied to the acquired image data, wherein the control means is configured to identify a region of interest, ROI, in the image in which the target object is included; associate the acquired point cloud data within the ROI with the target object recognised in the image, including selecting a subset of the plurality of points of the point cloud within the ROI that corresponds to the target object; and control the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object. (fig. 1)

Description

Field of the invention

[0001] The present disclosure relates to the autonomous or assisted driving of an industrial truck and, in particular, a system for the autonomous or assisted driving of an industrial truck, a method, a computer program and a controller.Technical scope

[0002] Industrial trucks, such as forklifts, tow trucks, dolly trucks, lift trucks, luggage trucks, hand trucks, and so on, move and transport materials in an industrial environment, such as a warehouse, port, logistic hub, industrial site, or the like. Industrial trucks may have an autonomous driving function that allows the truck to navigate the industrial environment without being manoeuvred by a person, and / or an assisted driving function that provides guidance to the human operator of the truck to correct the operation of the truck or to automatically adjust the operation of the truck.

[0003] To perform an autonomous driving or assisted driving function, the information relating to the position of the truck in the industrial environment must be accurate, to ensure that the industrial truck interacts with other elements of the industrial environment in an appropriate manner (e.g. apparatuses for loading / unloading the industrial truck) and that the risks of collision are reduced. Known systems for obtaining information about the position include LIDAR (optical radars, Light Detection and Ranging) sensors, satellite navigation receivers such as GNSS (Global Navigational Satellite System) or IMU (Inertial Measurement Unit) receivers. Based on the collected information relating to position, the system generates a control signal to execute the autonomous / assisted driving function, for example by making sure that the operator is provided with a visual and / or audible indication (e.g. via a display device, lights and / or speakers), or by automatically controlling the movement of the truck.

[0004] When performing a loading or unloading operation, the position of the industrial truck must be extremely precise, so as to guarantee a correct positioning of the industrial truck with respect to a loading / unloading apparatus included in the warehouse, on the industrial site or the like. The loading / unloading apparatus may be, for example, a crane, a bridge crane, a self-propelled wheeled crane, a shelving unit, a mobile shelf or the like. Normal navigation systems based, for example, on GNSS are not able to guarantee a sufficiently precise positioning of the industrial truck with respect to the loading / unloading apparatus, which is necessary during loading / unloading operations. Furthermore, since the loading / unloading apparatus can be mobile, it is difficult for the known navigation systems to determine under any circumstances the correct wished position with respect to the loading / unloading apparatus so as to perform the loading / unloading operations in collaboration with the loading / unloading apparatus.

[0005] Systems are also known that use optical markers applied to target objects (for example to a crane or a bridge crane), which are recognised by the industrial truck by processing images taken by a camera, where the markers are used by the controller of the industrial truck as a reference during alignment by detecting the position of the industrial truck with respect to the detected optical marker. A system of this type is described in European patent application EP4317049A1. In such known systems, however, there are some drawbacks, including: the environment in which the industrial truck operates (port, warehouse) must be provided with markers (i.e. the optical markers must be printed and positioned on the object with the industrial truck must be aligned); the markers need periodic maintenance; the marker detection algorithms may suffer from some problems related to the occlusion and orientation of the marker with respect to the camera; even under consideration of other characteristics, due to the use of monocular cameras it is difficult to obtain a correct estimate of the staircase / distance from a target object when the markers are partially detected or are not detected.

[0006] Systems using sensors such as depth sensors or LIDARs are also known. These types of sensors produce a very high amount of data points since they are intended to provide a data point by performing a scan both in the horizontal and vertical direction, with an increase in the amount of data points as the precision of the sensor increases. An approach that requires processing these data to detect objects in the scope requires a high amount of data processing and is subject to the errors that can be generated by these types of sensors.Summary of the invention

[0007] There is therefore a need to improve the autonomous / assisted driving function of an industrial truck, for example to obtain in a reliable manner a precise and correct position of the industrial truck with respect to a loading / unloading apparatus under different circumstances.

[0008] The present disclosure therefore proposes an industrial truck having an autonomous and / or assisted driving function for positioning the industrial truck with respect to a target object, comprising: means for handling a load; a video camera configured to generate image data relating to a visual field, a LIDAR device configured to generate point cloud data relating to the visual field, a control means connected with the video camera in order to receive image data generated by the video camera and connected with the LIDAR in order to receive point cloud data generated by the LIDAR device; wherein the control means is configured to: acquire image data from the video camera and point cloud data from the LIDAR device relating to the same visual field, recognise the target object in an image by image processing applied to the acquired data, wherein, preferably, the control means is configured to identify a region of interest, ROI, in the image in which the target object is included; associate the acquired point cloud data (for example within the ROI) with the target object recognised in the image; control the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object.

[0009] Preferably, controlling the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object comprises determining a distance between the industrial truck and the target object based on the point cloud data associated with the target object.

[0010] Preferably, associating the acquired point cloud data with the target object recognised in the image includes selecting a subset of the plurality of points of a point cloud depicted by the point cloud data corresponding to the target object recognised in the image (e.g., by selecting a subset of the corresponding plurality of points within the ROI).

[0011] Preferably, the industrial truck further comprising storage media in which one or more models of target objects are stored, each model of a target object comprising model point cloud data associated with the target object and / or image data associated with the target object.

[0012] Preferably, the control means is configured to compare the point cloud data associated with the recognised target object with the one or more stored models of target objects.

[0013] Preferably, the control means is configured to identify, based on the comparison, a correspondence between the point cloud data associated with the recognised target object and one of the stored models of target objects.

[0014] Preferably, the control means is configured to determine a distance between the industrial truck and the target object only if the control means identifies a correspondence of the point cloud data associated with the recognised target object to a stored model of a target object.

[0015] Preferably, the control means is configured to recognise, by image processing, one or more of the following target objects: a load to be handled by the means for handling a load, an unloading site where a load handled by the industrial truck is to be unloaded, a loading site from where a load to be handled by means of the industrial truck is to be picked up, a loading / unloading device with which the industrial truck is configured to interact in order to execute a loading / unloading function.

[0016] Preferably, the LIDAR device is mounted on the industrial truck in proximity to the video camera.

[0017] Preferably, the LIDAR device is configured to generate point cloud data including a distance associated with each of the points of a point cloud depicted by the point cloud data.

[0018] Preferably, the control means is configured to recognise the target object in the image (and / or identify the region of interest) without processing the point cloud data generated by the LIDAR device.

[0019] Thanks to the present disclosure, it is possible to estimate the distance between a target object and the vehicle by merging the information detected with a video camera with information from another sensor, i.e. a LIDAR. It is thus possible to find some characteristics from an image, combine them with the data of the LIDAR sensor and process them to obtain "mixed information". Preferably, the mixed data can be compared to a pre-existing dataset, containing for example the mixed characteristics of the target object and, if there is a good correspondence, the mixed data can be used to calculate the distance. This allows to obtain a precise recognition of the obstacles and / or of a target object, such as for example an apparatus with which the industrial truck is configured to interact for loading / unloading operations of the industrial truck. In particular, it is possible to precisely and accurately estimate the distance or relative position of the industrial truck with respect to an obstacle and / or a target object for alignment with that target object. In particular, it is possible to recognise an object through data from the video camera, and it is possible to obtain reliable data on the distances from the object through the LIDAR in order to be able to reliably control alignment during loading / unloading operations. The distance between the truck and a target object is derived directly from the LIDAR, therefore it has the highest precision available. The environment in which the industrial truck operates (port, warehouse) may therefore no longer be provided with markers. Furthermore, compared to the prior art, to extract features, algorithms can be used that may be stronger to occlusion in viewing an object feature and be more precise.Brief description of the drawings

[0020] The present invention will be illustrated with reference to exemplary embodiments described with reference to the accompanying drawings.

[0021] In the drawings: figure 1 illustrates an industrial truck according to the present disclosure; figure 2 illustrates a schematic view of the components of the industrial truck of figure 1; figure 3 illustrates an image taken from the video camera of the industrial truck; figure 4 illustrates a point cloud generated by the LIDAR device; figure 5 illustrates an association of the image data and the point cloud data; figure 6 illustrates a stored model of a target object; figure 7 illustrates a step of comparing the point cloud data associated to the image data with the stored model of a target object; figure 8 illustrates the steps of a method performed by the control means of the industrial truck; figure 9 illustrates a further embodiment of the method performed by the control means; figure 10 illustrates a possible implementation of a control unit of the industrial truck. Detailed description

[0022] With reference to the figures, an industrial truck 10 is shown in figure 1. The industrial truck 10 can be used in an industrial site, a port, a warehouse or the like. The industrial truck can be used to handle a load 14. For this purpose, the industrial truck 10 can include material handling means 11. For example, the industrial truck 10 can be a forklift truck, preferably a counterbalanced forklift truck. The forklift truck 10 can include an upright 12 on which a material handling device 13, for example a fork, can slide. Other types of material handling devices can also be used, such as a gripper, a strut, a slab material handling device or other. The industrial truck 10 preferably includes a cab for a driver. The industrial truck preferably includes a seat 15 for the user of the industrial truck. The industrial truck 10 includes control means 40, preferably capable of controlling the functions of locomotion and / or material handling. The industrial truck 10 includes a frame 20. The industrial truck 10 can for example include three or four wheels, but other configurations are also possible according to the prior art.

[0023] According to the present disclosure, the industrial truck 10 has an autonomous and / or assisted driving function for positioning the industrial truck with respect to a target object. For example, the target object may be one of the following: a load to be handled by the means for handling the load, an unloading site where a load handled by the industrial truck is to be unloaded, a loading site from where a load to be handled by means of the industrial truck is to be picked up, a loading / unloading device with which the industrial truck is configured in order to interact to execute a loading / unloading function (such as a bridge crane, a crane, RTG ("Rubber-Tired Gantry Cranes") or a QC (Quay Crane) typically employed in a port). The autonomous or assisted driving function can also be used to determine a positioning of the industrial truck with respect to an apparatus with which the industrial truck must interact in order to carry out a loading and / or unloading operation of the industrial truck, for example to ensure precise alignment with said apparatus. The target object according to an embodiment could also be a typical obstacle that can be encountered in an industrial site and that must be avoided during locomotion of the truck 10. An obstacle could be, for example, a load resting on the ground, another industrial truck, or a hauler. The autonomous or assisted driving function can also be used to determine a positioning of the industrial truck with respect to an obstacle.

[0024] The function of autonomous or assisted driving is carried out by control means 40 of the industrial truck 10.

[0025] The industrial truck 10 further includes a video camera (or camera) 22 configured to generate image data relating to a visual field; the image data represents an image taken by the video camera 22. The video camera 22 can be made according to any known technique. The video camera 22 may be mounted on the industrial truck, for example on the upright 12 of the material handling means; however, the video camera 22 could also be mounted in other positions, for example on the roof of the cab of the industrial truck. Advantageously, the video camera 22 is positioned so as to frame a visual field positioned in the forward direction of the industrial truck 10; in an alternative embodiment, an additional video camera could also be arranged so as to face the reverse direction of the industrial truck. Figure 3 shows an image 30 taken by the video camera representing a target object 50, for example a QC 50. The target object could also be different, for example a bridge crane, an RTG or other loading / unloading devices. According to one embodiment, the target object could advantageously be a device configured to interact with the industrial truck 10 to perform loading / unloading operations of the industrial truck or by the industrial truck 10.

[0026] The industrial truck 10 also includes a LIDAR ("Light Detection and Ranging") device 21 configured to generate point cloud data relating to the visual field. The LIDAR device, also known as "optical radar", can be made in itself according to any known technique. The LIDAR device 21 is configured to generate point cloud data including a distance associated with each of points of a point cloud depicted by the point cloud data. An example of a point cloud is shown in figure 4. A point cloud includes a plurality of points detected in a visual field of the LIDAR device. The point cloud data generated by the LIDAR device includes data relating to the points of the point cloud, where each point is associated with a distance from the LIDAR device and a direction with respect to the LIDAR device. In the example in figure 4, the target object 50 of figure 3 is in the visual field of the LIDAR device and a plurality of points of the point cloud 31 correspond to the target object 50, for example a QC-type crane. The circles 60 depicted in figure 4 represent different distances from the LIDAR 21. The LIDAR device 21 can be mounted on the industrial truck 10 in proximity to the video camera 22, as in the example in figure 2. For example, the LIDAR device 21 may be mounted on the upright 12 of the industrial truck. However, the LIDAR device could also be mounted in other positions, for example on the roof of the cab of the truck 10. The visual field (or field of action) of the LIDAR device 21 corresponds to the directions with respect to the LIDAR device 21 for which the points of the point cloud are detected. The visual field of the LIDAR device overlaps, at least in part, the visual field of the video camera 22. Consequently, points of the point cloud 31 may correspond to an object depicted in the image 30 taken by the video camera. Preferably, the visual field of the LIDAR device 21 is wider than and comprises the visual field of the video camera 22. As shown in figure 4, the LIDAR device 21 could also detect point cloud data at 270° with respect to a vertical axis, or in any case for a viewing angle greater than 180° with respect to a vertical axis.

[0027] As depicted in figure 2, the control means 40 is connected with the video camera 22 in order to receive image data generated by the video camera and connected with the LIDAR device 21 in order to receive point cloud data generated by the LIDAR device. The control means 40 is further connected with truck actuator means and / or truck user interface means 24, for sending control signals to said means 24 and 25.

[0028] In particular, the control means 40 is configured to acquire image data from the video camera 22 and point cloud data from the LIDAR device 21 relating to the same visual field. The image data corresponds for example to the image 30 of figure 3. The point cloud data corresponds for example to the point cloud of figure 4.

[0029] Further, the control means is configured to recognise a target object 50 in an image by means of an image processing applied to the acquired image data. For example, by means of any known image processing algorithm, the control means 40 is capable of recognising a target object 50 in the image. In one example, the control means identifies by image processing a region of interest 32 (Region of Interest, ROI) in the image 30, depicted in figure 5. Preferably, the control means 40 is configured to recognise by image processing one or more of the following target objects: a load to be handled by the load handling means (which may depend on the type of industrial or logistic site where the industrial truck operates, for example a pallet, a reel, slab material, or other types of loads that can be handled by the industrial truck according to the needs), an unloading site where a load handled by the industrial truck is to be unloaded (for example, a site on a shelving, a rack of a hauler, or the like), a loading site from where a load to be handled by means of the industrial truck is to be picked up (for example, a site on a shelving, a rack of a hauler, or the like), a loading / unloading device with which the industrial truck is configured to interact in order to execute a loading / unloading function (e.g. a bridge crane, a QC-type crane, or other).

[0030] However, the control means 40 could also be able to recognise other types of objects, such as for example obstacles that can typically be found in the environment (a port, a logistic hub, a warehouse and so on) where the industrial truck operates. For example, the control means could have a function of recognising objects in an image such as a load, another industrial truck, a hauler or other.

[0031] After identifying the target object 50 in the image 30, the control means 40 is configured to associate the acquired point cloud data with the target object 50 recognised in the image 30. In this step, the control means 40 may for example associate the acquired point cloud data with the target object 50 recognised in the image 30 by selecting a subset of the plurality of points of a point cloud 31 depicted by the point cloud data that corresponds to the target object 50 recognised in the image. This step is depicted in figure 5, in which according to an embodiment example the points of the point cloud 31' within the ROI 32 are selected to form a set (or subset) of point cloud data associated (or corresponding) to the target object 50 included in the ROI 32. In this way it is possible to obtain a second point cloud 31' associated with the target object 50, which includes a subset of all point cloud points 31 initially acquired by the LIDAR device 21 (shown in figure 4). Said second point cloud 31' may correspond to point cloud data associated with the target object 50 recognised in the image 30. In this way, the relevant point cloud points are filters, allowing to obtain relevant and precise information about the positioning of the target object 50 with respect to the industrial truck 10. According to the description above, the control means 40 can identify the target object 50 in the image 30 and / or the region of interest in the image 30 in which the target object 50 is included, without processing the point cloud data 31 initially acquired by the LIDAR device 21.

[0032] The control means 40 is further configured to control the autonomous and / or assisted driving function of the industrial truck 10 based on the point cloud data associated with the recognised target object. For example, the control of the autonomous driving function may be carried out by sending control signals to one or more actuator devices, such as for example one or more electric motors, locomotion actuators, actuators associated with the material handling means. In figure 2, the actuator means is schematically shown with reference numeral 25; it is understood that such actuator means 25 may include a plurality of actuating devices as mentioned above. The control of the assisted driving function may be carried out by the control means 40 by sending command signals to a user interface 24, which may comprise a display and / or other user interface devices, such as acoustic signal emitters, or also actuators that impose vibration of components in contact with the driver during operation of the industrial truck (e.g. a steering wheel, a joystick or the like). For example, a display of the user interface 24 could display information that assists the driver in driving the truck, such as a trajectory to follow, information about the distance to be travelled, a lifting height, or the like.

[0033] According to a preferred embodiment, controlling the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object comprises determining a distance between the industrial truck and the target object based on the point cloud data associated with the target object 50. The distance between the industrial truck 10 and the target object 50 can be determined in a precise manner by using the information relating to the distance provided by the LIDAR device in the point cloud data; this allows to carry out a control of the positioning of the industrial truck with respect to a target object, in a precise and reliable manner. For example, an operation of approach to a loading / unloading device of the truck (such as a QC-type crane) can be performed with precision based on the distance determined by the control means based on the point cloud data. In addition, a subset of the overall point cloud data is appropriately selected, since it has previously been associated with the target object recognised in the image taken by the video camera.

[0034] According to one embodiment, the industrial truck 10 also includes storage media 23 (shown in figure 2) in which one or more target object models are stored. Each target object model may comprise model point cloud data associated with the target object and / or image data associated with the target object. An example of a representation of a target object model 34 stored in the storage media 23 is shown in figure 6; in the example in figure 6, the target object model 34 refers to a QC-type crane, i.e. the same target object taken in the image 30 by the video camera. The storage media 23 may be realized with a non-volatile memory, or a database of the industrial truck. Different target object models relating to different target objects can be advantageously stored. For example, different target object models may refer to the different target objects identifiable by the control means 40 by image processing as described above, such as for example a loading site, a load, an apparatus for loading / unloading the industrial truck and so on. The target object model may be stored in the form of an association between image data and point cloud data, similarly to the point cloud data associated with the target object that are generated by the control means based on the image data and the point cloud data generated by the video camera 22 and the LIDAR device 21, respectively. The models of the target objects stored in the storage media 23 may be preset models loaded into the industrial truck by a programmer; alternatively they could also be the result of previous recognitions of target objects by the industrial truck in the environment in which it operates. The stored models can be updated automatically following different work cycles of the industrial truck.

[0035] In the case where the storage media 23 are present, the control means 40 is configured to compare the point cloud data 31' associated to the recognised target object with the one or more stored target object models. The comparison can be made by comparing, on the one hand, the combined data including image data relating to the ROI 32 and point cloud data associated with the target object, and on the other hand the target object model. This step is shown in figure 7. The point cloud data 31' associated with the target object 50 (and possibly the image data relating to the target object, for example the image data relating to the ROI 32) are compared with the target object model 34 relating to a QC-type crane. The processing means may compare the point cloud data 31' associated with the target object 50 (and possibly the image data relating to the target object, for example the image data relating to the ROI 32) with a plurality of previously stored models, so as to identify the model with the highest correspondence. The control means 40 may advantageously be configured to identify, based on the comparison, a correspondence between the point cloud data associated with the recognised target object (and possibly the image data relating to the target object, for example the image data relating to the ROI 32) with one of the stored target object models. In other words, the comparison can be made using combined data comprising image data relating to the recognised target object (for example the ROI) and the point cloud data associated with the target object (therefore those relating to the point cloud 31' associated for example with the ROI 32). For example, a level of correspondence or correlation between the point cloud data 31' associated with the target object and the stored model 34 may be determined by the control means through a correlation index between the stored data set and that generated by the data by the LIDAR device and the video camera. When the correlation index exceeds a predetermined threshold, the control means 40 may determine that there is a correspondence between the point cloud data associated with the target object (or the combined data described above) and the target object model.

[0036] Preferably, the control means 40 may be configured to determine a distance between the industrial truck and the target object, only if the control means 40 identifies a correspondence of the point cloud data associated with the recognised target object (or the above-described combined data) to a stored target object model. In this way, the strength of the control is further increased.

[0037] With reference to figure 8, a mode of operation of the industrial truck 10 is described, i.e. a method implemented by the control means 40 of the industrial truck: step S10: the control means 40 executes the step of acquiring image data from the video camera and point cloud data from the LIDAR device relating to the same visual field; step S11: the control means 40 executes the step of recognising the target object in an image by image processing applied to the acquired data; step S12: the control means 40 executes the step of associating the acquired point cloud data with the target object recognised in the image; step S13: the control means 40 executes the step of controlling the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object.

[0038] With reference to figure 9, a further mode of operation of the industrial truck 10 is described, namely a method implemented by the control means of the industrial truck: step S10: the control means 40 executes the step of acquiring the image data from the video camera and point cloud data from the LIDAR device relating to the same visual field; step S11: the control means 40 executes the step of recognising the target object in an image by image processing applied to the acquired image data; step S12: the control means 40 executes the step of associating the acquired point cloud data with the target object recognised in the image; step S14: the control means 40 executes the step of comparing the point cloud data associated with the recognised target object (or alternatively the point cloud data associated with the recognised target object combined with the image data relating to the recognised target object) with the one or more stored target object models; step S15: the control means 40 executes the step of verifying whether there is a correspondence with a stored target object model; in the event that there is a target object model corresponding to the point cloud data associated with the target object (or alternatively the point cloud data associated with the recognised target object combined with the image data relating to the recognised target object), the method continues with the control of the autonomous / assisted driving function referred to in step S13; conversely, a control based on the acquired point cloud data through the LIDAR device is not carried out and a further data acquisition step is carried out via video camera and LIDAR device; step S13: the control means 40 executes the step of controlling the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object.

[0039] Both the methods of figure 8 and 9 can be executed iteratively so as to periodically provide updated distance data to the control means 40, for optimal execution of the assisted / autonomous driving function.

[0040] The present disclosure also relates to a computer program which, when executed by a control unit of an industrial truck, causes the industrial truck to execute steps S10-S13 described above with reference to figure 8 or steps S10-S15 described above with reference to figure 9.

[0041] Fig. 10 shows a possible implementation of a control unit 40 (or controller or control means). In one embodiment, the control unit 40 includes at least one processor 42, a memory 43, and an I / O interface 41. The processor 42 is configured to execute instructions comprised in a computer program stored in the memory 43 in order to execute any of the functions of the control unit, as described above. The memory 43 is configured to store the computer program. During execution of the computer program instructions, the processor 42 receives information from other system elements (e.g., from the LIDAR device 21 or the video camera 22) via the I / O interface 41 and sends in output information to the other system elements via the I / O interface 41.

[0042] According to what has been found by the inventor, the LIDAR devices and the cameras are complementary sensors: one has millimetre precision but provides poor data (the LIDAR), while the other provides dense data (richer in information) but is not suitable for precision tasks in uncontrolled environments (the video camera). Given a vehicle, with camera and Lidar sensor, and given sensor data (i.e. an image and a point cloud) according to one aspect, the steps are as follows: 1. Project the LIDAR points on the image plane 2. Perform feature detection on the image and merge this information with the depth from the LIDAR 3. Compare mixed data with data already in a dataset 4. If the correspondence is good, calculate the relative distance between the vehicle and the target object. Modifications and variations

[0043] Embodiments of the software of the examples presented herein may be provided as a computer program or software, such as for example one or more programs having instructions or sequences of instructions, included in or stored in an article of manufacture as a machine-accessible or machine-readable medium, an instruction store, or a computer-readable storage device, each of which, in an example embodiment, may be non-transitory. The program or instructions on the non-transitory machine-accessible medium, machine-readable medium, instruction store, or computer-readable storage device may be used to program a computer system or other electronic device. The techniques described herein are not limited to any software configuration. They can be applied in any computing or processing environment. Furthermore, it is common in the art to speak of software, in one form or another (e.g., program, procedure, process, application, module, unit, logic, and so on), as if it were performing an action or producing a result. Such expressions are nothing more than a shorthand way of stating that the execution of software by a processing system causes the processor to execute an action to produce a result.

[0044] As used in this context, a control unit (or control means) may include one or more processors (e.g., a CPU with single / multiple cores (computational unit), a microprocessor), graphical processing unit (GPU), video processing unit (VPU), tensor processing unit 15 (TPU), a combination thereof, or other known types of processing means. The control unit may be replaced by processing means or processing unit.

[0045] The present disclosure also provides a method performed by control means of an industrial truck provided with an autonomous and / or assisted driving function for positioning the industrial truck with respect to a target object, wherein the industrial truck comprises: means for handling a load, a video camera configured to generate image data relating to a visual field, a LIDAR device configured to generate point cloud data relating to the visual field, and control means connected with the video camera in order to receive image data generated by the video camera and connected with the LIDAR device in order to receive point cloud data generated by the LIDAR device. The method comprises the steps of: acquiring image data from the video camera and point cloud data from the LIDAR device relating to the same visual field, recognising the target object (50) in an image by image processing applied to the acquired data; associating the acquired point cloud data with the target object recognised in the image; controlling the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object.

[0046] The present disclosure also relates to a computer program containing instructions that, when executed by a computer, cause the computer to execute the steps of any of the methods described above.

[0047] While various exemplary embodiments of the present invention have been described, it is intended that they have been presented by way of non-limiting example. It is apparent to persons skilled in the relevant art(s) that various changes in form and detail can be made in this context. Therefore, the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only based on the following claims and equivalents thereof.

[0048] Although the present specification contains many details of specific embodiments, these are not to be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of specific features of particular embodiments disclosed herein. Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features of the invention that are described in the context of a single embodiment may also be implemented separately or in any appropriate sub-combination. Furthermore, although the features may be described above as agents in certain combinations and even initially claimed as such, one or more features of a claimed combination may in some cases be excluded from the combination and the claimed combination may be directed to a sub-combination or a variation of a sub-combination.

Claims

1. An industrial truck (10) having an autonomous and / or assisted driving function for positioning the industrial truck with respect to a target object (50), comprising: - means for handling a load (12, 13); - a video camera (22) configured to generate image data relating to a visual field, - a LIDAR device (21) configured to generate point cloud data relating to the visual field, - a control means (40) connected with the video camera in order to receive image data generated by the video camera and connected with the LIDAR device in order to receive point cloud data generated by the LIDAR device; wherein the control means is configured to: - acquire image data from the video camera and point cloud data from the LIDAR device relating to the same visual field, - recognise the target object (50) in an image by image processing applied to the acquired data, wherein the control means is configured to identify a region of interest, ROI, (32) in the image in which the target object is included; - associate the point cloud data acquired within the ROI with the target object recognised in the image, including to select a subset of the plurality of points of the point cloud within the ROI that corresponds to the target object; and - control the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object.

2. An industrial truck (10) according to claim 1, wherein controlling the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object comprises determining a distance between the industrial truck and the target object based on the point cloud data associated with the target object.

3. An industrial truck (10) according to one or more of the preceding claims, further comprising storage media (23) in which one or more models of target objects are stored, each model of a target object comprising model point cloud data associated with the target object and / or image data associated with the target object.

4. An industrial truck (10) according to claim 3, wherein the control means (40) is configured to compare the point cloud data associated with the recognised target object and / or image data associated with the target object with the one or more stored models of target objects.

5. An industrial truck (10) according to claim 4, wherein the control means (40) is configured to identify, based on the comparison, a correspondence between the point cloud data associated with the recognised target object and one of the stored models of target objects.

6. An industrial truck (10) according to claim 5, wherein the control means (40) is configured to determine a distance between the industrial truck and the target object only if the control means identifies a correspondence of the point cloud data associated with the recognised target object to a stored model of a target object.

7. An industrial truck (10) according to one or more of the preceding claims, wherein the control means (40) is configured to recognise, by image processing, one or more of the following target objects: - a load to be handled by the means for handling a load, - an unloading site where a load handled by the industrial truck is to be unloaded, - a loading site from where a load to be handled by means of the industrial truck is to be picked up, - a loading / unloading device with which the industrial truck is configured to interact in order to perform a loading / unloading function.

8. An industrial truck (10) according to one or more of the preceding claims, wherein the LIDAR device (21) is mounted on the industrial truck in proximity to the video camera (22).

9. An industrial truck according to one or more of the preceding claims, wherein the LIDAR device (21) is configured to generate point cloud data including a distance associated with each of the points of a point cloud represented by the point cloud data.

10. An industrial truck according to one or more of the preceding claims, wherein the control means (40) is configured to recognise the target object (50) in the image (30) without processing the point cloud data generated by the LIDAR device.

11. Method for control means of an industrial truck having an autonomous and / or assisted driving function for positioning the industrial truck with respect to a target object, wherein the industrial truck comprises: means for handling a load, a video camera configured to generate image data relating to a visual field, a LIDAR device configured to generate point cloud data relating to the visual field, and control means connected with the video camera in order to receive image data generated by the video camera and connected with the LIDAR device in order to receive point cloud data generated by the LIDAR device, wherein the method comprises the steps of: - acquiring image data from the video camera and point cloud data from the LIDAR device relating to the same visual field, - recognising the target object (50) in an image by image processing applied to the acquired image data, wherein a region of interest, ROI, (32) is identified in the image in which the target object is included; - associating the point cloud data acquired within the ROI with the target object recognised in the image, including selecting a subset of the plurality of points in the point cloud within the ROI that corresponds to the target object; and - controlling the autonomous and / or assisted driving function based on the point cloud data associated with the recognised target object.

12. Control unit for an industrial truck having an autonomous and / or assisted driving function for positioning the industrial truck with respect to a target object, wherein the industrial truck comprises: means for handling a load, a video camera configured to generate image data relating to a visual field, a LIDAR device configured to generate point cloud data relating to the visual field, wherein the control unit is configured to be connected to the video camera in order to receive image data generated by the video camera and connected to the LIDAR device in order to receive point cloud data generated by the LIDAR device, wherein the control unit is configured to perform the method according to claim 11.

13. Computer program, preferably one or more processors included in the control unit according to claim 12, containing instructions which, when executed by a computer, cause the computer to execute the steps of the method according to claim 11.

Images