Method and control unit for marking a person

Abstract

Method for identifying a person (2) located in the vicinity (4) of a vehicle designed as a forklift truck, comprising the steps of capturing (K1) at least one image with at least one camera (22, 24, 26) arranged on the vehicle, wherein the at least one image contains image information relating to the person (2), capturing (R1) at least one point cloud with at least one radar sensor (32, 34, 36) arranged on the vehicle, wherein the at least one point cloud contains at least one point relating to the person, Recognition (K2) of the person (2) based on the image information of at least one image, Determining (R2) a distance of the detected person (2) from the vehicle based on the at least one point of the at least one point cloud, Assigning (S4) an area of ​​at least a partial representation of the environment (4) of the vehicle to the identified person (2) and Marking (S5) the area assigned to person (2) depending on the specified distance; wherein danger zones are defined in a coordinate system of the vehicle, wherein marking only takes place when the identified person is in such a danger zone, wherein the area is defined for a tool arranged on the vehicle.

Description

Technical field

[0001] The invention relates to the detection of a person located in the vicinity of a vehicle designed as a forklift truck by means of camera-based image capture. State of the art

[0002] To implement driver assistance systems, environmental sensors are used on a vehicle to detect surrounding situations. This allows the driver and / or the vehicle to perceive and recognize conditions in the vehicle's environment and react accordingly to hazards.

[0003] German patent DE 10 2015 209 857 A1 discloses an emergency braking system for a vehicle based on sensor-based environmental perception. This sensor-based environmental perception serves to detect and identify a pedestrian in order to initiate emergency braking. Furthermore, German patent DE 10 2017 120 498 A1 discloses another sensor-based environmental perception system for a vehicle used to perceive road conditions. For this purpose, sensor data from several sensors are combined to generate an environmental view.

[0004] Furthermore, DE 10 2014 014 662 A1 discloses a display device for a commercial vehicle with which a driver of the commercial vehicle can be informed about obstacles located in the vehicle's surroundings. Description of the invention

[0005] The present invention relates to a method for identifying a person located in the vicinity of a vehicle designed as a forklift truck.

[0006] The identification of a person in the vicinity of a vehicle designed as a forklift, such as a pedestrian, can be done visually. Such visual identification can be achieved through an image-based representation of the vehicle's surroundings or through an abstract representation of the vehicle's surroundings. The vehicle's surroundings can also be understood as the area around the vehicle. An image-based representation of the vehicle's surroundings can, for example, be at least a section or part of a captured 360° or panoramic image. An abstract representation can, for example, simply be an abstract representation of the detected person in relation to a vehicle coordinate system, without an image-based representation.

[0007] Visual identification of a person can, in principle, be achieved using any abstract information suitable for visual identification. This identification can involve signaling, marking, and / or highlighting the person or their surroundings within a vehicle environment display. The person can be identified in this way temporarily, cyclically, or permanently. By identifying a person in the vicinity of the vehicle in this manner, the person can be displayed to a driver or any other operator or user of the vehicle in an abstract way. This allows the procedure for warning of a person in the vicinity of the vehicle to be applied.

[0008] The vehicle can be a non-autonomous vehicle, meaning a manned vehicle, or an autonomous vehicle, meaning a driverless vehicle. If it is an autonomous vehicle, the procedure can be applied such that the identified person is automatically taken into account as additional information during the vehicle's autonomous operation. Such consideration can involve intervention in the vehicle's propulsion, steering, or a task being performed by the vehicle. For example, emergency braking can be initiated if the person presents an unavoidable obstacle, a deviation from a route can be made to avoid the person, or a task can be reassigned if, for instance, the person is blocking a work site.Such a reaction to a marked person can also be carried out by the driver of a vehicle.

[0009] One process step is the capture of at least one image with at least one camera mounted on the vehicle, wherein the at least one image contains image information relating to the person.

[0010] Image capture can involve recording or generating a digital image of an area surrounding the vehicle, corresponding to the camera's field of view. This field of view can encompass a spherical range of up to approximately 180 degrees using a wide-angle or fisheye lens. The camera can be any imaging camera sensor, such as an RGB camera, an infrared camera, or a 3D camera. It can be either a panoramic or an omnidirectional camera. The image captured by the camera can contain pixels with corresponding RGB, thermal, or depth information. For continuous recording of the person, any of the aforementioned cameras can also be configured as a video camera.

[0011] A further procedural step is the identification of the person based on the image information of at least one image.

[0012] Person recognition within image data can be achieved using established image processing algorithms, such as segmentation or matching algorithms. Semantic segmentation can be applied as a segmentation algorithm for person recognition in the image. Template matching can be used as a matching algorithm.

[0013] A further procedural step is to assign an area of ​​at least a partial representation of the vehicle's surroundings to the identified person and to mark the area assigned to the person.

[0014] The area within the environment display can be defined by an outline, contour, or silhouette of the depicted person. The area within the environment display can also be an area adjacent to the depicted person. Such an adjacent area can be either directly connected to or separated from the depicted person. The environment display itself can be any real or abstract visualization of the vehicle's surroundings, and can be displayed on a visual display device, such as a screen, as a still image, or as a video. In the case of an autonomous vehicle, the display can also be processed internally without being visible to the vehicle's user. The display can also be visualized separately from the vehicle and at a distance from the user of the autonomous vehicle.

[0015] Within the scope of the invention, a person captured by a camera or camera system can be visually highlighted not only by a mere visual reproduction of the person, but also by an additional visual marking of the person in order to visually warn the driver or user. An advantageous effect of such a warning system can be the reduction of accidents involving people and the vehicle.

[0016] The invention is based on the understanding that person recognition based on a real image, which is therefore solely the responsibility of the driver or observer, is disadvantageous because it can lead to dangerous situations when operating a vehicle. One concept of the invention is therefore that both person recognition and person identification are automatically provided on a vehicle in order to detect potentially hazardous situations immediately and automatically.

[0017] This is advantageous because the invention allows for a direct and essentially automatic warning only when a person is in the vehicle's vicinity, but not when another object is present. This minimizes the risk of creating a dangerous situation. This is particularly relevant when a person moves abruptly towards the vehicle or when the vehicle rapidly approaches a stationary person.

[0018] The method further comprises the acquisition of at least one point cloud by at least one radar sensor mounted on the vehicle, wherein the at least one point cloud contains at least one point relating to the person. The radar sensor can be a one-dimensional or point-based sensor, a two-dimensional or fan-shaped sensor, or a three-dimensional or scanning sensor. The point cloud can have a corresponding dimension, which in the one-dimensional case can already consist of a single point. The radar sensor can thus discretely capture the contour of the person with individual points, whereby a horizontally oriented fan-shaped radar sensor can capture a transverse plane as the contour of the person, and a vertically oriented fan-shaped radar sensor can capture a sagittal plane or a frontal plane as the contour of the person.Detecting people using a radar sensor is advantageous due to its high spatial and temporal resolution.

[0019] The procedure includes, as additional steps, determining the distance between the detected person and the vehicle based on at least one point in the point cloud, and marking the area assigned to the person depending on the determined distance. The distance determination can be based on the points recorded on the person, for example, at the center of gravity of these points, relative to any point on the vehicle. This allows the distance between the person and the vehicle to be determined using radar-based distance measurement. The determined distance can be taken into account in the marking of the person by scaling the marking depending on the distance. This can be achieved by using distance-dependent color coding of the marking, for example, based on a traffic light system.A person located in a danger zone around the vehicle, in its direction of travel, or within its working radius can be colored accordingly. Distance-dependent marking, and thus a fusion of radar and image data, is advantageous because it allows warnings to be issued about people who, due to their relative position to the moving vehicle, may pose a danger.

[0020] Based on at least one point in the point cloud relating to the person, their position relative to the vehicle in a vehicle coordinate system can also be determined. For this purpose, the position and orientation of the radar sensor in the vehicle coordinate system can be determined or established through calibration.

[0021] Alternatively or additionally to radar-based distance determination or position determination, a distance to a person detected by image processing, or their position, can be determined based on camera-based image acquisition. For this purpose, methods such as stereophotogrammetry can be applied, whereby at least two images from the same or different cameras at different image acquisition locations can be used. In addition to image-based person detection, a person can also be detected based on radar-based point cloud acquisition. For this, for example, movement patterns of individual points in point clouds measured at different times can be taken into account.

[0022] Alternatively or additionally, the embodiment can include, as further process steps, determining a direction angle of the detected person relative to a vehicle-specific reference direction based on the at least one point of the at least one point cloud, and marking the area assigned to the person depending on the determined direction angle. The vehicle-specific reference direction can, for example, be defined by the vehicle's longitudinal axis. Such consideration of the relative orientation of the detected person with respect to the vehicle can be advantageous, as it allows a person to be marked and warned of if, for example, they are located within an angle or sector related to the direction of travel.

[0023] Another embodiment includes, as a further process step, the visualization of at least a partial representation of the vehicle's surroundings based on the at least one captured image. The visualization of the vehicle's surroundings can be based on a single image from one camera or on a combination of several images from one or more cameras. Multiple images can be stitched together to create a bird's-eye view, a panoramic image, or a 360-degree view of the vehicle's surroundings using appropriate projection. Such a visual representation has the advantage that the vehicle's surroundings can be realistically depicted, and a person identified within them can be placed in an intuitive spatial relationship to the vehicle's environment.

[0024] In another embodiment, the at least partial representation of the vehicle's surroundings is a bird's-eye view image. The vehicle's surroundings, or a portion thereof, can be depicted from any perspective, such as a side view, a perspective view, or a top view. A bird's-eye view representation can be advantageous because it allows the vehicle itself to be shown, thus revealing spatial relationships between the person depicted and the vehicle. The bird's-eye view representation can be based on a central projection or a spherical projection of at least one image onto a display plane. Such a "bowl" representation has the advantage that virtually the entire vehicle's surroundings can be visualized in a single image.Alternatively or additionally, a panoramic image can also be generated using a spherical projection.

[0025] In another embodiment, the assigned area is formed based on the contour of the detected person. The marking of the person can thus refer to a line-like or an area-like marking based on the contour, that is, a marking based on an outline of the person in their pictorial representation. The contour visible in a pictorial representation of the person and / or the vehicle's surroundings can therefore be abstractly marked from any perspective and thus visually emphasized in the representation. The contour or the area enclosed by it can, for example, be colored. The contour can correspond to the person's silhouette or have a symbolic contour that may differ from the actual silhouette.

[0026] In a further embodiment, the associated area includes a segment with reference to a foot point of the detected person in the at least partial representation of the vehicle's surroundings. The segment can be any geometric shape suitable for identifying the person. For example, the segment can be bar-shaped, circular, or triangular. Such a segment can be positioned to identify the person in the surroundings representation such that it encompasses a projection point of the person onto a representation plane. The segment can also be positioned adjacent to or spaced apart from the projection point in the representation. Such a segmented representation can be advantageous because, unlike with a contour-based, area-based identification, the person cannot be covered by the segment but can be surrounded by it.This has the effect of combining a realistic representation of the person with an abstract representation of the same person in a single image. For example, the person can be overlaid or backed with a semi-transparent colored area segment. Completely covering the person, on the other hand, has the advantageous effect of generalizing a detailed representation of the surrounding environment.

[0027] Another embodiment includes, as a further process step, marking the area assigned to the person with a visual warning signal. The warning signal can be visual, consisting at least of coloring the assigned area and visualizing or indicating it. The coloring can be done with a signal color, for example, red, yellow, or green. The warning signal can be visualized temporarily (i.e., flashing), cyclically (i.e., blinking), or continuously (i.e., permanently illuminated).

[0028] The color of the area assigned to a person can be distance-dependent, meaning the color is chosen based on the distance between the vehicle and the person. The coloring can also depend on the person's angle relative to the vehicle's reference direction. Coloring the area assigned to a person with a signal color, such as red, can thus effectively warn of people who are in close proximity to the vehicle, in a direction of travel, or within the vehicle's operating radius. Conversely, the area assigned to a person can be colored with a different signal color, such as green, if that person poses no danger, meaning if they are not in close proximity to the vehicle, not in the direction of travel, or outside the vehicle's operating radius. The close proximity can encompass just a few meters.The direction of travel can be determined based on a route or a movement trajectory of the vehicle.

[0029] Another embodiment includes, as a further process step, determining the relative position of the person in relation to the vehicle based on at least one of the detection steps, i.e., based on the step of capturing an image with the camera and / or based on the step of capturing a point cloud with the radar sensor. The relative position can include the person's relative position in relation to the vehicle and / or a relative direction or angle between a vehicle-specific reference direction and the person. Considering the relative position is advantageous because the identification of the person or their associated area can be carried out independently of a higher-level coordinate system based on a vehicle-specific coordinate system. This allows a moving vehicle to issue a warning about a person in its vicinity autonomously and in near real time.

[0030] The embodiment includes, as an additional process step, the marking of the area assigned to the person depending on the determined relative position. Person marking or the marking of an area assigned to a person can thus be carried out depending on the person's position in a vehicle coordinate system. According to the invention, danger zones are defined in the vehicle's coordinate system, and marking only occurs when the person is located within such a danger zone. An advantageous effect of this is that, regardless of the vehicle's current absolute position, an area is defined for a tool mounted on the vehicle in which a person can be marked.

[0031] Another embodiment includes, as a further process step, the display of the determined relative position in the representation of the vehicle's surroundings. In addition to indicating the area associated with the person, the representation can also show the distance to the person, the distance between the vehicle and the person, and the angle of direction relative to the person. These parameters can also be used as a basis for operating an autonomous vehicle without being displayed. This embodiment has the advantage that additional visual information about people in the vehicle's vicinity can be provided to a driver or user.

[0032] Another embodiment includes, as a further process step, the acquisition of a multitude of point clouds with at least one radar sensor mounted on the vehicle, wherein each point cloud contains at least one point relating to the person. The acquisition of the point clouds can be performed synchronously with multiple sensors. This allows a large area of ​​the vehicle's surroundings to be captured from a single location of the vehicle. This has the advantage that, for example, fan-scanning radar sensors can be combined in such a way that a multitude of people can be simultaneously captured in different vertical and / or horizontal detection planes. Alternatively or additionally, the acquisition of the point clouds with one or more sensors can be performed continuously, i.e., sequentially, at specific time intervals. Thus, points relating to the same person can be captured, and the person can therefore be tracked.They can be tracked. This has the advantageous effect that, in addition to the person's location, their movement patterns and kinematics can also be determined. This, in turn, has the advantage that even a moving person can be continuously tagged in near real time.

[0033] The embodiment includes, as additional process steps, determining the movement behavior of a person based on the captured point clouds. This movement behavior includes at least one of the following: the person standing still, the person moving, the person moving in a certain direction, the person moving at a certain speed, and a change in any of these movement characteristics. Furthermore, the embodiment includes marking the area associated with the person based on the determined movement behavior. Marking the area associated with the person based on one or more of the aforementioned characteristics can thus visualize, i.e., represent, the person's movement. This movement can be related to a location or the movement of a vehicle, and a hazardous situation can be derived from this.For example, a warning can be issued for a person moving towards the vehicle using a signal color, such as red, while a person moving away from the vehicle can either not be marked or can be highlighted with a different signal color, such as green. Considering such movement patterns when marking a person has the advantage that a dynamically changing environment can be monitored and explicit warnings can be issued for people moving within it.

[0034] Another embodiment includes, as a further process step, the display of the determined movement behavior in a representation of the vehicle's surroundings. In addition to marking the area assigned to the person, one or more movement patterns of the person can be displayed. These movement parameters can also be used as a basis for operating an autonomous vehicle without being displayed. This embodiment has the advantage that additional visual information about the movement behavior of people in the vehicle's vicinity can be provided to a driver or user.

[0035] Another embodiment includes, as further process steps, classifying the detected person as either moving or stationary and marking the detected person based on this classification. For example, a person in the vicinity of a vehicle can be marked if they are moving. Marking the area associated with the person can alternatively or additionally be based on each of the aforementioned movement characteristics alone or, for example, on a corresponding truth table. The term "classifying" can also refer to categorizing, and "categorizing" can also refer to categorizing. Marking a person based on a classification is advantageous because it allows artificial intelligence to issue warnings only when a person actually poses a danger.This keeps the driver or user constantly aware of potential hazards, as unnecessary warnings about non-existent dangers can be minimized. Furthermore, it can minimize the driver or user consciously ignoring a supposedly non-existent danger.

[0036] Another embodiment includes, as a further process step, the output of a warning signal as a reaction to a detected person. The output of the warning signal can be determined based on the classification according to the person's movement behavior, i.e., whether the person has been classified as moving or stationary. Alternatively or additionally to the process step of marking the area assigned to the person with a visual warning signal, such a warning signal can be visualized or output separately. In principle, any other warning signal can be visualized or output to warn the driver or user of a person in the vehicle's vicinity. Any medium can be used for this purpose, such as a screen, a loudspeaker, or a display. The warning signal can be a visual warning signal, for example, a light signal from a rotating beacon.The warning signal can also be an acoustic signal, such as a horn sound. Furthermore, the warning signal can also be a haptic signal, for example, triggered by a vibration of the steering wheel or control lever of the vehicle. One or more such warning signals advantageously reinforce, in addition to identifying the person, a warning of a dangerous situation.

[0037] Another embodiment includes, as a further process step, intervention in the vehicle's drivetrain as a reaction to a detected person. If a person is tagged, possibly depending on one of the aforementioned classifications, intervention in the vehicle's drivetrain or powertrain can occur. An autonomously driving vehicle can thus be automatically braked or rerouted around the person as an obstacle. Furthermore, an emergency stop or emergency shutdown can be initiated as a consequence of the tagged person to avoid a collision. Particularly for the operation of autonomous vehicles, such a reaction can improve the safety of people in the vicinity of the vehicle.

[0038] Another embodiment includes, as a further process step, the detection of an object that is not a person, based on the image information of at least one image, wherein the at least one image contains image information relating to the object. Each process step of the invention or of one of its embodiments can, in principle, also be applied to any object recognizable in an image that is not a person. In addition to warning of people, warnings can thus be issued for individually defined objects or, alternatively, for essentially all objects in the vehicle's surroundings. This has the advantage that collisions with various objects and people can be minimized, thereby increasing the vehicle's level of automation.

[0039] The embodiment includes, as additional process steps, the assignment of an area of ​​the visualized, at least partial, representation of the vehicle's surroundings to the object recognized in the image information and the marking of the area assigned to the object. Each process step of the invention or one of its embodiments can, in principle, also be applied to the recognition and marking of objects that are not persons. In addition to marking persons, individually defined objects or, alternatively, essentially all objects in the vehicle's surroundings can be marked. For this purpose, one of the areas mentioned in connection with marking a person can also be assigned to an object. This has the advantage that the vehicle's surroundings can be monitored with regard to a large number of obstacles.

[0040] Another embodiment includes as further process steps a classification as a recognized person or a recognized object, and marking of the area assigned to the person and the area assigned to the object, depending on the classification. For example, a marking step of one of the aforementioned types can be performed if the object is a person, and a further marking step of another of the aforementioned types if it is an object, or the latter can be omitted. The classification of this embodiment can be combined with the classification based on the aforementioned movement characteristics. Marking a person depending on one of these classifications is advantageous because it further minimizes unnecessary warnings about non-existent dangers.

[0041] The present invention further relates to a control unit configured to perform the process steps of the invention or the process steps of one or more of its embodiments. The control unit can be a control device with which such programmed process steps can be controlled or regulated on the vehicle and / or in a user environment of a user of an autonomous vehicle.

[0042] The present invention further relates to a vehicle with such a control unit. The vehicle can be an autonomous, semi-autonomous, or non-autonomous vehicle. The vehicle can be any vehicle designed for transporting persons and / or goods.

[0043] The vehicle can be any manned or unmanned transport vehicle, for example, from the logistics sector, or any work vehicle. In one embodiment, the vehicle is an autonomous industrial truck. Such a truck could, for example, be a forklift. When operating a forklift, preventing accidents involving people in the vicinity of the forklift is of particular importance, as this increases workplace safety.

[0044] In another embodiment, the at least one camera mounted on the vehicle is located at the rear of the industrial truck. Positioning sensors in general, and the camera or radar sensor in particular, at the rear of an industrial truck, such as a forklift, is advantageous because the rear contains stationary vehicle parts. An alternative or additional arrangement of a sensor at the front, for example, on the mast of a forklift, or at the side, is also possible. However, these areas may contain moving vehicle parts, such as the forklift forks, which can complicate fixed positioning. Conversely, a lateral arrangement of at least one camera or radar sensor can also create a corresponding lateral detection range. Brief description of the drawings Fig. Figure 1 shows a flowchart with process steps of an exemplary embodiment of a method for marking a person located in the vicinity of a vehicle. Fig. Figure 2 shows an exemplary embodiment of a vehicle in a perspective view. Fig. Figure 3 shows exemplary embodiments of marking a person located in the vicinity of the vehicle to further explain the procedure according to Fig. 1 and of the vehicle according to Fig. 2. Fig. Figure 4 shows further examples of marking a person located in the vicinity of the vehicle to further explain the procedure according to Fig. 1 and of the vehicle according to Fig. 2. Detailed description of embodiments

[0045] In Fig. Figure 1 shows the process steps S1, S2, S3, S4, S5 in a temporal sequence for carrying out the procedure for marking a person 2 located in the vicinity 4 of a forklift 10 as a vehicle (see following figures).

[0046] In a first step S1, data is acquired using a multitude of cameras 22, 24, 26 and a multitude of radar sensors 32, 34, 36, which are arranged on the forklift 10. In a first sub-step K1, each of the cameras 22, 24, 26 captures an image of the forklift 10's surroundings 4. At least one of these images from one of the cameras 22, 24, 26 contains image information relating to person 2. In a second sub-step R1, each of the radar sensors 32, 34, 36 captures a point cloud relating to the forklift 10's surroundings 4, whereby at least one of these point clouds contains a point relating to person 2.

[0047] Following step S1, a person detection process is performed in a second step S2 based on the data acquired in step S1 using cameras 22, 24, 26 and / or radar sensors 32, 34, 36. In a first sub-step K2, person 2 is automatically detected using image processing based on image information from at least one image from cameras 22, 24, 26. In a second sub-step R2, the distance or movement pattern of person 2, detected in sub-step K2, is determined based on at least one point cloud from radar sensor 32, 34, 36. For this purpose, the image information from at least one of the images acquired by cameras 22, 24, 26 and at least one of the point clouds acquired by radar sensors 32, 34, 36 can be transformed into a common coordinate system to establish a spatial relationship between the image information and the point clouds.Through such a linking via transformation, image information, which includes at least one pixel or its value or values, and a point of a point cloud are assigned to the same person.

[0048] In a further step S3, the environment 4 of the forklift 10 is visualized based on the images captured by cameras 22, 24, and 26. For this purpose, the images from cameras 22, 24, and 26 are stitched together to create a single image 5 of the environment 4 of the forklift 10, showing the environment 4 from a bird's-eye view. In image 5 of the environment 4, the forklift 10 is centrally located, encompassing the center of a central projection underlying image 5.

[0049] After step S2 or step S3, in a fourth step S4, an area of ​​image 5 of the environment 4 of the forklift 10 is assigned to a person 2 identified in sub-step K2. The environment representation carried out in step S3 and the area assignment carried out in step S4 are performed with reference to the Fig. 3 and Fig. 4 explained in more detail.

[0050] Following step S4, in a fifth step S5, person 2, identified in sub-step K2, is signaled by marking the area assigned to them in step S4. The person marking carried out in step S5 is also referenced in the Fig. 3 and Fig. 4 explained in more detail.

[0051] In Fig. 2 is a forklift 10 as a vehicle with a control unit 50 for carrying out the process steps S1, S2, S3, S4, S5 according to Fig. 1 shown.

[0052] Three cameras 22, 24, 26 and three radar sensors 32, 34, 36 are arranged at the rear section 11 of the forklift 10. The cameras 22, 24, 26 are arranged on the roof 12 of the forklift 10, with the first camera 22 being configured as a rear-view camera, the second camera 24 as a side-view camera looking to the left, and the third camera 26 as a side-view camera looking to the right, with corresponding viewing angles to the rear, left, and right.

[0053] Cameras 22, 24, and 26 are designed as wide-angle cameras, each with a viewing angle of up to approximately 180 degrees. The first radar sensor 32 and the second radar sensor 34 are each located on a side frame 13 of an open driver's cab 14 of the forklift 10. The driver's cab 14 is bounded above by the roof 12 and to the rear by the two frame members 13.

[0054] The radar sensors 32, 34 are fan-shaped scanning radar sensors 32, 34 which form a scan plane horizontally, obliquely, or vertically at the height of their respective mounting location on the spars 13. In this embodiment, the radar sensors 32, 34 are mounted at approximately the same height on the spars 13, enabling them to detect point clouds in a common scan plane, for example, a horizontal plane.

[0055] Another radar sensor 36 is located below the other radar sensors 32, 34 on the chassis 15 of the forklift 10 in its rear area 11. This additional radar sensor 36 detects a point cloud in a scan plane further than that covered by the other two radar sensors 32, 34. The scan planes are parallel to each other, for example, horizontally aligned.

[0056] Cameras 22, 24, 26 and radar sensors 32, 34, 36 are all identical in construction.

[0057] In Fig. In the area 4 surrounding the forklift 10, persons 2 are identified in a so-called top-view view. The area assigned to each person 2 is defined by their depicted contour 3. This contour 3 is filled with a signal color, for example, red, to identify the person. Thus, person 2 is colored within their contour 3. The contour 3 of person 2 is extracted from a background image of the forklift's surroundings 4.

[0058] Image 5 of the environment 4 of the forklift 10, which shows the environment 4 from a bird's-eye view including the forklift 10 as a background, is generated from individual images of cameras 22, 24, 26 using stitching and central projection. This essentially visualizes the entire environment 4 of the forklift 10. The control unit 50 is used to identify the persons 2 in the environment 4 of the forklift 10 and also serves as the central evaluation unit 40 for generating image 5.

[0059] In Fig. Figure 3 shows a further embodiment where bar segments 6 are visualized as areas assigned to person 2. Depending on the distance of person 2 from the forklift 10, the bar segments 6 are colored with different signal colors, for example, red for nearby persons 2, yellow for persons 2 further away, and green for persons 2 even further away. Depending on the distance between the forklift 10 and a person 2, or depending on a direction angle in the forklift 10 relative to a person 2, based on the direction of travel or the longitudinal axis of the forklift 10, a plurality of the respective bar segments 6 are projected onto a foot point of the respective person 2 and thus shown in Figure 5 of the forklift 10's surroundings. The distances are calculated from the point clouds of the radar sensors 32, 34, 36 based on the points in the point clouds assigned to a person 2 by means of the evaluation unit 40.

[0060] The marking of a person 2 by means of one or more bar segments 6 is further represented on an object 9, which is not a person 2, by evaluation of the point clouds generated by the radar sensors 32, 34, 36 by the evaluation unit 40.

[0061] The in Fig. The four illustrated embodiments of marking a person 2 located in the vicinity 4 of the forklift 10 differ initially from those in Fig. The embodiments shown in Figure 3 are simplified in that an image 5 of the environment 4 of the forklift 10 is formed from an abstract background and from a panoramic image of persons 2 extracted by the evaluation unit 40 from cameras 22, 24, 26. The embodiments can therefore be described as simplified.

[0062] Another difference is that instead of using and coloring the outline 3 of the persons 2, each person 2 is assigned a circular segment 7, which is displayed below their feet. The center of each circular segment refers to the position or representation of the forklift 10. The respective circular segment 7 is colored according to the distance between a person 2 and the forklift 10, which is determined by radar. For example, the circular segment 7 assigned to a person 2 located close to the forklift 10 is colored red, a circular segment for a person located further away is colored yellow, and a circular segment for a person 2 located even further away is colored green.

[0063] The colored line segments 8 represent a scale for the distance-dependent coloring described above. The line segments 8 also define an area assigned to person 2, which is colored accordingly based on distance.

[0064] The in the Fig. 3 and Fig. The four illustrated embodiments explain how a person 2 in the vicinity 4 of a forklift 10 is indicated to a driver or user of the forklift 10 by means of abstract marking of an area assigned to them in an environmental representation with a signal color, and how they are thus warned of the person 2. Reference sign 2 Person 3 Contour 4 Environment 5. Image of the surroundings 6 bar segment 7 Circle segment 8 line segment 9 objects 10 forklifts 11 Rear area 12 Roof 13 Holm 14 Driver's cab 15 chassis 22 first camera 24 second camera 26 third camera 32 first radar sensor 34 second radar sensor 36 third radar sensor 40 evaluation units 50 control unit S1 Data Acquisition K1 Image Acquisition R1 Point Cloud Capture S2 Person Recognition K2 image-based person recognition R2 point cloud-based distance determination S3 Environment Display S4 Area Assignment S5 Personal identification

Claims

[1] Method for identifying a person (2) located in the vicinity (4) of a vehicle designed as a forklift truck, comprising the steps of capturing (K1) at least one image with at least one camera (22, 24, 26) arranged on the vehicle, wherein the at least one image contains image information relating to the person (2), capturing (R1) at least one point cloud with at least one radar sensor (32, 34, 36) arranged on the vehicle, wherein the at least one point cloud contains at least one point relating to the person, Recognition (K2) of the person (2) based on the image information of at least one image, Determining (R2) a distance of the detected person (2) from the vehicle based on the at least one point of the at least one point cloud, Assigning (S4) an area of ​​at least a partial representation of the environment (4) of the vehicle to the identified person (2) and Marking (S5) the area assigned to person (2) depending on the specified distance; wherein danger zones are defined in a coordinate system of the vehicle, wherein marking only takes place when the identified person is in such a danger zone, wherein the area is defined for a tool arranged on the vehicle. [2] Method according to claim 1, comprising visualizing (S3) the at least partial representation of the environment (4) of the vehicle based on the at least one captured image. [3] Method according to one of the preceding claims, wherein the at least partial representation of the environment (4) of the vehicle is a picture (5) of the environment (4) of the vehicle from a bird's-eye view. [4] Method according to one of the preceding claims, wherein the assigned area is formed based on a contour (3) of the recognized person (2). [5] Method according to one of the preceding claims, wherein the associated area comprises a segment (6, 7, 8) with reference to a foot point of the detected person (2) in the at least partial representation of the environment (4) of the vehicle. [6] Method according to one of the preceding claims, comprising marking (S5) the area assigned to the person (2) with a visual warning signal. [7] Method according to any one of the preceding claims, with Determining the relative position of the person (2) in relation to the vehicle based on at least one of the detection steps (K1, R1) and Identify (S5) the area assigned to person (2) depending on the determined relative position. [8] Method according to claim 7, with displaying the determined relative position in the representation of the environment (4) of the vehicle. [9] Method according to any one of the preceding claims, comprising Capturing (R1) a plurality of point clouds with at least one radar sensor (32, 34, 36) mounted on the vehicle, wherein the point clouds each have at least one point relating to the person, Determining the movement behavior of the person (2) based on the recorded point clouds, wherein the movement behavior exhibits at least one of the following: the person (2) standing still, the person (2) moving, the person (2) moving in a certain direction, the person (2) moving at a certain speed, and a change in one of these movement characteristics. Identify (S5) the area assigned to person (2) depending on the determined movement behavior. [10] Method according to claim 9, with displaying the determined movement behavior in the representation of the environment (4) of the vehicle. [11] Method according to any one of the preceding claims, with Performing a classification of the identified person (2) as a moving person or as a non-moving person and Identification (S5) of the identified person (2) depending on the classification. [12] Method according to any of the preceding claims, comprising issuing a warning signal as a reaction to a detected person (2). [13] Method according to any of the preceding claims, comprising intervening in the vehicle's drive train as a reaction to a detected person (2). [14] Method according to any one of the preceding claims, with Identifying an object (9) that is not a person (2) based on the image information of the at least one image, wherein the at least one image contains image information relating to the object (9), Assigning (S4) an area of ​​the visualized at least partial representation of the environment (4) of the vehicle to the object (9) recognized in the image information and marking (S5) the area assigned to the object (9). [15] Method according to claim 14, characterized by Making a classification as a recognized person (2) or as a recognized object (9) and Identify (S5) the area assigned to the person (2) and the area assigned to the object (9) depending on the classification. [16] Control unit which is configured to perform the process steps according to any of the preceding claims. [17] Vehicle with a control unit (50) according to claim 16, wherein the vehicle is designed as a forklift truck. [18] Vehicle according to claim 17, wherein the vehicle is an autonomous industrial truck. [19] Vehicle according to claim 18, wherein the at least one camera (22, 24, 26) arranged on the vehicle is arranged at a rear area (11) of the industrial truck.

Images