DEVICE AND METHOD FOR NAVIGING AND / OR GUIDING A VEHICLE AND VEHICLE

Description

[0001] The invention relates to a device and a method for navigating and / or guiding a vehicle, in particular a wheelchair, as well as a vehicle.

[0002] US 4 407 393 reveals a wheelchair that is controlled via sensors using signals originating from a part of the passenger's body, namely the mouth.

[0003] From EP 3 646 127, a special wheelchair control system for an electric wheelchair is known. This system comprises an input element, for example, smart glasses worn on the passenger's head, and an adapter box for transmitting the data from the input element to an input / output module of the electric wheelchair. To generate driving commands, when using the smart glasses, the passenger's head movements are detected by motion sensors and assigned to the desired directions and speeds of travel. Control signals generated from these driving commands are then transmitted to the wheelchair's input / output module to control the wheelchair.

[0004] EP 3 076 906 relates to a control unit for a personal vehicle. Using two independent motion sensors, the control unit can provide a control signal for the vehicle by evaluating the relative orientations of these sensors to each other. This allows the system to recognize a passenger's desired driving command even in the presence of disturbances such as inclines, declines, or unpaved roads. If the evaluation of the orientation sensors identifies previously known gestures, these are used as the vehicle control signal. Additional external sensors can be used for configuration and to determine the vehicle's position and orientation.

[0005] The described state of the art is problematic in many respects. While motion- or orientation-based control can be implemented in many cases using data from at least one primary motion sensor, and some embodiments even allow the recognition of movement patterns to initiate an emergency stop, emergency call, or the launch of external applications, a failure of the primary motion sensor can already lead to a failure of the vehicle control system. For example, in a vehicle control system based solely on head orientation, even a cramp or a temporary stiffening of the neck can result in the passenger losing control. Furthermore, unwanted control commands from the passenger, triggered, for example, by errors in the primary motion sensor, can be transmitted to the vehicle's actuators, causing it to move or maintain its current position.

[0006] Based on this prior art, the object of the present invention is to provide a device and a method that creates increased safety and comfort during the operation of a vehicle. In particular, it aims to enable increased reliability, prevent the transmission of unwanted control commands to the vehicle, and improve comfort when using the vehicle. This object is achieved by the device according to claim 1, the vehicle according to claim 14, and the method according to claim 15.

[0007] The problem is solved in particular by a device for navigating and / or guiding a vehicle, especially a wheelchair, wherein the device comprises at least: A first sensor, in particular an inertial navigation sensor, configured and arranged to detect at least one first body part, in particular its position and / or rotation and / or translation, of a passenger of the vehicle and to output first sensor signals. A second sensor, in particular an image-based sensor, configured and arranged to detect at least the first body part of the passenger and / or its features, in particular its position and / or rotation and / or translation, and to output second sensor signals.A control unit designed to receive the first and second sensor signals, to determine, based at least on the first sensor signals, first control signals for controlling the vehicle, to determine, based at least on the second sensor signals, whether the first control signals meet at least first reliability criteria, and to enter a safety mode if the control unit determines that the first control signals do not meet at least the first reliability criteria.

[0008] One aspect of the invention arises from the fact that every sensor is subject to a non-eliminable error or failure rate. Particularly in applications with significant negative consequences from sensor failure, it is advantageous to provide suitable measures for dealing with sensor errors. The present invention addresses this by not using first sensor signals from at least one first sensor directly to determine a control signal for the vehicle, but rather by using at least second sensor signals, in particular image-based sensor signals, from a second sensor to determine at least the first reliability criteria. A safety mode is then activated if at least the first control signals determined using the first sensor signal do not meet the first reliability criteria.

[0009] A safety mode can be understood as a restricted operating range of the device, the extent of which is determined, on the one hand, by sensory-detectable physical conditions of a passenger. On the other hand, reduced accuracy and / or reliability of the sensors and actuators used by the device can also lead to a restriction of the device's operating range, e.g., driving at reduced speed.

[0010] The reliability criteria are designed to define thresholds above which the safety mode, i.e., the restricted operating range of the device, must be activated. With regard to the detectable physical states of the passenger, these thresholds can be general or individualized. Individualized thresholds may be necessary if a passenger's externally perceptible expressions of physical states, such as facial expressions or gestures, are significantly impaired. This can be the case, for example, with (partial) paralysis of a body part, such as the face. In such cases, the device may learn these thresholds through repeated exposure to specific facial expressions or gestures.It is also possible to adjust the threshold values ​​during operation of the device, for example if it is known that a passenger usually begins to tremble during prolonged operation of the vehicle as a result of the general exertion.

[0011] Reliability criteria can also define threshold values ​​for the required accuracy and / or reliability of individual sensors. Furthermore, permissible tolerances for deviations of the same physical parameters, determined by different sensors, can be defined. An example of this is the permissible positional deviation of the iris in a feature-based vehicle control system, where two different sensors detect the passenger's eye.

[0012] Using at least the second sensor signals, it is possible to determine whether the passenger is capable of controlling the vehicle. How this determination is made depends on the chosen embodiment of the present invention. It is possible to determine whether the passenger is capable of controlling the vehicle by comparing the second sensor signals with predefined, first reliability criteria.

[0013] For example, a passenger's head can be strongly tilted to one side over a prolonged period by at least the second sensor, due to progressively decreasing neck muscle strength. With head orientation-based control applied, this could lead to a potentially dangerous circular motion. In such a case, the safety mode can adjust the permissible operating range of the device so that the vehicle brakes to a standstill by issuing a second control signal.

[0014] Braking to a standstill is a possible measure that influences driving dynamics in response to sensorily detectable physical conditions.

[0015] It is also possible to determine a safe stopping position before the vehicle comes to a complete stop, using advanced position sensors or an environment-sensing sensor assembly, and then to control the vehicle to this stopping position. In this case as well, the device generates second control signals that are different from the first control signals.

[0016] The first and second control signals can be driving dynamics-influencing control signals transmitted to vehicle actuators. According to the invention, it is also possible for the control signals to provide data for device or passenger monitoring, or selection options for vehicle modes, to a portable computer system (wearable) or a passenger's mobile phone. The control signals can also be configured for forwarding to a remotely located computer system (backend) for remote device or passenger monitoring.

[0017] In safety mode, the device allows the operating range to be adjusted to initiate an emergency call. This enables designated individuals to be notified of the activated safety mode. If necessary, the passenger's recorded physical condition can also be transmitted, for example, to perform a remote human diagnosis of the passenger's physical condition.

[0018] A safety mode may also be initiated due to insufficient sensor capability. It is conceivable that one or more of the sensors provide the control unit with their own accuracy or reliability analysis data, in which case one or more sensors may not be used to determine the initial control signals or reliability criteria.

[0019] If the device generates first or second control signals that are classified as valid, these are transmitted to vehicle actuators, in particular controllable motors, wherein the vehicle actuators are arranged to drive the vehicle and are designed to receive and process the control signals of the device.

[0020] According to the invention, a first body part can be any part of the human body that is capable of performing movements.

[0021] In one embodiment, the first body part is the passenger's head. In this embodiment, the device comprises, as the first sensor, an inertial navigation unit arranged on or in the wearable computer system, which is configured to detect at least the head, in particular its orientation and / or position and / or rotation and / or translation, and to output first sensor signals.

[0022] The second sensor that detects the passenger's head can be an image-based sensor, such as a front-facing or rear-facing sensor from a mobile phone. Alternatively, it can be an additional image-based sensor mounted on the vehicle that detects the passenger's head. The control unit can be configured to receive the first and second sensor signals and, based on the first sensor signals, to generate initial control signals for controlling the vehicle.

[0023] Based on the second sensor signals, it can be determined whether the first control signals meet at least the first reliability criteria. The first reliability criteria are determined based on the second sensor signals. Safety mode can be activated if the control unit determines that the first control signals do not meet at least the first reliability criteria.

[0024] Basic data can be determined from the second set of sensor signals. This basic data can include geometric dimensions and / or size ratios and / or facial features and / or gestures, and the first reliability criteria are then determined using this basic data.

[0025] These geometrically or mathematically expressed properties of the basic data can characterize facial expressions perceived by humans and allow inferences to be drawn about the passenger's emotional state during vehicle operation. On the one hand, the first reliability criteria should be met if the passenger is highly likely to be in a state of happiness, relaxation, or a similar state associated with a positive mood. On the other hand, the first reliability criteria should be not met if the passenger is highly likely to be in a state of worry, surprise, anxiety, or a similar state associated with a negative mood.

[0026] According to the invention, image-based analysis can also detect redness, enlargement or constriction of the eyes, including the pupils, as well as the opening or closing of the eyes or mouth and lips, including inferable characteristics such as shortness of breath, increased pulse, or increasing, dangerous physical exertion of the person. The detection of such physical conditions also means that the first reliability criteria are not met.

[0027] Furthermore, initial reliability criteria may not be met if the first control signals detect control commands from the passenger that contradict at least the second sensor signals, i.e., if one sensor detects a desired movement to the left, while at least one other sensor detects a desired movement to the right.

[0028] If the initial reliability criteria are not met, the safety mode is initiated. According to the invention, it is therefore possible to prevent a control signal already generated from being transmitted to the vehicle actuators responsible for vehicle control due to a violation of the initial reliability criteria. This arrangement ensures that clearly recognizable, critical physical conditions of the passenger are detected, the vehicle is stopped, and / or an emergency call mode, following the safety mode, enables rapid assistance for the passenger by contacting an emergency contact. Furthermore, this prevents further physical harm to third parties and the passenger.

[0029] In one embodiment, the control unit is (furthermore) configured to determine the first control signals using not only the first sensor signals from the inertial navigation unit attached to or integrated with the wearable, but also the second sensor signals from the image-based sensor. The safety mode is activated if either the first or second reliability criteria are not met. In this embodiment, the second reliability criteria are based on an accuracy and / or reliability analysis of at least the first and second sensor signals.

[0030] Accuracy and / or reliability analysis can be performed using self-diagnostic data provided by the sensors along with the sensor signals.

[0031] Furthermore, the accuracy and / or reliability analysis can be determined using the first and second sensor signals of one or more of the features location and / or position and / or rotation and / or translation of the passenger's head, and the reliability of the two sensors can be estimated based on their comparison.

[0032] If the comparison of the feature(s) used to control the vehicle is within a predefined tolerance(s), the feature values ​​of the inertial navigation unit are used to determine the first control signal. Should one or more of the tolerances be exceeded, the self-diagnostic data of the image-based sensor is used, and in case of poor reliability, for example due to poor image quality, a signal indicating this poor reliability is transmitted to the passenger via the wearable device or their mobile phone. Generating control signals based on the initial sensor signals remains possible. If the image-based sensor has sufficient reliability, a safety mode is initiated, as the control unit assumes a fault in the inertial navigation unit. In this case, the safety mode initiates an immediate stop.Continuous analysis of the sensors ensures safe operation of the vehicle and allows for immediate measures to minimize damage in the event of sensor malfunctions.

[0033] In a further embodiment, the fulfillment of the first reliability criteria is determined by implementing machine learning, in particular classification, preferably by a support vector machine (SVM), or by implementing a neural network, in particular a convolutional neural network (CNN). This embodiment has the advantage of allowing the image-based facial recognition to be individually adjusted to the specific passenger. Furthermore, the aforementioned artificial intelligence methods generally achieve better hit rates and are more reliable in the field of image-based facial recognition than conventional classification methods, such as decision trees.

[0034] These artificial intelligence approaches require pre-labeled image data, particularly images with associated facial expressions, to classify the passenger's expressed mood. This labeled image data can consist of a set of images of randomly selected individuals, an individual image set of the passenger, or a mixed set.

[0035] In a further embodiment, a third sensor, in particular a vital signs sensor, is arranged on or between a body part to detect vital signs, in particular heart rate, of the passenger and to output third-party sensor signals. The control unit receives the third-party sensor signals, and if the third-party sensor signals show values ​​outside a certain range of vital signs, the second reliability criteria are not met. Consequently, the system enters safety mode.

[0036] Since vital signs can be measured at any part of the human body, the location where the vital signs sensor is attached can be any part of the body. However, placement on the neck, in the ear, around the wrist, around the chest, or around the abdomen is particularly desirable.

[0037] The vitality data sensor can also be a two-part sensor, for example a chest strap and a transmission unit on the wrist.

[0038] Furthermore, vitality data is not limited to heart rate. Continuous monitoring of blood pressure, blood oxygen levels, blood count, blood sugar, saliva, or other directly or indirectly measurable bodily parameters can also be achieved through the vitality data sensor.

[0039] In another embodiment, the first sensor is an eye feature detection unit instead of the inertial navigation unit. This unit is attached to or integrated into the wearable and is designed to detect at least one eye of the passenger and / or its features, in particular its orientation and / or position and / or rotation and / or translation. The eye feature detection unit can output first sensor signals, which the control unit receives and uses to determine the first control signals for controlling the vehicle. The eye feature detection unit can be an RGB image-based eye feature detection unit or an infrared light-based eye feature detection unit.

[0040] By analyzing the accuracy and / or reliability of the first sensor signals from the eye feature detection unit and the second sensor signals from the image-based sensor, it can be verified whether the second reliability criteria are met. In particular, the control unit must determine whether there are any discernible deviations between the first and second sensor signals capturing the eye features.

[0041] Any deviation of one or more of the characteristics relevant for generating the first control signals, namely the position and / or rotation and / or translation of at least one eye of the passenger, must be within the permissible deviations or tolerances of the second reliability criteria.

[0042] Should one or more of the tolerances be exceeded, the system uses the self-diagnostic data from the image-based sensor and, in case of poor reliability (e.g., due to poor image quality), sends a signal indicating this poor reliability to the passenger's wearable device or mobile phone. Generating initial control signals based on the initial sensor signals remains possible.

[0043] If tolerances are not met despite the image-based sensor's sufficient reliability, a safety mode is initiated because the control unit assumes a fault in the eye feature detection unit. This safety mode, when activated, at a minimum, results in the vehicle stopping.

[0044] Continuous sensor analysis ensures safe vehicle operation and allows for immediate action to minimize damage in the event of sensor malfunctions. Safety is further enhanced by the third sensor, the vitality data sensor. If the signals from this third sensor fall outside a defined range of vitality parameters, the second reliability criteria are not met.

[0045] In one embodiment, the first sensor is a speech input sensor attached to a head support mechanism, a wearable device, or the vehicle to enable the passenger to make voice inputs. The speech input sensor is configured to output first sensor signals, wherein the control unit receives the first sensor signals and uses them to determine the first control signals for controlling the vehicle.

[0046] The control unit is designed to use an implementation of neural networks, preferably recurrent neural networks (RNNs), such as an LSTM or a GRU, or convolutional neural networks (CNNs), to process the initial sensor signals and determine the first control signals. Particularly for passengers with limited speech abilities, the self-learning neural networks allow speech recognition to be adapted to the passenger's voice. Control commands given via voice input can, on the one hand, result in initial control signals for starting or stopping the vehicle. On the other hand, voice input can be used to operate a menu system, which can be displayed to the passenger by the wearable device via a head-up display.This allows further commands to be issued, for example to open doors, order elevators, press buttons, operate switches or other external applications.

[0047] The control unit is further configured to determine the first control signals using not only the first sensor signals from the speech input sensor, but also second sensor signals from a second sensor, preferably an image-based sensor. The second sensor is directed towards the passenger's head and is configured to detect features of the position and / or rotation and / or translation of one or both eyes and to output the second sensor signals.

[0048] The third sensor, a vital signs sensor, is positioned on or between a body part to record the passenger's vital signs, particularly heart rate, and to output third-party sensor signals. The control unit receives and processes these third-party sensor signals. If the third-party sensor signals fall outside the specified range of vital signs, the second reliability criteria are not met, and the system enters safety mode.

[0049] A fourth sensor, an eye feature detection unit attached to or integrated into the wearable, is used. This unit is configured like the second sensor to detect at least one eye of the passenger and / or its features, in particular its position, orientation, rotation, and / or translation, and to output fourth sensor signals. The control unit receives these fourth control signals and uses them to determine the first control signals for controlling the vehicle. The eye feature detection unit can be an RGB image-based or an infrared light-based eye feature detection unit. Consequently, in this embodiment, the first, second, and fourth sensor signals are used to determine the first control signals.

[0050] This results in a redundant system, whereby the use of an infrared-based eye feature detection unit leads to diverse redundancy in eye feature detection by the second and fourth sensors. In this embodiment, the safety mode is activated if the first or second reliability criteria are not met.

[0051] The control unit can determine basic data from the second sensor signals. This basic data can include geometric dimensions and / or size ratios and / or facial features and / or gestures, and the first reliability criteria are determined using this basic data.

[0052] These geometrically or mathematically expressed properties of the basic data characterize the facial expressions perceived by humans and allow inferences to be drawn about the passenger's emotional state during vehicle operation. On the one hand, the first reliability criteria are intended to be met if the passenger is highly likely to be in a state of happiness, relaxation, or a similar state associated with a positive mood. On the other hand, the first reliability criteria are intended to be not met if the passenger is highly likely to be in a state of concern, surprise, anxiety, or a similar state associated with a negative mood.

[0053] It is also possible that, upon detecting a positive mood, feedback occurs within the control unit, initiating a recording mode. At least the passenger's head movements, captured by the second, image-based sensor, are recorded and used to train a machine learning algorithm. This algorithm will later recognize the passenger's learned gestures and interpret them as control commands. Such a gesture could, for example, be a rapid head shake intended to trigger the vehicle's stop. This form of gesture recognition makes the vehicle experience more personalized for the passenger.

[0054] Fulfillment of the first reliability criteria is verified by implementing machine learning, in particular classification, preferably using a support vector machine (SVM), or by implementing a neural network, in particular a convolutional neural network (CNN). At least when using CNNs, the control unit also evaluates the probabilities of the recognized facial expressions to determine whether the first reliability criteria are met.

[0055] Both machine learning methods and neural network implementations are based on image-based, labeled training data, which also includes images of faces and facial features. This training data can also contain images of the passenger's face. This allows the algorithm for capturing facial expressions to be adapted to the specific passenger.

[0056] According to the invention, image-based analysis can also detect redness, enlargement or constriction of the eyes, including the pupils, as well as the opening or closing of the eyes or mouth and lips, including inferable characteristics such as shortness of breath, increased pulse, or increasing, dangerous exertion of the person. The detection of such physical conditions also means that the first reliability criteria are not met.

[0057] In this embodiment, the second reliability criteria are based on accuracy and / or reliability analysis of the first, second, and fourth sensor signals. In particular, an accuracy comparison is performed between the detected eye or iris features of the second, image-based sensor and those of the fourth sensor, the eye feature detection unit.

[0058] Based on the accuracy and / or reliability analysis of the second sensor signals of the image-based sensor and the fourth sensor signals of the eye feature detection unit, the control unit can check whether the deviation of one or more of the features location and / or position and / or rotation and / or translation of the passenger's eye is within one or more tolerances.

[0059] Should one or more of the tolerances be exceeded, the system uses the self-diagnostic data from the image-based sensor and, in case of poor reliability (e.g., due to poor image quality), sends a signal indicating this poor reliability to the passenger via their wearable device or mobile phone. Generating control commands based on the fourth sensor signals remains possible.

[0060] If, when one or more of the tolerances are exceeded, low reliability of the second, image-based sensor cannot be determined, the safety mode is activated and the vehicle is at least brought to a stop. The reason for this is that, if the second sensor is sufficiently reliable, the control unit assumes a fault in the eye-tracking unit.

[0061] If the first and second reliability criteria are not violated, feature-based vehicle control can be initiated by assuming an initial state, such as fixating on a point in the center of the eyes. A sequence of gestures, such as blinking three times or certain predefined eye movement patterns, can also initiate feature-based control. The first voice input sensor can also be used to detect a start command.

[0062] The vehicle is then controlled by matching a gaze direction to the desired direction of movement. It can be stopped by detecting a predefined gesture, such as closing the eyes or blinking, with the second or fourth sensor. The vehicle can also be stopped by voice command using the first sensor, the voice input sensor.

[0063] Continuous sensor analysis ensures safe vehicle operation and allows for immediate action to minimize damage in the event of sensor malfunctions. Safety is further enhanced by the use of a third sensor, the vital signs sensor. If the signals from this third sensor fall outside a defined vital signs parameter range, the second reliability criteria are not met, and the system enters safe mode.

[0064] In one embodiment, instead of the fourth sensor, the eye feature detection unit, a fifth sensor, a brain-control unit interface or input device, is arranged on and / or in the passenger's head. This device is configured to output fifth sensor signals, which the control unit receives and uses to determine the first control signals for controlling the vehicle. The brain-control unit interface serves as the primary data source for generating these first control signals. Furthermore, at least the first reliability criteria are determined using the image-based sensor, and if these criteria are not met, the safety mode is initiated.

[0065] Predefined control commands are assigned to the fifth control signals in the control unit, so that a passenger's thought detected by the fifth sensor signals is assigned to a desired control command. Furthermore, abstract thought patterns of the passenger detected by the fifth sensor signals can be provided to a machine learning algorithm in the control unit. The learning algorithm learns from the provided data and determines the first control signals based on recognized control commands from the passenger.

[0066] However, the signals can also be used to determine whether the passenger has panicked, for example, because he or she is in a dangerous physical condition.

[0067] By combining it with the first voice input sensor, the second image-based sensor, and the third sensor of this embodiment, the vitality data sensor, dangerous situations can be classified more easily, and driving the vehicle becomes safer.

[0068] In one embodiment, all previously used sensors are combined to provide an even safer and more reliable solution. The first sensor, the inertial navigation unit attached to or integrated into the wearable, is used to detect the orientation, position, rotation, and / or translation of the passenger's head and transmit initial sensor signals to the control unit. These signals are then used to determine the first control signals for controlling the vehicle. The second, image-based sensor can detect the passenger's head and transmit further sensor signals to the control unit. These signals are then used to determine the first reliability criteria and to generate the first control signals for controlling the vehicle.

[0069] A third sensor, the vital signs sensor, transmits third-party sensor signals to the control unit. These signals reflect the passenger's vital signs, particularly heart rate. If these third-party sensor signals fall outside a defined range of vital signs, the second reliability criteria are not met.

[0070] A fourth sensor is an eye feature detection unit, attached to or in the wearable, and configured to detect one or both eyes of the passenger, in particular their position and / or rotation and / or translation, and to output fourth sensor signals, wherein the control unit receives the fourth sensor signals and uses them to determine the first control signals for controlling the vehicle.

[0071] Based on the accuracy and / or reliability analysis of the first sensor signals of the inertial navigation unit and the second sensor signals of the image-based sensor, the control unit can check whether the deviation of one or more of the features location and / or position and / or rotation and / or translation of the passenger's head is within one or more tolerances.

[0072] Based on the accuracy and / or reliability analysis of the second sensor signals of the image-based sensor and the fourth sensor signals of the eye feature detection unit, the control unit can verify whether the deviation of one or more of the features location and / or position and / or rotation and / or translation of the eye or eyes of the passenger is within one or more tolerances.

[0073] The fifth sensor, specifically the brain-control unit interface or input device, is located on and / or inside the passenger's head and is configured to output fifth-sensor signals. The control unit receives these fifth-sensor signals and uses them to determine the first-level control signals for controlling the vehicle. The fifth-sensor signals may include control commands from the passenger to start and stop the vehicle.

[0074] The sixth sensor used is the voice input sensor, which can be, in particular, a head-holding mechanism voice input sensor, a wearable voice input sensor, or a vehicle voice input sensor, and can be configured to output sixth sensor signals. The control unit receives these sixth sensor signals and uses them to determine the first control signals for controlling the vehicle. Passenger commands to start or stop the vehicle can be transmitted to the control unit via the sixth sensor signals from the voice input sensor.

[0075] If one or more of the tolerances are not met, the second reliability criteria are not fulfilled. If the first or second reliability criteria are not fulfilled, the device enters safety mode.

[0076] If the reliability criteria are met, the control unit can transmit initial control signals for the vehicle's movement to the vehicle actuators. The fourth sensor signals, as part of the initial control signals, define the direction of travel, while the first sensor signals define the desired speed in that direction.

[0077] In a further embodiment, a first environment-sensing sensor assembly, in particular an ultrasonic assembly and / or a LIDAR sensor assembly and / or an image-based sensor assembly and / or a RADAR sensor assembly, is attached to the vehicle for sensing the environment and is configured to output first sensor assembly signals, wherein the control unit receives first sensor assembly signals and uses them to determine the first control signals.

[0078] The environment can also be captured by an image-based rear sensor and / or a wearable image sensor, and the corresponding sensor signals can be transmitted to the control unit. Using these signals, the control unit can determine the initial control signals.

[0079] The control unit can evaluate the captured environmental information and activate safety mode when approaching obstacles, slopes, inclines, or similar features with a critical risk.

[0080] In one embodiment, the wearable and / or a mobile phone is designed and arranged to output seventh sensor signals, in particular passenger destination input sensor signals, wherein the control unit has transmitted previously available destination input selection data to the wearable and / or the mobile phone for passenger destination input.

[0081] The target selection data is generated by the control unit evaluating the sensor signals of at least the first environment-sensing sensor assembly. The targets can, in principle, be any elements whose contour can be detected by a radar and / or lidar and / or ultrasound and / or image-based sensor assembly and which are located within the passenger's field of vision.

[0082] Image-based sensor data can be used to train convolutional neural networks (CNNs). The image-based training dataset required for training one or more CNNs comprises a variety of different, labeled image data captured both indoors and outdoors by various cameras under different lighting conditions. This allows the CNNs to recognize a variety of objects detected by the environment-sensing sensor assembly and provide them to the passenger as target input selection data.

[0083] Destination input selection data is projected into the passenger's field of vision, similar to a head-up display, when the wearable device is used. Mobile phones allow for screen display. The control unit receives the seventh sensor signals and uses them to determine the first control signals.

[0084] The control unit's task is to calculate a trajectory to the destination, provide suitable initial control signals, and, during the journey to the destination, to make adjustments to the trajectory based on the real-time data from the environment-sensing sensors, for example due to the changing environment, and to provide adapted initial control signals.

[0085] When using LiDAR or radar sensor assemblies, three-dimensional, environment-imaging maps are generated. Machine learning can then assign image features from captured images of the environment to the three-dimensional point clouds of these maps. This makes it possible to generate a three-dimensional trajectory to the target object in the environment. The vehicle moves along this trajectory by sending the generated initial control signals to the vehicle's actuators to minimize the distance to the target location or object. When the minimum distance to the target is reached, the vehicle stops.

[0086] It is also possible to project destination selection data onto the passenger's head-up display in the wearable device in real time. This is achieved by arranging the wearable's image sensor and / or at least one, preferably image-based, environment-sensing sensor assembly to align itself with the passenger's gaze direction and transmit initial sensor assembly signals to the control unit. Using these initial sensor assembly signals, the control unit determines destination selection data and transmits it to the wearable for projection onto the head-up display. This preferably occurs via an LTE or 5G connection.

[0087] This allows the passenger to select a target object directly using real-time images or video, and seventh sensor signals, specifically passenger target input sensor signals, are transmitted from the wearable to the control unit. The control unit can then use these seventh sensor signals to determine the initial control signals.

[0088] The second, third, and fifth sensor signals are used by the control unit to provide feedback on whether the vehicle is on the correct path to the destination from the passenger's perspective. The passenger's facial expression, heart rate, and thoughts are monitored, and the system activates a safety mode if the sensor signals detect a critical physical condition. The automated navigation to a selected target can also be interrupted or terminated at any time by the passenger through manual override.

[0089] In one embodiment, an eighth sensor, in particular a position sensor, for example a GPS sensor or a sensor for locating within buildings, is arranged and configured to output eighth sensor signals, wherein the control unit receives the eighth sensor signals and uses them to determine the first control signals.

[0090] The position sensor allows a map to be displayed showing the current position of the vehicle or passenger via the wearable's head-up projection or on their mobile phone. Using a map in conjunction with the displayed destination selection data allows for the selection of destinations or objects that are no longer detectable by ambient sensors. Destination selection can be made by moving a virtual mouse pointer using eye tracking in the wearable's head-up projection or by making a selection on the mobile phone.

[0091] Based on the passenger's destination input, the seventh passenger destination sensor signals are provided to the control unit. Using the map and the eighth sensor signals from the position sensor, a trajectory to the destination or object can be determined, and corresponding initial control signals can be provided.

[0092] The control unit is therefore tasked with calculating a trajectory to the destination, providing suitable initial control signals, and, during the journey to the destination, making adjustments to the trajectory based on real-time data from the environmental sensors, for example, due to changing circumstances, and providing adjusted initial control signals. Upon reaching the destination or target object, the vehicle stops.

[0093] The second, third, and fifth sensor signals can be used by the control unit to provide feedback on whether the vehicle is on the right track to its destination from the passenger's perspective. This involves monitoring the passenger's facial expression, heart rate, and thoughts, with the system activating safety mode if the sensor signals indicate a critical physical condition of the passenger.

[0094] The automated control of a selected target object can also be interrupted or terminated at any time by manual override by the passenger.

[0095] In one embodiment, the device comprises a remote communication device, for example an LTE or 5G communication device or a mobile telephony communication device, which is arranged on the vehicle and configured to perform signal exchange with the control unit and external mobile network operator devices. The wearable is also connected to the control unit via an LTE or 5G connection, enabling (video) communication or the sharing of images or videos captured by the wearable image sensor with others via personal messages or social networks.

[0096] Furthermore, in the event of an initiated safety mode, for example due to a detected critical physical condition of the passenger, at least one emergency contact can be contacted automatically.

[0097] In one embodiment, the device also includes vehicle dynamics sensors, in particular wheel speed sensors and / or inertial navigation units with integrated accelerometers and / or yaw rate sensors and / or compasses, arranged on the vehicle and configured to output ninth sensor signals, wherein the control unit receives the ninth sensor signals and uses them to determine the first control signals. This sensor system allows, in particular, the control of vehicle dynamics by feeding actual vehicle dynamics sensor values ​​back to the control unit, in order to enable a more precise implementation of the desired control commands compared to a conventional controller.

[0098] In another embodiment, the image-based front sensor for detecting the passenger's face is replaced by an image-based vehicle sensor, which transmits image-based sensor signals to the control unit and uses these signals to determine the first control signals and assess whether the first reliability criteria are met.

[0099] Further advantageous embodiments are described in the dependent claims.

[0100] The invention will below be described with regard to further features and advantages using exemplary embodiments, which are explained in more detail with reference to the figures.

[0101] This shows: Fig. 1 a flowchart of the device according to an exemplary embodiment; Fig. 2 a vehicle with wearable, inertial navigation unit, image-based sensor, control unit and vehicle actuators; Fig. 3 the vehicle according to Fig. 2 with vital data sensor; Fig. 4 the vehicle according to Fig. 3 without inertial navigation unit, with eye feature detection unit; Fig. 5 the vehicle according to Fig. 4 , with wearable voice input sensor; Fig. 6 the vehicle according to Fig. 3 , without wearable, without inertial navigation unit, with brain control unit interface or input device, without wearable voice input sensor, with vehicle voice input sensor; Fig. 7 the vehicle according to Fig. 6 , with wearable, with inertial navigation unit, with eye feature detection unit, with voice input sensor; Fig. 8 the vehicle according to Fig. 7 , with wearable image sensor, with environment-sensing sensor assembly; and Fig. 9 the vehicle according to Fig. 8 with position sensor, with remote communication device, with wearable voice input sensor, without vehicle voice input sensor.

[0102] In the following description and drawings, the same reference symbols are used for identical and equivalent parts.

[0103] Fig. 1 shows a flowchart of a device 80 for navigating a wheelchair 200 (cf. Fig. 2 ).

[0104] In one embodiment, the device 80 comprises two sensors. The first sensor is an inertial navigation unit 11, which is attached to a portable computer system, wearable 10, on the head of a wheelchair user of a wheelchair 200 (e.g. Fig. 2 ) is attached. For example, it is part of Google Glass, which is worn by the wheelchair user. The inertial navigation unit 11 detects the position and rotation of the wheelchair user's head and outputs this data as the first sensor signals S1. The second sensor is an image-based front sensor 21 of a mobile phone 20 ( Fig. 2 ), wherein the mobile phone 20 is fixed to a component of the wheelchair 200 such that the front sensor 21 detects the head of the wheelchair user. Image data generated by the front sensor 21 are output by the mobile phone 20 as second sensor signals S2.

[0105] A control unit 100 is communicatively connected to Google Glass, e.g., via Bluetooth, and receives the first sensor signals S1. The second sensor signals S2 can be transmitted to the control unit 100, specifically to a sensor signal receiver unit 101, via a USB connection. In a control signal detection unit 102 of the control unit 100, initial control signals ST1 for controlling the wheelchair 200 are generated based on the initial sensor signals S1. Based on the second sensor signals S2, a reliability criteria check unit 103 determines whether the initial control signals ST1 meet the initial reliability criteria.

[0106] A safety mode is entered when the reliability criteria test unit 103 determines that the first control signals ST1 do not meet the first reliability criteria.

[0107] The reliability criteria test unit 103 determines basic data based on the second sensor signals S2. This basic data includes geometric dimensions, proportions, and facial features. These geometrically and mathematically expressed properties of the basic data characterize a person's perceptible facial expressions and thus allow for the classification of the feelings or emotions experienced by the wheelchair user while operating the wheelchair 200.

[0108] Devices for classifying a person's emotions based on recognized geometric dimensions, proportions, and facial features are known. For example, IN 00554CH2014 A describes a method and a device with which a change in facial expression compared to previously defined, neutral expressions can be detected and assigned to one or more emotions.

[0109] The teaching method of IN 00554CH2014 A uses so-called constrained local modeling (CLM) methods to recognize faces and then determine the dimensions, proportions, and shapes of features such as the eyes, nose, mouth, or chin. Furthermore, a support vector machine (SVM) can be trained on a large number of previously labeled image data of faces to derive actions, such as closing an eye or opening a mouth, from the previously recognized geometric facial features. Finally, a statistical procedure (discriminative power concept) is used to determine the probability of a person exhibiting a specific emotion by evaluating the likelihood of an action if a particular emotion is present, minus the probability of that action if the emotion is absent.

[0110] Specifically, this could mean that if upward-turned corners of the mouth are detected as opposed to a known, neutral mouth position, a state of happiness in the wheelchair user is determined, since none of the other known emotions are characterized by upward-turned corners of the mouth. The IN 00554CH2014 A specifies the emotions anger, fear, happiness, surprise, disgust, and sadness as recognizable. Furthermore, the probabilities for each of these emotions are provided.

[0111] This embodiment offers the possibility and thus the advantage of individually adjusting the image-based facial feature recognition to the specific wheelchair user, provided the support vector machine (SVM) is also trained with labeled image data of the wheelchair user's facial muscle movements. Furthermore, the captured facial features underlying the emotion classification can be evaluated in real time. For this purpose, these features are superimposed on the image data sequence (video sequence) of the second sensor signals S2 of the image-based front sensor 21.

[0112] Alternatively, according to a further embodiment, Fig. 1 The classification of the wheelchair user's emotions is based on an implementation of a convolutional neural network (CNN), which is applied to the second sensor signals S2 in the reliability criteria test unit 103. This artificial intelligence approach requires pre-labeled image data—that is, image data with assigned emotions of a person—to classify the wheelchair user's expressed emotion. The geometric features of the face, including its dimensions and proportions, are not visible during training due to the abstract representation of the trained weights of the neural network.

[0113] The accuracy rate, and thus the quality of emotion recognition, is assessed using a test dataset. The convolutional neural network is designed such that each emotion to be recognized by the wheelchair user is assigned a probability value. The use of this neural network has the advantage that, similar to human perception of emotions in faces, it does not consider one or a few facial features, but rather determines the wheelchair user's emotion through the interplay of all image data transmitted by the second sensor signals (S2). This allows person-specific facial features, such as laugh lines, to be implicitly considered for emotion recognition through the CNN training data, even without being explicitly characterized in the training data.

[0114] Regardless of the choice of method for classifying emotions, SVM or CNN, the first reliability criteria should mean that the first reliability criteria are met if there is a high probability of a state of happiness, relaxation or a similar state associated with positive emotion in the wheelchair user.

[0115] However, a detected positive mood does not only mean that the initial reliability criteria are met. In addition, the control signal detection unit 102 also records the position and rotation data of the wheelchair user's head over a time series, which is then fed into a machine learning algorithm. This algorithm can capture and process the individual movement sequences of the wheelchair user's head as depicted in the data, thereby adapting the wheelchair controls to the individual user's needs.

[0116] It is possible to determine the maximum forward / backward and left / right head tilts, as well as the corresponding rotational speeds, for each wheelchair user when operating the wheelchair. For example, even a slight head tilt in one direction can result in maximum wheelchair movement in the desired direction for a wheelchair user with some remaining but severely limited neck mobility. Furthermore, recording the rotational speeds via the head angles makes it possible to avoid misinterpreting head tilts caused by illness-related tremors or nervousness as steering commands. This also allows for a smoother, more individualized wheelchair driving experience for users with involuntary or uncontrolled jerky head movements.

[0117] Regression algorithms are suitable machine learning algorithms for this task. This means that a control characteristic desired by the wheelchair user does not have to be based solely on a single, recorded position-time or rotational velocity-position function. Instead, several functions recorded at different times can be used to model an averaged, individual control characteristic of the wheelchair.

[0118] Once the initial reliability criteria are met, the wheelchair is controlled by tilting the head upwards, which, depending on the preset parameters, moves the wheelchair forwards or backwards. Tilting the head to the left or right causes the wheelchair to move left or right, respectively. To stop the wheelchair, the head is returned to a previously defined normal or neutral position.

[0119] The first reliability criteria are defined as a failure to meet the first reliability criteria if there is a high probability of the wheelchair user being unable to control the wheelchair 200, such as in a state of worry, surprise, anxiety, dissatisfaction, or a similar state associated with negative emotions. For example, downturned corners of the mouth signal dissatisfaction. Surprise and anxiety can be identified by wide-open eyes or mouth. An inability to control the wheelchair 200 is inferred from closing the eyes for a period longer than the blink rate. The training data provided to the convolutional neural network (CNN) takes this mapping of facial expressions to emotions and states into account.The same applies to the exemplary implementation for the classification of emotions by the Support Vector Machine (SVM).

[0120] The safety mode is a restricted operating range of the device 80, in which second control signals ST2 are output by the reliability criteria test unit 103. If the wheelchair 200 cannot be controlled, for example if the wheelchair user has closed their eyes, the second control signals ST2 will cause the wheelchair to brake and stop.

[0121] If the probability of the wheelchair user experiencing surprise, anxiety, or worry increases, the reliability criteria test unit 103 uses the first sensor signals S1 of the inertial navigation unit 11 to determine the second control signals ST2. If the determined position or rotation of the wheelchair user's head is within an impermissible range, the second control signals ST2 cause the wheelchair 200 to stop. Conversely, if the position and rotation of the wheelchair user's head are within an permissible range, only the maximum speed of the wheelchair 200 is reduced, as the reliability criteria test unit 103 assumes that this speed was previously inappropriately high for the wheelchair user.

[0122] The second control signals ST2 are different from the first control signals ST1, but both can be output by the device 80 via the control signal output unit 104. Only the control signals that are valid at any given time are output.

[0123] This arrangement ensures that any recognizable negative emotions of the wheelchair user or an inability to control the wheelchair are detected and the safety mode is activated. Furthermore, this prevents further physical harm to third parties and the wheelchair user.

[0124] If the first reliability criteria are met, the first control signals ST1 are determined by the control signal determination unit 102 of the wheelchair 200 using the first sensor signals S1. Depending on the preset, tilting the wheelchair user's head upwards means moving forwards or backwards, while tilting the head to the side means orienting the wheelchair 200 to that side.

[0125] In another embodiment of the wheelchair 200 in Fig. 2 The control unit 100 is further configured to determine the first control signals ST1 using not only the first sensor signals S1 of the inertial navigation unit 11 attached to the wearable 10, but also the second sensor signals of the image-based front sensor 21. Safety mode is activated if either the first or the second reliability criteria are not met. Safety mode is also activated if both reliability criteria are not met. In this embodiment, the second reliability criteria are based on accuracy and reliability analysis of the first sensor signals S1 and the second sensor signals S2 by the reliability criteria test unit 103.

[0126] Accuracy and reliability analysis is performed using the first sensor signals S1 and the second sensor signals S2, which include feature values ​​for the position and rotation of the wheelchair user's head. The reliability of the inertial navigation unit 11 and the image-based front sensor 21 is determined by comparing these signals. If the position and rotation features used to control the wheelchair 200 are within predefined tolerances, the feature values ​​of the inertial navigation unit 11 are used to determine the first control signals ST1. The wheelchair 200 is controlled as described in the preceding embodiment.

[0127] Should one or more of the tolerances be exceeded, the system again uses the second sensor signals S2 of the image-based front sensor 21. In case of poor accuracy and reliability, for example due to poor image quality, a signal indicating poor reliability is transmitted to the wheelchair user on their mobile phone 20 and displayed there. The poor image quality can be detected because the algorithm used to recognize the wheelchair user's emotion assigns only low probabilities to each emotion, e.g., happiness or dissatisfaction. The generation of initial control signals ST1 based on the initial sensor signals S1 remains possible.

[0128] If the front sensor 21 is sufficiently reliable, the safety mode is initiated because the reliability criteria test unit 103 assumes a fault in the inertial navigation unit 11. The wheelchair 200 is stopped.

[0129] The continuous comparison of the characteristic values ​​of the inertial navigation unit 11 and the image-based front sensor 21 enables safe operation of the wheelchair 200 and immediately initiates measures to minimize the consequences of damage in the event of sensor errors.

[0130] Fig. 3 Figure 1 shows a further embodiment in which a vitality data sensor 40 is arranged on the wheelchair user's wrist as a third sensor to record the wheelchair user's heart rate and output third sensor signals. The control unit 100 receives the third sensor signals, transmitted, for example, via a Bluetooth connection, by means of the sensor signal receiver unit 101. If the third sensor signals indicate that the heart rate exceeds a permissible range, the reliability criteria test unit 103 also checks whether the wheelchair user is highly likely to be experiencing anxiety or fear. If this is also the case, the second reliability criteria are deemed not to have been met. Consequently, the safety mode is activated. The second control signals ST2 cause the wheelchair 200 to stop.

[0131] Fig. 4 Figure 1 shows another embodiment, which is similar to the preceding example. However, instead of the inertial navigation unit 11, this embodiment has an eye feature detection unit 12 as the first sensor. This unit is attached to the wearable 10 and connected to it, for example via cabling, and is designed to detect the position and translation of the iris of one eye of the wheelchair user and to output first sensor signals S1. The sensor signal receiver unit 101 receives the first sensor signals S1 after forwarding them by the wearable 10 and uses them to determine the first control signals ST1 for controlling the wheelchair 200.

[0132] The reliability criteria test unit 103 uses an accuracy and reliability analysis of the first sensor signals S1 of the eye feature detection unit 12 and the second sensor signals S2 of the image-based front sensor 21 to verify whether the second reliability criteria are met. The reliability criteria test unit 103 determines whether there are deviations in the feature values ​​of position and translation of the iris of an eye.

[0133] Deviations of these characteristic values ​​must lie within predefined, permissible tolerances of the second reliability criteria.

[0134] Should one or more tolerances be exceeded, the sensor signals of the image-based front sensor 21 are used, as in the preceding embodiment. In case of poor accuracy and reliability, for example due to poor image quality, a signal indicating poor reliability is projected into the wheelchair user's field of vision via the wearable 10. The information is also transmitted to the user's mobile phone 20 by the control signal output unit 104, for example via a Bluetooth connection. The generation of initial control signals ST1 based on the initial sensor signals S1 is also enabled.

[0135] If the tolerance is not maintained, i.e., if the permitted positional deviation of the iris position determined by both sensors is not adhered to despite sufficient reliability of the image-based front sensor 21, the safety mode is initiated because the reliability criteria test unit 103 assumes errors in the eye feature detection unit 12. The safety mode leads to the wheelchair 200 being stopped.

[0136] If the iris positions and translations of one eye, determined by the second and fourth sensors, are within tolerance, the control signal detection unit 102 determines a final position value by calibrating the eye movement range. This involves determining the maximum eye movement to the left, right, up, and down to enable eye-position-proportional control of the wheelchair 200. If a reliable iris position cannot be determined, the second reliability criteria are not met, the safety mode is activated, and the wheelchair 200 is stopped.

[0137] If the first and second reliability criteria are not violated, the feature-based control of the wheelchair 200 is initiated by assuming an initial state, e.g., by fixating on a point in the center of the eyes. A sequence of gestures, such as blinking or a predefined eye movement pattern, can also trigger the feature-based control via the control signal detection unit 102. An eye movement pattern is a sequence of gaze directions defined and executed by the wheelchair user.

[0138] The wheelchair 200 is controlled by associating a gaze direction with the desired direction of movement. The direction of movement of the wheelchair 200 is controlled by looking to the left, resulting in a leftward rotation of the wheelchair 200, and looking to the right, resulting in a rightward rotation. The wheelchair 200 is stopped by detecting a predefined gesture, such as closing the eyes or blinking several times, by the first sensor, the eye feature detection unit 12, or by the second sensor, the image-based front sensor 21.

[0139] Continuous sensor analysis ensures the safe operation of the wheelchair 200 and immediately initiates measures to minimize damage in the event of sensor malfunctions. Safety is further enhanced by the vitality data sensor 40. If the third sensor signals fall outside a defined vitality parameter range, the second reliability criteria are not met, and the system activates safety mode.

[0140] Fig. 5 Figure 1 shows a further, modified embodiment in which the first sensor is a wearable speech input sensor 61, which is attached to the wearable 10 and connected to it, for example, by a cable. The wearable speech input sensor 61 is configured to output first sensor signals S1, wherein the sensor signal receiving unit 101 receives the first sensor signals S1 after forwarding them by the wearable 10 and uses them to determine the first control signals ST1 for controlling the vehicle 200 in the control signal detection unit 102.

[0141] The control signal detection unit 102 is designed to use an implementation of a recurrent neural network (RNN) when processing the first sensor signals S1 to determine the first control signals ST1. These algorithms for analyzing complete, sequential speech sensor signals allow for more time-efficient training compared to fully connected neural networks (FCNs) and improved recognition of the wheelchair user's speech input with comparable computational effort.

[0142] Especially for wheelchair users with limited speech abilities, the self-learning neural networks allow speech recognition to be adapted to the wheelchair user's voice. Control commands given via voice input can, on the one hand, lead to the (starting) or stopping of the wheelchair 200 via the first control signals ST1. On the other hand, voice input is used to operate a menu system, which is displayed to the wheelchair user by the wearable 10 via a standard head-up display. This allows commands to be given, for example, to open doors, order elevators, press buttons, and operate switches.

[0143] The control unit 100 is further configured to determine the first control signals ST1 using not only the first sensor signals S1 of the wearable speech input sensor 61, but also the second sensor signals S2 of the image-based front sensor 21. The second sensor is directed towards the head of the wheelchair user and is configured to detect the position and translation of both eyes or their irises and to output the second sensor signals S2.

[0144] As a third sensor, the vitality data sensor 40 is arranged on the wrist of the wheelchair user to determine the heart rate, wherein this sensor and the control unit 100 are designed to provide the functionalities according to the preceding embodiments.

[0145] The fourth sensor is the eye feature detection unit 12, attached to the wearable 10. Like the second sensor, the image-based front sensor 21, the eye feature detection unit 12 is designed to detect the position or translation of the wheelchair user's eye or iris and to output fourth sensor signals. The sensor signal receiver 101 receives these fourth control signals, and the control signal determination unit 102 uses them to determine the first control signals ST1 for controlling the vehicle 200. The eye feature detection unit 12 is an infrared light-based eye feature detection unit.

[0146] Consequently, in this embodiment, the first sensor signals S1, the second sensor signals S2, and the fourth sensor signals are used to determine the first control signals ST1. This results in a redundant system, with the use of the infrared light-based eye feature detection unit 12 leading to diverse redundancy in eye feature detection by the second and fourth sensors.

[0147] In this embodiment, the safety mode is activated when the reliability criteria test unit 103 determines that the first or the second reliability criteria are not met. The fulfillment of the first reliability criteria is determined in the reliability criteria test unit 103 at least using the basic data of the second sensor, the image-based front sensor 21, as described in the preceding embodiments.

[0148] The evaluation of the second reliability criteria, as in previous embodiments, includes a reliability and accuracy analysis. Based on the second sensor signals S2 of the image-based front sensor 21 and the fourth sensor signals of the eye feature detection unit 12, the reliability criteria test unit 103 checks whether the deviation of the position or translation of an eye of the wheelchair user is within the respective permissible tolerance.

[0149] Should one or more of the tolerances be exceeded, the sensor signals of the image-based front sensor 21 are used, as described in the preceding embodiments. In case of poor accuracy and reliability, for example due to poor image quality, a signal indicating poor reliability is transmitted to the wheelchair user via the wearable 10 or to their mobile phone 20. The generation of first control signals ST1 using the fourth sensor signals by the control signal detection unit 102 is still possible, and the first control signals ST1 are output by the control signal output unit 104.

[0150] If, when one or more of the tolerances are exceeded, no low reliability of the image-based front sensor 21 is detected, the safety mode is activated and the wheelchair 200 is stopped. This is because, if the image-based front sensor 21 has sufficient reliability, the reliability criteria test unit 103 assumes that there is a fault in the eye feature detection unit 12.

[0151] The wheelchair is controlled as in the previous embodiment. In addition, the first sensor, the wearable speech input sensor 61, can be used to detect a start command.

[0152] The wheelchair 200 can also be stopped by voice input using the first sensor, the wearable voice input sensor 61.

[0153] Continuous sensor analysis ensures the safe operation of the wheelchair 200 and promptly initiates measures to minimize damage in the event of sensor malfunctions. Safety is further enhanced by the use of the third sensor, the vitality data sensor 40. If the third sensor signals fall outside a defined vitality parameter range, the reliability criteria test unit 103 determines that the second reliability criteria are not met. The safety mode is then activated according to the conditions described in the preceding embodiments.

[0154] Fig. 6 Figure 1 shows a further embodiment of the wheelchair 200 according to the invention, wherein, instead of the fourth sensor, the eye feature detection unit 12, a fifth sensor, a brain-control unit interface or input device 50, is arranged on the head of the wheelchair user. This device is configured to output fifth sensor signals, wherein the sensor signal receiving unit 101 receives the fifth sensor signals, e.g., via a Bluetooth connection, and uses them to determine the first control signals ST1 for controlling the wheelchair 200 in the control signal detection unit 102.

[0155] A brain-control interface is known from US patent 2017 / 0042439A1. In this device, an electrode array is arranged around a person's head to measure and subsequently process brainwaves. This allows for the determination of the person's mental state or emotion.

[0156] The brain-control unit interface, or input device 50, is used as the primary control data source for generating the first control signals ST1. Furthermore, the first reliability criteria are determined using the image-based front sensor 21, and if these criteria are not met, the safety mode is activated. If an elevated heart rate is additionally detected by the vitality data sensor 40, the reliability criteria test unit 103 assumes a critical physical condition of the wheelchair user, and the second reliability criteria are also not met. The wheelchair 200 is stopped.

[0157] In the control signal detection unit 102, the fifth sensor signals are recorded, so that a thought of the wheelchair user recorded by the fifth sensor is assigned to a desired first control signal ST1.

[0158] The brain-control unit interface or input device knows 50 predefined control commands, which are output as fifth sensor signals when the wheelchair user thinks of them.

[0159] However, the fifth sensor signals do not only include predefined control commands.

[0160] As part of the fifth sensor signals, the abstract thought patterns of the wheelchair user, captured by the fifth sensor, are provided to a machine learning algorithm in the control signal detection unit 102. The learning algorithm learns from the data so that, upon recognition of a previously known thought pattern that can be associated with a control command from the wheelchair user, the first control signals ST1 are determined based on the recognized control command.

[0161] The machine learning algorithm is trained by recording the wheelchair user's emotion in response to a detected initial control signal ST1, as described in the preceding examples, and also providing this information to the control signal detection unit 102. The wheelchair user's heart rate is also provided. This allows the machine learning algorithm to learn which control commands, determined based on thought patterns, the wheelchair user considers appropriate. This enables the thought-based control of the wheelchair 200 to be individualized for each wheelchair user.

[0162] However, determining the emotions of the wheelchair user is not limited to training the machine learning algorithm in practice.

[0163] Furthermore, the fifth sensor signals are also used to determine whether the wheelchair user has panicked, for example, because they are in a dangerous physical condition. By combining these signals with the first sensor, the vehicle voice input sensor 60, the second sensor, the image-based front sensor 21, and the third sensor in this embodiment, the vitality data sensor 40, dangerous situations can be classified more easily, making wheelchair driving 200 safer.

[0164] Fig. 7 Figure 1 shows a further embodiment of the wheelchair 200, in which all previously used sensors are combined to provide an even safer and more reliable solution. The inertial navigation unit 11, attached to the wearable 10, is used as the first sensor to detect the position and rotation of the wheelchair user's head and to transmit initial sensor signals S1 to the sensor signal receiver 101. Using these signals, the control signal detection unit 102 determines the initial control signals ST1 for controlling the wheelchair 200. The second sensor, the image-based front sensor 21, detects the wheelchair user's head and its features and transmits additional sensor signals S2 to the control unit receiver 101. Using these signals, the reliability criteria test unit 103, as known from previous embodiments, checks whether the initial reliability criteria are met.In addition, the first control signals ST1 for controlling the wheelchair 200 are determined using the second sensor signals S2.

[0165] The third sensor, the vitality data sensor 40, is arranged and configured as in previous embodiments. The fourth sensor, an eye feature detection unit 12, is attached to the wearable 10 and is configured, as known from previous embodiments, to detect the position and translation of an eye or iris of the wheelchair user. The fourth sensor signals are provided to the control unit 100, which then uses them to determine the first control signals ST1.

[0166] Based on the accuracy and reliability analysis of the first sensor signals S1 of the inertial navigation unit 11 and the second sensor signals S2 of the second sensor of the image-based front sensor 21, the reliability criteria test unit 103 checks, as known from previous embodiments, whether the deviations of the position or rotation of the wheelchair user's head are within the respective permissible tolerances.

[0167] In the accuracy and reliability analysis of the second sensor signals S2 of the second sensor, the image-based front sensor 21 and the fourth sensor signals of the fourth sensor, the eye feature detection unit 12, the reliability criteria test unit 103 checks, as known from previous embodiments, whether the deviation of the features position or translation of an eye of the wheelchair user is within the respective tolerances.

[0168] The fifth sensor, the brain-control unit interface or input device 50, is arranged and configured as known from the preceding embodiments. Using the transmitted fifth sensor signals, the control signal detection unit 102 determines control commands from the wheelchair user to start and stop the wheelchair 200.

[0169] The sixth sensor used is the vehicle voice input sensor 60, which is configured to output sixth sensor signals. The control unit 100 receives these sixth sensor signals and uses them to determine the first control signals ST1 for controlling the wheelchair 200. Commands to start or stop the wheelchair 200 can be transmitted to the sensor signal receiver unit 101 via the vehicle voice input sensor 60 using these sixth sensor signals.

[0170] If one or more of the tolerances are not met, the second reliability criteria are not fulfilled. If the first or second reliability criteria are not fulfilled, the device 80 enters safety mode. If they are fulfilled, first control signals ST1 for moving the wheelchair 200 can be transmitted from the control signal output unit 104 to the vehicle actuators 70.

[0171] Using the fourth sensor signals, the desired direction of travel is determined by the control signal detection unit 102, and using the first sensor signals, the desired speed in that direction is determined. From this, the first control signals ST1 are derived and output by the control signal output unit 104 to the vehicle actuators 70.

[0172] Looking forward by the wheelchair user causes the wheelchair to move forward, while looking to the side causes it to move 200 degrees in that direction. Depending on the preset, a downward tilting motion increases or decreases the wheelchair speed, while an upward tilting motion has the opposite effect.

[0173] This embodiment also has the advantage over the previous ones that the safety of the wheelchair user is increased, since he has to look in the direction of travel.

[0174] In Fig. 8 Another embodiment is shown, which additionally includes a first environment-sensing sensor assembly 90, a LIDAR sensor assembly. This is attached to the wheelchair 200 for sensing the environment and is configured to output first sensor assembly signals, wherein the control signal acquisition unit 101 receives first sensor assembly signals via a high-speed data line, for example an Ethernet connection, and uses them to determine the first control signals ST1.

[0175] The environment is also perceived by an image-based rear sensor 22 of the mobile phone 20 and a wearable image sensor 13, so that the first sensor signals ST1 are determined by the control signal detection unit 102 using these. The wearable image sensor 13 is connected to the wearable 10, for example, via data lines, so that the wearable 10 transmits its sensor signals to the sensor signal receiving unit 102.

[0176] The captured environmental information is evaluated by the control signal detection unit 102 and the reliability criteria testing unit 103, and the safety mode is activated when the wheelchair approaches obstacles, slopes, inclines, or similar features critically. This causes the wheelchair 200 to stop.

[0177] In a further embodiment, the wearable 10 and the mobile phone 20 are configured to output seventh sensor signals, in particular passenger destination input sensor signals, wherein the control unit 100 has transmitted previously available destination input selection data to the wearable 10 and the mobile phone 20 for passenger destination input.

[0178] The target input selection data is generated by evaluating the sensor signals of the first environment-sensing sensor assembly 90 by the control unit 100. The targets can, in principle, be any elements whose contours can be detected by a LIDAR sensor assembly 90 in combination with the other environment-sensing sensors 13, 22 and which are located in the vicinity of the wheelchair user.

[0179] Destination input selection data is projected into the wheelchair user's field of vision by the wearable 10, similar to a head-up display. The information is also displayed on the screen of the mobile phone 20. The sensor signal receiver 101 receives the seventh sensor signals and uses them to determine the first control signals ST1.

[0180] The control signal determination unit 102 has the task of calculating a trajectory to the destination, providing suitable first control signals ST1, and making adjustments to the trajectory during the journey to the destination based on the real-time data of the environment-sensing sensors 13, 22 and the LIDAR sensor assembly 90, for example due to the changing environment, and providing adapted first control signals ST1.

[0181] When using the LIDAR sensor assembly 90, three-dimensional, environment-imaging maps are generated. Image features from captured images of the environment are assigned to the three-dimensional point clouds of these maps using machine learning. This makes it possible to generate a three-dimensional trajectory to the target object in the environment. The wheelchair 200 moves along this trajectory by providing the generated initial control signals ST1 to the vehicle actuators 70 to minimize the distance to the target location or object. When the minimum distance to the target is reached, the wheelchair 200 stops.

[0182] The wheelchair user's head-up display (WIDAR) 10 projects destination selection data in real time. This is achieved by the wearable's image sensor 13 transmitting its sensor signals to the control unit 100. The environment-sensing LIDAR sensor assembly 90 is positioned to align itself with the wheelchair user's gaze direction and transmit initial sensor assembly signals to the control unit 100. Using these signals, the control unit determines the destination selection data and provides it to the wearable 10 for projection into the head-up display.

[0183] This allows the wheelchair user to select a target object directly based on real-time images or video, and seventh sensor signals, in particular passenger target input sensor signals, are transmitted from the wearable 10 to the sensor signal receiver unit 101. The control signal detection unit 102 uses the seventh sensor signals to determine the first control signals ST1.

[0184] The second, third, and fifth sensor signals are used by the control signal detection unit 102 to provide feedback on whether the wheelchair 200 is on the correct path to the destination from the wheelchair user's perspective. The wheelchair user's facial expression, vital signs, and thoughts are monitored, and the system activates a safety mode if the sensor signals indicate a critical physical condition. Furthermore, the automated control of a selected target object can be interrupted or terminated at any time by manual override from the wheelchair user, for example, based on sensor signals from the inertial navigation unit 11, the eye feature detection unit 12, or the image-based front sensor 21.

[0185] Fig. 9 Figure 1 shows a further embodiment of the invention. In this embodiment, an eighth sensor, a GPS position sensor 106, is arranged on the wheelchair 200 and is designed to output eight sensor signals, wherein the sensor signal receiving unit 101 receives the eighth sensor signals and uses them to determine the first control signals ST1.

[0186] The position sensor 106 allows a map showing the current position of the wheelchair 200 or the wheelchair user to be displayed via the head-up projection of the wearable 10 and on their mobile phone 20. Using a map in conjunction with the displayed target input selection data allows the selection of destinations or objects that cannot be detected by the environment sensors 13, 22, or the environment sensor assembly 90. A target is selected by moving a virtual mouse pointer using eye control in the head-up projection of the wearable 10 or by making a selection on the mobile phone 20.

[0187] Based on the wheelchair user's destination input, the seventh passenger destination input sensor signals are provided to the control signal determination unit 102. Using the map and the eighth sensor signals from the GPS-based position sensor 106, a trajectory to the destination location or object is determined, and corresponding first control signals ST1 are provided.

[0188] Thus, the control signal detection unit 102 has the task of calculating a trajectory to the destination, providing suitable initial control signals ST1, and, during the journey to the destination, making adjustments to the trajectory based on the real-time data from the environment-sensing sensors 13, 22 and the environment-sensing sensor assembly 90, for example due to the changing environment, and providing adjusted initial control signals ST1. After reaching the destination or target object, the wheelchair 200 stops.

[0189] The second, third, and fifth sensor signals are used by the control signal detection unit 102, as described in the preceding embodiment, to provide feedback on whether the wheelchair 200 is on the correct path to the destination from the wheelchair user's perspective. If the first or second reliability criteria, known from previous embodiments, are violated, the safety mode is activated.

[0190] The automated control of a selected target object can be interrupted or terminated at any time by manual override by the wheelchair user.

[0191] Fig. 9 Figure 1 shows a further embodiment in which a remote communication device 107 for transmitting LTE or 5G signals is also arranged on the wheelchair 200. This device exchanges signals with the control unit 100 and external mobile network operator devices. The wearable 10 is also connected to the control unit 100 via an LTE or 5G connection, enabling (video) communication or allowing images or videos captured by the wearable image sensor 13 to be shared with others via personal messages or social networks.

[0192] Furthermore, if a safety mode is activated, for example due to a detected critical physical condition of the wheelchair user, at least one emergency contact can be contacted automatically. This involves sending an automated text message and initiating a video call.

[0193] The Fig. 9 Furthermore, another embodiment of the wheelchair 200 can be taken from the figure, in which a further inertial navigation unit with integrated accelerometers, gyroscopes, and compass, arranged on the wheelchair 200 and designed to output ninth sensor signals, is used as the vehicle dynamics sensor 71. The sensor signal receiver unit 101 receives the ninth sensor signals and uses them to determine the first control signals ST1. This sensor system allows, in particular, the control of the vehicle dynamics by feeding the vehicle dynamics sensor values ​​back to the control signal determination unit 102, in order to enable a more precise implementation of the desired control commands compared to a purely control-based system.

[0194] In another one, which Fig. 9In the removable embodiment, the image-based front sensor 22 for detecting the face of the wheelchair user is replaced by an image-based vehicle sensor 14, which transmits image-based sensor signals to the control signal receiver unit 101 via a connection, e.g. USB connection, and using these, the control signal determination unit 102 determines the first control signals ST1, and the reliability criteria test unit 103 evaluates the fulfillment of the first reliability criteria.

[0195] It should be noted at this point that all parts described above, in particular the individual embodiments and examples, are to be regarded individually – even without additional features described in the respective context, even if these have not been explicitly identified as optional features in the respective context, e.g., by using: in particular, preferably, for example, e.g., parentheses, etc. – and in combination or any sub-combination as independent embodiments or further developments of the invention as defined in particular in the introduction and the claims. Deviations from this are possible. Specifically, it should be noted that the word "in particular" or parentheses do not denote features that are mandatory in the respective context. Reference symbol list

[0196] 10 Wearable computer system 11 Inertial navigation unit 12 Eye feature detection unit 13 Wearable image sensor 14 Image-based vehicle sensor 20 Mobile phone 21 Image-based front sensor 22 Image-based rear sensor 40 Wearable vitality data sensor 41 Ear vitality data sensor 50 Brain control unit interface or input device 60 Vehicle voice input sensor 61 Wearable voice input sensor 70 Vehicle actuators 71 Vehicle dynamics sensors 80 Device for navigating and / or guiding and / or stabilizing a vehicle 90 Environment sensing sensor assembly 100 Control unit 101 Sensor signal receiving unit 102 Control signal detection unit 103 Reliability criteria test unit 104 Control signal output unit 106 Position sensor 107 Remote communication device 200 Vehicle, in particular wheelchair ST1 First control signals ST2 Second control signals S1 First sensor signals S2 Second sensor signals

Claims

1. A device for navigating and / or guiding the path and / or stabilizing a vehicle (200), the device (80) comprising: - at least one first sensor, in particular inertial navigation unit (11), which is designed and arranged to detect at least one first body part of a passenger of the vehicle (200) and to output first sensor signals (S1); - at least one second sensor, in particular image-based sensor (21, 22), which is designed and arranged to detect at least the first body part of the passenger and / or their features, and to output second sensor signals (S2); and - control unit (100) designed to a) receive the first and second sensor signals (S1, S2); b) determine first control signals (ST1) for controlling the vehicle (200) based at least on the first sensor signals (S1); c) determine whether the first control signals (ST1) meet at least first reliability criteria based at least on the second sensor signals (S2); d) adopt a safety mode if the control unit (100) determines that the first control signals (ST1) do not meet at least the first reliability criteria.

2. The device of claim 1, characterized in that the first body part is the passenger's head, wherein a) the control unit (100) determines basic data on the basis of geometric dimensions and / or size ratios and / or features of the face and / or gestures detected by at least the second sensor; b) the first reliability criteria are determined using the basic data.

3. The device according to claim 1 or 2, characterized in that the control unit (100) is further designed to determine the first control signals (ST1) using at least the second sensor signals; and e) to adopt the safety mode when the control unit (100) determines that first control signals (ST1) do not meet at least the first and / or second reliability criteria, wherein the first and / or second reliability criteria are determined by accuracy and / or reliability analysis of at least the first sensor signals (S1) and the second sensor signals (S2).

4. The device according to one of the preceding claims, characterized in that the control unit determines the fulfillment of the first reliability criteria by implementing machine learning.

5. The device according to claim 3, characterized in that a vitality data sensor is arranged on a body part or between two body parts and detects vitality parameters as characterizing features and outputs third sensor signals, wherein the control unit receives the third sensor signals and, if a vitality parameter value range is not met by the third sensor signals, the second reliability criteria are not met.

6. The device according to one of the preceding claims, characterized in that at least one first sensor assembly (90) is mounted on the vehicle (200) for sensing the environment and is designed to output first sensor assembly signals, wherein the control unit (100) receives first sensor assembly signals and determines the first control signals (ST1) using them.

7. The device according to one of the preceding claims, characterized in that a remote communication device (107) is arranged on the vehicle and designed to perform signal exchange with the control unit and mobile telephony provider devices for performing video communication and / or remote health condition monitoring of the passenger.

8. The device according to one of the preceding claims, characterized in that the first sensor is mounted on or in a wearable (10) and is designed to detect the first body part or its features of the passenger of the vehicle (200) and to output first sensor signals, wherein, by using said signals, the first control signals (ST1) for controlling the vehicle (200) are determined.

9. The device according to claim 8, characterized in that a fourth sensor, in particular an eye feature detection unit (12) is mounted on or in the wearable (10) and is designed to detect second body part or its features of the passenger of the vehicle (200) and to output fourth sensor signals, wherein, by using said signals, the first control signals (ST1) for controlling the vehicle are determined.

10. The device according to one of the preceding claims, characterized in that a first sensor, in particular a brain control unit interface or input device (50), is arranged on and / or in the head of the passenger and is designed to output fifth sensor signals, wherein, by using said signals, the first control signals (ST1) for controlling the vehicle (200) are determined.

11. The device according to one of the preceding claims, in particular according to claim 9, characterized in that a wearable speech input sensor (61) or a vehicle speech input sensor (60) is designed to output sixth sensor signals, wherein the control unit (100) receives the sixth sensor signals and, by using said signals, determines the first control signals (ST1) for controlling the vehicle (200).

12. The device according to one of claims 9-11, characterized in that the wearable (10) and / or a smartphone (20) is designed and arranged to output seventh sensor signals, in particular passenger destination input sensor signals, wherein the control unit (100) transmits available destination input selection data to the wearable (10) and / or the smartphone (20) for passenger destination input as well as receives the seventh sensor signals and, by using said signals, determines the first control signals (ST1).

13. The device according to one of the preceding claims, characterized in that an eighth sensor, in particular a position sensor (106) is arranged and designed to output eighth sensor signals, wherein the control unit (100) receives eighth sensor signals and, by using said signals, determines the first control signals (ST1) and / or vehicle dynamics sensors (71) are arranged on the vehicle (200) and are designed to output ninth sensor signals, wherein the control unit (100) receives the ninth sensor signals and, by using said signals, determines the first control signals (ST1).

14. A vehicle comprising a - device (80) according to one of the preceding claims; - vehicle actuators (70) arranged to drive the vehicle (200) and designed to receive and process the control signals of the device (80).

15. A method for controlling a device for navigating and / or guiding the path and / or stabilizing a vehicle (200), in particular according to claim 14, comprising the steps of a) receiving at least first sensor signals (S1), in particular inertial navigation-based sensor signals, which describe at least a first body part of a passenger of the vehicle; and receiving second sensor signals (S1, S2) from an image-based front sensor (21), which is designed and arranged to describe at least the first body part or its characterizing features of the passenger of the vehicle (200); b) determining first control signals (ST1) for controlling the vehicle (200) based at least on the first sensor signals (S1); c) determining whether the first control signals (ST1) meet at least first reliability criteria based on at least the second sensor signals (S2); d) adopting a safety mode when the control unit (100) determines that the control signals (ST1) do not meet at least the first reliability criteria.