A method for generating a control signal for a driving dynamic functional device, a computer program product, a non-transitory computer-readable storage

Abstract

A method for generating a control signal for a driving dynamic functional device 14 of an at least in part automatically operated motor vehicle 10 by an assistance system 12 of the motor vehicle. Comp

Description

[0001] The present invention relates to the field of automobiles. More specifically, the present invention relates to a method for generating a control signal for a driving dynamic functional device of an at least in part automatically operated motor vehicle by an assistance system of the motor vehicle. Furthermore, the present invention relates to a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as to a corresponding assistance system. BACKGROUND INFORMATION

[0002] Automated steering, also known as in particular autonomous or self-driving motor vehicles, is a rapidly developing technology that has the potential to revolutionize the way of travelling. Autonomous motor vehicles use a combination of sensors, cameras, and for example artificial intelligence algorithms to navigate roads and make decisions without human intervention. Also known from the state of the art is the semi-automatic driving which is also known as partial automation or as a driver assistance system, which refers to the vehicle that assist drivers with certain tasks but still require human intervention or supervision. These systems are designed to improve safety, comfort, and convenience by automatic specific driving functions, such as steering, acceleration, and braking. Some examples of the semi-automatic driving features are, for example, the adaptive cruise control. This system uses sensors or cameras to maintain a safe following distance from the vehicle in front, automatically adjusting the speed to match the traffic flow. Lane departure warning and lane keeping assistance are also known. These systems alert drivers when they are drifting out of their lane and can provide steering assistai keep the vehicle on track. Parking assist systems use sensors and cameras to driver into a parking space, automatically controlling the steering, acceleration, braking. Traffic jam assist system combines adaptive cruise control with lane keeping assist to enable the vehicle to follow the car in front in heavy traffic, reducing driver stress. Blind spot detection system uses sensors to monitor the blind spots on either side of the vehicle, altering the driver when another vehicle is present and providing visual or audio warnings. Semi-automatic driving features are becoming increasingly common in modern motor vehicles: from budget cars to luxury models. They offer many benefits for divers, including reduced fatigue, improved safety, and increased convenience. However, it is important to note that these systems still require human supervision and intervention and should not be relied upon as a substitute for attentive driving. Driver must remain alert in control of their vehicles at all times and be prepared to take over if necessary.

[0003] Therefore, there is a need in the art to provide semi-automatic driving in a more comfortable way. SUMMARY OF THE INVENTION

[0004] It is an object of the invention to provide a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding assistance system, by which a more comfortable driving of an at least in part automatically motor vehicle can be provided.

[0005] This object is solved by a method, a corresponding computer program product, a non-transitory computer-readable storage medium, as well as a corresponding assistance system according to the independent claims. Advantageous embodiments are presented in the dependent claims.

[0006] One aspect of the invention relates to a method for generating a control signal for a driving dynamic functional device of an at least in part automatically operated motor vehicle by an assistance system of the motor vehicle. The eyes of an operator of the motor vehicle are captured by a capturing device of the assistance system. A current gaze of the operator is determined depending on the captured eyes by an electronic computing device of the assistance system. The control signal for controlling the driving dynamic functional device is generated depending on the determined gaze and the electronic computing device.

[0007] Therefore, the comfort for the operator / driver of the motor vehicle in ai part automatically operated motor vehicle can be raised. In particular, for exam depending on the gaze a steering of the motor vehicle can be provided. For ex; lane change of the motor vehicle can be automatically provided depending on the gaze.

[0008] Therefore, for example, two sets of camera sensors, for example, near-infrared sensors are installed on a dashboard of the motor vehicle to detect the operator’s eyeball movement. The operator can take certain actions to activate these functions such as blinking eyes three times continuously. When the eye-control-vehicle function is on, the driver can look in a direction to control, for example, the motor vehicle in a left turn, a right turn or drive straight. Looking at a rear camera view may move the vehicle backwards.

[0009] According to an embodiment, a control signal for a steering device of the motor vehicle as the driving dynamic functional device for adapting a direction of the motor vehicle is generated.

[0010] In another embodiment, a current situation in which the motor vehicle is situated is taken into consideration for generating the control signal.

[0011] In another embodiment, a current speed of the motor vehicle is taken into consideration for generating the control signal.

[0012] In another embodiment, a maximum steering angle depending on the current speed is determined and depending on the maximum steering angle the control signal is generated.

[0013] In another embodiment, a current traffic situation is taken into consideration for generating the control signal.

[0014] In another embodiment, at least two driving direction signs are projected on a windshield of the motor vehicle, and the determined gaze is assigned to one of the at least two driving direction signs and the control signal is generated depending on the driving direction sign the gaze assigned to.

[0015] In particular, the present invention is a computer-implemented method. Therefore, another aspect of the invention relates to computer program product comprising program code means for performing a method according to the pre aspect.

[0016] A still further aspect of the invention relates to a non-transitory computer-readable storage medium comprising at least the computer program product according to the preceding aspect.

[0017] Furthermore, the present invention relates to an assistance system for generating a control signal for a driving dynamic functional device of an at least in part automatically operated motor vehicle, comprising at least one capturing device and one electronic computing device, wherein the assistance system is configured for performing a method according to the preceding aspect. In particular, the method is performed by the assistance system.

[0018] Furthermore, the present invention relates to an at least in part automatically operated motor vehicle comprising at least the assistance system according to the preceding aspect.

[0019] Advantageous embodiments of the method are to be regarded as advantageous embodiments of the computer program product, the non-transitory computer-readable storage medium, the assistance system, as well as the motor vehicle. The assistance system comprises means for performing the method.

[0001] A computing unit / electronic computing device may in particular be understood as a data processing device, which comprises processing circuitry. The computing unit can therefore in particular process data to perform computing operations. This may also include operations to perform indexed accesses to a data structure, for example a look-up table, LUT.

[0002] In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or । said units.

[0003] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0004] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0020] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0022] The drawings show in:

[0023] Fig. 1 a schematic side view according to an embodiment of a motor vehicle comprising an embodiment of an assistance system;

[0024] Fig. 2 a schematic flow chart according to an embodiment of the meth

[0025] Fig. 3 another schematic flow chart according to an embodiment of th<

[0026] In the figures the same elements or elements having the same function are indicated by the same reference signs. DETAILED DESCRIPTION

[0027] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0028] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0029] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0030] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0031] Fig. 1 shows a schematic side view according to an embodiment of a i vehicle 10. The motor vehicle 10 is at least in part automatically operated. Thei motor vehicle 10 comprises an assistance system 12 as well as one driving dyr functional device 14, for example, a steering device. Furthermore, the assistance system 12 comprises at least one electronic computing device 16 and one capturing device 18 for capturing a gaze 20 of an operator 22 of the motor vehicle 10.

[0032] Fig. 2 shows a schematic flow chart according to an embodiment of the method. In particular, Fig. 2 shows a method for generating a control signal for the driving dynamic functional device 14 of the at least in part automatically operated motor vehicle 10 by the assistance system 12. The eyes of the operator 22 are captured by the capturing device 22 of the assistance system 12. The current gaze 20 is determined depending on the captured eyes by the electronic computing device 16. The control signal for controlling the driving dynamic functional device 14 is generated depending on the determined gaze 20 by the electronic computing device 16.

[0033] In particular, Fig. 2 shows in a first step S1 that the method starts. In a second step S2, an eyeball tracking is provided. In a third step S3, it is checked if the assistance system 12 is calibrated. If not, a fourth step S4 is performed wherein the calibration and motion management mitigation is provided. After the third step S3, for example, if the assistance system 12 is calibrated, or after the calibration in the fourth step S4, a fifth step S5 is provided, wherein safety arbitration is provided. This safety arbitration is further shown in the Fig. 3. After the fifth step S5, a sixth step S6 is performed, wherein an actuation trigger is provided. This is further shown in the Fig. 4. After the sixth step S6, a seventh step S7 is performed, wherein the control signal is generated. After the seventh step S7 an eighth step S8 is performed, wherein in the eighth step S8 the method is ended.

[0034] In particular, for example, two sets of camera sensors and near-field infrared light illumination modules are installed on a dashboard of the motor vehicle 10 to detect the driver’s eyeball movement. A head-up system may handle display driving direction signs on the windshield. The driver may take certain actions to activate this function such as blinking eye three times continuously. When the eye control function is on, the operator 22 can look at the head-up sign to control the vehicle left turn, right turn, or drive straight. Looking at the rendering camera, a view on the head-up may move the motor vehicle 10 backwards.

[0035] Therefore, according to Fig. 2 a control signal for the steering device c motor vehicle 10 as the driving dynamic functional device 14 for adapting a dire the motor vehicle 10 is generated. Furthermore, a current situation in which the vehicle 10 is situated is taken into consideration for generating the control signal. For example, a current speed of the motor vehicle 10 is taken into consideration for generating the control signal. Furthermore, a maximum steering angle depending on the current speed is determined and depending on the maximum steering angle the control signal is generated. Furthermore, a current traffic situation is taken into consideration for generating the control signal.

[0036] For example, when the motor vehicle 10 is at higher speed, the maximum steering angle may be lower than when the motor vehicle 10 is at lower speed. Furthermore, if another motor vehicle behind and left and no space to do a lane shift, then the front angle for steering may be limited. For the eyeball tracking, for example a processor may control the near field illumination module to emit near field illumination. The light reflection is captured by a camera sensor. The processor may process the image and find out the vector. A calculation to determine the location of the point of gaze 20, for example, what the operator 22 is looking at, is provided with a great accuracy. In particular, the eye tracker take images for example every 0.833 ms and allow to determine eye movements of the amplitude less than 0.1 degree of visual angle.

[0037] For calibration of the assistance system 12, there may be two possibilities. For example, a stand still calibration may be provided wherein an eye tracker starts the calibration and each spot the operator 22 has to look at inside of the motor vehicle 10, for example on a mirror, on the windshield, at a rear view mirror is lighted up by a specific order .The operator 22 may stare at the light up spot by sequence. The eye tracker saves the gaze point coverage and non-volatile memory mapping. Another example may be the dynamic calibration. The eye tracker starts to calibrate and lively IMU data is used. The driver manually drives the motor vehicle 10 and a motion management mitigation is provided. The eye tracker generates the gaze 20. And the eye tracker saves the gaze point deviation in the non-volatile memory mapping.

[0038] In particular, since road surface situation is un-predictable and the operator’s eyeballs are not fixed, motion mitigation is hard to implement.

[0039] In particular, for the calibration and motion management, specific light used inside the motor vehicle 10. Furthermore, a head-up unit of the motor veh may also display spots on the windshield.

[0040] Fig. 3 shows another schematic flow chart according to an embodiment of the method. In particular, Fig. 3 shows the so-called safety arbitration. In a nineth step S9, automatic driving sensors such as cameras, radar, ultrasonic or lidar are used, and in a tenth step S10 automatic driving safety feature algorithms are used, for example, blind spot monitor, active emergency braking, emergency steering system or furthermore. In an eleventh step S11, safety accepted criteria are provided. For example, a front wheel angle limitation, front wheel angle rate limitation, IGN status, four wheel speed limitation, yaw / pitch / role rate limitation, the automatic driving controller status, an acceleration rate limitation may be determined. In a twelfth step S12, it is checked, if the eye control request meets the safety criteria. If not, a thirteenth step S13 is provided, wherein the eye control pauses. In a fourteenth step S14, for example, it is checked if the pause is performed for three seconds. If yes, a fifteenth step S15 is provided, wherein the method ends. If no, a sixteenth step S16 is provided, wherein an eyeball tracking is performed. In a seventeenth step S17, the gaze 20 is determined. In an eighteenth step S18, an eyecontrol vehicle actuation request is generated. Coming from the eighteenth step S18, the twelfth step S12 may be performed.

[0041] Fig. 4 shows another schematic flow chart according to an embodiment of the method, In particular, Fig. 4 shows the actuation trigger. In a nineteenth step S19, automatic driving functions are running. In a twentieth step S20, it is checked if the eye tracking sensors are healthy. If not, a forty-third step S43 is performed, in which the method ends. If there are healthy, a twenty-first step S21 is performed, wherein the safety arbitration is provided. After the safety arbitration, a twenty-second S22 is provided wherein it is checked if the eyes were blinking three times in order to activate the function. In a twenty-third step S23, the gaze 20 is determined.

[0042] The following description is now described for a gaze direction in the middle. This is provided with the branch 24. Furthermore, a branch 26 for a left gaze 20, a branch 28 for a right gaze 20, and a branch 30 for the reverse gaze 20 is shown. The following blocks for the mid gaze 20 are the same as for the left gaze 20, the right gaze 20 and the reverse gaze 20. Therefore, this is just shown schematically.

[0043] In a first block after determining the gaze direction it is checked in a tv step S24, the gaze 20 is in the same direction as in the previous cycle. If not, a accumulated value for -7 is provided. If yes, a twenty-fifth S25 is performed wk middle accumulator is provided and the value is added by 10. In particular, a cycle time of every 10 ms is performed. In a twenty-sixth step S26 it is checked if the accumulated value is higher than 300. If yes, it is checked in a twenty-seventh step S27, if the accumulated value is lower than 450. Coming from the twenty-sixth step S26, if the accumulated value is lower than 300, it is checked in a step S28 if the accumulated value is higher than 0. If not, a twenty-nineth step S29 is performed, wherein the value 0 is outputted. If yes, the thirtieth step S30 is performed, wherein the output of the value is provided. Coming from the twenty-seventh step S27, if the value is lower than 450, the thirtieth step S30 is performed. If the value is higher than 450, the value 450 is outputted in a thirty-first step S31. The steps S26 to S31 are provided in a block 34.

[0044] In particular, this mechanism is to control the accumulated value floating between 0 to 450, no overflow in the threshold of the accumulated value is also tunable. In particular, after beginning the accumulator is 0, once this process starts running, for example every ten milliseconds will add or minor certain value on the accumulator, in particular like kind of a water tank. For example, at 0 ms, the gaze point is in the middle, the mid accumulator is 0. After 10 ms, the gaze point is in the middle, the mid accumulator is 10, left / right / reverse accumulator is 0. After 50 ms, the gaze point is in the middle, and the mid accumulator is 50, left / right / reverse accumulator is 0. After 60 ms, the gaze point is right, the mid accumulator is 43 / drain7, the right accumulator is 10 left / reverse is 0. Compare all these accumulators every 10 ms and determine the one which has the largest value, then it is the driver’s intention.

[0045] In the thirty-second step S32, each values from the left gaze 20, the middle gaze 20, the right gaze 20 and the reverse gaze 20 are outputted. In a thirty-third step S33, the largest accumulated value is determined and matched to a direction. In a thirty-fourth step S34, a front wheel angle weight map is provided and the front angle request is generated. Coming from the thirty-third step S33, the accumulated value is provided and in a thirtysixth step S36 an array which has five individual accumulated values every cycle is provided. In a thirty-seventh step S37 a differential value of the array is calculated. In a thirty-eighth step S38, it is checked if the differential value is lower than 0. If yes, the thirty-nineth step S39 is performed, wherein the deceleration weight map is provided. After that, a fortieth step S40 is provided, wherein a vehicle deceleration reque generated. If the differential value is not lower than 0, a forty-first step S41 is p wherein the acceleration weight map is generated. After the forty-first step S41 second step S42 is provided wherein the vehicle acceleration request is provided. signs motor vehicle assistance system driving dynamic functional device electronic computing device capturing device gaze Operator branch branch branch branch block block steps of the method

Claims

1. A method for generating a control signal for a driving dynamic functional device (14) of an at least in part automatically operated motor vehicle (10) by an assistance system (12) of the motor vehicle (10), comprising the steps of:- capturing the eyes of an operator (22) of the motor vehicle (10) by a capturing device (18) of the assistance system (12);- determining a current gaze (20) of the operator (22) depending on the captured eyes by an electronic computing device (16) of the assistance system (12); and - generating the control signal for controlling the driving dynamic functional device (14) depending on the determined gaze (20) by the electronic computing device (16).

2. The method according to claim 1, characterized in thata control signal for a steering device of the motor vehicle (10) as the driving dynamic functional device (14) for adapting a direction of the motor vehicle (10) is generated.

3. The method according to claim 1 or 2, characterized in thata current situation in which the motor vehicle (10) is situated is taken into consideration for generating the control signal.

4. The method according to claim 3, characterized in thata current speed of the motor vehicle (10) is taken into consideration for generating the control signal.

5. The method according to claim 2 and 4, characterized in thata maximum steering angle depending on the current speed is determined and depending on the maximum steering angle the control signal is generated.

6. The method according to claim 3, characterized in thata current traffic situation is taken into consideration for generating the control signal.

7. The method according to any one of claims 1 to 6, characterized in thatat least two driving direction signs are projected on a windshield of the motor vehicle (10) and the determined gaze (20) is assigned to one of the at least two driving direction signs and the control signal is generated depending on the driving direction sign the gaze (20) assigned to.

8. A computer program product comprising program code means for performing a method according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium comprising at least the computer program product according to claim 8.

10. An assistance system (12) for generating a control signal for a driving dynamic functional device (14) of an at least in part automatically operated motor vehicle (10), comprising at least one capturing device (18) and one electronic computing device (16), wherein the assistance system (12) is configured for performing a method according to any one of claims 1 to 7.

Images