Controlling a function on board a motor vehicle
The electromyographic sensor-based control system allows for intuitive and complex control of motor vehicle functions by detecting hand postures and gestures, addressing the challenge of accessing and operating vehicle systems, thereby enhancing user experience and interaction.
Patent Information
- Application Number
- PCT/DE2024/100330
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2024-04-17
- Publication Date
- 2024-11-21
AI Technical Summary
Operating or controlling functions on board a motor vehicle can be difficult due to the variety of available elements and complex arrangements of seats, with many functions only accessible from the first row of seats and some reserved for the driver, limiting user interaction and comfort.
A control system using an electromyographic sensor to detect muscle stimulation and determine hand postures or gestures, allowing for composite inputs that can control vehicle functions, optionally combined with traditional controls or a menu-based interface, enabling intuitive and complex control by anyone on board.
Enables easy and reliable control of vehicle functions, improving user experience by allowing anyone on board to interact with the vehicle systems, enhancing comfort and immersion through intuitive and playful interaction.
Smart Images

Figure DE2024100330_21112024_PF_FP_ABST
Abstract
Description
[0001] Control of a function on board a motor vehicle
[0002] The present invention relates to the control of a motor vehicle. In particular, the invention relates to the control of a function of a motor vehicle.
[0003] A motor vehicle includes a number of systems or subsystems that can be controlled by a person on board. For example, an entertainment system may be provided that can display visual or audible information via one or more screens and / or one or more speakers. A communications system may allow interactive data exchange with a remote location. A ventilation system can heat, cool, or supply fresh air to different locations within the interior. Advanced or luxury motor vehicles may have a significant number of functions that can be controlled by a person.
[0004] A motor vehicle not only serves to transport its occupants to a predetermined destination, but should also provide a comfortable space for those on board. A positive experience on board a motor vehicle can be achieved if a person has the opportunity to control the vehicle's functions or use these functions, for example, to work on a task alone or together with another person during their journey.
[0005] Operating or controlling functions on board a motor vehicle can be difficult due to the large number of available elements, the layout of seats within the interior, or the assignment of functions to people. Often, many functions can only be controlled by people in the front row of seats, and some functions are reserved for the driver of the vehicle.
[0006] An object underlying the present invention is to provide an improved operating concept on board a motor vehicle. The invention solves this problem by means of the subject matter of the independent claims. Subclaims specify preferred embodiments. A proposed control system on board a motor vehicle comprises an electromyographic sensor attached to a person on board the motor vehicle for detecting stimulation of a muscle for moving a hand of the person; and a processing device. The processing device is configured to determine a posture or gesture of the hand; and to determine an input composed of multiple postures or gestures.
[0007] By combining multiple hand positions or hand gestures into a composite input, even complex control of a function on board a motor vehicle can be achieved simply and reliably. The input can be made by any person on board. The input can be given using one or both of the person's hands. Optionally, a composite input can also be determined from gestures or hand positions of several people. The composite inputs can, for example, enable improved communication between one person on board and another person. A learned sequence of operating inputs for achieving a predetermined result can be replaced by a combination of hand positions or gestures that is easier to learn or remember.
[0008] The control system can be designed as a standalone system or as an additional system for controlling the function. If necessary, traditional operation of the function is also possible, for example, using an assigned control element such as a switch, a lever, a button, or a touch-sensitive surface. Alternatively, the function can also be controlled using a menu-based user interface or a dynamically configurable multifunction input device.
[0009] The electromyographic sensor may comprise an electrode for application to the person's skin. More preferably, at least two electrodes are provided, between which an electrical voltage can be determined. Preferably, a plurality of electrodes is used, which can be applied to a predetermined area of the skin. Voltages, each indicative of stimulation of a muscle, can be determined between pairs of electrodes. An electrode may comprise a surface electrode that can rest loosely on the skin. In another embodiment, the sensor may be embedded in a device or a piece of clothing worn by the person. Operation of the control element can be easy and elegant for the person. Wearing the electromyographic sensor can be very comfortable. The person's skin does not have to be injured.
[0010] A normal stimulation leads to a perceptible movement, known as isotonic stimulation. However, the sensor can also detect muscle activation even if the activation is not accompanied by any movement. A change in activation may be so slight that it can be detected by the sensor, but trigger no or no noticeable movement of any part of the hand. A change in the force exerted against an object or the stimulation of two opposing muscles can also be detected without necessarily involving any movement. In this case, it can be referred to as isometric stimulation.
[0011] When a muscle is stimulated, it contracts and exerts a force on a joint or limb. Different types of contraction can be distinguished. An isometric ("equal measure") contraction or isometric muscle contraction occurs when a muscle only undergoes a change in tension but no change in length. In an isotonic ("equally tensioned") contraction, the muscle shortens without a change in force. This can happen, for example, when a weight or object is lifted a short distance very slowly. An auxotonic ("differently tensioned") contraction involves a change in both force and length. This is the most common type of contraction and accompanies a normal movement.
[0012] Based on a determination of the stimulation, different types of contraction can be distinguished. This makes it possible to determine not only the position or posture of a limb connected to the muscle, but also its tension. For example, the hand can grasp an object and hold the object with varying degrees of strength without any significant change in posture. Similarly, the hand can be pressed against an object, for example, the surface of an object. Two sections of the hand can be pressed against each other with different degrees of strength, for example two fingers, one of which can encompass the thumb. The person can also stimulate two muscles with different degrees of strength that counteract each other at the same limb or joint. For example, one muscle may comprise a flexor and the other an extensor of a finger of the hand.
[0013] Different types or intensities of stimulation can be considered when determining a posture or gesture. These parameters can be evaluated alone or in combination with, for example, a position, an orientation, or a nearby object. Additional sensor signals, such as those indicating a position, acceleration, or orientation, can also be considered. This allows selection from a wide range of parameters that are characteristic for determining a predetermined action or posture.
[0014] In particular, when many signals are collected, for example when many electrodes are used, when different types or strengths of stimulation or contraction of a muscle are recorded, when signals from another sensor or another scanning device are taken into account, or when a sequence of observations is to be evaluated, the recognition of the gesture or posture can be carried out using a learning technique.
[0015] For example, an artificial neural network (ANN) can be trained to recognize a predetermined gesture or posture based on a large number of such observations. The training can take place outside the vehicle. For training, different test subjects can perform a number of gestures or postures to be recognized, while the aforementioned observations are recorded. During training, the ANN can be configured, for example using back propagation, to recognize a posture or gesture in different people. Training usually requires a large number of measurements, each with associated gestures or postures to be recognized. These must be presented to the ANN very frequently in order to achieve good recognition results.
[0016] The required recognition performance of the ANN can depend on a controlled function. Thresholds can be selected for a false positive and a correct positive rate. If the first rate is exceeded and the second rate is simultaneously exceeded, training can be terminated, and the ANN can be used for an evaluation on board a motor vehicle as described herein. Optionally, the ANN can also be further trained while it is being used to recognize gestures or postures on board the motor vehicle. This allows the ANN's function to be adapted to a specific person.
[0017] A posture can be determined in particular by an orientation of the hand in space and / or the alignment of one or more fingers on the hand. A muscle can be activated differently in a predetermined hand posture, depending on how the hand is held. For example, if the hand hangs downwards, a different activation can affect a muscle for a finger than if the hand is stretched forwards. The sensor or the processing device is preferably configured to determine the posture of the hand independently of its orientation in space. A gesture is generally understood to be a transition from one posture to another. A gesture can relate to only one section of the hand, in particular a finger. In this way, several gestures can be evaluated simultaneously.
[0018] The electromyographic sensor is preferably designed for attachment to the subject's wrist or forearm. Muscles required to move a finger or the entire hand may be located above the wrist, allowing particularly insightful data about hand posture or movement to be collected in this area.
[0019] The electromyographic sensor may comprise a wristband for attachment to a person's wrist. The wristband may be included in an electronic device, such as a smartwatch or a fitness tracker. The sensor may also be incorporated into another object that the person can place near the hand. For example, the sensor may be provided in a finger ring or a glove. The person may wear the wristband on a left or right wrist. The effect of a gesture may depend on whether it is performed with a left or right hand. The selection of the left or right wrist may be based on the person's preference or where the person is seated in a motor vehicle.
[0020] The bracelet can be aesthetically pleasing so that it can be perceived as jewelry by the person or another person. In a further embodiment, the sensor can be integrated into a device worn on a bracelet, for example a smartwatch or a fitness tracker. The bracelet is preferably configured to hold one or more electrodes of the electromyographic sensor on the surface of the person's skin. For this purpose, the bracelet can be elastic or comprise an elastic element. Alternatively, an active device can be provided to press an electrode against the skin, for example a pneumatic or thermal actuator. The bracelet can be aesthetically pleasing and can also serve as jewelry. The bracelet can preferably be worn on a right or left extremity of the person. Several people on board the motor vehicle can each wear one or more electromyographic sensors.
[0021] An acceleration sensor may be attached to the wristband. The processing device may be configured to determine a position, orientation, gesture, or posture of the hand based on acceleration data from the acceleration sensor.
[0022] It should be noted that sensor values from multiple sensors can be combined. For example, the position of the wristband or hand can be determined based on camera data from a camera mounted on the vehicle and additionally based on acceleration data from an acceleration sensor mounted on the wristband or hand.
[0023] Preferably, the plurality of postures or gestures are in a predetermined temporal sequence relative to one another. In other words, the composite input can comprise a predetermined sequence of multiple postures or gestures. To detect an input, it can be determined whether the predetermined temporal sequence is adhered to. If, for example, a predetermined posture or gesture follows another posture or gesture with a time delay other than a predetermined one, the combination cannot be accepted as a predetermined composite input. However, the temporal correlation between elements of the input can support a predetermined tolerance. Furthermore, a dynamic evaluation is possible, so that an input can, for example, be executed quickly overall or slowly overall. If the speed at which postures or gestures are given changes, acceptance of the input can be refused.
[0024] Preferably, the processing device is configured to control a function associated with the composite input. The function can be executed by the motor vehicle or a system or subsystem included in the motor vehicle. Any actuators, devices, or equipment of the motor vehicle can be used for this purpose. An association between a composite input and a function can be predetermined or determined by the person.
[0025] It is particularly preferred that the function is composed of multiple controls. In other words, the function can comprise the control of multiple processes on board the motor vehicle. For example, a first function can be provided which prepares the motor vehicle for a summer journey. In this case, a convertible top can be opened, a heater or air conditioning system can warm the shoulder area of the driver and front passenger, interior ventilation can be turned off, and the playback volume of an entertainment system can be limited. Another exemplary function can prepare the motor vehicle for a winter journey. In this case, a heater can initially be directed specifically at a windshield, a heated windshield or rear window can be activated, exterior lighting can be switched on, and a drive program can be set for winter conditions.
[0026] The controls can also be arranged in a predetermined temporal sequence. In this case, a control can be synchronized with an element of the composite input. This allows for direct and complex interaction between the person and the vehicle. Control sequences that would otherwise be very complex can be achieved simply and playfully. The person's control of the vehicle can be enhanced by increased immersion. The interaction can be intense and intuitive, allowing the person to perceive the vehicle as a controllable part of their environment. The person or an observer may have the impression of thought-oriented or magical control of the vehicle.
[0027] In a further embodiment, a detection device is provided for determining the position of the person's hand. The processing device can be configured to additionally determine the composite input depending on the position. Thus, a predetermined posture or gesture made at different locations in the motor vehicle can have different meanings. For example, waving the hand in the area of a footwell can lead to the activation of a light, while the same movement in the area of a ventilation system (heating, ventilation, air conditioning, HVAC) can cause a changed diffusion of outgoing air. In a continuation of this idea, the pointing direction of the hand can also be evaluated. The hand posture or hand gesture can be evaluated taking the pointing direction into account, so that the repertoire of available elements of the composite input can be further expanded.The pointing direction can be evaluated with respect to the person, another person, the motor vehicle or an element on board the motor vehicle to which the person is pointing.
[0028] Similarly, a detection device for determining a person's line of sight can also be provided. The processing device can be configured to additionally determine the composite input depending on the line of sight.
[0029] It should be noted that both the pointing direction and the gaze axis can change during the composite input. A direction that changes during the provision of a composite input can be evaluated in any way. For example, the direction that was pointed or looked at while the respective element was being detected can apply to each element of the composite input. In another embodiment, a specific direction can apply to several or all elements of the composite input. In particular, the direction can be determined at the beginning or end of the input.
[0030] The processing device may be configured to determine the degree to which a composite input corresponds to a predetermined input. A degree of correspondence may be reported back to the person. The degree of correspondence may indicate a measure of the difference between a predetermined gesture and a gesture performed by the person.
[0031] The person can thus learn to provide the compound input in a clearer, faster, or more concise manner. Conversely, the degree of correspondence can be used to improve the control system's ability to recognize a posture, gesture, or compound input. Determining the degree of overlap or correspondence can, for example, be used for a game that requires the person's dexterity. The better the person performs a predetermined compound input, the more successfully they can participate in the game. The game can also function between several people on board the motor vehicle, with several players, for example, attempting to replicate a predetermined compound input or each providing a synchronized compound input, which is then compared with one another.
[0032] The control system may further comprise a display device for providing an indication regarding the compound input. In one embodiment, the determined degree of match can be provided in this way. The indication regarding the input can be provided before, during, or after. In one embodiment, the recognized compound input can be shown on the display device. The representation can be in the form of an animated hand. A viewing angle or playback speed can be changeable. Furthermore, an indication can be shown for an input indicating which function it is associated with. While a compound input is being provided, the elements already recognized can be analyzed, and the display device can show which functions can be controlled by a compound input that begin with the input already made.This allows the person to better navigate through a decision space that can be traversed when breaking down a complex input.
[0033] It is particularly preferred that the processing device is configured for machine learning. In particular, the processing device can learn a posture, a gesture, or, for example, a temporal relationship between different postures or gestures. For example, a predetermined composite input can always be slightly modified in the same way by a predetermined user, so that the difference can be learned by the processing device. The learning process can be person-specific, so that different people can perform different modifications, which the control system can each recognize without error.
[0034] According to a further aspect of the present invention, a motor vehicle comprises a control system as described herein. The motor vehicle preferably comprises a motorcycle, a passenger car, a truck, or a bus.
[0035] A method for controlling a motor vehicle comprises steps of detecting stimulation of a muscle for moving a hand of a person on board the motor vehicle by means of an electromyographic sensor attached to the person; determining a posture or gesture of the hand; and determining an input composed of a plurality of postures or gestures.
[0036] The method can be carried out by means of a control system described herein. The control system can comprise a preferably electronic processing device, which is more preferably embodied as a microcomputer or microcontroller. The method can be in the form of a computer program product with program code means. The computer program product can also be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to the control system, or vice versa. The invention will now be described in more detail with reference to the accompanying drawings, in which:
[0037] Figure 1 shows a control system on board a motor vehicle;
[0038] Figure 2 shows an example composite input;
[0039] Figure 3 temporal relations of elements of a composite input;
[0040] Figure 4 shows a mapping of composite inputs to controllers
[0041] Figure 5 shows an exemplary application of a composite input; Figure 6 shows another exemplary application of a composite input; and
[0042] Figure 7 illustrates a flowchart of a process.
[0043] Figure 1 shows a control system 100 on board a motor vehicle 105. The control system 100 is configured to facilitate or enable the operation of a predetermined element 115 for a person 110 on board the motor vehicle 105. The element 115 is illustrated here as an example as a ventilation grille and symbolizes a ventilation, heating, or air conditioning system. The person 110 can in particular comprise a driver or a passenger. As a passenger, the person 110 can be located in a first, second, or further row of seats in the motor vehicle 105. The element 115 can serve to control the motor vehicle 105, for example a drive, a lighting system, or a door system, or to perform an additional function, for example a navigation function, air conditioning, heating, or an entertainment system.
[0044] The control system 100 comprises a detection device 120, a processing device 125, an electromyographic sensor 130, and optionally an acceleration sensor 132. The sensors 130, 132 are mounted in the region of a hand 135 of a person 110 on board the motor vehicle 105. The detection device 120 is configured to determine a position of the hand 135 on board the motor vehicle 105. Preferably, an orientation of the hand 135 is also determined so that an overall pose is present. The detection device 120 can, for example, comprise a camera with which the hand 135 can be recognized and positioned. In another embodiment, the detection device 120 can position the electromyographic sensor 130 or an element connected thereto instead of the hand.
[0045] The electromyographic sensor 130 can be attached to a surface of the skin of the person 110. Using the electromyographic sensor 130, stimulation of a muscle that can move or hold the hand 135 can be determined. Even small changes in the state of tension can be resolved. The electromyographic sensor 130 can be configured to determine stimulation of multiple muscles, each of which can control different parts of the hand 135.
[0046] In this case, the sensor 130 is attached to a wristband 140, which the person 110 can wear on a right or left wrist. A person 110 can also wear two wristbands 140 on different wrists, allowing relative or combined postures or gestures to be determined. In the illustration of Figure 1, both persons 110 each wear a wristband 140; the function of the control system 100 is explained by way of example with reference to the person shown on the right.
[0047] The acceleration sensor 132 is configured to determine an acceleration of the hand 135 or a wristband 140 attached thereto. Based on the acceleration, a movement and / or position of the hand 135 or the wristband 140 can be determined.
[0048] The detection device 120 or another detection device can be configured to determine a gaze axis 145 of the person 110. For this purpose, a position and an orientation of the head of the person 110 can be determined. Furthermore, a position of one or both eyes of the person 110 can be determined. The gaze axis 145 follows a gaze of the person 110 and originates from their head. A link between a composite gesture and a vehicle feature or a vehicle function can preferably be created or modified with the support of a graphical user interface (GUI). For example, elements of a composite gesture can be selected in this way and arranged in a desired order. Furthermore, a feature or a function can be selected to which the composite gesture can be assigned.
[0049] In a corresponding manner, the detection device 120 or another detection device can be configured to determine a pointing direction 150 of the person 110. The person 110 can point with their hand 135 at an element 115 in the motor vehicle 105. In doing so, they typically move the hand 135 away from their body. The pointing direction 150 typically runs through the hand 135 and, in various embodiments, can originate in the region of an elbow joint, a shoulder, or the head of the person 110. In general, the pointing direction 150 runs through an element 115 to which the person 110 is pointing.
[0050] The processing device 125 is configured to receive information from the detection device 120 and the sensor 130 and optionally to determine the viewing axis 145 and / or the pointing direction 150. Based on this information, the processing device 125 can determine an element 115 near which the hand 135 of the person 110 is located, which element is intersected by the viewing axis 145 and / or the pointing direction 150. Furthermore, the processing device 125 can determine a posture or gesture of the hand 135 based on information from the sensor 130. The processing device 125 can receive corresponding sensor information via a preferably wireless communication device 155.
[0051] An output device 160 can be used to assist the selection of an element 115, the selection of a function of the element 115 and / or the control of this function by outputting feedback to the person 110. In the illustrated embodiment, the output device 160 is shown as a panoramic head-up display (BMW Panoramic Vision). In other embodiments, any other elements on board the motor vehicle 105 can be used to output feedback to the person 110. This includes, for example, a lighting element, an acoustic element or a haptic element of the motor vehicle 105. An output device 160 can also be provided on the wristband 140. The output device 160 can be controlled in such a way that the person 110 can interactively control a predetermined function using the control system 100. An output on the output device 160 can, in particular, be a hint orprovide feedback about a selectable or selected, actuatable or actuated element to which a predetermined function is assigned.
[0052] The processing device 125 can control the element 115 of the motor vehicle 105 via an interface 165. Optionally, feedback can be provided via the interface 165, which may, for example, concern a state or an option of the element 115. The feedback can be further processed to operate the element 115 or to inform the person 110.
[0053] Figure 2 shows an example composite input 200. The input 200 comprises three elements, which are explained in more detail below.
[0054] As can be seen in a left-hand area of Figure 2, a first element consists of a clapping motion of hand 135 onto another hand of person 110. Hand 135 is stretched out flat, and all fingers are held straight. The position of the other hand is, for example, such that the tips of the thumb and index finger touch. The remaining fingers are curved in accordance with the index finger. Since the position of the left hand is difficult to determine without electromyographic sensor 130, the left hand can also be held differently. The clapping motion shown can be regarded as the first element 205 of the composite input 200.
[0055] A second element 210, shown in the right-hand area of Figure 2, involves removing the hand 135 from the other hand. A third element 215, also shown in the right-hand area, involves snapping the fingers of the hand 135. The illustrated composite input 200 can be used to control any function on board the motor vehicle. The function can be executed in different ways depending on the nuances of the elements 205 to 215. For example, the volume or speed of the clapping of the first element 205 can determine a parameter that can be evaluated in the control of the function. For example, the more intense the clapping was detected, the more intensively the function can be executed.
[0056] Figure 3 shows exemplary temporal relationships of elements of a composite input 200. From top to bottom, a first example 305, a second example 310, a third example 315, and a fourth example 320 are shown. A time is plotted in the horizontal direction. In each example 305 to 320, a gesture of a first hand 135 is shown in an upper area, and another hand 135 of the same person 110 is shown in a lower area. This division is purely exemplary and may require the use of multiple electromyographic sensors 130 on the person. Purely as an example, it is assumed that a gesture or posture of a left hand is shown at the top and a right hand is shown at the bottom.
[0057] In the first example 305, a first element 205 is expressed with the left hand 135, and a second element 210 is expressed with the right hand 135. The start and end times of the elements 205, 210 are the same, so one can speak of synchronous execution.
[0058] In the second example 310, there is a pause between the end of the first element 205, which is played with the left hand 135, and the beginning of the second element 210, which is played with the right hand 135. In this case, one can speak of consecutive or sequential elements.
[0059] In the third example 315 there is a similar constellation as in the second example
[0060] 310, however, the second element 210 begins while the first element 205 is still ongoing. In this case, the gestures overlap. The fourth example 320 is a sequence of the first example 305 and the third example 315. The two examples 305 and 315 are executed sequentially, similar to the second example 310. Such an input 200 can be classified as complex.
[0061] Figure 4 shows an exemplary mapping of composite inputs 200 to controls of elements 115 on board the motor vehicle 105. In a left-hand area, three exemplary composite inputs 200 are shown one above the other. In a right-hand area, exemplary controls 405 to 425 are shown.
[0062] According to the purely exemplary assignment shown, a composite input 200 shown at the top can lead to the actuation of controls 405, 410, and 420. A composite input 200 shown in the middle can lead to the isolated triggering of control 415. A composite input 200 shown at the bottom can act on controls 410 and 425.
[0063] Controls 405 through 425 that can be activated by the same composite input 200 are also referred to herein as functions. Accordingly, the upper composite input 200 can control a function that includes controls 405, 410, and 420. The middle composite input 200 can control a function that includes only control 415. The lower composite input 200 can control another function that includes controls 410 and 425.
[0064] It should be noted that in different embodiments, the depicted image may be fixedly predetermined or modifiable by a person 110. In one embodiment, the person 110 may also create the image entirely themselves.
[0065] Figure 5 shows an exemplary application 500 of a compound input 200. In a left-hand pane, the input 200 is depicted as a sequence of elements 205, 210, and 215. The first element 205 may include spreading all fingers of the hand 135. The second element 210 may include snapping fingers of the hand 135. The third element 215 may include a grasping movement of the hand 135. The compound input 200 may, during or after its execution, cause one or more controls 405 to 435, which are depicted in the right-hand pane of Figure 5.For example, the first control 405 relates to interior lighting, the second control 410 to music or an acoustic effect, the third control 415 to a representation of fire, for example on the output device 160, the fourth control 420 to a vibration or massage of a seat on which the person 110 is sitting, a fifth control 425 to the raising or lowering of a side window, the sixth control 430 to the activation of a sun visor, and the seventh control 435 to a "levitation" of the motor vehicle 105. Levitation can be a function that includes multiple controls. The attempt is made to make the motor vehicle 105 as smooth, quiet, and / or vibration-free as possible. For a person 110 on board the motor vehicle 105, the impression of floating or weightless travel can arise.To achieve this effect, the motor vehicle can drive straight ahead without acceleration, and acoustic sources on board the motor vehicle 105 can be switched off as far as possible.
[0066] By cleverly combining a composite input 200 with a function 400 consisting of multiple controls 405 to 435, a complex control can be achieved that may be reminiscent of magic, as symbolically indicated in the center of Figure 5. It should be noted that, as with performing a magic trick, a convincing effect occurs particularly when there is sufficient practice to provide the required gestures or elements 205 to 220 as perfectly as possible.
[0067] In a further embodiment, another system for interacting with the motor vehicle 105 can also be integrated into the operating concept. For example, selecting or controlling a function can also be supported by a function of a voice assistant (e.g., the BMW Intelligent Personal Assist IPA). Functions of the IPA and the device 100 can be interlinked, or a sequence of inputs can be performed using different systems.
[0068] Figure 6 shows another exemplary application of a composite
[0069] Input 200. One or more people 110 can play the well-known game of rock-paper-scissors. An input 200 shown on the left symbolizes a piece of paper, a middle input 200 symbolizes scissors, and a right input 200 symbolizes a rock. According to a predetermined winning matrix, a zero-sum game is played, in which it is determined in each round which input 200 beats which other input 200. A draw is also possible. The game can also be played between a person 110 and the control system 100 or the processing device 125. A score can be displayed, for example, on the output device 160. In another embodiment, an outcome, an intermediate score, or a state of a game can be displayed by means of another output device on board the motor vehicle 105. For example, an aspect of the interior experience can be changed for this purpose, for example, an audio effect or a lighting effect.Depending on a game parameter, access to or control of a function of the motor vehicle 105 can also be controlled. For example, a winner of a game can select music or a film that can then be played on board.
[0070] Other games can also be executed using the control system 100 on board a motor vehicle 105. For example, a person 110 can be trained to recreate a predetermined compound input 200. For example, two people 110 can play against each other, each attempting to imitate a compound input 200 of the other person 110. Other games can be based on a driving condition, a location of the motor vehicle, or an element or event in the environment of the motor vehicle.
[0071] Figure 7 shows a flowchart of a method 700 that can be performed with the control system 100.
[0072] The left area shows the steps of a training phase and the right area shows the steps of a usage.
[0073] In the training phase, a predetermined composite input 200 can be acquired in a step 702. In a step 704, a composite input 200 determined by the user or by a person 110 can be acquired. In a step 706, inputs from steps 702 and 704 can be combined.
[0074] In a training process 708, which can preferably be executed on a powerful computer, for example, on a server or on a cloud service, data can first be preprocessed in a step 710. In a step 712, the data can be transformed, and in a step 714, features can be captured.
[0075] The input data from step 706 and the training data from step 708 may be used to train a machine learning model in a step 716. In a step 718, an output of the model may be predicted, and in a step 720, the predicted data may be compared to the training data.
[0076] Overall, in the training phase of steps 702 to 720, the machine learning model can be trained to recognize a predetermined gesture or a predetermined posture of a hand 135 of a person 110. Furthermore, it can be trained to recognize a predetermined sequence of hand postures or hand gestures that form a composite input 200.
[0077] In a usage section shown in the right-hand area of Figure 7, a composite input 200 performed by a person 110 on board the motor vehicle 105 can be detected in a step 730. In a step 732, the input can be interpreted and / or classified. From this, a composite input can be determined in a step 734.
[0078] In a step 736, a predetermined composite input 200 may be determined based on the trained learning model, which may be compared with the input from step 734 in a step 738.
[0079] In a step 740, it may be determined that the comparison is positive and the composite inputs correspond to one another. In a step 742, a function associated with the composite input 200 may be determined. The function may be controlled with elements 115 of the motor vehicle 105 in a step 744. The comparison in step 738 may alternatively be determined to be negative in a step 746. The composite input 200 of the person 110 may be rejected in a step 748. Alternatively, no control may be performed in a step 750, or an indication of the failed recognition may be provided in a step 752.
[0080] Reference symbol
[0081] 100 Tax system
[0082] 105 Motor vehicle
[0083] 110 people
[0084] 115 elements
[0085] 120 recording device
[0086] 125 processing facility
[0087] 130 electromyographic sensor
[0088] 132 Accelerometer
[0089] 135 Hand
[0090] 140 Bracelet
[0091] 145 Line of sight
[0092] 150 Pointing direction
[0093] 155 Communication device
[0094] 160 Output device
[0095] 165 Interface
[0096] 200 compound input
[0097] 205 first element
[0098] 210 second element
[0099] 215 third element
[0100] 220 fourth element
[0101] 305 first example
[0102] 310 second example
[0103] 315 third example
[0104] 320 fourth example
[0105] 400 Function
[0106] 405 first control
[0107] 410 second control
[0108] 415 third control
[0109] 420 fourth control
[0110] 425 fifth control 30 sixth control
[0111] 435 seventh control
[0112] 500 applications
[0113] 700 procedures
[0114] 702 capture predetermined compound input
[0115] 704 Capture user-controlled compound input
[0116] Summarize 706 entries
[0117] 708 Training process
[0118] 710 Preprocess data
[0119] 712 Transform data
[0120] Record 714 features
[0121] 716 machine learning
[0122] Predict 718 issue
[0123] Compare 720 data
[0124] 730 capture compound input
[0125] 732 interpret, classify
[0126] 734 determine compound input
[0127] 736 determine known input
[0128] Compare 738
[0129] 740 Determine comparison positively
[0130] 742 determine assigned function
[0131] 744 Control function
[0132] 746 Determine comparison negative
[0133] 748 Reject input
[0134] 750 no control
[0135] 752 Failed detection indication
Claims
Claims 1. A control system (100) on board a motor vehicle (105), the control system (100) comprising the following elements: an electromyographic sensor (130) attached to a person (110) on board the motor vehicle (105) for detecting stimulation of a muscle for moving a hand (135) of the person (110); a processing device (125) configured to determine a posture or gesture of the hand (135); and to determine an input (200) composed of a plurality of postures or gestures (205-220).
2. The control system (100) of claim 1, wherein the electromyographic sensor (130) comprises an electrode for application to the skin of the person (110).
3. The control system (100) of claim 1 or 2, wherein the electromyographic sensor (130) is configured to be mounted on a wrist or forearm of the person (110).
4. The control system (100) of any preceding claim, wherein the electromyographic sensor (130) comprises a wristband (140) for attachment to a wrist (135) of the person (110).
5. The control system (100) according to claim 4, wherein an acceleration sensor (132) is attached to the wristband (140); wherein the processing device (125) is configured to determine a position, an orientation, a gesture, or a posture of the hand (135) depending on acceleration data from the acceleration sensor (132).
6. Control system (100) according to one of the preceding claims, wherein the plurality of postures or gestures (205-220) are in a predetermined temporal sequence.
7. Control system (100) according to one of the preceding claims, wherein the processing device (125) is configured to control a function (400) associated with the composite input (200).
8. Control system (100) according to claim 7, wherein the function (400) is composed of a plurality of controllers (405-435).
9. The control system (100) of claim 8, wherein the controls (405-435) are in a predetermined time sequence.
10. The control system (100) according to any one of the preceding claims, further comprising a detection device (120) for determining a position of the hand (135) of the person (110); wherein the processing device (125) is configured to additionally determine the composite input (200) as a function of the position.
11. The control system (100) according to any one of the preceding claims, further comprising a detection device (120) for determining a viewing axis of the person (110); wherein the processing device (125) is configured to additionally determine the composite input (200) as a function of the viewing axis.
12. The control system (100) of any preceding claim, wherein the processing means (125) is configured to determine a degree to which a composite input (200) corresponds to a predetermined input (200).
13. The control system (100) of claim 12, wherein the processing means (125) is configured to provide feedback about the determined degree.
14. The control system (100) of any preceding claim, further comprising a display device (160) for providing an indication regarding the composite input (200).
15. Control system (100) according to one of the preceding claims, wherein the processing device (125) is configured to learn a posture, a gesture (205-220) or a relationship between different postures or gestures (205-220).
16. Motor vehicle (105) comprising a control system (100) according to one of the preceding claims.
17. A method (700) for controlling a motor vehicle (105), the method comprising the following steps: detecting (730) a stimulation of a muscle for moving a hand (135) of a person (110) on board the motor vehicle (105) by means of an electromyographic sensor (130) attached to the person (110); Determining (732) a posture or gesture (205-220) of the hand (135); and Determining (738) an input (200) composed of several postures or gestures (205-220).
Citation Information
Patent Citations
Wearable electromyography-based controllers for human-computer interface
US20090326406A1
Detecting and Using Body Tissue Electrical Signals
US20180153430A1
