Method and system for detecting a hand in an area of an image displayed on a screen of a vehicle
The method and system enhance driver safety by using a time-of-flight sensor and machine learning to detect hand poses on a vehicle's touch screen, reducing the need for manual control and improving attention on driving.
Patent Information
- Application Number
- FR2023005455
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2043-05-31
Smart Images
Figure 00000010_0000 
Figure 00000010_0001 
Figure 00000011_0000
Abstract
Description
Title of the invention: Method and system for detecting a hand in an area of an image displayed on a screen of a vehicle Technical field of the invention
[0001] The invention relates to a method and a system for detecting the presence of a user's hand in at least one area of an image displayed on a touch screen on board a vehicle, in particular in a motor vehicle. State of the prior art
[0002] Touch screens are widely implemented in motor vehicles. They can be used to control various services or equipment such as geolocation services, telephony services or air conditioning and / or heating equipment. However, it can be difficult for the driver to concentrate on driving and simultaneously control these services or equipment from the touch screen. Presentation of the invention
[0003] The aim of the present invention is to propose a low-cost method for detecting the presence of a hand in at least one area of an image displayed on a touch screen in order to facilitate the control of services by this touch screen.
[0004] Advantageously, this method can be used to highlight the area close to the hand, make part of the control buttons disappear and thus reduce the mental load on the user. Advantageously, the driver can give a greater part of his attention to his driving. The implementation of the detection method allows safer driving.
[0005] The present invention also aims to propose a corresponding detection system. Summary of the invention
[0006] The subject of the present invention is a method for detecting the presence of at least one part of a user's hand in front of at least one area and / or marker of an image displayed on a touch screen embedded in a vehicle, the method being implemented by a detection system comprising a touch screen, at least one time-of-flight sensor having a field of vision which extends substantially parallel to the mean plane of the touch screen and a processing unit comprising a machine learning model, the processing unit being capable of receiving distance data generated by the time-of-flight sensor, the processing unit being adapted to transmit at least one image to the touch screen, the method comprising the following steps: - display at least one image on the touch screen, - capturing a plurality of distance data using the time-of-flight sensor, - detecting, by the machine learning model, the presence of at least a part of a hand in a first predefined pose in front of at least one area and / or marker of the image displayed on the touch screen from the captured distance data, said area and / or marker of the image comprising an element representative of a control button.
[0007] The features set out in the following paragraphs may, optionally, be implemented. They may be implemented independently of one another or in combination with one another: - The method comprises a learning phase adapted to train the machine learning model; the learning phase comprising the following steps: - define zones and / or markers on at least one image, - display said at least one image on the touch screen, - for each zone and / or landmark, capturing a plurality of distance data using the time-of-flight sensor, said distance data being representative of at least a portion of a hand in the first predefined pose, said at least a portion of a hand being located at different locations relative to the zones and / or landmarks defined on the displayed image, and - for each area and / or landmark, training the machine learning model from the plurality of captured distance data to associate said areas and / or landmarks with the captured distance data when the at least one part of a hand is in the first predefined pose. the method further comprises the following steps: - upon detecting the presence of at least one part of a hand in the first predefined pose, processing said at least one displayed image to highlight a part of the image comprising or adjacent to the area and / or the marker associated with the captured distance data, and - displaying said at least one processed image on the touch screen. - The first preset pose is representative of a straight index finger and other fingers folded towards a palm. The step of processing a part of the image comprises at least one of enlarging a part of the image, highlighting a part of the image, illuminating a part of the image and disappearing a complementary part of the image. - The at least one time-of-flight sensor is disposed adjacent to an edge of the touchscreen, the field of view of the time-of-flight sensor extending in front of the touchscreen. The learning phase also includes the following steps: - for each zone and / or marker, capturing a plurality of distance data using the time-of-flight sensor; said distance data being representative of at least one part of a hand in a second predefined pose, said at least one part of a hand being located at different locations relative to the zones and / or markers defined on the displayed image, - for each area and / or landmark, training the machine learning model from the plurality of captured distance data to associate said areas and / or landmarks with the captured distance data when the at least part of a hand is in the second predefined pose. The second predefined pose is representative of a straight palm and fingers. The second predefined pose is representative of a folded palm and fingers. The present invention also relates to a detection system suitable for being embedded in a vehicle, in particular a motor vehicle, the system comprising: - a touch screen capable of displaying at least one image, - at least one time-of-flight sensor capable of capturing a plurality of distance data, the time-of-flight sensor being disposed adjacent an edge of the touchscreen, the time-of-flight sensor having a field of view that extends substantially parallel to the midplane of the touchscreen, and - a processing unit capable of receiving distance data generated by the time-of-flight sensor, the processing unit being adapted to transmit at least one image to the touch screen, the processing unit comprising a machine learning model adapted to detect the presence of at least one part of a hand in a first predefined pose in front of at least one zone and / or a marker of the image displayed on the touch screen from the captured distance data, said zone and / or said marker of the image comprising an element representative of a control button. Brief description of the figures
[0008] [Fig.l] is a schematic view of a detection system according to the invention;
[0009] [Fig.2] is a flowchart representing steps in a learning phase of a detection method according to the invention;
[0010] [Fig.3] is a flowchart representing steps of the detection method according to the invention;
[0011] [Fig.4] is a schematic view representing areas and / or markers of an image displayed on a display screen.
[0012] [Fig.5] is a schematic view of a hand in a first predefined pose;
[0013] [Fig.6] is a schematic view of an example of a hand in a second pose predefined; and
[0014] [Fig.7] is a schematic view of another example of a hand in a second predefined pose. Detailed description of the invention
[0015] An example of a detection system 2 according to the invention has been shown schematically in [Fig.l]. This system is suitable for implementing the detection method according to the invention. This detection system 2 is installed in a vehicle, preferably in a motor vehicle.
[0016] This detection system 2 comprises a touch screen 4, a time-of-flight sensor 6 and a processing unit 10 connected to the touch screen and the time-of-flight sensor. Generally, the touch screen is located on the upper part of the central console of the vehicle. The touch screen is for example a plasma screen or a liquid crystal screen.
[0017] The time-of-flight sensor 6 is arranged adjacent to an edge 7 of the touch screen such as for example the lower edge, the side edge or the upper edge. It has a field of vision 8 which extends substantially parallel to the mean plane of the touch screen. In particular, the field of vision of the time-of-flight sensor extends in front of the touch screen. The time-of-flight sensor 6 comprises a light-emitting diode capable of emitting radiation in the field of vision, a sensor adapted to receive the radiation reflected by a hand or a part of a hand located in the field of vision and a processor capable of calculating the distance between the hand and the sensor. The time-of-flight sensor 6 is capable of delivering distance data which are transmitted to the processing unit. The emitted radiation is, for example, infrared radiation. A sensor with 16 pixels can be used, for example.
[0018] The processing unit 10 is constituted by a processor or a microprocessor.
[0019] The processing unit 10 comprises a memory 12 and a learning model automatic 14.
[0020] The automatic learning model is called “machine learning” in English. The memory 12 of the on-board display device comprises a random access memory, denoted RAM and a read-only memory denoted ROM. The memory 12 is capable of storing images 16 and instructions for implementing the detection method according to the present invention, when these instructions are executed by the processor.
[0021] The machine learning model 14 is based on one or more types of models known to those skilled in the art, such as neural networks, XGBoost®, SVM® and K-means®.
[0022] The processing unit 10 is adapted to transmit at least one image to the touch screen. In particular, the processing unit is adapted to transmit at least one image representative of a control panel comprising several elements 15 illustrating control buttons or images making it possible to control or select equipment or services.
[0023] Alternatively, the detection system 2 may comprise several time-of-flight sensors.
[0024] The detection method begins with a learning phase illustrated in [Fig.2].
[0025] This learning phase includes a step 20 during which zones 18 and / or markers 22 are defined by the processing unit on an image intended to be displayed on the touch screen.
[0026] An example of such an image 16 has been shown in [Fig. 4]. This image represents different elements 15 illustrating control buttons or actuation images. In this example, ten zones 18 have been defined on the image. In the illustrated example, these zones 18 form horizontal bands. On the fifth zone, five markers 22 have also been defined. In a step 24, the image is displayed on the touch screen. In a step 26, a hand arranged in a first pose is positioned in an area 18 and distance data is captured using the time-of-flight sensor.
[0027] Advantageously, the first pose has been defined on the one hand to increase the reliability of the detection method and on the other hand so that the detection method only detects a real desire of the user to make a command. Thus, if the user accidentally positions his hand in the field of vision 8 in a position other than the first defined pose, the detection method will not detect the user's hand. Thus, advantageously, the detection method detects an intention to make a command on the touch screen.
[0028] The first pose is predefined. For example, the first pose of the hand is representative of a straight index finger and other fingers folded towards a palm, as illustrated in [Fig.5].
[0029] Step 26 is repeated for different orientations and different locations of the hand in front of area 18.
[0030] Step 26 is also repeated for different hands, in particular having different sizes or morphologies. The different hands are all in the first pose.
[0031] Preferably, distance data is also captured for all different orientations, locations, and hand sizes for each marker 22 within a given area.
[0032] During a step 28, the machine learning model 14 is trained from the plurality of captured distance data to associate said areas and possibly landmarks with the captured distance data when a hand or part of a hand is in the first predefined pose.
[0033] Preferably, the learning phase further comprises steps 30 and 32.
[0034] In a step 30, a hand or part of a hand is arranged in front of the image in a second pose and distance data is captured using the time-of-flight sensor. Distance data is captured for different orientations and locations of the hand in front of image 16. Distance data is also captured for different hands. The different hands are all in the second pose. The second pose is for example representative of a straight palm and fingers, as illustrated in [Fig.6]. The second pose can also be representative of a folded palm and fingers, as shown in [Fig.7].
[0035] During a step 32, the machine learning model 14 is trained from the plurality of captured distance data to associate the second predefined pose with the captured distance data.
[0036] Advantageously, the machine learning model 14 is trained not to detect a desire to control the touch screen when the hand is in the second predefined pose.
[0037] The learning phase is then complete.
[0038] With reference to [Fig. 3], the detection method begins with a step 34 during which an image 26 is displayed on the touch screen 4.
[0039] Then, during a step 36, distance data is captured by the time-of-flight sensor. The distance data is transmitted to the machine learning model 14 of the processing unit 10.
[0040] During a step 38, the machine learning model 14 analyzes the received distance data.
[0041] If the machine learning model 14 does not detect the presence of a hand or part of a hand in the first predefined pose in front of an area 18 and / or a marker 22 of the image displayed on the touch screen, the method returns to step 34, during a step 40.
[0042] If the machine learning model 14 detects the presence of a hand or part of a hand in the second predefined pose in front of an area 18 and / or a marker 22 of the image displayed on the touch screen, the method also returns to step 34.
[0043] If the machine learning model 14 detects the presence of a hand or part of a hand in the first predefined pose in front of an area 18 and / or a marker 22 of the image displayed on the touch screen, the method continues with a step 42. During step 42, the processing unit 10 processes the displayed image to highlight a portion of the image comprising or adjacent to the area and / or the marker associated with the captured distance data. In particular, a portion of the image comprising the element 15 representative of a control button is highlighted. This step of processing a part of the image can, for example, consist of enlarging a part of the image or highlighting a part of the image or illuminating a part of the image or even making the other part of the image disappear.
[0044] This processing step may also consist of searching for a part of the image recorded in the memory 12 comprising one of the processing operations mentioned above.
[0045] During step 44, the processed image 16 is displayed on the touch screen.
[0046] Alternatively, distance data may also be captured for all different positions, orientations, locations, and hand sizes only for each marker 22.
Claims
1. Claims A method for detecting the presence of at least one part of a user's hand in front of at least one area (18) and / or a marker (22) of an image (16) displayed on a touchscreen (4) on board a vehicle, the method being implemented by a detection system (2) comprising a touchscreen (4), at least one time-of-flight sensor (6) having a field of vision (8) which extends substantially parallel to the mean plane of the touchscreen and a processing unit (10) comprising a machine learning model (14), the processing unit being capable of receiving distance data generated by the time-of-flight sensor, the processing unit being adapted to transmit at least one image to the touchscreen, the method comprising the following steps: - displaying (34) at least one image on the touchscreen, - capturing (36) a plurality of distance data using the time-of-flight sensor, - detecting (38, 42),by the machine learning model (14), the presence of at least a part of a hand in a first predefined pose in front of at least one zone (18) and / or a marker (22) of the image displayed on the touch screen from the captured distance data, said zone and / or said marker of the image comprising an element (15) representative of a control button, characterized in that the method comprises a learning phase adapted to train the machine learning model (14); the learning phase comprising the following steps: - defining (20) zones (18) and / or markers (22) on at least one image (16), - displaying (24) said at least one image (16) on the touch screen, - for each zone and / or marker, capturing (26) a plurality of distance data using the time-of-flight sensor,said distance data being representative of at least a portion of a hand in the first predefined pose and at least a portion of a hand in a second pose, said at least a portion of a hand being located at different locations relative to the areas and / or landmarks defined on the displayed image, and, - for each area and / or landmark, training (28) the machine learning model (14) from the plurality of captured distance data to associate said areas and / or landmarks with the captured distance data when the at least one part of a hand is in the first predefined pose, and when the at least one part of a hand is in the second predefined pose.
2. A detection method according to claim 1, wherein the method further comprises the following steps: - upon detection of the presence of at least one part of a hand in the first predefined pose, processing (42) said at least one displayed image to highlight a part of the image comprising or adjacent to the area and / or the marker associated with the captured distance data, and - displaying (44) said at least one processed image on the touch screen.
3. A detection method according to any one of claims 1 and 2, wherein the first predefined pose is representative of a straight index finger and other fingers folded towards a palm.
4. A detection method according to any one of claims 2 and 3, wherein the step of processing (42) a portion of the image comprises at least one of enlarging a portion of the image, highlighting a portion of the image, illuminating a portion of the image, and removing a complementary portion of the image.
5. A detection method according to any one of claims 1 to 4, wherein the at least one time-of-flight sensor (6) is arranged adjacent to an edge (7) of the touchscreen (4), the field of view (8) of the time-of-flight sensor extending in front of the touchscreen.
6. The detection method of claim 1, wherein the second predefined pose is representative of a straight palm and fingers.
7. The detection method of claim 1, wherein the second predefined pose is representative of a folded palm and fingers.