Method for detecting the position of a hand, associated device and system comprising such a device

Abstract

A method for detecting the position of a hand belonging to a vehicle occupant comprises at least one iteration including the following steps: - acquisition of an image (IMG) including the occupant by means of an image sensor (5); - estimation, by processing the acquired image (IMG), of the respective three-dimensional coordinates (Xi, Yi, Zi) of a plurality of points on a part of the occupant's body including said hand; - determination, based on the respective three-dimensional coordinates (Xi, Yi, Zi) of the plurality of points on said body part, of a plurality of probabilities of presence (Pj) of the hand relative respectively to a plurality of predetermined areas of the vehicle's interior. A detection device and an associated system are also described. Figure for the abstract: Figure 1

Description

Title of the invention: Method for detecting the position of a hand, associated device and system comprising such a device. Technical field of the invention

[0001] The present invention relates to the technical field of vehicle driving assistance.

[0002] It relates in particular to a method for detecting the position of a hand, an associated device and a system comprising such a device. State of the art

[0003] It has already been proposed to detect the position of certain parts of the driver's body in order, for example, to determine whether the position adopted by the driver is ergonomic or whether the driver is able to resume driving the vehicle after a phase of autonomous driving of the vehicle.

[0004] Some solutions propose extracting the desired characteristics from an image of the driver using an artificial neural network. However, these solutions require the use of large artificial neural networks and consequently the handling of a significant amount of data, both during the training phase of the artificial neural network and during its operation.

[0005] These solutions are therefore not suitable for real-time operation within a vehicle. Presentation of the invention

[0006] In this context, the present invention proposes a method for detecting the position of a hand of a vehicle occupant, comprising at least one iteration including the following steps:

[0007] - acquisition of an image including the occupant by means of an image sensor;

[0008] - estimation, by processing the acquired image, of the three-dimensional coordinates respective of a plurality of points of a part of the occupant's body including said hand;

[0009] - determination, as a function of the respective three-dimensional coordinates of the plurality of points of said body part, of a plurality of probabilities of hand presence relative respectively to a plurality of predetermined zones of the vehicle's passenger compartment.

[0010] Using a representation based on the three-dimensional coordinates of points on only a part of the body reduces the processing required. Such a solution is therefore suitable for real-time processing within a vehicle's interior.

[0011] The method may include a plurality of iterations (each comprising the steps of acquisition, estimation and determination and) each producing a plurality of probabilities of presence respectively relating to said plurality of predetermined zones and an identifier of the zone for which the probability of presence of the hand is maximal among the predetermined zones.

[0012] The detected position can then be the area identified by the most frequent identifier among the identifiers produced for the different iterations of the plurality of iterations.

[0013] The image sensor is, for example, an RGB camera. Such an RGB camera, directed towards the aforementioned occupant (for example, the driver of the vehicle), is increasingly common in vehicle interiors. The solution proposed by the invention is therefore easily integrated into such a vehicle.

[0014] According to one embodiment, the three-dimensional coordinates of the plurality of points can be estimated by processing the acquired image using a posture estimation algorithm. In practice, the posture estimation algorithm can also produce the respective three-dimensional coordinates of points covering the entire body of the occupant (thus enabling efficient estimation), the three-dimensional coordinates of only a subset of these points covering the entire body then being used in the step of determining the plurality of probabilities of presence.

[0015] The probabilities of hand presence relating respectively to the plurality of predetermined zones can be determined from said three-dimensional coordinates by means of a classification algorithm.

[0016] The use of a limited number of zones (for example less than 20 zones, or even less than 10 zones) makes it possible (in combination in particular with the use of points from only one part of the body at this stage, as already mentioned) to lighten the processing carried out by the classification algorithm (that is to say to reduce the means necessary for the implementation of this algorithm).

[0017] The method may then include a step of determining a level of distraction as a function of the detected position.

[0018] Alternatively or in combination, the method may include a step of generating an alarm based on the detected position, for example an alarm to signal that the occupant's hand (here the driver) is not present on the steering wheel of the vehicle while an autonomous driving phase of the vehicle is coming to an end, or an alarm to signal discomfort or a medical problem related to the occupant.

[0019] The invention also proposes a device for detecting the position of a hand of a vehicle occupant, the device comprising:

[0020] - an estimation unit, by processing an image including the occupant, of the respective three-dimensional coordinates of a plurality of points on a part of the occupant's body including said hand;

[0021] - a unit of determination, based on three-dimensional coordinates respective of the plurality of points of said part of the body, of a plurality of probabilities of presence of the hand relating respectively to a plurality of predetermined zones of the vehicle's passenger compartment.

[0022] This detection device may further include:

[0023] - a module designed to activate the estimation unit and the unit multiple times of determination so as to produce each time (output of the determination unit) a plurality of probabilities of presence of the hand relating respectively to said plurality of predetermined zones and (for example output of a selection unit) an identifier of the zone for which the probability of presence of the hand is maximal among the predetermined zones;

[0024] - a module for storing said identifiers;

[0025] - a decision unit designed to provide as detected position the area identified by the most frequent identifier among the stored identifiers.

[0026] The invention finally proposes a system comprising such a device and an image sensor designed to acquire said image.

[0027] Of course, the various features, variants, and embodiments of the invention can be combined with one another in various ways, provided they are not incompatible or mutually exclusive. Detailed description of the invention

[0028] Furthermore, various other features of the invention will become apparent from the following description, made with reference to the accompanying drawings which illustrate a non-limiting embodiment of the invention and where:

[0029] [Fig.1] represents an example of a system conforming to the teachings of the invention;

[0030] [Fig.2] represents a model used within a posture estimation algorithm;

[0031] [Fig.3] represents an example of defining predetermined zones within a vehicle interior; and

[0032] [Fig.4] is a flowchart showing the main steps of an example process in accordance with the teachings of the invention.

[0033] Fig. 1 represents an example of a system conforming to the teachings of the invention.

[0034] This system is an electronic system, embedded here in a vehicle (such as a motor vehicle), and includes an image sensor 5, an electronic detection device 10 and, optionally, another electronic device 100.

[0035] This other electronic device 100 is configured to receive and use position information R from a hand of a vehicle occupant provided by the electronic detection device 10 as explained below.

[0036] This other electronic device 100 (made for example in the form of an electronic control unit) can be for example a driver monitoring system (or DMS for "Driver Monitoring System") or an autonomous driving system.

[0037] The image sensor 5 is here an RGB camera. Such an image sensor 5 is designed to acquire a sequence of images, each representing the environment facing the image sensor (here a part of the vehicle's interior where an occupant is located, as seen by way of example in [Fig.3]) at different times.

[0038] The image IMG produced by the image sensor 5 for each acquisition period is represented by at least one matrix comprising light quantity values ​​respectively associated with a plurality of pixels in the image. In the case where the image sensor 5 is an RGB camera, the image can be represented by three such matrices corresponding respectively to the three color components (here red, green, blue).

[0039] The electronic detection device 10 aims to detect the position of a hand of the occupant present in the IMG image acquired by the image sensor 5. This occupant is, for example, the driver of the vehicle.

[0040] The processing performed to detect one of the occupant's hands is described in detail below. A similar processing method can be used to detect the occupant's other hand, as also specified later.

[0041] The electronic detection device 10 comprises a receiving unit 12, an estimation unit 14, a determination unit 16, a selection unit 18, a control module 20, a memorization module 30 and a decision unit 40.

[0042] The electronic detection device 10 is for example made by means of a processor and a memory: this memory then forms the memory module 30 and also stores computer program instructions designed to implement the functionalities of the control module 20 and of the units 12, 14, 16, 18, 40 when these computer program instructions are executed by the aforementioned processor.

[0043] The control module 20 and / or each of the units 12, 14, 16, 18, 40 can thus be implemented through the cooperation of computer program instructions (stored in memory) and the processor (on which these computer program instructions are executed).

[0044] Alternatively, one or more of the aforementioned unit(s) and / or the aforementioned module may be implemented by means of a dedicated electronic circuit, for example another processor or an application-specific integrated circuit.

[0045] The receiving unit 12 is designed to receive (at each acquisition time) the IMG image acquired by the image sensor 5 and including the occupant. The receiving unit 5 thus receives, for example, three matrices of light quantity values ​​(each matrix corresponding to a color component).

[0046] The estimation unit 14 is designed to process the IMG image, here by means of a posture estimation algorithm, in order to estimate the respective three-dimensional coordinates Xi, Y, Z of a plurality of points of a part of the occupant's body.

[0047] In the example described here, the estimation unit 14 estimates (using the posture estimation algorithm applied to the IMG image) the three-dimensional coordinates of points Ao, ..., A32 covering the entire body of the occupant (here of 33 points covering the entire body of the occupant), that is to say in particular of points Ao, ..., A») of the head of the occupant and / or of points An, Ai2, A23, A24 of the trunk of the occupant and / or of points An, ..., A22 of the upper limbs of the occupant and / or of points A23, ..., A32 of the lower limbs of the occupant.

[0048] The positioning of each of these points on the body according to a potentially usable model is represented in [Fig.2].

[0049] The three-dimensional coordinates mentioned here are, for example, expressed in a frame linked to the image sensor 5, or in another frame whose position and orientation with respect to the image sensor 5 are known (or determined by initial calibration).

[0050] The posture estimation algorithm is, for example, implemented using a specifically trained artificial neural network (for example, for the type of vehicle concerned) to provide the three-dimensional coordinates of points A0, ..., A32 of the occupant's body based on an image of the vehicle's interior including the occupant (as shown as an example in [Fig. 3]). An algorithm such as the MediaPipe Pose Landmarker offered by Google can be used, for example.

[0051] It is noted that such an artificial neural network has a relatively lightweight structure (relatively simple and shallow) and is therefore suitable for real-time operation.

[0052] The estimation unit 14 outputs the three-dimensional coordinates X, Y, Z of points on a part of the occupant's body, including the hand of which one wishes detect the position, here for example the three-dimensional coordinates Xi5 Y;, Z of the points of the arm including the hand concerned.

[0053] For example, to detect the right hand, the three-dimensional coordinates X, Y, Z of each of the points A12, AM, Ai6, Ai8, A20, A22 are produced as output from the estimation unit 14. The estimation unit 14 can thus produce as output the three-dimensional coordinates of each of at least two points among the following points: shoulder (Ai2), elbow (Au), wrist (Ai6), little finger (A18), index finger (A20), thumb (A22).

[0054] Specifically, the estimation unit 14 produces the following output data in this example: X12, Y12, Z12, Xu, Yu, Zu, X16, Y16, Z16, X18, Y18, Z18, X20, Y20, Z20, X22, Y22, Z22.

[0055] If the detection of the left hand is to be carried out in parallel, the estimation unit 14 can produce as output the three-dimensional coordinates of each of the points Ai i, Ai 3, Ai 5, Ai 7, A19, A21. The processing carried out to obtain the position of the left hand on the basis of these three-dimensional coordinates is identical to that carried out for the right hand and will therefore not be described in the following.

[0056] The determination unit 16 receives as input the respective three-dimensional coordinates Xi, Y, Z of the points on the part of the occupant's body including the hand in question. The determination unit 16 therefore receives here the three-dimensional coordinates X, Y, Z of each of the points Ai2, A14, Ai6, Ai8, A20, A22 of the occupant's arm including the hand in question (here the right hand).

[0057] The determination unit 16 is designed to determine, as a function of these different three-dimensional coordinates Xi5 Y;, Z and here by means of a classification algorithm, a plurality of probabilities Pj of presence of the hand relating respectively to a plurality of predetermined (or, in other words, predefined) zones (or regions) Rj of the vehicle's passenger compartment.

[0058] Figure 3 represents an example of defining such predetermined zones within from inside the vehicle:

[0059] - a Ri zone corresponding to the steering wheel;

[0060] - an R2 zone corresponding to the central consonant;

[0061] - an R3 zone corresponding to a central display;

[0062] - an R4 zone corresponding to the lower part of the occupant's body (to his usual position in the passenger compartment);

[0063] - an R5 zone corresponding to the upper part of the occupant's body (to his usual position in the passenger compartment);

[0064] - an R6 zone corresponding to the interior rearview mirror;

[0065] - an R7 zone corresponding to an adjacent seat (for example, the passenger seat when (the occupant is the driver);

[0066] - an R8 zone corresponding to the occupant's door.

[0067] Other divisions of the passenger compartment into predetermined zones (or regions) can of course be envisaged, depending on the nature and precision of the detection that one wishes to achieve.

[0068] The classification algorithm used is, for example, a classifier using a gradient-boosted decision tree (or GBDT for "Gradient Boosted Decision Tree"), such as XGBoost.

[0069] In the example described here, the classification algorithm therefore receives as input the 16 data (Xn, Yn, Z12, X14, Y14, Z14, Xi6, Y16, Z16, X18, Y18, Z18, X20, Y20, Z20, X22, Y22, Z22) corresponding to the three-dimensional coordinates of the points of the part of the body including the hand concerned, and produces as output the probabilities Pb ..., P8 respectively associated with the predetermined zones ..., R8.

[0070] The probabilities Pj determined by the determination unit 16 are transmitted to a selection unit 18 which deduces the most probable zone RK, that is to say the zone RK for which the probability PK is maximum among the predetermined zones RB ..., R8, and produces as output an identifier (here the index K) of this most probable zone RK and the probability PK associated with this most probable zone RK.

[0071] In other words, we have: K = arg maxj Pj.

[0072] The control module 20 is designed to activate the receiving unit 12, the estimating unit 14, the determining unit 16 and the selecting unit 18 a number N of times.

[0073] According to one possible embodiment, the number N is predetermined (with for example N = 16).

[0074] According to another possible embodiment, the number N depends on the frame rate of the video sequence produced by the image sensor 5. For example, if the frame rate is less than or equal to 8 frames per second, the number N is equal to the frame rate multiplied by 2, and if the frame rate is strictly greater than 8 frames per second, the number N is equal to the frame rate.

[0075] Furthermore, it can be envisaged, for example, that all the images in the image sequence produced by the image sensor 5 are successively applied to the input of the detection device 10 (i.e., received by the receiving unit 12) and processed by units 14, 16, 18 (during the N different activations of units 12, 14, 16, 18, the process repeating in this case after N activations as explained later). Alternatively, however, only a portion of the images in the image sequence produced by the image sensor 5 (for example, one image every M images, with M a predefined integer greater than or equal to 2) could be applied to the input of the detection device 10 (i.e. received by the receiving unit 12) and processed by units 14, 16, 18.

[0076] At each activation of units 12, 14, 16, 18 by the control module 20, the determination unit 16 produces as output a plurality of probabilities Pj of presence of the hand relating respectively to the predetermined zones Rj so that the selection unit 18 can produce as output the identifier (here the index K) of the zone RK for which the probability of presence of the hand is maximum among the predetermined zones and the probability PK relating to this zone RK.

[0077] The memory module 30 stores the data successively produced by the selection unit 18 during its N activations, namely the identifier (here the index) K representing the most probable RK zone identified at each activation and, in association with this identifier K, the corresponding PK probability.

[0078] After the N activations, the memory module 30 thus memorizes N identifiers Ki, ..., Kn representing the most probable zones identified successively during the N activations and the corresponding N probabilities P' b ..., P'N (the probability P'n being the probability determined by the determination unit 16 for the most probable zone ZKn during the nth activation).

[0079] The decision unit 40 is designed to provide, as the detected position of the hand concerned (or hand position information), the area R identified by the most frequent identifier among the identifiers Kb ..., KN stored in the memory module 30. (If several identifiers have the same frequency, the decision unit 40 can, for example, provide as output the area identified first among these identifiers.)

[0080] In the example described here, the decision unit 40 also outputs a probability P associated with this detected position. The decision unit 40 determines this probability P as equal to the average of the probabilities P'n stored in the memory module 30 in association with an identifier Kn equal to the most frequent identifier. In other words, the probability P produced as output by the decision unit is equal to the average of the probabilities relating to the area R identified by the most frequent identifier (among the stored identifiers Kb ..., KN) and obtained for the activations for which the probability of this area R was maximal within the predetermined areas Zb ..., Z8.

[0081] Fig. 4 is a flowchart showing the main steps of an example of a process in accordance with the teachings of the invention.

[0082] This process is implemented here in the system of [Fig. 1].

[0083] The process in [Fig. 4] begins with a step E2 of acquiring an IMG image including the vehicle occupant by means of image sensor 5.

[0084] The process then includes a step E4 of estimation, by processing the acquired image IMG (here within the estimation unit 14), of the respective three-dimensional coordinates Xi5 Y;, Z of a plurality of points Ao, ..., A32 of the body of the occupant.

[0085] As already indicated in the context of the description of [Fig. 1], the processing of the acquired IMG image is carried out here by processing using a posture estimation algorithm and makes it possible to obtain the three-dimensional coordinates Xi5 Yi, Z; respective of points Ao, ..., A32 (for example of at least 20 points) covering the whole body of the occupant (and therefore including in particular points Ao, ..., A10 of the head of the occupant and / or points An, A12, A23, A24 of the trunk of the occupant).

[0086] The method then includes a step E6 of filtering the three-dimensional coordinates Xi5 Y, Z (here within the estimation unit 14) so ​​as to use in the remainder of the method (here in step E8) only the three-dimensional coordinates X, Y, Z relating to a plurality of points (here relating to points Ai2, Am, Ai6, Ai8, A20, A22) of a part of the body (here the arm) of the occupant including the hand whose position is to be determined. This part of the body includes, for example, 10 points or fewer (here 6 points) among the points covering the entire body as mentioned above.

[0087] The process continues with a step E8 of determining, based on the respective three-dimensional coordinates X, Y, Z of the plurality of points on this part of the occupant's body (i.e., based on the three-dimensional coordinates retained in step E6) and here without taking into account the other three-dimensional coordinates (three-dimensional coordinates discarded in step E6), a plurality of probabilities Pj of hand presence relative respectively to a plurality of predetermined zones Rb ..., R8 of the vehicle's passenger compartment. Step E8 is implemented here by the determination unit 16.

[0088] As already indicated, the probabilities Pj of presence of the hand relating respectively to the plurality of predetermined zones Rh ..., R8 are determined here from said three-dimensional coordinates X;, Y;, Z by means of a classification algorithm.

[0089] It is noted that the use of points from the whole body in step E4 allows us to obtain a good estimate of the three-dimensional coordinates Xi5 Y;, Z; of these points, while the use of points from only a part of the body in step E8 allows us to lighten the processing carried out by considering only the data relevant to the localization of the hand (here the three-dimensional coordinates Xi5 Y;, Z; of the points of the occupant's arm).

[0090] The process of [Fig.4] then continues with a step E10 of selection (here by the selection unit 18) of the most probable zone ZK according to the probabilities determined in step E8, that is to say of selection of the zone ZK (or equivalently of the index K) for which (or for which) the probability PK is maximum among the probabilities Pj of all the zones Zj.

[0091] The process of [Fig.4] continues with a step E12 of memorization (here within the memorization module 30 and under the action of the selection unit 18) of an identifier (here the index K) corresponding to the most probable zone ZK (selected in step E10) and, in addition here, of the associated probability PK.

[0092] The control module 20 then determines at step E14 whether N iterations of steps E2 to E12 have been carried out in such a way as to memorize N indices Kn (representing the most probable zones ZK determined by each of the N iterations) and, here, the N respective associated probabilities PK „•

[0093] In the negative at step E14 (NEG arrow in [Fig.4]), the process loops back to step E2 so as to perform a new iteration of steps E2 to E12.

[0094] In case of positive determination at step E14 (POS arrow in [Fig.4]), N identifiers (here N indices Kn) and, here, N corresponding probabilities PKn are stored in the storage module 30 and the process then continues to step E16.

[0095] The decision unit 40 can thus determine at step E16 the position of the hand concerned as equal to the zone R identified by the most frequent identifier among the identifiers ..., KN produced for the aforementioned N iterations (identifiers stored in the storage module 30 during the N successive passes to step E12).

[0096] In the example described here, the decision unit 40 also determines during step E16 a probability P associated with the current detection by calculating the average of the probabilities P'n associated with an identifier Kn equal to the most frequent identifier.

[0097] It is noted that the implementation of several iterations and the choice of the most frequently obtained area as the most probable area allows for robust processing and a stable result.

[0098] The process then loops back to step E2 to begin a new detection of the position of the hand concerned (the iteration counter being reset to zero so as to perform N iterations of steps E2 to E12 again for this new detection).

[0099] In parallel, the position R obtained in step E16 (as well as possibly the associated probability P) can be used by the other electronic device 100 of the vehicle.

[0100] For example, during an El8 step, a level of distraction of the occupant (here the driver of the vehicle) can be determined by using in particular the position R (among other parameters) within a driver monitoring system.

[0101] According to another example (possibly combinable with the previous one), an autonomous driving system can determine during an E20 step whether the driver is ready to take back control of the vehicle based in particular on the R position and / or issue an alert if the R position does not indicate that the driver's hand is on the steering wheel when it is planned to end (in the short term) the autonomous driving phase.

[0102] In other variants, the R position could allow the detection of physical discomfort or a medical problem; such detection could then lead to the generation of an alert signal and / or the display of assistance information.

Claims

Demands

1. Method for detecting the position of a hand of a vehicle occupant, comprising at least one iteration including the following steps: - acquisition (E2) of an image (IMG) including the occupant by means of an image sensor (5); - estimation (E4), by processing the acquired image (IMG), of the respective three-dimensional coordinates (Xi5 Y;, Zj) of a plurality of points (An, Au, Ai6, Ai8, A20, A22) of a part of the occupant's body including said hand; - determination (E8), as a function of the respective three-dimensional coordinates (X;, Y;, Z) of the plurality of points (Ai2, Aw, Ai6, A[8, A20, A22) of said part of the body, of a plurality of probabilities of presence (Pj) of the hand relating respectively to a plurality of predetermined zones (RB R2, R3, R4, R5, R6, R7, R8) of the vehicle's passenger compartment.

2. A method according to claim 1, comprising a plurality of iterations each producing a plurality of probabilities of presence (Pj) respectively relating to said plurality of predetermined zones (Rb R2, R3, R4, R5, R6, R7, R8) and an identifier (K) of the zone for which the probability of presence of the hand is maximal among the predetermined zones, wherein the detected position (R) is the zone identified by the most frequent identifier among the identifiers (Kb KN) produced for the different iterations of the plurality of iterations.

3. Method according to claim 1 or 2, wherein the image sensor (5) is an RGB camera.

4. A method according to any one of claims 1 to 3, wherein said three-dimensional coordinates (X, Y, Z) of said plurality of points (Ai2, Am, Ai6, Ai8, A20, A22) are estimated by processing the acquired image (IMG) using a posture estimation algorithm.

5. A method according to any one of claims 1 to 4, wherein the probabilities of presence (Pj) of the hand relating respectively to the plurality of predetermined zones (Rb R2, R3, R4, R5, R6, R7, R8) are determined from said three-dimensional coordinates (Xi5 Y;, Z) by means of a classification algorithm.

6. A method according to any one of claims 1 to 5, comprising a step (El8) of determining a level of distraction as a function of the detected position.

7. A method according to any one of claims 1 to 6, comprising a step (E20) of generating an alarm based on the detected position.

8. Device for detecting the position of a hand of a vehicle occupant, the device comprising: - an estimation unit (14), by processing an image (IMG) including the occupant, of the respective three-dimensional coordinates (Xi5 Y;, Z; ) of a plurality of points (Ai2, AM, Ai6, A[8, A2o, A22) of a part of the occupant's body including said hand; - a determination unit (16), as a function of the respective three-dimensional coordinates (Xi5 Y;, Zj) of the plurality of points (Ai2, Am, Aie, Aïs, A2q, A22) of said part of the body, of a plurality of probabilities of presence (Pj) of the hand relating respectively to a plurality of predetermined zones (RB R2, R3, R4, R5, R6, R7, R8) of the vehicle's passenger compartment.

9. A detection device according to claim 8, comprising: - a module designed to activate the estimation unit (14) and the determination unit (16) multiple times so as to produce each time a plurality of probabilities of presence (Pj) of the hand relating respectively to said plurality of predetermined zones (RB R2, R3, R4, R5, R6, R7, R8) and an identifier (K) of the zone for which the probability of presence of the hand is maximum among the predetermined zones; - a storage module (30) of said identifiers (Kb KN); - a decision unit (40) designed to provide as detected position (R) the zone identified by the most frequent identifier among the stored identifiers (Kb KN).

10. System comprising a device according to claim 8 or 9, and an image sensor (5) designed to acquire said image.

Images