Determining the configuration of an articulated structure

Abstract

A computer-implemented method is proposed, comprising: determining the configuration of an articulated structure by taking into account: the positions of key points of the articulated structure, and at least one topological relationship between key points. Abstract: Figure 1

Description

Title of the invention: Determination of a configuration of an articulated structure. Technical field

[0001] This disclosure falls within the field of analysis and interpretation of data from articulated structures. More specifically, it relates to a method for determining a configuration of an articulated structure, a corresponding system, computer program, and recording medium. Previous technique

[0002] Existing systems for recognizing dynamic patterns or gestures generally rely on approaches based on image or video data. Some approaches use artificial intelligence algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformers. These neural networks are trained to detect static gestures in images, or dynamic gestures in ordered sequences of images. While effective under certain conditions, these approaches often suffer from a high computational cost and require significant hardware resources, which limits their adoption in resource-constrained environments.

[0003] Other approaches exploit key points extracted from the articulated structure to represent configurations. These key points are used in the form of vectors or tensors and are processed by models such as multilayer perceptrons (MLPs). However, these methods do not always allow for the accurate recognition of complex configurations or dynamic movements.

[0004] In this context, there is a need for a technique to overcome these limitations, offering accurate and robust recognition of configurations and dynamic movements, while optimizing the necessary hardware resources. Summary

[0005] This disclosure improves the situation.

[0006] In one aspect, a computer-implemented method is proposed, comprising: a determination of a configuration of an articulated structure taking into account: positions of key points of the articulated structure, and at least one topological relationship between key points.

[0007] According to another aspect, a system is proposed comprising: a module for determining the configuration of an articulated structure, taking into account: of key point positions of the articulated structure, and at least one topological relationship between key points.

[0008] According to another aspect, a computer program is proposed comprising instructions which, when the program is implemented by a processor, lead to the implementation of the process as defined herein. According to another aspect, a non-transient, computer-readable recording medium is proposed on which such a program is recorded.

[0009] The system, computer program and recording medium described are capable of implementing all embodiments of the described process.

[0010] The proposed technique has many advantages. Thus, it can contribute, at least in certain embodiments, to: improved accuracy in recognizing complex configurations by taking into account the topological neighborhood of key points (i.e., local spatial relationships between adjacent key points), increased frugality in terms of computational resources, making the proposed technique suitable for constrained or real-time environments, extensibility for dynamic applications, as the proposed technique can be repeated over time to identify dynamic movements of the articulated structure, and compatibility with various data capture devices, such as 2D or 3D cameras or motion capture systems, this compatibility facilitates broad integration into a variety of existing systems.

[0011] The features described in the following paragraphs may optionally be implemented independently of each other or in combination with each other.

[0012] In at least one embodiment, the positions of the key points are expressed in a first coordinate system and at least one topological relation is expressed in a second coordinate system.

[0013] In at least one embodiment, the first coordinate system is a Cartesian coordinate system and the second coordinate system is a polar, cylindrical or spherical coordinate system.

[0014] In at least one embodiment, the articulated structure includes a hand and a wrist.

[0015] In at least one embodiment, the articulated structure is divided into a plurality of articulated substructures, each substructure comprising at least one joint and / or at least one end.

[0016] In at least one embodiment, a topological relation includes a distance and / or an angle.

[0017] In at least one embodiment, a topological relation is at least one element of a list comprising: a relationship of proximity between key points, a relationship between key points belonging to the same articulated substructure, or a relationship between key points belonging to distinct articulated substructures.

[0018] In at least one embodiment, the determined configuration of the articulated structure is chosen from a discrete set of possible configurations.

[0019] In at least one embodiment, the determination of the configuration is repeated over time, the method comprising: a determination of a dynamic movement of the articulated structure on the basis of the determined configurations.

[0020] In at least one embodiment, the determination of the dynamic movement of the articulated structure includes: a construction of a sequence of symbols, a symbol representing a determined configuration, and a detection of a pattern in the sequence of symbols.

[0021] In at least one embodiment, the configuration of the articulated structure is determined using a convolutional neural network applied to the data obtained.

[0022] In at least one embodiment, the data are structured in a form that allows at least one topological relation to be deduced.

[0023] In at least one embodiment, the method includes determining a user command based on a similarity between the determined configuration and a configuration associated with said command. Brief description of the drawings

[0024] Other features, details and advantages will become apparent from the detailed description below, and from the analysis of the accompanying drawings, cited by way of simple non-limiting examples, on which: Fig. 1

[0025] [Fig.1] shows, in an example of an embodiment, a method for determining a configuration of an articulated structure. Fig. 2

[0026] [Fig.2] shows, in an example of an embodiment, a static configuration of an articulated structure. Fig. 3

[0027] [Fig.3] shows, in an example of an embodiment, a result of a change of coordinate system applied to the key points of an articulated structure. Fig. 4

[0028] [Fig.4] shows, in an example of an embodiment, a tensor structuring key point data. Fig. 5

[0029] [Fig.5] shows, in an example of an embodiment, a method for determining a configuration of an articulated structure. Fig. 6

[0030] [Fig.6] shows, in an example of an embodiment, a dynamic configuration of an articulated structure. Description of the implementation methods

[0031] In the drawings, identical reference numerals designate identical elements or elements having similar functions.

[0032] Some specific terms are now clarified for a better understanding of the proposed technique.

[0033] An articulated structure is an entity composed of segments connected by joints that allow relative movement between the segments. For example, a human hand is an articulated structure comprising several substructures, such as the fingers (each finger being a substructure) and the wrist (a common reference point for the whole, for example). An articulated substructure is an identifiable part of an articulated structure, comprising at least one joint (e.g., finger joint) and / or one extremity (e.g., fingertip).

[0034] A static configuration of an articulated structure is a specific arrangement or posture of the segments of the articulated structure at a given moment. This configuration is defined by the relative positions of the key points of the articulated structure, expressed in particular as a function of the spatial relationships (e.g., distances, angles, and / or alignments) between these key points. For a human hand, a static configuration might correspond, for example, to an open hand, a closed fist, or a pointing finger. In the case of a pointing finger, the joints of the pointing finger are aligned, while those of the other fingers are folded towards the palm. For a robotic structure, a static configuration might correspond to a resting position or a posture adopted to perform a specific task (e.g., a robotic arm extended forward). A static configuration is immobile and does not vary over time.It provides a snapshot of the articulated structure at a specific moment, without taking into account any movements that may precede or follow it.

[0035] A dynamic configuration of an articulated structure is a sequence of successive static configurations that can evolve over time and form a movement or gesture (prolonged immobility in the same static configuration can be considered a gesture in certain embodiments). A dynamic configuration is characterized by the temporal variation of the positions of key points and the spatial relationships that connect them. For a human hand, a dynamic configuration may correspond to a closing movement of the hand (from an open position to a closed fist). Another dynamic configuration may be a gesture of approval (raising the thumb from a closed hand). For a robotic structure, a dynamic configuration may correspond to the movement of a robotic arm from a grasping point to a dropping position.In a dynamic configuration, the topological relationships between key points, such as distances and angles, evolve continuously or in discrete steps. Dynamic configurations can be analyzed to identify specific patterns, such as gestures, trajectories, or complex movements.

[0036] Determining a configuration of an articulated structure means identifying, analyzing, and / or recognizing a particular arrangement of the segments and joints of the articulated structure from obtained data. This process may include classification, for example, assigning the detected configuration to a predefined category. For a hand, determining a configuration may mean recognizing whether it is open or closed by analyzing the relative positions of the joints and fingertips.

[0037] Key points are specific locations defined on an articulated structure to represent segments, joints, extremities, or other features of the articulated structure. For a hand and wrist, key points may include, for example, the center of the wrist, the proximal, intermediate, and distal joints of the fingers, or the fingertips.

[0038] The position of a key point can be defined in a two-dimensional or three-dimensional space, with various coordinate systems. For example, in a three-dimensional space, the position data of key points can be expressed in the form of Cartesian coordinates (x, y, z), cylindrical coordinates (r, 0, z), and / or spherical coordinates (r, 0, z). For example, in a two-dimensional space, the position data of key points can be expressed in the form of Cartesian coordinates (x, y) and / or polar coordinates (r, 0).

[0039] A reference point is a point defined in a two-dimensional or three-dimensional space, used as a basis for expressing spatial or functional relationships between the key points of an articulated structure. The reference point may, for example, be chosen so as to be stable and representative of the structure as a whole or of a specific substructure. For example, for a hand, the center The wrist can serve as a common reference point for all key points, as it remains relatively immobile compared to finger movements. In a robotic arm, a reference point can be placed at the base of the main joint to express the positions and orientations of the segments.

[0040] A reference direction is a direction used as a basis for expressing angular relationships in space. It can be chosen to correspond to a geometric or functional characteristic of the articulated structure. For example: the longitudinal axis of the articulated structure, such as the axis of the arm for a hand, or a direction orthogonal or parallel to a segment defined by two key points (for example, between a proximal joint and an intermediate joint), or an absolute direction in a global coordinate system (for example, the x, y or z axis of a three-dimensional Cartesian coordinate system).

[0041] A relative distance between two key points, or between a key point and a reference point, can be expressed in several ways. The Euclidean distance F12 between a first point with Cartesian coordinates Xp y} and a second point with Cartesian coordinates (X2, yZ2} in a three-dimensional space can be expressed as a scalar value, calculated according to the relation The relative distance between two key points, or between a key point and a reference point, can be normalized, that is, expressed as a proportion of a reference length (for example, the total length of a hinged structure or a portion of the hinged structure).

[0042] The orientation of a key point can be expressed as an angular deviation between the vector connecting a key point to the reference point and the reference direction. For example, in a polar or cylindrical coordinate system, the orientation of a key point can be expressed as an angle between the key point, the reference point, and an axis chosen as the reference direction. For example, in a spherical coordinate system, the orientation of a key point can be expressed as a solid angle formed by a vector defined by a segment (e.g., wrist-tip of the middle finger) with respect to a global or local direction.

[0043] The reference point can be placed at the origin of a coordinate system, in particular a polar, cylindrical, or spherical coordinate system. In this case, the relative distance between a key point and the reference point corresponds to the radius r, expressing the Euclidean distance between these two points. The relative orientation of the key point is expressed, in a polar or cylindrical system, by the angle between the vector connecting the key point to the reference point and a reference direction defined at starting from the reference point. The orientation of a key point is expressed, in a spherical system, as a first angle between the projection of the vector connecting the key point to the reference point onto a first plane and a first principal axis chosen as the reference direction in this first plane, and a second angle between the projection of the vector connecting the key point to the reference point onto a second plane orthogonal to the first plane and a second principal axis orthogonal to the first principal axis and chosen as the reference direction in this first plane.

[0044] In the case of a human hand, if the reference point is the center of the wrist, and if a first reference direction is the longitudinal axis of the forearm (z-axis, oriented from the elbow to the wrist) and a second reference direction is the transverse axis of the plane defined by the forearm and the hand in a neutral position (x-axis, oriented perpendicular to the z-axis and aligned with the width of the hand at the center of the wrist), the spherical coordinates of a key point, such as the tip of a finger, make it possible to capture: the distance r, which describes the distance of the fingertip from the center of the wrist. angle 0, which describes the horizontal orientation of the fingertip relative to the center of the wrist and the x-axis, and the angle q>, which describes the vertical inclination of the fingertip relative to the center of the wrist and the z-axis.

[0045] The topology of an articulated structure refers to the logical and spatial organization of key points within that articulated structure, as well as the relationships between them. It does not necessarily refer to a strict mathematical definition of topology, but rather serves to describe the order in which key points are connected or arranged (for example, the sequence of joints in a finger) and / or the spatial relationships between key points, for example in the form of distances, angles, and / or alignments. In a human hand, the topology captures the organization of the fingers and joints, defining the logical connections between the wrist, the finger joints and their tips, as well as the relative arrangement of the fingers with respect to one another.

[0046] A topological relation refers to information describing a functional interaction or a spatial relationship between at least two key points of an articulated structure. A topological relation can be a proximity relation between key points, for example, an adjacency relation, that is, a direct relation between two key points connected by a segment or a joint, for example, the relative arrangement of a proximal joint and an intermediate joint of a finger. A topological relation can also be a relation internal to a substructure, that is, a relation between key points belonging to the same substructure, by For example, a finger, for instance, the relative arrangement of a distal joint and the tip of a finger. A topological relation can be a relation between distinct substructures, that is, a relation between key points belonging to different substructures, for example, the relative arrangement of the fingertips in a hand.

[0047] In a static context, a topological relationship between two key points, or between a key point and a reference point, can be expressed in the form of one or more elements of the following list: a distance, i.e. a spatial proximity, an angle, i.e. an orientation relative to a reference direction, and a hierarchical relationship, such as a functional or structural dependency, for example a direct connection or adjacency.

[0048] Topological relationships can extend to sets of three or more key points, allowing for the capture of complex spatial and functional features. A topological relationship may, for example, include local curvature or relative symmetry. Local curvature is a measure of the deviation between successive key points on an articulated substructure. For example, in a finger, curvature can be expressed as the angle formed by the segments connecting three joints (proximal, intermediate, and distal). High curvature indicates a bent finger, while low curvature characterizes an extended finger. Relative symmetry describes correspondences between distinct substructures in an articulated structure. For example, in an open hand, the index and ring fingers may exhibit approximate symmetry in position and orientation.This symmetry can be expressed in the form of geometric relationships, such as similar distances or angles with respect to a central axis (e.g., the axis of the middle finger).

[0049] In a dynamic context, topological relationships are not fixed and can vary over time to reflect movements (or a lack of movement) of the articulated structure. Each key point can have a defined trajectory in space, represented by a sequence of successive positions. Thus, the topological relationship between the tip of a finger and the wrist can, for example, be described by a series of distances and / or angles that change over time.

[0050] The topological neighborhood of a key point refers to the set of topological relations that define its interaction with other nearby key points in the articulated structure. These relations can be direct or indirect. For example, in a hand, the topological neighborhood of the middle finger's intermediate joint can include, but is not limited to: adjacency relations with the proximal joint of the middle finger and with the distal joint of the middle finger (i.e., with key points of the same substructure), a spatial proximity relationship with the intermediate joint of the ring finger (therefore with key points of another substructure).

[0051] The terms "topology", "topological relation", and "topological neighborhood" are used in this document as abstractions to describe spatial relations, without necessarily implying a strict geometric or mathematical structure. These notions allow the characterization, depending on the desired application, of static and / or dynamic configurations of an articulated structure.

[0052] This disclosure relates to a technique for determining the configuration of an articulated structure.

[0053] In the field of one-hand gesture detection, existing systems can be divided into two main categories.

[0054] A first category of systems relies on image analysis algorithms. For static gesture detection, a convolutional neural network (CNN) is often used to extract visual features (such as contours, textures, or shapes) directly from provided images. A recurrent neural network (RNN) can be used in conjunction with the convolutional neural network (CNN) to process a video sequence and thus detect dynamic gestures. Such systems require significant computing power, are sensitive to variations in lighting and background, and are highly dependent on the quality of the provided images.

[0055] A second category of systems relies on classification algorithms. Multilayer perceptrons (MLPs) are often used to process keypoints representing hand joints or fingertips and to recognize static configurations. Recurrent neural networks with short- and long-term memory (LSTMs) can be used in addition to analyze temporal sequences of keypoints and recognize dynamic configurations. These approaches often lack precision for complex configurations or subtle movements.

[0056] Unlike existing gesture detection systems based on an image or video of a hand, the proposed technique relies on the use of data representing the positions of key points and their topological relationships, rather than on pixel-by-pixel analysis of an image. Thus, the proposed technique is independent of variations in lighting, background, or image quality, and is less computationally intensive, making it suitable, at least in certain embodiments, for embedded or real-time constrained systems.

[0057] Unlike existing gesture detection systems based on key points of a hand, the proposed technique explicitly takes into account topological relationships between key points, thereby improving the accuracy and reliability of complex configuration recognition. Optionally, the proposed technique uses a convolutional neural network to exploit topological relationships and further improve configuration classification.

[0058] Thus, the proposed technique differs from the state of the art, in particular from existing gesture detection systems.

[0059] The proposed technique is not limited to the hand, but applies to any articulated structure, such as arms, legs or parts of the human skeleton, articulated robotic structures, or even animal structures (tails, paws, etc.). Thanks to this, the method is independent of the specific nature of the articulated structure, which allows its use in various fields (biomechanics, robotics, sports, etc.). The proposed technique can be applied, for example, to determining the configuration of an entire human body; the configuration thus determined can then be used to determine a person's activity.

[0060] Some concepts specific to artificial neural networks are now presented.

[0061] Artificial neural networks (ANNs) are computational models inspired by the biological structure of the brain. They consist of layers of interconnected neurons that transform inputs into outputs through adjustable weights and activation functions. A network comprises input, hidden, and output layers. Each layer includes neurons configured to perform linear or nonlinear transformations on the data provided to them. The weights of the connections between neurons are adjusted during a so-called training phase, via algorithms such as backpropagation, which minimizes a cost function. Training can be supervised or unsupervised. The output of a neural network is generally a probability vector or numerical scores associated with predefined classes.In the context of this document, classes are possible configurations of an articulated structure.

[0062] Among the different types of artificial neural networks, there are in particular: convolutional neural networks (CNNs), recurrent neural networks (RNNs), recurrent neural networks with short-term and long-term memory (LSTMs), and multi-layer perceptrons (MLPs).

[0063] CNNs are designed to process grid-structured data. CNNs apply convolutional filters that slide across the input grid to extract relevant local features. These filters enable pattern detection. specific features, such as textures, contours, or spatial structures, are analyzed by examining local relationships within the data. At each convolutional layer, the extracted features become increasingly abstract, progressing from basic patterns (e.g., contours) to complex concepts (e.g., parts of an articulated structure).

[0064] In a typical use of a CNN, images captured by a camera are converted into pixel matrices. For example, a grayscale image is represented by a 2D grid, where each cell contains a light intensity value (e.g., 0 for black, 255 for white). A color image is encoded into a 3D grid with three channels (red, green, blue), each channel containing a grid of intensities for the corresponding color. The CNN analyzes this grid or these grids to extract patterns useful for the task, such as recognizing a static hand gesture. Before being processed by the CNN, the images may be normalized, resized, or encoded.

[0065] In an application of a CNN according to an embodiment of the proposed technique, keypoint data are provided as input in the form of structured tensors. A tensor is a multidimensional structure (e.g., 2D, 3D, or more) organized to reflect the characteristics of the input data. For an articulated structure, each keypoint can be represented by a set of values ​​(e.g., its coordinates in one or more systems). For example, a set of values ​​representing a keypoint can be a triplet, comprising the position (x, y) of the keypoint in a Cartesian coordinate system and the distance (r) between the keypoint and the origin of the coordinates in a polar coordinate system. Alternatively, a set of values ​​representing a keypoint can be a quadruplet, further including the polar angle (0).These sets of values ​​can be organized into a tensor to reflect the topological relationships between key points (e.g., adjacent points located in close boxes). In addition to the key point positions, the tensor can include explicit topological relationships, such as distances and angles. Alternatively, the tensor's structure itself can be chosen so that the CNN implicitly infers these relationships from the data arrangement. This embodiment of the proposed technique allows the CNN to directly analyze the data from the articulated structure, reducing complexity compared to the known use of a CNN for image analysis.

[0066] RNNs are designed to process data sequences, using recurrent connections that allow them to retain a memory of previous states. At each time step, the RNN takes as input a piece of data from the sequence (for example, a static configuration detected by a CNN) and updates its internal state. This internal state captures the history of previous data, allowing the RNN to Modeling temporal relationships. Simple RNNs can struggle to capture temporal relationships over long sequences due to gradient vanishing during training. In a system combining CNNs and RNNs to detect dynamic gestures, the CNN determines successive static patterns from input images, and the RNN analyzes these patterns over time to detect dynamic patterns or gestures.

[0067] MLPs are fully connected networks where each neuron in each layer is connected to all the neurons in the preceding layer. MLPs are well suited for processing feature vectors, where each feature is an input. The data is transformed through several layers, each transformation enabling the detection of increasingly complex patterns. MLPs can classify static configurations of the articulated structure (e.g., "open hand", "closed fist") based on the positions of key points. They are often used for tasks where temporal relationships are not required.

[0068] LSTMs are a variant of RNNs, designed to process long sequences by overcoming vanishing gradient problems. LSTMs use memory cells and gate mechanisms (in, forget, out) to control which information is stored, updated, or forgotten at each time step. This allows them to capture complex temporal relationships over long sequences. LSTMs can analyze sequences of static configurations to detect complex dynamic gestures, such as a greeting or a fluid hand-closing motion. They are particularly useful for modeling subtle gestures that require consideration of long-term temporal relationships.

[0069] Reference is now made to figures 1 and 2.

[0070] Figure 1 represents a possible example of a flowchart of a process suitable for implementing the proposed technique. This flowchart shows different logic modules, each defined by a specific function: an input module 100, a processing module 200, a structuring module 300, a configuration determination module 400, and an output module 500.

[0071] Figure [Fig.2] represents a human hand as a possible example of an articulated structure for which twenty-one key points are defined as follows.

[0072] A key point 0 is located in the center of the wrist. Four key points 1, 2, 3, 4 are located at the proximal, middle, and distal knuckles and at the tip of the thumb. Four key points 5, 6, 7, 8 are located respectively at the proximal, middle, and distal knuckles and at the tip of the index finger. Four key points 9, 10, 11, 12 are located respectively Four key points 13, 14, 15, and 16 are located at the proximal, middle, and distal joints and at the tip of the ring finger, respectively. Four key points 17, 18, 19, and 20 are located at the proximal, middle, and distal joints and at the tip of the little finger, respectively.

[0073] The input module 100 is configured to obtain the positions of the key points of the structure in a coordinate system, for example in the form of (xi, Yj) doublets where xi and Yi are the horizontal and vertical positions of a key point 1 in an image formed by a pixel grid. The doublets are concatenated to form, in the example of [Fig. 2], a 42-dimensional vector (21 key points and 2 values ​​per key point). Obtaining the positions of the key points can be implemented using various methods known per se.

[0074] The processing module 200 and the structuring module 300 are configured to respectively process and structure the positions of the key points obtained by the input module in order to prepare them for further processing by the configuration determination module 300.

[0075] The processing by the processing module 200 may include one or more operations aimed at transforming and / or enriching the positions obtained.

[0076] For example, the treatment may include a change of reference frame. The positions of the key points may be expressed in a new coordinate system, for example by placing a specific point (such as key point 0, the center of the wrist) at the origin. The position of key point 1 in this coordinate system can be calculated, in Cartesian coordinates, as (xi~x0, Y} ~ Yq), where and Yq are the horizontal and vertical positions of key point 0 as obtained by input module 100 and xi, Yj are the horizontal and vertical positions of key point J.

[0077] For example, the processing may include a change of coordinate system. The position of key point 1 expressed in Cartesian coordinates in a system originating at key point 0 may, for example, be converted into polar coordinates (^i. 0^ where: = arctan(y. - y Q, - x0)'

[0078] Fig. 3 illustrates a result of such a change of coordinate system for key point 5. The conversion to polar coordinates is particularly useful here to allow the 300 module to directly analyze distances and / or angles between key points and the center of the wrist.

[0079] For example, the processing may include enriching the given positions obtained by adding topological relations derived or calculated from the obtained positions. The distance and angle 0i are examples of topological relations derived from the obtained positions -^o, xi, Y q and y

[0080] Other, non-exhaustive, examples of topological relations include: the distance rij between key points i, j with respective coordinates (xi, Y) and (xj, Yj), and an angle having its vertex at the point j and formed between the vectors and jJ, where k is a point with coordinates (xk, Y ^).

[0081] The key point data represent information derived or calculated from the positions obtained by the input module 100.

[0082] When the processing module 200 implements a processing of the obtained positions, the keypoint data includes the positions transformed and / or enriched by such processing. Alternatively, in the absence of the processing module 200, the keypoint data are simply the positions obtained by the module 100, without transformation or enrichment.

[0083] The structuring module 300 is configured to structure or organize the key point data from the processing module 200 into a structure usable by the determination module 400.

[0084] The structuring may include the generation of a multidimensional tensor grouping the key point data.

[0085] The structuring may include a reordering, or reorganization, of keypoint data to reflect topological relationships through their order. According to an example of natural ordering, the keypoints are organized according to their membership in a finger, for example (1,2,3,4) for the thumb, (5,6,7,8) for the index finger, etc. This ordering reflects the adjacency of the keypoints within a substructure (a finger). According to an example of alternative ordering, the points are grouped according to specific relationships, for example (4,8,12,16,20) groups the fingertips, (3,7,11,5,19) groups the proximal joints, etc.

[0086] If module 300 is absent, the natural order of positions or keypoint data from the preceding modules can be used directly. A concatenated vector of positions obtained by module 100 or of keypoint data from module 200 may suffice to implicitly convey topological relations, such as in the order (1, 2, 3, 4), (5, 6, 7, 8), etc.

[0087] The determination module 400 is configured to analyze the structured data produced by the module 300 (or directly by the preceding module(s) if the module 300 is absent) in order to determine a static configuration of the articulated structure.

[0088] Thus, the 400 module can be configured to receive as input data: a tensor grouping the structured data of the key points, a vector of concatenated positions, or a vector of concatenated key point data.

[0089] In one embodiment, the 400 module uses a convolutional neural network (CNN) to analyze the input data.

[0090] The CNN can be configured to extract local features from the input data (e.g., relationships between adjacent key points), combine extracted local features to identify global patterns representing configurations (e.g., "open hand", "closed fist"), and classify the configurations into predefined categories, each category corresponding to a specific static configuration.

[0091] The CNN can be configured to determine a probability or score associated with each possible configuration category, for example in the form of a probability vector ("open hand": 95%, "closed fist": 5%).

[0092] In one embodiment, the articulated structure is a human hand as represented by 21 key points as illustrated in [Fig.2], the input data is structured in the form of a tensor 600, as illustrated in [Fig.4] and the module 400 analyzes the input data using a convolutional neural network 700 as illustrated in [Fig.5].

[0093] The first dimension dim^ of the 600 tensor corresponds to the characteristics associated with each key point. For example, in Figure 4, each key point is described by 4 values ​​(xi_ xo, Yj ~ Yq, i, 0^), so dim^ = 4. The second dimension dim^ of the 600 tensor corresponds to the number of key points per articulated substructure. For example, in Figure 4, each finger is described by 4 key points; for example, key points 1, 2, 3, 4 represent the thumb, so din^ = 4. In this example, key point 0 is conventionally placed at the origin and does not belong to any articulated substructure. The third dimension dim^ of the 600 tensor corresponds to the number of articulated substructures. For example, in figure 4, the hand has 5 fingers, so dilïl3 = 5. The tensor is thus, in this example, of dimensions 4x4x5.

[0094] The 700 convolutional neural network is configured to analyze structured keypoint data and determine a static hand configuration by exploiting both local and global relationships between these keypoints. In one embodiment, the 700 convolutional neural network comprises at least: a 710 convolutional layer configured to extract local patterns from 610 data representing key points of the same articulated substructure, and a 720 convolutional layer configured to extract local patterns from 620 data representing key points belonging to different articulated substructures but sharing topological relationships.

[0095] To isolate, from the tensor 600, the key point data 610 belonging to the same specific articulated substructure (a finger), the index of the dimension dim^ is fixed. For example, the key point data of the thumb, indexed to ) = 1' are 1], the notation * indicating that all values ​​of the dimensions dim^ and dim2 are included. This produces a 4x4 matrix where the 4 rows represent the key point features and the 4 columns represent the 4 key points describing the thumb (joints and tip).

[0096] The convolutional layer 710 applies convolutional filters sliding across this 4x4 matrix. Each filter analyzes the relationships between the key points of the same finger, detecting local patterns such as a linear or curved arrangement of these key points. The presence, in the data 610, of distance values ​​Fi or angles 0i facilitates the detection of these local patterns by the convolutional layer 710. If, for example, the key points of the thumb form a characteristic curve, this curve can be an indicator for recognizing a global gesture such as "open hand." Based on the detected local patterns, the convolutional layer 710 determines a local feature map representing the patterns detected within each finger.

[0097] To isolate from the tensor 600 the data 620 of key points belonging to different articulated substructures but sharing topological relations, that is to say in this example the data of one of the following four groups of key points: a group comprising key points 4, 8, 12, 16, 20 located at the tips of the fingers, a group comprising key points 3, 7, 11, 15, 19 located at the distal joints, a group comprising key points 2, 6, 10, 14, 18 located at the joints intermediaries and A group comprising key points 1, 5, 9, 13, 17 located at the proximal joints, the index of the dimension dîlïly is fixed. For example, key point data for the fingertips, indexed at = 4', are [*, 4, *]• This produces a 4x5 matrix where the 4 rows represent the characteristics of key points and the 5 columns represent the 5 key points describing the fingertips.

[0098] The convolutional layer 720 applies sliding convolutional filters to this 4x5 matrix. Each filter analyzes the relationships between the key points of the same group, detecting global patterns such as relative symmetry or spacing between fingertips. The presence of Fj distance values ​​or 0^ angles in the 620 data facilitates the detection of these global patterns by the 720 convolutional layer. A spread-out arrangement of the fingertips may indicate an open hand, while a close spacing of the fingertips may indicate a closed fist. Based on the detected local patterns, the 720 convolutional layer determines a global feature map representing the detected relationships between the fingers.

[0099] In one possible example of the architecture of the convolutional neural network 700, the outputs of the convolutional layers 710 (local features) and 720 (global features) are flattened into 1D vectors, and then the 1D vectors are concatenated to form a single global vector. A Relu activation function is applied to the global vector to introduce nonlinearity and allow the learning of complex relationships. Successive fully connected layers transform the global vector into an output vector. Finally, a sigmoid activation function is applied to the output to produce a probability vector 800, where each value represents the probability of a category or class from among a set of n predefined classes, that is, from among a discrete set of possible configurations.

[0100] An example of probability vectors may, for example, contain the following information: class 1 (“closed fist”): 5%, class 2 (“open hand”): 95%, other classes: 0%.

[0101] The description of key points of a hand by quadruplets (xi~x0, Y^~ Y q, F i, Qj) representing both positional data and topological adjacency relationships, the structuring of the set of key points as a 4x4x5 tensor highlighting topological relationships between key points belonging or not to the same articulated substructure, and the use of a configured CNN to analyze this structured data each contribute to improving the accuracy of the proposed method. Together, in certain experiments conducted by the inventors, these advances increase the correct classification rate by more than 8% compared to existing methods for determining the static configuration of a hand.Thus, in at least some embodiments, the proposed technique combines the precision of image-based methods with the simplicity and efficiency of keypoint methods, thereby offering a high-performance and cost-effective solution.

[0102] For example, module 500 can be configured to translate a probability vector determined by module 400 into a unique configuration. This translation This can involve selecting the class corresponding to the highest probability. For example, if module 400 produces a probability vector indicating that the "open hand" configuration is associated with a 95% probability and the "closed fist" configuration with a 5% probability, module 500 interprets these results to determine that the hand is in the "open hand" position. Once this interpretation is complete, module 500 can convert this determined class into various formats suited to specific use cases. For instance, it can generate descriptive text such as "configuration detected: open hand," which can be used in diagnostic systems or educational environments to provide detailed information about the detected configurations.The 500 module can also produce graphical representations, for example in the form of images or 3D models illustrating the detected configuration, which can be displayed in a user interface or used to simulate movements in an augmented or virtual reality environment.

[0103] In addition to descriptive texts and graphical representations, the 500 module can be configured to convert detected configurations into logical commands suitable for interactive systems. These commands can be used to activate specific actions in human-machine interfaces or robotic systems. For example, if the 400 module detects an "open hand" configuration, the 500 module can interpret this configuration as a "select" command in a user interface, allowing the user to point or click on an element displayed on the screen. Alternatively, if the detected configuration is that of a "closed fist," the 500 module can interpret this as a "grasp" command in a robotic system, for example, to activate the grasping action of a robotic arm. These commands can also be associated with dynamic gestures when a sequence of static configurations is identified.For example, the dynamic gesture corresponding to a click, consisting of a succession of configurations "index finger raised", "index finger half lowered", "closed fist", "index finger half lowered", "index finger raised", can be interpreted by the 500 module as a "virtual click" command in a user interface.

[0104] The 500 module can also transmit output data to external systems for various applications. For example, in an interactive context with a human-machine interface, descriptive text such as "configuration: open hand" or a logical command "select" can be sent to a navigation system to point to or select an element displayed on the screen. In robotic environments, a "grasp" command corresponding to a "closed fist" configuration can be transmitted to a robotic arm to enable it to manipulate an object. In an augmented reality environment, a A graphical representation of the detected configuration can be displayed to the user to visualize the state or movement of the hand. Furthermore, the 500 module can integrate these results into educational or training systems, generating detailed reports on detected gestures or providing a visual representation of the configurations to aid in learning joint movements.

[0105] In addition to these interactive actions and visual representations, the 500 module can be configured to produce structured data streams for analysis or monitoring systems. For example, it can transmit detected static or dynamic configurations as symbols or codes. Within a complex human-machine interface, these symbols can be combined into sequences to represent more complex dynamic gestures. For example, the 500 module can associate the symbol "I" with a "raised index finger" configuration, "P" with a "closed fist" configuration, and generate a sequence such as "liPil" to indicate a click. These sequences can then be used by external systems to execute complex commands or be displayed as descriptive text, for example, "command detected: click".This flexibility makes it possible to meet the varied needs of interactive systems, whether for applications in robotics, home automation, virtual or augmented reality, or even medical systems requiring contactless interaction.

[0106] Module 500 can therefore be seen as an interface allowing the results of module 400 to be linked to concrete applications, by translating these results into formats adapted to the requirements of users and / or connected systems.

[0107] Fig. 6 illustrates an example of a dynamic 910 configuration of a hand, formed by a succession of distinct elementary static configurations, numbered 900, 901, 902, 903 and 904. These static configurations represent intermediate positions of the hand, captured at different times in time.

[0108] In this example, the dynamic configuration 910 corresponds to a complex gesture simulating a virtual click, consisting of the following actions: lowering the index finger, forming a closed fist, and then raising the index finger. Each step of this gesture can be represented by an elementary static configuration. The first static configuration 900 corresponds to an initial position where the index finger is raised, indicating a waiting or preparing posture. The next configuration 901 captures an intermediate step where the index finger is half-lowered, representing the beginning of the clicking movement. Configuration 902 corresponds to a closed fist, representing the culmination of the dynamic, where the index finger is fully lowered. Configuration 903 returns to an intermediate position similar to 901, but in a release phase, and finally, configuration 904 corresponds to the return to the initial position with the index finger raised again.

[0109] This decomposition of a dynamic gesture into elementary static configurations allows a modular approach for the recognition of dynamic gestures. The 400 determination module can be configured to independently detect and identify static configurations 900, 901, 902, 903, and 904. The 500 output module can then be configured to associate these configurations with distinct symbols (e.g., "I" for raised index finger, "i" for half-lowered index finger, and "P" for closed fist) and generate a symbol sequence representing the full dynamics of the gesture. This symbol sequence can be interpreted to detect movement or the absence of movement by applying a predefined criterion. For example, the absence of movement can be detected if the same symbol is repeated at least a certain number of consecutive times in the sequence (e.g., "PPPPPPPP" for a held closed fist). Conversely, the presence of several distinct symbols within a sequence of a given length can indicate movement.The size of a sequence is defined as the total number of symbols it contains. In practice, a sequence can be very long, which can make its complete analysis more complex. To simplify this analysis, a sequence can be divided into smaller portions, each portion corresponding either to an identified or unidentified movement, or to the absence of movement. This division allows dynamic gestures to be treated as a series of elementary analytical units. For example, a sequence "IlPPPiiiPPP" can be interpreted as corresponding to a dynamic gesture comprising several steps: a raised index finger ("II"), a closed fist ("PPP"), a half-lowered index finger ("iii"), and then another closed fist ("PPP"). Each portion can be analyzed to determine its contribution to an overall gesture.

[0110] For example, for the dynamic configuration 910, the symbols associated with the elementary static configurations are as follows: configuration 900 is associated with the symbol "I" (index finger raised), configuration 901 is associated with the symbol "i" (index finger half lowered), configuration 902 is associated with the symbol "P" (closed fist), configuration 903 is associated with the symbol "i" (index finger half lowered), and configuration 904 is associated with the symbol "I" (index finger raised).

[0111] The resulting sequence, “liPil”, is then analyzed by the output module 500, which recognizes it as a virtual click.

[0112] The use of such a symbol sequence facilitates the management of variations in the execution of dynamic gestures. For example, if the gesture is performed more slowly or more quickly, resulting in repetitions or deviations in the detected configurations (for example, "IliiPPiilI" instead of "liPil"), or in the event of a one-off error in the recognition of a static configuration by the 400 module, the sequence can always be recognized thanks to the use of regular expressions. In other words, the use of such a sequence of symbols offers increased robustness in the recognition of a dynamic configuration of an articulated structure.

[0113] Regular expressions allow for the description of flexible patterns for searching for sequences in a stream of symbols. For example, in the case of the gesture corresponding to a click (ideal sequence: "liPil"), a regular expression can be designed to identify sequences that respect the order of the steps of the gesture (index finger raised then index finger half lowered then closed fist then return), tolerating unintentional repetitions, for example several consecutive "I" or "i" (e.g. "IliiPPiilI") and ignoring insignificant or poorly detected intermediate configurations (e.g. a symbol "_" inserted between two configurations).

[0114] For example, for the "click" gesture, a possible regular expression could be "I +i*P+i*I+", where 1+ denotes one or more consecutive occurrences of "I" (raised index finger), i* denotes zero, one or more occurrences of "i" (half-lowered index finger), and P+ denotes one or more occurrences of "P" (closed fist).

[0115] Thus, module 500 can be configured to search for a match between a sequence of symbols obtained (for example "IliiPPiilI" and the regular expression "I+i*P+i*I+" and, upon detection of such a match, generate a command corresponding to a click.

[0116] The match search can be based on a similarity measure between the detected sequence and one or more reference sequences, such as pre-recorded sequences (for example, a regular expression). Various distance or similarity calculation algorithms can be used depending on the embodiment. For example, these may be distance or similarity calculation algorithms in multi-dimensional spaces such as those for symbol sequences, in particular the Levenshtein distance, dynamic time warping (DTW) algorithms, etc.

[0117] The ability to recognize a dynamic structure through the use of a sequence of symbols associated with specific static configurations is not limited to cases where the dynamic configuration corresponds to a click, but can be applied to many dynamic configurations that may or may not be interpreted as commands. For example, a "zoom in" command can be triggered upon detection of a sequence of static configurations where the fingers gradually move apart, while a "zoom out" command can be triggered upon detection of a sequence of static configurations where the fingers move together. Similarly, commands such as "drag" or "rotate" can be triggered upon detection of sequences of static configurations. elementary elements reflecting intermediate stages of a corresponding movement. Other regular expressions can thus be defined for various supported dynamic configurations.

[0118] Alternatively, it is possible to proceed directly to the analysis of successive static configurations in order to identify a dynamic configuration without going through an explicit step of conversion into symbols.

[0119] In a first example, the elementary static configurations detected by the 400 module can be directly processed as temporal feature vectors. Each static configuration is represented by a set of numerical features (e.g., keypoint coordinates, relative distances, angles, or other topological relationships). These feature vectors are then grouped into a temporal structure, such as a sequence or a matrix, which is analyzed by a temporal classifier such as a recurrent neural network (RNN) or a variant such as an LSTM. These models are configured to detect patterns in the variation of temporal features, thus enabling the recognition of a dynamic configuration such as a click, a zoom, or a swipe, without prior conversion of the static configurations into symbols.

[0120] In a second example, a dynamic configuration can be recognized using a statistical approach based on probabilistic models. Here, each elementary static configuration is associated with a conditional probability that depends on the preceding and following static configurations in the time sequence. A model such as a Hidden Markov Model (HMM) can be trained to represent the probable transitions between static configurations within a given dynamic gesture. Once the model is trained, the dynamic configuration can be determined by identifying the most probable sequence of transitions corresponding to the observed static configurations. For example, for a click, the HMM model can capture the high probability of a transition from "index finger raised" to "index finger half-down," then to "closed fist," and finally back to "index finger raised."

[0121] In a third example, the dynamic configuration can be recognized using a 3D convolutional neural network (3D CNN), designed to directly process temporal sequences of static configurations. In this approach, the keypoint data of each static configuration are structured as three-dimensional tensors where an additional dimension represents the evolution over time. The 3D CNN extracts spatiotemporal features by simultaneously analyzing the relationships between keypoints at a given time and their variation over time. This approach enables robust gesture recognition. dynamics taking into account both local characteristics (within a static configuration) and global characteristics (between successive configurations). Industrial application

[0122] The technical solutions proposed in this disclosure can find applications in numerous fields where they contribute to improving human-machine interaction, the efficiency of automated systems, and / or the accuracy in recognizing articulated configurations. These solutions can be integrated into gesture control systems, augmented or virtual reality environments, and / or advanced robotic systems.

[0123] Furthermore, this disclosure is not limited to the embodiment examples described above, which are provided for illustrative purposes only. It encompasses all variations and modifications that a person skilled in the art could envision within the scope of the claims and the protection sought. These variations include, but are not limited to, adaptations to different types of articulated structures, the use of combinations of several neural networks, and / or various types of structuring and processing of keypoint data.

[0124] In particular, although the examples described focus on a two-dimensional representation of the positions of key points, a three-dimensional representation is also possible. In this case, the position data of a key point can be represented by three Cartesian coordinates and one, two, or three additional coordinates in another coordinate system (for example, a distance and zero, one, or two angles).

Claims

Demands

1. Method for determining a configuration of an articulated structure, the method being implemented by computer, and the determination of the configuration of the articulated structure taking into account: positions of key points of the articulated structure, and at least one topological relationship between key points.

2. A method according to claim 1, wherein the positions of the key points are expressed in a first coordinate system and at least one topological relation is expressed in a second coordinate system.

3. A method according to claim 2, wherein the first coordinate system is a Cartesian coordinate system and the second coordinate system is a polar, cylindrical or spherical coordinate system.

4. A method according to any one of claims 1 to 3, wherein the articulated structure comprises a hand and a wrist.

5. A method according to any one of claims 1 to 4, wherein the articulated structure is divided into a plurality of articulated substructures, each substructure comprising at least one joint and / or at least one end.

6. A method according to any one of claims 1 to 5, wherein a topological relation includes a distance and / or an angle.

7. A method according to any one of claims 1 to 6, wherein a topological relation is at least one element of a list comprising: a proximity relation between key points, a relation between key points belonging to the same articulated substructure, or a relation between key points belonging to distinct articulated substructures.

8. A method according to any one of claims 1 to 7, wherein the determined configuration of the articulated structure is chosen from a discrete set of possible configurations.

9. A method according to any one of claims 1 to 8, wherein the configuration determination is repeated over time, the method comprising: a determination of a dynamic movement of the articulated structure based on the determined configurations.

10. A method according to claim 9, wherein the determination of the dynamic movement of the articulated structure comprises: a construction of a sequence of symbols, a symbol representing a determined configuration, and a detection of a pattern in the sequence of symbols.

11. A method according to any one of claims 1 to 10, wherein the configuration of the articulated structure is determined using a convolutional neural network (700).

12. A method according to claim 11, wherein the convolutional neural network (700) is applied to data structured in a form allowing at least one topological relation to be deduced.

13. A method according to any one of claims 1 to 12, comprising determining a user command based on a similarity between the determined configuration and a configuration associated with said command.

14. Determination module (400) of a configuration of an articulated structure, the determination taking into account: positions of key points of the articulated structure, and at least one topological relation between key points.

Images