Control system and method

Abstract

Control System and Method This description relates to a control system (100) comprising: - an intent detection device (104) including a first camera (105) operated by a user (106) and configured to estimate the first coordinates of a first gaze fixation point, in an environment, of a user in a first frame of reference associated with the image of the first camera; - a robotic system (102) including a second camera (110) configured to determine, on the basis of the first estimated coordinates of the first point in the first frame of reference, the second coordinates of a second point, in a second frame of reference associated with the image of the second camera, of the user's gaze fixation point, the robotic system further including a robotic system (116, 118) configured to perform an action towards a point in the environment, on the basis of the second coordinates in the second frame of reference. Figure for the abbreviation: Fig. 1A

Description

Title of the invention: Control system and method technical field

[0001] The present description relates in general to the control of a robotic system, and in particular to control based on intention detection by eye tracking. Previous technique

[0002] Intention detection can be used for the control of robotic systems by users such as people with physical limitations. For example, these robotic devices include motorized exoskeletons, motorized auxiliary arms, etc., the control of which is managed by a processing unit configured to determine a user's intention.

[0003] For example, intention detection can be implemented by an eye-tracking device that estimates a point in space targeted by the user's gaze. The detection device is coupled to the robotic system configured to perform an action toward the point targeted by the user.

[0004] However, since the device configured for intention detection is, for example, mobile with the user, and the robotic system is, for example, fixed relative to the user, it is important that the coupling between the two devices be reliable so that the robotic system acts correctly at the actual position of the target point.

[0005] There is a need to improve systems configured for controlling robotic systems based on intention detection. Summary of the invention

[0006] One embodiment provides a control system comprising: - an intention detection device comprising a first camera operated by a user and configured to estimate first coordinates of a first point of fixation of the gaze, in an environment, of a user in a first frame associated with the image of the first camera; - a robotic system comprising a second camera configured to determine, on the basis of the first estimated coordinates of the first point in the first frame, the second coordinates of a second point, in a second frame associated with the image of the second camera, of the user's gaze fixation point, the robotic system further comprising a robotic system configured to perform an action towards a point in the environment, on the basis of the second coordinates in the second frame.

[0007] According to one embodiment, the above control system further comprises at least one projection device configured to point to the point in the environment whose position corresponds to the second coordinates in the second frame of reference, the action being carried out on the basis of a validation, by the user, of the pointed point.

[0008] According to one embodiment, the projection device includes at least one laser pointer.

[0009] According to one embodiment, the robotic system is further configured to correct a servo error between the target point and the pointed point.

[0010] According to one embodiment, the robotic system is further configured to estimate third coordinates of a third point, in a third frame associated with the robotic system, on the basis of the second coordinates of the second point, the position of the point at which the action is carried out corresponding to the third coordinates, in the third frame.

[0011] According to one embodiment, the first reference frame is movable relative to the second reference frame.

[0012] According to one embodiment, the above control system is further configured: - to determine one or more characteristic points in a neighborhood of the first point on the image of the first camera, each characteristic point being associated with one or more characteristic measurements; and - determine the coordinates in the second frame, associated with the image of the second camera, of the characteristic point(s) on the basis of the characteristic measurement(s).

[0013] According to one embodiment, the above control system is configured to determine the second coordinates of the second point in the second frame further on the basis of the coordinates of the characteristic point(s) in the first frame and the coordinates of the characteristic point(s) in the second frame.

[0014] According to one embodiment, the above control system further includes a projection device configured to project one or more symbols, each symbol identifying an action to be performed, the robotic system being configured to perform a first action identified by a first symbol from among one or more symbols, based on a selection, by the user, of the first symbol.

[0015] According to one embodiment, the selection by the user of the first symbol is carried out via the intention detection device.

[0016] According to one embodiment, the robotic system is configured to: - perform object recognition based on the image recorded by the second camera and the second coordinates of the second point; - to command the projection device to project one or more symbols associated with the recognized object and identifying one or more actions associated with the object recognized.

[0017] According to one embodiment, object recognition based on the image recorded by the second camera and the second coordinates of the second point is performed by a neural network included in the robotic system.

[0018] According to one embodiment, the intention detection device is an eye tracking device comprising a plurality of eye cameras, the first camera being a front camera, the eye tracking device being configured to: - record, for a period of time and via the plurality of eye cameras, a plurality of fixation points of the user's gaze; - determine a plurality of coordinates corresponding to the coordinates of each point of the plurality of fixation points in the first frame of reference associated with the image of the first camera; and - determine the first coordinates of the first point (310), in the image of the first camera, by calculating the average of the plurality of coordinates.

[0019] According to one embodiment, the second camera is an RGB camera or an RGBD camera.

[0020] According to one embodiment, the robotic device is an exoskeleton or an extra limb. Brief description of the drawings

[0021] These features and advantages, as well as others, will be described in detail in the following description of particular embodiments, given by way of non-limiting example, in relation to the accompanying figures, among which:

[0022] Fig. 1A illustrates an example of a robotic system controlled by an eye-tracking device;

[0023] Fig. 1B illustrates another example of a robotic system controlled by an eye-tracking device;

[0024] [Fig.2] is a flowchart representing steps in a process for controlling a robotic system, according to an embodiment of the present description;

[0025] [Fig.3A] is a flowchart representing steps for estimating a user's gaze fixation point, according to one embodiment of the present description;

[0026] [Fig.3B] illustrates an example of the use of an eye-tracking device in performing fixation point estimation;

[0027] [Fig.4A] is a flowchart representing steps for pairing between the eye tracking device and the robotic system, according to an embodiment of the present description;

[0028] [Fig.4B] illustrates an example of a region of interest used for pairing between the eye-tracking device and the robotic system;

[0029] [Fig.5A] is a flowchart representing steps for estimating the position of the fixation point by the robotic system, according to an embodiment of the present description;

[0030] [Fig.5B] illustrates an example of estimating the position of the fixation point;

[0031] [Fig.6A] is a flowchart representing steps in estimating the position of the fixation point in the user's environment, according to an embodiment of the present description;

[0032] Fig. 6B illustrates an example of the steps in estimating the position of the fixation point in the user's environment;

[0033] [Fig.7A] is a flowchart representing steps for visualizing the estimated position of the point in the environment, according to an embodiment of the present description;

[0034] [Fig. 7B] illustrates an example of the correction of a control error; and

[0035] [Fig. 8] illustrates an example of abbreviations identifying actions that can be performed by the robotic system. Description of the implementation methods

[0036] The same elements have been designated by the same reference numerals in the different figures. In particular, the structural and / or functional elements common to the different embodiments may have the same reference numerals and may have identical structural, dimensional and material properties.

[0037] For the sake of clarity, only the steps and elements useful for understanding the described embodiments have been shown and are detailed. In particular, the operation of intention detection devices, such as eye-tracking devices like eye-tracking glasses or eye-tracking bars, is not described in detail.

[0038] Unless otherwise specified, when referring to two elements connected together, this means directly connected without intermediate elements other than conductors, and when referring to two elements coupled together, this means that these two elements can be connected or linked through one or more other elements.

[0039] In the following description, when reference is made to absolute position qualifiers, such as the terms "front", "back", "top", "bottom", "left", "right", etc., or relative position qualifiers, such as the terms "above", "below", "superior", "inferior", etc., or to orientation qualifiers, such as the terms "horizontal", "vertical", etc., reference is made, unless otherwise specified, to the orientation of the figures.

[0040] Unless otherwise specified, the expressions "approximately", "roughly", and "on the order of" mean to within 10% or 10°, preferably to within 5% or 5°.

[0041] Figure 1A illustrates an example of a control system 100 for a robotic system 102. The system 100 comprises the robotic system 102 and an eye-tracking device 104. By way of example, the eye-tracking device 104 is integrated into glasses worn by a user 106. In another example, the eye-tracking device 104 is contained within a bar to be placed horizontally on the forehead and above the eyes of the user 106. In particular, the eye-tracking device 104 includes, for example, a front-facing camera 105 directed towards an environment viewed by the user 106. The eye-tracking device 104 further includes an eye-tracking system configured to determine the position, in the images recorded by the front-facing camera 105, of a point targeted 108, or fixed, by the user 106 in the environment, i.e., a fixation point. from the user's perspective. In the example illustrated by [Fig.[lA], point 108 is a point on the upper rim of a cup.

[0042] The robotic system 102 includes a camera 110, fixed in the robotic system's frame of reference, directed towards the environment. For example, the camera 110 is an RGB (Red Green Blue) color camera. In another example, the camera 110 is an RGB-D (Red Green Blue - Depth) color camera further configured to provide a depth map characterizing the environment. However, since the eye-tracking device 104 is mobile, for example, with the head of the user 106, the images recorded by the front camera 105 and the camera of the robotic system 102 are not identical, because the camera 105 and the camera of the robotic system are separated from each other by a certain distance. In particular, this distance is unknown and generally variable.Furthermore, the eye-tracking device 104, and in particular the front camera 105, is mobile relative to the robotic system 102. The viewing angle of the two cameras 105 and 11 therefore also differs.

[0043] In the example illustrated by [Fig.1A], the robotic system 102 includes an additional arm 112. By way of example, the additional arm 112 is a motorized device, configured to perform one or more actions towards the point 108. By way of example, the actions that can be performed by the additional arm 112 are to push and / or grasp an object, although other actions are possible.

[0044] The system 100 further includes one or more pointers 114 included in the robotic system 102. Each of the pointers is, for example, a projection device configured to emit light, and corresponds, for example, to a laser pointer.

[0045] According to one embodiment, the eye-tracking device 104 is configured to estimate a point in the environment that is fixed by the gaze of the user 106. In particular, the eye-tracking device 104 is configured to estimate a point in the image of the front camera 105. By way of example, this estimation is performed based on the images recorded by the front camera 105 and on the eye-tracking of the user 106, recorded by the eye-tracking device 104. For example, eye-tracking techniques based on the movement of one or both eyes of a user are known to those skilled in the art and will not be described in detail. The estimated point is then defined by coordinates in a frame of reference associated with the eye-tracking device 104. In other words, the estimated point is then defined by coordinates in the image recorded by the front camera 105.

[0046] According to one embodiment, the system 100 is then configured to convert the coordinates of the estimated point into coordinates in a frame of reference associated with the robotic system 102, in particular in the image recorded by the camera 110. The system 100 is then configured to convert the coordinates in the image of the camera 110 into coordinates in the environment of the robotic system 102 and thus of the user. Once the position of the point in the environment is determined, the system 100 is, for example, configured to point or project a light, a symbol, etc., via the pointer(s) 114, a point in the environment to the estimated position of the target point 108. By way of example, an action is performed by the auxiliary arm 112, towards the point pointed to in the environment, following validation by the user 106.For example, when the estimated position is incorrect, or not suitable for the user 106, the latter, for example via the eye-tracking device 104, has the possibility of invalidating the projected point, for example by means of an action detectable by the eye-tracking device 104. In this case, the system 100 is configured for example to perform a new estimation.

[0047] By way of example, the system 100 further includes a processing unit 115 (CPU, from the English "Central Processing Unit") configured to perform image processing operations to locate, in the image of the camera 110, the target point 108 from the image recorded by the front camera 105 and from eye-tracking data acquired by the eye-tracking device 104. In another example not shown, the eye-tracking device 104 includes a processing unit configured to process the data it acquires, and the robotic system 102 further includes another processing unit configured to receive data, such as images, processed by the eye-tracking device 104, and to estimate, from this data, the position in the images recorded by the front camera 105, from point 108 in the frame of the robotic system 102 and therefore the environment.

[0048] By way of example, the robotic system 102 includes a support 116 on which are fixed the camera 110, the auxiliary arm 112 and a motorized turret on which are fixed the pointers 114. By way of example, the camera 110 is fixed on the support 116 so as to be at the eye level of the user 106. In another example, at least one pointer 114 is fixed to the wrist of the auxiliary arm.

[0049] Figure 1B illustrates a control system 100' similar to system 100 except that a motorized exoskeleton 116 replaces the auxiliary arm 112. In this example, system 100' is, for example, configured to guide the user's hand 106, via the exoskeleton 118. In this example, the exoskeleton 118 allows the user 106 to perform an action, such as grasping, pushing, etc.

[0050] By way of example, the eye-tracking device 104 further includes a gyroscope (not shown in [Fig. 1A] and [Fig. 1B]). The validation, or invalidation, of the position pointed to, or projected, by the pointer 114 is carried out, for example, on the basis of recognition of head movements, for example, vertical or lateral movements, recorded by the gyroscope. In other examples, the validation or invalidation is carried out on the basis of eye movements detected by the eye-tracking device 104.

[0051] Figure 2 is a flowchart representing steps in a method for controlling the robotic system 102, according to an embodiment of the present description. The steps in Figure 2 are, for example, carried out by the eye-tracking device 104 and the robotic system 102, or by the eye-tracking device 104, the robotic system 102, and the processing unit 115.

[0052] In a step 201 (FOCUSING POINT ESTIMATION), the position of the point targeted by the user's gaze 106 is estimated by means of the eye-tracking device 104. For example, images recorded by the front camera 105, as well as eye-tracking data recorded by the eye-tracking device 104, are provided to the processing unit 115. The processing unit 115 is then configured to estimate the position of point 108 in one or more of the images captured by the front camera 105. For example, the processing unit 115 is configured to determine the coordinates of a first point in one or more of the images recorded by the camera 105. The first point corresponds, for example, to the estimated position of point 108 in the image(s) recorded by the front camera 105.In another example, the eye-tracking device 104 is configured to estimate, itself, the position of point 108 in the image of the front camera 105.

[0053] In step 202 (AREA OF INTEREST & PAIRING), a region of interest is determined, for example by the processing unit 115. Step 202 is described in more detail in relation to [Fig. 4A]. The region of interest corresponds, for example, to an area including the first point estimated in step 201. At least three characteristic points in the region of interest are identified. By way of example, characteristic points are easily recognizable points. As an example, these points are associated with color changes, contours, etc., in the image from the front camera 105. In another example, the region of interest and the characteristic points are determined by the eye-tracking device 104 or by the robotic system 102. The processing unit 115, or the robotic system 102, is further configured, for example, to match the image recorded by the front camera 105 with the image recorded by the camera 110.In particular, in order to match the two images, the processing unit 115 is configured, for example, to search in the image from camera 110 for points whose characteristics are similar to those of the characteristic points 407 in the image from the front camera 105.

[0054] In a step 203 (FOCUSING POINT ESTIMATION), the processing unit 115, or the robotic system 102, is configured to determine the coordinates of a second point in the image recorded by the camera 110, the second point corresponding to an estimation of the point 108 in the image recorded by the camera 110. In particular, the coordinates of the second point in the image of the camera 110 are determined on the basis of the pairing performed between the images of the cameras 105 and 110.

[0055] In a step 204 (FIND 3D POINT), the processing unit 115, or the robotic system 102, is configured, for example, to determine the coordinates of a third point in the coordinate system associated with the robotic system 102, the third point corresponding to an estimate of the position of point 108 in the environment. For example, if the camera 110 is an RGB-D camera, this step is performed based on the coordinates of the second point in the image from camera 110 and a depth map of the environment recorded by camera 110.

[0056] In step 205 (VISUALIZATION), the position of the third point in the environment is, for example, indicated by the pointer(s) 114. In another example, an image, such as a symbol, is projected toward the position of the third point. In yet another example, the position of the third point in the environment is indicated by one of the laser pointers 114, and one or more symbols are projected, for example, near the third point. By way of example, the projected symbol(s) represent actions that can be performed by the robotic system 102.

[0057] In step 206 (USER VALIDATION), user 106 validates, or invalidates, the pointed-to position. For example, validation or invalidation is performed by via the eye-tracking device 104. For example, the eye-tracking device 104 is configured to detect a validation or invalidation action performed by the user. For example, when the user 106 validates the position of the pointed-to point, the robotic system 102 is configured to perform an action toward the point. For example, when the robotic system 102 includes an auxiliary arm 112, the robotic system 102 is configured to move the auxiliary arm 112 toward the pointed-to point in the environment. For example, based on a request by the user 106, the auxiliary arm 112 is configured to perform an action. In the example illustrated by Figures IA and IB, the action could be to grasp the cup, push the cup in a direction, etc.

[0058] In the event that user 106 invalidates the pointed position in the environment, for example because the pointed position does not correspond to the position targeted by user 106 when carrying out step 201, the process resumes, for example, in a new embodiment of the piloting process from step 201.

[0059] Figures 3A to 7B illustrate in more detail the achievements of steps 201 to 205.

[0060] Figure 3A is a flowchart representing steps for estimating a The focal point of a user's gaze, according to one embodiment of this description. In particular, [Fig. 3A] is a flowchart representing substeps performed during the execution of step 201 of [Fig. 2]. Specifically, the execution of step 201 includes, for example, the execution of substeps 301, 302, and 303.

[0061] Fig. 3B illustrates an example of the use of the eye-tracking device 104 in carrying out substeps 301, 302 and 303.

[0062] In a substep 301 (FOCUS POINT SAVING) of [Fig. 3A], the eye-tracking device 104 is configured, for example, to record fixation points of the user's gaze. By way of example, this recording is performed by a plurality of eye cameras 304 of [Fig. 3B]. By way of example, each lens of the eye-tracking device 104 comprises a plurality of eye cameras 304, for example, arranged on its inner periphery. The plurality of eye cameras 304 on the right lens is, for example, configured to record the movement of the user's right eye 306, and the plurality of eye cameras 304 on the left lens is, for example, configured to record the movement of the user's left eye 308.The pair of eye-tracking cameras 304 allows a fixation point to be determined, for example defined by the direction of the user's gaze 106, and in particular by the right and left eye poses. As an example, eye movements are recorded by the eye-tracking device 104 for a period of time between one second and five seconds, for example for two seconds. Thus, during the recording time period, a plurality of fixation points are recorded.

[0063] In a substep 302 (PROJECTION POINT) of [Fig. 3A], the fixation points determined in step 301 are, for example, projected onto the image recorded by the front camera 105. An example of such an image is shown in Figure 310 of [Fig. 3B]. By way of example, steps 301 and 302 are carried out in parallel. For each determination of a fixation point, it is projected onto the image recorded by the front camera 105 at that time. Another fixation point is determined and projected onto the current image of the front camera 105. Each fixation point is then characterized by coordinates on the image plane of the front camera 105.

[0064] In a substep 303 (MEAN COMPUTATION) of [Fig. 3A], the coordinates of a mean point 310 in the image from the front camera 105 are determined, for example. For instance, the coordinates of the mean point 310 correspond to the average coordinates of all the recorded fixation points. For instance, the coordinates of the mean point 310 are coordinates on the last image acquired by the front camera 105 during recording. In the example shown in [Fig. 3B], the last recorded image is an image 312. The image 312 and the coordinates of the point 310 are then saved, for example, by the eye-tracking device 104 and / or by the processing device 115. In the example shown in [Fig. 3B], the mean point is located on the image of a beverage can placed on a table.In other examples, instead of determining the coordinates of the mean point on the last recorded image, the coordinates are recorded on the first image, or on any other image acquired during recording.

[0065] The execution of substeps 301, 302 and 303 makes it possible, for example, to prevent the oscillation of the gaze of the user 106.

[0066] By way of example, the front camera 105 is a 50Hz camera. Thus, for a recording duration of 2 seconds, the mean point 310 is obtained from 100 fixation points projected onto the 100 recorded images. In this example, the position of the mean point is calculated on the 100th image of the recording.

[0067] By way of example, substeps 301, 302, and 303 are performed by the eye-tracking device 104. In another example, the eye-tracking device 104 is configured to provide the processing device 115 with the recordings made by both the eye cameras 304 and the front camera 105. In this example, the processing device 115 is configured to determine the coordinates of each fixation point and to project them onto each associated image acquired by the camera front camera 105, and to calculate the coordinates of the average point 310 in the last image recorded by the front camera 105.

[0068] By way of example, the eye-tracking device 104 is further configured to detect, for example by means of a gyroscope, any movement of the user's head 106 that would distort the estimation of the mean point. For example, following any head rotation with an angle greater than a threshold angle, for example between 0.5 and 2°, the eye-tracking device 104 is configured to reset the fixation point recording.

[0069] Figure 4A is a flowchart representing the steps for estimating a fixation point, according to one embodiment of this description. In particular, Figure 4A is a flowchart representing the substeps carried out during the execution of step 202 of Figure 2. In particular, the execution of step 202 includes, for example, the execution of substeps 401, 402, and 403 of Figure 4A.

[0070] Fig. 4B illustrates an example of a region of interest used for pairing between the eye-tracking device 104 and the robotic system 102 during the execution of step 202.

[0071] Following the completion of step 201 of [Fig.2], and in the case where the eye tracking device 104 is configured to carry out steps 301, 302 and 303 of [Fig.3A], the image 312 and the coordinates defining the position of the mean point 310, in the image 312, are for example sent to the processing unit 115 or to the robotic system 102.

[0072] In a substep 401 (AREA OF INTEREST DEFINITION), a region of interest around the mean point in the transmitted image is, for example, determined by the processing device 115 or by the robotic system 102. By way of example, the region of interest is a predetermined surface including the mean point. In another example, the region of interest is determined according to the image and the mean point. By way of example, the processing device 115 or the robotic system 102 is configured to determine whether the mean point is contained within a surface delimited by a contour and to define the region of interest as a sub-image of the transmitted image, including the object, or at least a part of the object including the mean point, delimited by this contour.

[0073] In the example of [Fig.4B], the region of interest is a part 405 of the image 312. The region of interest 405 is a square-shaped sub-image of the image 312 and includes the image of the can on which the mean point 310 is located.

[0074] In a substep 402 (CHARACTERISTIC POINTS), the processing unit 115, or the robotic system 102, is configured, for example, to determine feature points belonging to the region of interest. The feature points are represented by descriptors that allow for the identification of these points from one image to another. For example, the variation in intensity gradient between neighboring pixels is a descriptor. For example, the processing device 115, or the robotic system 102, is configured to implement one or more extraction methods such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Feature), or ORB (Oriented FAST and Robust BRIEF) to determine characteristic points in the region of interest.

[0075] In the example illustrated by [Fig.4B], the characteristic points extracted from the sub-image 405 are points 407. As an example, the points 407 are all located on the image of the can and correspond, for example, to points on a logo, a text, an image, on the can.

[0076] In a substep 403 (PAIRING) of [Fig. 4A], the processing unit 115, or the robotic system 102, is configured, for example, to search for the feature points extracted in step 402 in the image recorded by the camera 110. By way of example, geometric and / or statistical filters are applied to the image from the camera 11, for example, to eliminate outliers. The feature points are searched for in the image from the camera 110 based on their characteristics, or descriptors, such as a color profile, etc.

[0077] When a feature point from the region of interest in the image of the front camera 105 is found in the image of the camera 110, its position in the images of cameras 105 and 110 is, for example, matched. As an example, the positions of the successfully matched points in the image of camera 105 and in the image of camera 110 are stored respectively in a first and a second matrix. For example, the first and second matrices are of size 3 x n, where n is the number of feature points found in the image of camera 110. Thus, two columns of the first matrix contain the coordinates of a feature point in the image of camera 105, for example, in the last image recorded during step 301. Two columns of the second matrix then contain the coordinates of a feature point in the image of camera 110.For example, the first row of the first matrix and the first row of the second matrix contain the x-coordinates of the characteristic points. The second row of the first matrix and the second row of the second matrix contain the y-coordinates of the characteristic points. For example, the third row of the first matrix and the third row of the second matrix consist of only one value, for example, only the value 1. In particular, for an integer less than or equal to n, the coordinates stored in the jth columns of the first and second matrices are those of the same characteristic point.

[0078] In the example illustrated in [Fig. 4B], all the characteristic points 407 extracted from the sub-image 405 are found in an image 409 of the camera 110. The characteristic points in the image 409 are represented by points 411. By way of example, the coordinates of the points 407 are stored in a plml matrix and the coordinates of the points 411 are stored in a plm2 matrix. The pairing corresponds to the matching of the points in the two matrices plml and plm2.

[0079] Figure 5A is a flowchart representing substeps for estimating the position of the fixation point by the robotic system 102, according to an embodiment of this description. In particular, Figure 5A is a flowchart representing substeps performed during step 203 of Figure 2. Specifically, the execution of step 203 includes, for example, the execution of substeps 501, 502, and 503.

[0080] Fig. 5B illustrates an example of a position of the fixation point estimated by the robotic system 102 during the execution of step 203.

[0081] In a substep 501 (HOMOGRAPHIC MATRIX) of [Fig. 5A], a homographic matrix is, for example, calculated by the processing unit 115 or by the robotic system 102, based on the first and second matrices. The homographic matrix then represents the geometric transformation between the views of cameras 105 and 110. In another example, an essential matrix, instead of a homographic matrix, is calculated.

[0082] In relation to the example illustrated by figures 4B and 5B, a homographic matrix is ​​the matrix H such that plml = H*plmï.

[0083] In a step 502 (PROJECTION ON REP2) of [Fig.5A], the processing device 115, or the robotic system 102, is configured to determine the coordinates of the mean point 310, in the image of the camera 110.

[0084] In relation to the example illustrated by Figures 4B and 5B, the Pr2 coordinates of a point 504 represent the coordinates of the mean point 310 in the image frame of the camera 110. The Pr2 coordinates of point 504 are, for example, obtained from the matrix H. In particular, the Pr2 coordinates are such that Pr2 = H * Prl, where Prl represents the coordinates of the mean point 310 in the image frame of the front camera 105. Point 504 then corresponds to an estimate of the point aimed at by the user 106 projected onto the image 409 acquired by the camera 110.

[0085] In the example where the camera 110 is not configured to acquire a depth image, a substep 503 (LINE COMPUTING) of [Fig. 5A], performed by the processing unit 115 or the robotic system 102, comprises determining the coefficients of a line perpendicular to the focal plane of the camera 110 and passing through point 504 in the coordinate system associated with the camera 110. This line allows the depth to be estimated, where is the distance between the camera 110 and point 504. When the camera 110 is configured to establish a depth map, the distance to point 504 is already estimated during image acquisition by camera 110, and sub-step 503 is then, for example, omitted.

[0086] Figure 6A is a flowchart representing substeps in estimating the position of the fixation point in the user's environment, according to an embodiment of this description. In particular, Figure 6A is a flowchart representing substeps performed during the execution of step 204 of Figure 2. Specifically, the execution of step 204 includes, for example, the execution of substeps 601 and 602 in the case where the camera 110 is an RGB-D camera.

[0087] Fig. 6B illustrates an example of estimating the position of the fixation point in the environment of user 106. In particular, in the example illustrated by Fig. 6B, the camera 110 used is an RGB-D camera.

[0088] In the example where the camera 110 is not configured to acquire a depth image, a substep 601 (INTERSECTION) of [Fig.6A], carried out by the processing device 115 or the robotic system 102, includes the determination of the coordinates of the point at the intersection between the line determined in substep 403 and a point cloud of the depth map acquired by the camera 110.

[0089] Fig. 6B illustrates an example of a 603 depth map acquired by camera 110, when camera 110 is an RGB-D camera.

[0090] In a substep 602 (PROJECTION ON REP_ROBOT) of [Fig. 6A], the processing unit 115 or the robotic system 102 is, for example, configured to project the coordinates of the point determined in substep 601 onto the control frame of the robotic system 102. For example, this projection is performed by applying a transformation matrix to the coordinates of the point determined in substep 601. This transformation matrix is, for example, predetermined during the installation or calibration of the system 100 or 100'. For example, this matrix is ​​stored in a memory of the processing unit 115 or the robotic system 102. The resulting coordinates then correspond to the position, in the frame associated with the robot, of an estimate of the point targeted by the user 106.In particular, the frame of reference associated with the robot, for example associated with the additional arm 116 or the exoskeleton 118, is a frame of reference describing a 3-dimensional space, unlike the frames of reference of the images acquired by the cameras 105 and 110 which are two-dimensional frames of reference.

[0091] As an example, coordinates Xr= Rrobot*Pr3, where Rrobot is the transition matrix and Pr3 are the 3-dimensional coordinates of the point at the intersection of the line from point 504 and the point cloud of image 603 of [Fig.6B], define the position in the frame 605 of the robot, of an estimate 607 of the point targeted 108 by the user 106.

[0092] In the case where the camera 110 is an RGB camera and not an RGB-D camera, the coordinates of the point in the robot's frame are obtained, for example, by another stereovision system, i.e., via an additional RGB camera or an additional projection device. In another example, the coordinates of the point in the robot's frame are obtained by a triangulation calculation between a laser pointer included in the pointer(s) 114. In this example, the system is pre-calibrated to allow the triangulation calculation to be performed. A two-dimensional visual servoing step in the image space of the camera 110 is then carried out to converge the laser pointer either towards the fixation point 504 of the image of the fixed camera 110, or towards the fixation point 310 of the image of the front camera 105.

[0093] Figure 7A is a flowchart representing substeps for visualizing the estimated position of the point in the environment, according to one embodiment of this description. In particular, Figure 7A is a flowchart representing substeps carried out during the execution of step 204 of Figure 2. In particular, the execution of step 204 includes, for example, the execution of substeps 701 and / or 702.

[0094] Fig. 7B illustrates an example of the correction of a control error carried out during the execution of step 205, and more particularly during the execution of substep 702.

[0095] In a substep 701 (LASER), the robotic system 102 is, for example, configured to orient the pointer(s) 114 towards the point in the environment whose coordinates in the frame of reference of the robotic system 102 were calculated in the embodiment of step 204. When the camera 110 is an RGB-D camera, the robotic system is, for example, configured to orient the pointer(s) 114 towards the point 607 at the coordinates Pr3 obtained in the embodiment of substep 602. The orientation of the pointer(s) 114 is, for example, achieved on the basis of a Cartesian servo control of the turret on which the pointer(s) 114 are fixed. This type of servo control is well known to those skilled in the art.

[0096] In a substep 702 (VISUAL SERVOING TASK), visual servoing of the turret 116 in the image from the camera 110 is, for example, performed by the processing unit 115 or by the robotic system 102. In particular, substep 702 is omitted when the estimation of the coordinates of point 607 during the execution of substep 602 is based on the camera 110 and when it is an RGB-D camera. By way of example, substep 702 is only performed when the Pr3 coordinates are not known, or are estimated with low accuracy, for example, in the case where the camera 110 is an RGB camera, without a depth component. A servo law is, for example, implemented in the image of the Camera 110. The control law allows, for example, for zero statistical error control between the point targeted during substep 701 and the point determined during substep 602. Visual control allows, for example, the piloting of turret 116 by moving the position of the point targeted during substep 701. As an example, a control law is defined by: [Math 1] s) (Prl-Ptl} where ^116 is the velocity vector of the axes of turret 116, J is the Jacobian matrix of turret 116, C[(s) is the controller, and where Pr2 and P12 are respectively the coordinates of the point determined in step 502 of [Fig. 5A] and the coordinates of the point projected in the image of camera 110. For example, the projected point is a bright and easily detectable laser point. The controller is, for example, a proportional controller or a proportional-integral controller. For example, the controller C^s) is a proportional controller when an integral term is present in the velocity controller of the robotic device 102. Otherwise, the controller is, for example, a proportional-integral controller to ensure zero static error. The robotic system 102 is then configured, for example, to correct the orientation of the laser pointer following visual servoing.The laser pointer then points to a point at a corrected position in the environment. In another example, the control is achieved in the same way, but in the image from the front camera 105.

[0097] In the example illustrated in [Fig. 7B], the point 108 targeted by the user is located on the handle of a cup. Following the implementation of steps 201 to 205 of [Fig. 2], the robotic system 102 orients, for example, a laser pointer so that it projects a point 703 onto the cup. If the three-dimensional coordinates Pr3 of point 607 are not known, the control error then corresponds to the distance 705 between these two points. Following correction of the control error, a corrected point 707 is pointed to by the laser pointer.

[0098] User 106 validates, or invalidates, the position of point 607 or corrected point 707 by an action, and the processing unit 115 or robotic system 102 is, for example, configured to detect the action, for example, via the eye-tracking device 104 with or without a gyroscope. For example, to validate the position of point 607 or 707, user 106 blinks their right or left eye. In another example, to validate the position of point 607 or 707, user 106 tilts their head to the right, to the left, or forward. For example, to invalidate the position of point 607 or 707, user 106 close your eyes for a period of time, for example, 2 or 3 seconds. For example, when the three-dimensional coordinates Pr3 of point 607 are unknown, the pointer is configured to project a first pattern before the servo error correction, and then to project a second pattern once the servo error is corrected. This way, user 106 knows whether the pointed-to point is corrected or not. For example, the pattern projected before correction is a near-infrared pattern. Therefore, user 106 does not see the point before the servo error correction.

[0099] In one example, the robotic system 102 is further configured to display one or more symbols. For example, the symbol(s) are projected at the position of point 607 or 707 before user confirmation or rejection. In another example, the symbol(s) are projected after user confirmation 106. In one example, the symbol(s) are projected in place of point 607 or 707. In another example, when point 607 or 707 is confirmed, the symbol(s) are projected into the environment. For example, the robotic system 102 is configured to recognize a surface in the image from camera 110, which, for example, has a uniform color, and to project the symbol(s) onto it.

[0100] The acronym(s) represent, for example, actions that can be carried out by the robotic system 102, for example by the motorized auxiliary arm or by the motorized exoskeleton.

[0101] Fig. 8 illustrates an example of a set of acronyms identifying actions that can be performed by the robotic system 102.

[0102] By way of example, symbols 801 to 806 illustrate pushing actions and symbols 807 to 812 illustrate grasping actions. By way of example, symbols 801 to 806 are projected in a first color and symbols 807 to 812 are projected in a second color, different from the first color. Symbols 801 and 807 represent, for example, a pushing and grasping action, respectively, performed from the back of the stitch. Symbols 802 and 808 represent, for example, a pushing and grasping action, respectively, performed from the front of the stitch. Symbols 803 and 809 represent, for example, a pushing and grasping action, respectively, performed from the bottom of the stitch. Symbols 804 and 810 represent, for example, a pushing and grasping action, respectively, performed from the left of the stitch. The abbreviations 805 and 811 represent respectively, for example, a pushing and grasping action, performed from above the point.The symbols 806 and 812 represent, for example, a pushing and grasping action respectively, performed from the right of the point.

[0103] By way of example, the symbols 801 to 806 are projected one after the other. In one example, for each symbol projected, the user validates or invalidates the action, and the processing unit 115 or the robotic system 102 is configured, for example, to detect The action, for example, is performed via the eye-tracking device 104. For instance, each symbol is projected for a set period of time, for example, 3 to 5 seconds, and if the user 106 does not confirm the identified action, the next symbol is projected. In yet another example, the user 106 can scroll through the symbols, for example, by moving their head. For example, the user 106 scrolls through the symbols by moving their head, for example, to the right to advance, or to the left to go back, or vice versa. The eye-tracking device 104 then includes a gyroscope to detect head movements. In yet another example, the robotic system 102 or the eye-tracking device 104 includes a microphone, and the confirmation or rejection of an action is carried out based on a voice command by the user 106.In yet another example, the validation or invalidation of the actions to be performed, and / or the position of the corrected point, is carried out via a joystick manipulated, for example, by the mouth, or via a push button, or via a touch sensor located on the user's puck 106. In yet another example, the validation or invalidation of the actions to be performed and / or the corrected position is carried out by lip reading. In this example, the system 100 or 100' includes a camera configured for lip reading. In yet another example, the validation, or invalidation, of the actions to be performed and / or the corrected position is carried out via information from facial deformation sensors.

[0104] By way of example, the processing device 115 or the robotic system 102 is configured to determine the nature of the object designated by gaze, by a voice command, or by a reference text or image. By way of example, the processing device 115 or the robotic system 112 is configured to perform image recognition operations, for example, based on a database stored on a server or in the memory of the processing device 115 or the robotic system 102. In another example, the recognition of the type of the pointed-to object is performed based on a neural network implemented in the processing device 115 or the device 102. By way of example, for each type of object that can be recognized, one or more actions are associated. The robotic system 102 is then configured to project one or more actions that can be performed depending on the recognized object.For example, if a cup or a can is recognized, one suggested action is to pick it up. As an example, image recognition is performed on the image from camera 110 and / or on the image from the front camera 105 in parallel with at least one step 201 to 206.

[0105] By way of example, the pointer(s) 114 include an image projector for projecting information about the action to be performed. For example, the result of the action is projected.

[0106] An advantage of the described embodiments is to provide a robust method for estimating a point or object chosen by a user, through an intention detection device, such as an eye-tracking device.

[0107] Another advantage of the described embodiments is that the user has the possibility to participate, by validating or invalidating the pointed point and choosing an action to perform.

[0108] Various embodiments and variations have been described. Those skilled in the art will understand that certain features of these various embodiments and variations could be combined, and other variations will become apparent to them. In particular, during steps 202, 203, and 205, the image used can be replaced by that of the front camera 105. In this case, steps 204 and 205, including substep 702, are reversed, and a step is added to locate the point pointed to in the image of the camera 110, for example, by detecting a projected point of light during step 205. The point pointed to is then the point determined during step 201. The eye-tracking device 104 then includes, for example, an inertial measurement unit to ensure that the eye-tracking device is horizontal.

[0109] In another variant, step 204 is omitted and a visual servoing step on the robot on the projected pattern is added following validation by the user 106. In this variant, the additional arm, or the exoskeleton includes for example a camera and the additional servoing step is carried out on the image of this camera.

[0110] Finally, the practical implementation of the embodiments and variants described is within the reach of a person skilled in the art, based on the functional indications given above.

Claims

Demands

1. Control system (100, 100') comprising: - an intention detection device (104) comprising a first camera (105) operated by a user (106) and configured to estimate first coordinates of a first gaze fixation point (310) in an environment of a user in a first frame associated with the image of the first camera; - a robotic system (102) comprising a second camera (110) configured to determine, on the basis of the first estimated coordinates of the first point in the first frame, second coordinates of a second point (504) in a second frame associated with the image of the second camera of the user's gaze fixation point, the robotic system further comprising a robotic system (116, 118) configured to perform an action towards a point in the environment, on the basis of the second coordinates in the second frame.

2. Control system (100, 100') according to claim 1, further comprising at least one projection device (114) configured to point to the point (607) in the environment whose position corresponds to the second coordinates in the second frame, the action being carried out on the basis of a validation, by the user, of the pointed point (703).

3. Control system (100, 100') according to claim 2 wherein the projection device (114) comprises at least one laser pointer.

4. Control system (100, 100') according to any one of claims 1 to 3, wherein the robotic system is further configured to correct a servo error (705) between the target point (108) and the point being pointed (607).

5. A control system (100, 100') according to any one of claims 1 to 4, wherein the robotic system (102) is further configured to estimate, from the third coordinates of a third point (607), in a third frame associated with the robotic system (116, 118), the position of the point at which the action is carried out corresponding to the third coordinates, in the third frame of reference.

6. Steering system (100, 100') according to any one of claims 1 to 5, wherein the first reference frame is movable relative to the second reference frame.

7. A control system (100, 100') according to any one of claims 1 to 6, further configured: - to determine one or more characteristic points (407) in a neighborhood (405) of the first point (310) on the image of the first camera (105), each characteristic point being associated with one or more characteristic measurements; and - to determine the coordinates in the second frame, associated with the image of the second camera (110), of the characteristic point(s) (411) on the basis of the characteristic measurement(s).

8. Control system (100, 100') according to claim 7, wherein the robotic system (102) is configured to determine the second coordinates of the second point (504) in the second frame further on the basis of the coordinates of the characteristic point(s) in the first frame and the coordinates of the characteristic point(s) in the second frame.

9. A control system (100, 100') according to any one of claims 1 to 8, further comprising a projection device (114) configured to project one or more symbols (801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812), each symbol identifying an action to be performed, the robotic system (116, 118) being configured to perform a first action identified by a first symbol from among one or more symbols, based on a selection, by the user, of the first symbol.

10. Control system (100, 100') according to claim 9, wherein the selection by the user of the first symbol is carried out via the intention detection device (104).

11. Control system (100, 100') according to claim 9 or 10, wherein the robotic system (102) is configured to: - perform object recognition on the basis of the image (409) recorded by the second camera (110) and the second coordinates of the second point (504); - command, to the projection device (114), the projection of one or more symbols (801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812) associated with the recognized object and identifying one or more actions associated with the recognized object.

12. Control system (100, 100') according to claim 11 wherein object recognition based on the image (409) recorded by the second camera (110) and the second coordinates of the second point (504) is carried out by a neural network included in the robotic system (102).

13. A control system (100, 100') according to any one of claims 1 to 12, wherein the intention detection device (104) is an eye-tracking device comprising a plurality of eye cameras (304), the first camera (105) being a front camera, the eye-tracking device being configured to: - record, for a period of time and via the plurality of eye cameras, a plurality of fixation points of the user's gaze; - determine a plurality of coordinates corresponding to the coordinates of each point of the plurality of fixation points in the first frame associated with the image of the first camera (115); and - determine the first coordinates of the first point (310), in the image of the first camera (105), by calculating the average of the plurality of coordinates.

14. Control system (100, 100') according to any one of claims 1 to 13, wherein the second camera (110) is an RGB camera or an RGBD camera.

15. Piloting system (100, 100') according to any one of claims 1 to 14, wherein the robotic device is an exoskeleton or an extra limb.

Images