A method for detecting and tracking the faces of individuals wearing glasses in a video stream.

Abstract

The present invention relates to a method for tracking a face (125) of an individual (120) in a video stream captured by an image capture device (130), the face being wearing eyeglasses (110). The tracking method includes a step of evaluating parameters of a facial representation including a model of glasses and a model of a face, such that said representation of the face is superimposed on an image of the face in a video stream, said parameters being evaluated with respect to a plurality of feature points of said representation of the face previously detected in an image of the video stream, referred to as the first image, and all or part of the parameters of the representations being evaluated by taking into account at least one proximity constraint between at least one point of the face model and at least one point of the model of glasses.

Description

[Technical Field]

[0001] The field of this invention is the field of image analysis.

[0002] More precisely, the present invention relates to a method for detecting and tracking the face of an individual wearing glasses in a video stream.

[0003] The present invention has found applications, in particular, for the virtual wearing of eyeglasses. The present invention has also found applications, in particular, for augmented reality, or uncombined miniaturized reality, for faces wearing eyeglasses, which become obscured in images of eyeglasses actually worn by an individual, with added elements consisting of lenses, jewelry, and / or structures. The present invention has also found applications for performing ophthalmic measurements (PD, monoPD, height, etc.) of eyeglasses actually or virtually worn by an individual. [Background technology]

[0004] Techniques that enable the detection and tracking of individual faces in video streams are known from prior art.

[0005] These techniques generally rely on detecting and tracking distinctive facial features, such as the corners of the eyes, nose, or mouth. The quality of face detection generally depends on the number and location of the feature points used.

[0006] These techniques are generally reliable when detecting and tracking the faces of individuals without accessories in video streams.

[0007] Such techniques are described in particular in the French patent published in French Patent Application Publication No. 2955409 of the present patent applicant company and in the international patent application published in International Publication No. 2016 / 135078.

[0008] However, the quality of face detection tends to decrease when an individual is wearing glasses with corrective lenses, because some feature points used during detection, generally the edges of the eyes, are typically distorted by the lenses built into the frame, or even masked if the lenses are tinted. Furthermore, even if the lenses are not tinted, the frame may mask some of the feature points used for detection. When some feature points are obscured or their position in the image is distorted, the faces represented and detected by the model may generally be out of position and / or orientation, or scaled incorrectly, relative to the actual face.

[0009] None of the current systems can meet all the requirements simultaneously; in other words, none offer a technique for tracking a face wearing actual glasses that is more accurate and robust to an individual's movements in order to provide an improved augmented reality representation. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] French Patent Application Publication No. 2955409 [Patent Document 2] International Publication No. 2016 / 135078 [Patent Document 3] International Publication No. 2013 / 139814 [Patent Document 4] International Publication No. 2018 / 002533 [Patent Document 5] International Publication No. 2019 / 020521 [Overview of the project]

[0011] The present invention aims to improve all or some of the aforementioned drawbacks of the prior art.

[0012] For this purpose, the present invention relates to a method for tracking a person's face in a video stream acquired by an image acquisition device, the face being wearing glasses, and the video stream including a plurality of successively acquired images.

[0013] The tracking method includes evaluating parameters of a face representation including a model of the glasses and a model of the face, and causing the representation of the face to be superimposed on an image of the face in the video stream.

[0014] According to the present invention, at least one proximity constraint between at least one point of the face model and at least one point of the glasses model is taken into account when evaluating all or part of the parameters of the representation.

[0015] As an example, the proximity constraint can be defined, for example, as the arms of the glasses being placed on the upper surface of the joint between the auricle and the skull, i.e., the helix.

[0016] In other words, the proximity constraint is defined between a zone of the face model and a zone of the glasses model, and the zones can be points such as surfaces or ridges, or sets of points.

[0017] Proximity means a distance of zero or less than a predetermined threshold, for example on the order of a few millimeters.

[0018] Therefore, using the proximity constraint during the evaluation of the parameters of the face representation makes it possible to obtain a more faithful pose of the face representation with respect to the camera with a limited number of calculations. Therefore, real-time tracking of the person can be performed more robustly with respect to unexpected movements of the person with respect to the image acquisition device.

[0019] Furthermore, by using both a glasses model and a face model, facial positioning can be improved, particularly compared to tracking faces without glasses. This is because, in the latter case, the position of temple feature points is generally inaccurate. By superimposing the glasses arms onto an individual's temples, it becomes possible to obtain more accurate information about feature points detected within the image zone containing the individual's temples. Therefore, tracking the glasses allows for a more accurate estimation of the pose of the facial representation.

[0020] Preferentially, the parameters of the representation include external values ​​for the representation of the face and internal values ​​for the representation of the face, the external values ​​including the 3D position and 3D orientation of the representation of the face relative to the image acquisition device, and the internal values ​​including the 3D position and 3D orientation of the model of the glasses relative to the model of the face, and the parameters are evaluated with respect to a number of previously detected feature points of the representation of the face in an image of a video stream called a first image, or in a set of images including the first image, which are acquired simultaneously by multiple image acquisition devices.

[0021] In other words, a facial representation, referred to as an avatar, includes external position and orientation parameters in a three-dimensional environment, as well as relative internal position and orientation parameters between the facial model and the eyeglasses model. Other internal parameters can also be added, such as the constituent parameters of the eyeglasses, including the frame type, frame size, and material. The constituent parameters may also include parameters related to the deformation of the eyeglasses frame, particularly the arms, when the eyeglasses are worn on an individual's face. Such constituent parameters may be, for example, the opening and closing angles of the arms relative to a reference plane, such as the principal or tangent plane of the eyeglasses on the face.

[0022] Facial representation includes three-dimensional models of the face and glasses.

[0023] In certain embodiments of the present invention, all or some of the parameters of the representation are updated with respect to the location of all or some of the tracked or detected feature points in a second image of a video stream or in a series of second images acquired simultaneously by multiple image acquisition devices, the set of second images including the second image.

[0024] Therefore, updating the representation parameters, particularly the relative position and orientation values ​​between the glasses model and the face model, as well as the configuration parameters, makes it possible to obtain more robust and accurate tracking of individual faces.

[0025] Advantageously, the second image or set of second images presents a view of the individual's face from a different angle than the first image or set of first images.

[0026] In certain embodiments of the present invention, when evaluating all or some of the parameters of the representation, at least one proximity constraint is also taken into consideration between a three-dimensional point of one of the models included in the facial representation and at least one point included in at least one image of the video stream, or a horizon line.

[0027] In certain embodiments of the present invention, when evaluating all or some of the parameters of the representation, at least one dimensional constraint of one of the models included in the representation of the face is also taken into consideration.

[0028] In certain embodiments of the present invention, the method includes the step of pairing two distinct points, each belonging to a model separate from the models included in the facial representation, or to a model separate from the models included in the facial representation.

[0029] Pairing two points makes it possible to constrain the distance relationship between them, particularly their proximity and known dimensions. Known dimensions include, for example, the interpupillary distance of a face, the width of a frame, the characteristics or average size of the iris, or any combination of these values ​​that follows one or more distribution rules centered on a known mean of one of these values.

[0030] In a particular embodiment of the present invention, this method includes a preliminary step of pairing a point from one of two models included in a facial representation with at least one point from an image acquired by an image acquisition device.

[0031] The pairing of model points with image points or sets of points such as contour lines is generally performed automatically.

[0032] In a particular embodiment of the present invention, during the evaluation of the parameters of the representation, the alignment of the glasses model with the image of the glasses in the video stream is performed sequentially with the alignment of the face model with the image of the face in the video stream.

[0033] In a particular embodiment of the present invention, the alignment of a face model is performed by minimizing the distance between the facial feature points detected in the face image and the feature points of the face model projected into the image.

[0034] In certain embodiments of the present invention, the alignment of the eyeglasses model is performed by minimizing the distance between at least a portion of the contour of the eyeglasses in the image and a similar contour portion of the eyeglasses model projected into the image.

[0035] In fact, it is emphasized that the model of the glasses is a 3D model. Therefore, a projection of this 3D model is performed on the image in order to determine a similar contour that is used to calculate the minimization of the distance to the contour of the glasses detected in the image.

[0036] In certain embodiments of the present invention, the representation parameters also include a set of configuration parameters for a face model and / or a set of configuration parameters for a glasses model.

[0037] The configuration parameters for a face model or eyeglasses model can be, for example, morphological parameters that characterize the shape and size of the face model or eyeglasses model, respectively. The configuration parameters may also include deformation features of the model, particularly in the context of eyeglasses, to account for deformations of the arms, deformations of the eyeglasses surface, and even deformations of the opening and closing of each arm relative to the front of the eyeglasses.

[0038] In the context of face models, configuration parameters may also include parameters related to the opening and closing of the eyelids or mouth, or parameters related to the deformation of the facial surface due to facial expressions.

[0039] In certain embodiments of the present invention, the parameters of the expression are as follows: - 3D position of facial expression, - Three-dimensional orientation of facial expression, - Size of the eyeglass model, - Size of the face model, - The relative three-dimensional position between the glasses model and the face model. - The relative three-dimensional orientation between the glasses model and the face model. - One or more parameters of the eyeglass model configuration, - One or more parameters of the facial model configuration, - Includes all or some of one or more camera parameters.

[0040] In a particular embodiment of the present invention, the tracking method - A step of detecting multiple points of a face in the first image of the video stream, - An initialization step of a set of parameters for a face model relating to the face image in the first initial image, - A detection step of multiple points on glasses worn on an individual's face in a second image of a video stream, called a second initial image, wherein the second initial image is after or before the first initial image in the video stream, or is identical to the first image in the video stream. - Includes the step of initializing a set of parameters for the eyeglasses model for the eyeglasses image in the second initial image.

[0041] In a particular embodiment of the present invention, the initialization of the parameters of the face model is performed by a deep learning method that analyzes all or some of the detected points of the face.

[0042] In a particular embodiment of the present invention, the deep learning method also determines the initial position of the face model in a 3D reference frame.

[0043] In a particular embodiment of the present invention, the tracking method also includes the step of determining the scale of an image of eyeglasses worn on an individual's face by the dimensions of the known size elements of the eyeglasses in the image.

[0044] In a particular embodiment of the present invention, the scale is determined by pre-recognizing the eyeglasses worn on the individual's face.

[0045] In a particular embodiment of the present invention, the image acquired by the second image acquisition device is used to evaluate the parameters of the representation.

[0046] In certain embodiments of the present invention, the model of the eyeglasses in representation corresponds to the pre-modeling of the eyeglasses, differing only in deformation.

[0047] Since the shape and size of the eyeglass model remain constant, it becomes possible to obtain a better solution in a shorter computation time.

[0048] The present invention also, - A step of acquiring at least one stream of images of an individual wearing glasses on their face using at least one image acquisition device, - A step of tracking an individual's face using a tracking method according to any one embodiment of the above-described embodiments, and tracking the position and orientation of the facial expression, - A step of modifying all or part of the image of an image stream or one of the image streams, called the main video stream, which is acquired by an image acquisition device or by one of the image acquisition devices called the main image acquisition device, by superimposing a facial representation of an individual's face in real time on the main video stream. - Also relating to augmented reality methods, including the step of displaying the previously modified main video stream on the screen.

[0049] It is emphasized that the steps of the augmented reality method are advantageously implemented in real time.

[0050] The present invention also relates to an electronic device including a computer memory that stores instructions for a tracking or augmented reality method according to any one embodiment of the embodiments described above.

[0051] Advantageously, the electronic device includes a processor capable of processing instructions of the method.

[0052] Other advantages, purposes, and specific features of the present invention will become apparent from the following non-limiting description relating to at least one specific embodiment of the device and method to which the present invention is an object, with reference to the accompanying drawings. [Brief explanation of the drawing]

[0053] [Figure 1] Figure 1 is a schematic diagram of an augmented reality device that implements an embodiment of the detection and tracking method according to the present invention. [Figure 2]Figure 2 is a block diagram of the detection and tracking method implemented by the augmented reality device shown in Figure 1. [Figure 3] Figure 3 shows the mask of the glasses (partial figure a) and the distribution of points of the mask contour according to category (partial figures b and c). [Figure 4] Figure 4 shows perspective views of a glasses model's face with and without an external envelope (partial figures b and a, respectively). [Figure 5] Figure 5 shows the regression step of the method in Figure 2, which involves extracting images acquired by an image acquisition device of the device in Figure 1 with a model of eyeglasses superimposed on it. [Figure 6] Figure 6 shows the placement constraints between the glasses model and the face model. [Figure 7] Figure 7 is a perspective view of a parametric model (3DMM) of eyeglasses. [Figure 8] Figure 8 is a simplified view of the surface of the parametric model in Figure 7. [Modes for carrying out the invention]

[0054] This description is given in a non-limiting manner, and each feature of the embodiment can be advantageously combined with any other feature of any other embodiment.

[0055] Please note that the drawings are not currently scaled accurately.

[0056] Examples of specific embodiments Figure 1 shows an augmented reality device 100 used by an individual 120 wearing glasses 110 on their face 125. The glasses 110 typically comprise a frame 111 including a front 112 and two arms 113 extending on either side of the individual's face 120. Furthermore, the front 112 allows for the holding of lenses 114, which are positioned inside two rims 115 configured within the front 112. Two pads (not shown in Figure 1) protrude and are fixed to the ends of the separate rims 115 so that they can rest on the nose 121 of the individual 120. When the glasses 110 are fitted to the individual's face 120, a bridge 117 connecting the two rims 115 straddles the nose 121.

[0057] Device 100 includes a main image acquisition device, in this case a camera 130, which acquires a plurality of sequential images that form a video stream displayed in real time on the screen 150 of Device 100. A data processor 140 included in Device 100 processes the images acquired by the camera 130 in real time according to instructions of a subsequent method according to the present invention, and the images are stored in the computer memory 141 of Device 100.

[0058] Optionally, device 100 may also include at least one secondary image acquisition device, in this case at least one secondary camera 160, which can be oriented similarly or differently to camera 130, enabling the acquisition of a second stream of images of the faces 125 of individual 120. In this case, it is emphasized that the position and relative orientation of the secondary camera 160 or each secondary camera relative to camera 130 are generally advantageously known.

[0059] Figure 2 shows a method 200 for tracking the face of individual 120 in a video stream acquired by camera 130, in block diagram form.

[0060] Firstly, it should be emphasized that the tracking method 200 is generally performed in a loop of images, generally consecutive, within a video stream. For each image, several iterations of each step can be performed, particularly to converge the algorithm used.

[0061] Method 200 includes a first step 210 of detecting the presence of the face of an individual 120 wearing glasses 110 in an image of a video stream, called an initial image.

[0062] This detection can be performed in several ways, either from a learning base of faces wearing glasses using a deep learning algorithm, also known as "deep learning," which has been previously trained on a database containing images of faces wearing glasses, or by using a 3D model of a face wearing glasses, which is required to be matched to the face image in the initial image by determining the orientation and dimensions of the 3D model relative to camera 130. The matching between the face model and the face image in the initial image can be performed, in particular, by projecting the model of the face wearing glasses onto the initial image. It is emphasized that this matching can be performed even when part of the face or part of the glasses are hidden in the image, such as when the face is facing the camera, or when elements such as glasses or hair are superimposed on the face, or when elements such as hair are superimposed on the glasses.

[0063] Alternatively, step 210, which detects the face of an individual 120 wearing glasses 110 in the initial image, can be carried out by first detecting one of two elements, for example, the face, and then the other element, namely, the glasses. The face is detected, for example, by detecting facial feature points in the image. Methods for detecting such faces are well known to those skilled in the art. The glasses can be detected, for example, by a deep learning algorithm, also known by the English term "deep learning," which has been previously trained on a database of images of glasses that are preferentially fitted to faces.

[0064] It is emphasized that the detection step 210 can be performed only once for multiple images in the video stream.

[0065] As shown in Figure 3, the learning algorithm enables the calculation of a binary mask 350 for each acquired image.

[0066] The contour points of the mask shown in p2D are, - Outer contour of the mask 360°, - Generally, the internal contour 370 of the mask corresponds to the contour of the lens. - Mask top contour 380, - The contour of the bottom of the mask, 390, is associated with at least one category.

[0067] Alternatively, the mask contour point p2D is calculated using a robust distance between the feature points of the glasses detected in the image and the contour point of the mask, i.e., a distance that hardly changes between two consecutive iterations.

[0068] After detecting the face of individual 120 wearing glasses 110, method 200 includes a second step 220 in which a representation of the individual's face, hereafter referred to as "avatar," is aligned with the image of individual 120's face in the initial image. Here, it is advantageous that the avatar includes two parametric models, one corresponding to a model of the face without glasses and the other corresponding to a model with glasses. It is generally emphasized that the origin of the reference frame of the parametric model is located in a virtual space corresponding to camera 130. Therefore, the camera's reference frame will be described.

[0069] The combined use of these two parametric models improves regression performance and allows for a more accurate estimation of the position of an individual's face relative to the camera.

[0070] Furthermore, the two parametric models of the avatar are favorably linked by relative orientation and position parameters. Firstly, the relative orientation and position parameters correspond, for example, to the standard orientation of the parametric model of the glasses relative to the parametric model of the face, i.e., the frames face the individual's eyes and rest on the nose, with the arms extending along the individual's temples and resting on the individual's ears. This standard orientation is calculated, for example, by the average position of glasses naturally positioned on an individual's face. It is emphasized that, to a greater or lesser extent, the glasses may rest on the nose for each individual.

[0071] In a non-limiting example of the present invention, the parametric model of eyeglasses is a model that includes a three-dimensional frame whose envelope has a thickness that is non-zero in at least cross-section. Advantageously, the thickness of each part of the cross-section of the frame is non-zero.

[0072] Figure 4 shows two views of a surface 300 of a parametric model of eyeglasses. The first view, indicated by 4a, corresponds to a skeletal view of surface 300 without an outer envelope. The second view, indicated by 4b, corresponds to the same view but with an outer envelope 320. As shown, the parametric model of eyeglasses can be represented by a series of contours 330, each having a cross-section perpendicular to the core 340 of the eyeglass frame. Thus, the contours 330 form the framework of the outer envelope 320. This parametric model consists of a thick 3D type.

[0073] It is emphasized that parametric models of eyeglasses can be advantageously provided with a predetermined number of numbered compartments such that the positions of compartments around the frame are identical in two different models of eyeglasses. Thus, compartments corresponding to points on the frame, such as the bottom point of the rim, the top point of the rim, the joint between the rim and the bridge, or the joint between the rim and the tenon that holds the hinge using the arm, have the same number in two different models. This makes it easier to adapt the eyeglasses model to the dimensional markings of the frame. These markings, usually referred to in English as "frame markings," define the lens width, bridge width, or arm length. This information is useful, for example, when defining constraints between two points corresponding to the center or end of two compartments selected according to their position on the frame. In this way, the eyeglasses model can be modified while adhering to dimensional constraints.

[0074] Examples of parametric models of eyeglasses used in this method are shown in more detail in the following section titled "Examples of Parametric Models of Eyeglasses".

[0075] In an alternative embodiment of the present invention, the parametric model of the eyeglasses includes a three-dimensional frame with zero thickness. This is a thicknessless 3D type model.

[0076] All parameters used to define the form and size of eyeglasses are called configuration parameters.

[0077] It is emphasized that the initial form of the parametric model frame can favorably correspond to the form of previously modeled eyeglass frames, for example, in the French patent published in French Patent Application Publication No. 2955409, or in the international patent application published in International Publication No. 2013 / 139814.

[0078] The parametric model of eyeglasses is also generally formed from an elastically deformable material, for example, the arms or front, which can be advantageously deformed. The deformation parameters are included in the constituent parameters of the eyeglasses model. For example, if the eyeglasses model is known through pre-modeling of eyeglasses 110, it is advantageous that the size and form of the eyeglasses model remain unchanged during decomposition. Then, only the deformation of the eyeglasses model is calculated. This reduces the number of parameters to be calculated, shortens the computation time, and yields good results.

[0079] In order to align the two parametric models—one representing the face in the image of the glasses and the other representing the face in the initial image—a regression of the points in the parametric models is performed during the second step 220, so that the parametric models correspond to the glasses 110 worn by individual 120 and to the face of individual 120, respectively, in terms of form, size, position, and orientation.

[0080] Therefore, the avatar parameters processed by regression are, in this example which does not limit the present invention, - The 3D position of the avatar, i.e., the set {glasses model, face model}, - Avatar's 3D orientation, - Size of the eyeglass model, - Size of the face model, - The relative three-dimensional position between the glasses model and the face model. - The relative three-dimensional orientation between the glasses model and the face model. - Optionally, the configuration parameters of the eyeglass model, - Optionally, constituent parameters of a face model, such as morphological parameters for defining the form, size, and position of various elements constituting the face, particularly the nose, mouth, eyes, temples, and cheeks, and also including parameters for opening and closing the eyelids or mouth, and / or parameters related to the deformation of the face surface due to facial expressions, - These are optional camera parameters such as focal length and judgment criterion calibration parameters.

[0081] Alternatively, only some of the avatar parameters mentioned above are processed by regression.

[0082] Camera parameters can be calculated more favorably if the 3D shape of the glasses model is known, for example, if glasses 110 worn by individual 120 are recognized. Adjusting camera parameters helps to more accurately estimate avatar parameters, resulting in better tracking of faces in images.

[0083] Here, regression is performed favorably in two stages. First, the feature points of the face model are minimized using the feature points detected on the initial image to obtain the estimated position of the avatar in the camera's reference frame.

[0084] Secondly, the avatar parameters are fine-tuned by performing a regression of contour points of the glasses model for the glasses seen in the first image of the video stream. The contour points of the glasses model considered during the regression are generally obtained from the glasses frame.

[0085] For this purpose, as shown in Figure 5, a point 410 considered for the contour of the glasses model 420 is a point whose normal vector 430 is perpendicular to the axis between the corresponding point 410 and the camera. Points of the glasses contour on the initial image are associated with each point 410 considered for the contour of the glasses model, and a point 440 is searched for along the normal vector 430 that has the highest gradient in a given color spectrum, such as the gray level. The contour of the glasses can be determined by a deep learning method, also known as "deep learning" in English, which has been previously trained on segmented images of glasses that are preferentially fitted to the face. By minimizing the position between the model contour points on the initial image and the points of the glasses, it becomes possible to fine-tune the avatar's parameters in the camera's reference frame.

[0086] For clarity, it should be emphasized that only five points, 410, are shown in Figure 5. The number of points used in regression is generally quite large. Point 410 is represented by a circle in Figure 4, and point 440 corresponds to the vertex of a triangle sliding along the normal vector 430.

[0087] The association between the contour points of the glasses model in the image and the contour points of glasses 110 corresponds to the pairing of 3D points of the glasses model with 2D points in the image. It is emphasized that this pairing is evaluated preferentially for each iteration and even for each image, because corresponding points in the image may be shifted from one image to another.

[0088] Furthermore, if one or more categories of contour points in an image are known, pairing these points with 3D points of the glasses model can be performed more effectively by pairing points that belong to the same category. In fact, it is emphasized that the points of the glasses model can also be classified according to the same categories as the contour points of the glasses mask in the image.

[0089] To improve regression regarding the position of the glasses model, the contours of the parcels are favorably associated with most of the points considered for the contours of the glasses model. The parcels associated with a point generally correspond to the edges of the frame containing that point. Each parcel is defined by a polygon consisting of a predetermined number of ridges. Thus, during regression, the calculation of normals is improved by becoming more accurate, allowing for a more accurate estimation of the pose of the glasses model relative to the image. This improvement is particularly true when using parametric models of thick 3D glasses.

[0090] Furthermore, it is emphasized that positional constraints between the face model and the glasses model are advantageously considered during regression to reduce computation time while improving pose quality. These constraints indicate, for example, point contact between a part of the face model and a part of the glasses model. These constraints express, for example, the fact that the rims of the glasses are placed on the nose and the arms on the ears, with or without pads. In general, positional constraints between the face model and the glasses model make it possible to parameterize the position of the glasses on the face with a single parameter, such as the position of the glasses on an individual's nose. Between two positions on the nose, the glasses translate along a 3D curve corresponding to the ridge of the nose, or even rotate on an axis perpendicular to this central plane of symmetry. Locally between two nearby points, the translation of the glasses along the 3D curve can be considered to follow the local plane of symmetry of the nose.

[0091] In other words, constraints are represented by pairings of points in the face model and points in the glasses model. It is emphasized that, in order to allow one of the two models to freely translate relative to the other two axes, the pairings between the two points can consist of a partial type, i.e., only one type of coordinate system, such as the X-axis alone.

[0092] Furthermore, the two parametric models included in the avatar—namely, the face model and the glasses model—can conveniently be used as constraints on known dimensions, such as the interpupillary distance previously measured for the face or the characteristic dimensions of the frame previously recognized. Therefore, by performing pairing between two points of the same model, the distance between these two points can be constrained for known dimensions.

[0093] For mathematical details of the algorithm, please refer to the section titled "Details of the Methods Used" below.

[0094] It is emphasized that if at least one secondary camera is available, several views of the individual's face, even if wearing glasses, become available, which allows for improved regression calculations of the avatar's parameters. This is because, since various views are captured at distinct angles, it becomes possible to improve the recognition of the individual's face by displaying parts that are hidden on the image captured by the main camera.

[0095] Figure 6 shows the position of the parametric model 610 of the glasses on the parametric model 620 of the avatar's face, which can be seen in the perspective view of subfigure a. The reference frame used is shown in subfigure e of Figure 6. The movement of the parametric model 610 of the glasses is parameterized here according to the movement of the arm 630 on the ear 640, which corresponds to translational movement along the Z axis (subfigure c of Figure 6). The corresponding translational movement along the Y axis can be seen in subfigure b of Figure 6. Rotation about the X axis is shown in subfigure d of Figure 6.

[0096] For example, to avoid misplacement of the glasses model on a face model, such as an arm within an individual's eye, non-contact constraints can be added between specific parts of the face model and specific parts of the glasses model.

[0097] One difficulty overcome by the present invention is managing the hidden portions of the eyeglasses in the initial image, which can cause errors in the regression of the parametric model of the eyeglasses, particularly with respect to the position and orientation of the parametric model relative to the eyeglasses 110 actually worn by individual 120. These hidden portions generally correspond to the parts of the frame that are masked by the individual's face, for example, when the face is turned to the camera to check the side of the face, or directly by the eyeglasses, for example, by tinted lenses. It is also emphasized that the arm portions placed on each ear are generally hidden by individual 120's ears and / or hair, regardless of the orientation of individual 120's face.

[0098] These hidden parts can be estimated during detection, for example, by considering a frame segmentation model and / or points of the contours of these hidden parts. The hidden parts of the glasses can also be estimated by calculating the pose of the parametric model of the glasses relative to the estimated positions of the individual's face. The parametric model used here can be the same as the one used for the avatar.

[0099] By aligning the parametric model of the eyeglasses, it becomes possible to recognize the model of the eyeglasses 110 actually worn by individual 120. This is because point regression makes it possible to obtain an approximate 3D contour of at least a part of the eyeglasses 110. This approximate contour is then recorded in a database and compared with the contour of eyeglasses that have been previously modeled. The images contained in the contour can also be compared with the appearance of eyeglasses recorded in the database for better recognition of the model of eyeglasses 110 worn by individual 120. In fact, it is emphasized that the eyeglass models stored in the database are generally modeled with textures and materials.

[0100] A parametric model of eyeglasses can be deformed and / or articulated to best accommodate eyeglasses 110 worn by individual 120. Generally, the arms of an eyeglasses model initially form an angle of about 5° between them. This angle can be adjusted by modeling the deformation of the eyeglasses according to the form of the frame and the stiffness of the material used for the arms, or the material used for the front of the eyeglasses frame, which may differ from the material of the arms. A parametric approach can be used to model the deformation of a parametric model of eyeglasses.

[0101] Real-time tracking of the face and / or glasses in the video stream on the image following the initial image is performed during the third step 230 of method 200 shown in Figure 2.

[0102] Real-time tracking can be based, for example, on tracking feature points within a sequence of images of a video stream using an optical flow method.

[0103] Since the updates to the image parameters of a video stream are generally performed with respect to alignment parameters calculated from previous images, this tracking can be performed, in particular, in real time.

[0104] To improve the robustness of tracking, the avatar's orientation relative to the individual's face is considered satisfactory, and this constraint is applied to images representing a view of the face with a similar orientation to the face in the key image, using key images, usually referred to in English as "keyframes." In other words, the key image for the selection of images in the video stream, also called the reference image, generally corresponds to one of the selected images, where the score associated with the avatar's orientation is the highest with respect to the individual's image. Such tracking is described in detail, for example, in the international patent application, International Publication No. 2016 / 135078.

[0105] It is emphasized that key image selection can be performed dynamically, and that image selection can correspond to a continuous sequence of video streams.

[0106] Furthermore, tracking can advantageously utilize multiple key images, each corresponding to a different orientation of the individual's face.

[0107] Furthermore, it is emphasized that combined tracking of the face and glasses can yield better and more robust results because it is based on a larger number of feature points. In addition, relative positional constraints of the parametric models of the face and glasses are generally used during tracking, which enables more accurate tracking of the individual's head, and therefore better avatar posture, in real time.

[0108] Furthermore, because eyeglasses contain landmarks that are clearly identifiable in the image, such as the ridges of the arms, the ridges of the face, or the rims on the front of the frame, tracking eyeglasses, which are manufactured products, is generally more accurate than tracking the face alone.

[0109] Tracking eyeglasses without a parametric model is characterized by low robustness and the need for a large amount of computation per image. Therefore, given the computing power currently available, performing such tracking in real time is more difficult. However, since processor capabilities are increasing regularly, tracking without a parametric model of eyeglasses is conceivable if the processor capabilities are sufficient for such applications.

[0110] It is also emphasized that it is possible to track individuals based solely on a parametric model of the eyeglasses. The optimization of the eyeglasses model's orientation relative to the camera, i.e., the alignment of the eyeglasses model with respect to the image, is performed for each image.

[0111] Next, during step 235, simultaneously with the tracking step 230, the parametric models of the face and glasses, and the alignment parameters with the image, are updated for each new image in the video stream acquired by camera 130.

[0112] Alternatively, the alignment parameters of the parametric models of the face and glasses are updated for each key image.

[0113] This alignment parameter update may also include pose parameters for the parametric model of the glasses on the parametric model of the face to improve the estimation of the position of the individual's face relative to the camera. This update can be performed in particular when the individual's face is facing a different direction relative to the camera, thus providing a different angle of view of the face.

[0114] Fine-tuning of the parametric model can be performed during the fourth step 240 of Method 200 by analyzing the reference key image used during tracking. This fine-tuning makes it possible to complete the parametric model of the eyeglasses using details of the eyeglasses 110 that were not previously captured. These details include, for example, reliefs, embossing, and silkscreen printing specific to the eyeglasses.

[0115] The key image analysis is performed using a cluster adjustment method, also known in English as "bundle adjustment," which allows for fine-tuning of the 3D coordinates of geometric models describing objects in the scene, such as glasses or a face. The "bundle adjustment" method is based on minimizing the reprojection error between the observed points and the model points.

[0116] Therefore, it is possible to obtain a parametric model that better fits the face of an individual wearing glasses.

[0117] The "bundle adjustment" method used here utilizes facial feature points or eyeglass points that can be more accurately identified within the key image. These points can be facial contour points or eyeglass points.

[0118] The "bundle adjustment" method, in general terms, emphasizes processing a scene defined by a set of 3D points that can move between two images. The "bundle adjustment" method makes it possible to simultaneously solve for the 3D position of each 3D point in the scene within a given reference frame (e.g., the reference frame of the scene), the parameters of the scene's relative motion to the camera, and the optical parameters of one or more cameras that acquired the image.

[0119] Sliding points related to contours of a face or glasses, calculated by the optical flow method, can also be used in the "bundle adjustment" method. However, since optical flow is generally calculated between two consecutive different images or two key images within a video stream, the matrix obtained during the "bundle adjustment" method of points from optical flow is generally hollow. To compensate for this lack of information, points of the contour of glasses can be advantageously used by the "bundle adjustment" method.

[0120] It is emphasized that new information can be obtained for a new key image that enables improvements to the parametric model of the face or the parametric model of the glasses. Furthermore, to supplement or replace the points used by the “bundle adjustment” method, a new detection of faces wearing glasses can be performed on this new key image, as described in step 210. To ensure that the fine-tuning of the parametric model is brought closer to the current image of the video stream, solution constraints with higher weights can be associated with the newly detected points.

[0121] Slide points on the contour of the glasses, corresponding to all points of the glasses model where the normal vector is at 90 degrees, can be paired with the 3D model of the glasses on the horizontal line of the contour of the glasses.

[0122] In an example of the embodiment of the present invention, the key images correspond to images of the face of individual 120 wearing the glasses 110 facing forward, and / or images of individual 120's face turned to the left or right relative to the natural position of the head at an angle of approximately 15 degrees with respect to the sagittal plane. In these key images, new parts of the face 125 and glasses 110 are visible. Thus, the parameters of the face model and the glasses model can be determined more accurately. The number of key images can be arbitrarily fixed to a number between 3 and 5 in order to obtain satisfactory results in training the face 125 and glasses 110 to build the corresponding models.

[0123] The size of the glasses 110 worn by individual 120 may also be introduced in method 200 in step 250 to obtain scene determination criteria and, in particular, to define a scale for determining optical measurements of an individual's face that can be defined as average sizes, such as interpupillary distance and iris size.

[0124] The size of eyeglasses 110 can be defined statistically in relation to a previously defined list of eyeglasses, or it can correspond to the actual size of eyeglasses 110.

[0125] An interface can be provided to indicate to Method 200 which "frame marking" is shown on the glasses 110. Alternatively, automatic reading in the image can be performed by Method 200 to recognize the characters of the "frame marking" and automatically obtain the associated value.

[0126] It is emphasized that knowing the "frame markings" is advantageous, especially if the eyeglasses 110 have been modeled previously, as it allows for the determination of the parametric model of the eyeglasses 110.

[0127] If eyeglass size information is unavailable, for example, if "frame markings" are unknown, the first parametric model of the eyeglasses used is typically a standard parametric model that includes the statistical average values ​​of eyeglasses used by the individual. This statistical framework makes it possible to obtain satisfactory results that are close to the models of eyeglasses 110 actually worn by individual 120, and each new image improves the parameters of the eyeglasses model.

[0128] A depth camera can also be used during Method 200 to fine-tune the shape and position of the face.

[0129] It should be emphasized that a depth camera is a type of depth sensor, commonly known by the English term "depth sensor." Furthermore, depth sensors generally operate using infrared light emission, but due to issues of refraction, transmission, and / or reflection, particularly those caused by the lens and / or the material of the front of the glasses, sufficient accuracy cannot be obtained to capture the contour of glasses 110 worn by an individual 120. In some cases, lighting conditions, such as the presence of a strong light source in the camera's field, can hinder the correct operation of an infrared depth camera by introducing high noise that interferes with reliable measurements. However, depth measurement can also be used on the visible parts of the face, such as on a face model or even a glasses model, to ensure a standard of measurement and a more accurate estimation of size and form.

[0130] Assuming that the face of individual 120, or at least the face of the glasses 110 only, is tracked by the method 200 described above, the removal of the glasses 110 worn by individual 120 in the video stream can be carried out by referring in particular to the technique described in the international patent application published in International Publication No. 2018 / 002533. Furthermore, the virtual fitting of new glasses can also be carried out.

[0131] Tracking method 200 is more effective, and because this tracking method allows for a more accurate determination of the position of the glasses relative to the camera, it is emphasized that removing the glasses from the image by concealing the glasses being worn is a more realistic approach.

[0132] The tracking methods described herein also make it possible to modify all or part of the eyeglasses worn by an individual, for example, by changing the color or tint of the lenses or by adding elements such as silkscreen printing.

[0133] Therefore, the tracking method 200 can be included in the augmented reality method.

[0134] It is emphasized that tracking method 200 can also be used as a method for measuring optical parameters, as described in the international patent application published in International Publication No. 2019 / 020521. By using tracking method 200, the measurement of optical parameters can be more accurate because the parametric models of the glasses and the face are solved concurrently within the same reference frame, which is not the case with the prior art where each model is optimized independently without considering the relative positional constraints between the glasses model and the face model. Details of the methods used

[0135] The algorithms presented in this section correspond to a general implementation of some of the tracking methods that are the objectives of the examples described in detail earlier. This part specifically corresponds to solving the parameters of the face model and glasses model for points detected in at least one image stream, in particular the parameters of the pose and configuration / morphology decomposition (step 220 above) and their updating (step 235 above). It is emphasized that these two steps are generally based on the same equations solved under constraints. The morphological modes of the face model and glasses model can also be solved in this part.

[0136] The advantage of solving for both the face model and the glasses model simultaneously is that it provides new contact or proximity constraints between the two models. This is because it not only guarantees that the two meshes, each corresponding to a separate model, do not intrude upon one another, but also guarantees that there are at least some points of contact or proximity between the two meshes, particularly in the ears and nose of an individual. One of the main problems in solving for the pose of a face model is determining the location of points on the temples, which are rarely accurately determined by the point detectors typically used. In many cases, it is advantageous to use the arms of the glasses that are better visible in the image and physically touch the temples.

[0137] It is emphasized that establishing a contact algorithm within minimization is difficult because the two models used are parametric and therefore deformable. Since the two models deform in each iteration, the contact points are distinguishable from each other between iterations.

[0138] In a non-limiting example of the present invention, consider n calibrated cameras, each acquiring p views, i.e., p images. It is emphasized that the unique parameters of each camera and their relative positions are known. Nevertheless, the position and orientation of the face are determined for each view. f The 3D parametric model of the face used, as shown, is α k,k=1..v This is a mesh composed of 3D points p3D that are linearly deformable by v parameters shown. Therefore, each 3D point in this mesh is described in the form of a linear combination.

[0139] [Formula 1] TIFF0007879156000001.tif16170

[0140] TIFF0007879156000002.tif35170

[0141] [Formula 2] TIFF0007879156000003.tif19170

[0142] Here, β k,k=1..μ Glasses M g This corresponds to the μ parameters of the parametric model.

[0143] The 3D face is initially replaced with a 3D reference frame called the world reference frame after every p acquisitions. The world reference frame can correspond to, for example, the camera's reference frame or the reference frame of one of two models. The position and orientation of the face model are initially unknown and are therefore determined during the minimization phase, which corresponds to the regression phase of the face model using feature points detected in the image.

[0144] Before performing this regression, the model M of the glasses g is placed on the model M of the face f For this purpose, the point p3D_g of the glasses model can be written in the face reference frame, taking into account the 3D rotation matrix R_g and the translation vector T_g.

[0145] [Equation 3] TIFF0007879156000004.tif13170

[0146] Next, through regression, the orientation and translational pose of the face model in the reference frame of the view l of one of the cameras corresponding to the world reference frame here are obtained.

[0147] [Equation 4] TIFF0007879156000005.tif13170

[0148] Here, R is the 3D rotation matrix, T is the translation vector, and l represents the view of the camera.

[0149] The projection function of the model p3D in the image i used in this method is shown as follows.

[0150] [Equation 5] TIFF0007879156000006.tif10170

[0151] Here, K i corresponds to the calibration matrix of the image i. R i and T i correspond to the rotation matrix and the translation vector between the world reference frame and the reference frame of the camera that acquired the image i, respectively. The symbol ~ in that part indicates equivalence within the scale factor. This equivalence can be expressed, in particular, by the fact that the last component of the projection is equal to 1.

[0152] Once the pose of the face representation model is solved, the following five types of constraints, namely, - 2D face constraints, - 2D glasses constraints, - 3D face - glasses constraint, - For example, 3D face constraints that correspond to interpupillary distance (PD), distance between temples, average iris size, or a mixture of distributions of several size constraints, where the mixture of distributions may correspond to a mixture of two Gaussian distributions for iris size and interpupillary distance, and combining these constraints may require the formulation of a gh filter type. For example, there are 3D constraints for eyeglasses that correspond to known dimensions, arising from markings on the frame, commonly referred to in English as "frame markings."

[0153] The 2D constraints on the face are based on pairing points in the 3D model with 2D points in the face images of at least one viewer and at least one camera. Preferably, this pairing is done per view and per camera. It is emphasized that pairings can be fixed for face points not included in the face contour in the image, or slid along the horizontal line of face contour points. This degree of freedom in pairing face contour points with image points makes it possible to improve the stability of the pose of the 3D model of the face relative to the image, and thus provides better continuity of the pose of the 3D model of the face between two consecutive images.

[0154] The pairing of points in a 3D model of a face with 2D points in an image can be mathematically expressed by the following equation.

[0155] [Formula 6] TIFF0007879156000007.tif12170

[0156] Here, φ j,i,l And, σ j,i,l These represent the 3D point index of the parametric facial model Mf and the 2D point index of the face in the images of view i and camera l, respectively.

[0157] The 2D constraints of the glasses are based on pairing the 3D points of the glasses model with the 2D points of the glasses in the image, particularly using the contour of the mask in the image.

[0158] [Equation 7] TIFF0007879156000008.tif10170

[0159] Here, φ j,i,l And, ω j,i,l These represent the 3D point index of the parametric model Mg of the eyeglasses and the 2D point index of the eyeglasses in the images of view i and camera l, respectively.

[0160] The constraints on 3D face-glasses models are based on pairing 3D points of the face model with 3D points of the glasses model, and their distance is defined by proximity constraints, even for contact (zero distance). By applying an influence function, the contact distance can be calculated with greater weighting for negative distances relative to the normal to the surface of the face model that are pointed outwards from the face model. It should be emphasized that for some points, constraints may be imposed on only certain coordinates, such as the axis of the relationship between the temples of the face and the arms of the glasses.

[0161] The pairing of 3D points in the face model and 3D points in the glasses model can be mathematically expressed by the following equation.

[0162] [Formula 8] TIFF0007879156000009.tif10170

[0163] Here ,ρ j and, τ j These represent the 3D point indices of the parametric model Mf for the face and the 3D point indices of the parametric model Mg for the glasses, respectively.

[0164] The 3D constraints of the face are based on previously measured, known distances of the face, such as the interpupillary distance (the distance between the centers of each pupil, which also corresponds to the distance between the centers of rotation of each eye). Therefore, the reference distance can be paired with a pair of points.

[0165] [Formula 9] TIFF0007879156000010.tif12170

[0166] Here, t j and u j Each of these represents an index of an individual 3D point in the parametric facial model Mf.

[0167] The 3D constraints of eyeglasses are based on known distances of the eyeglasses model worn by the individual, such as lens size, bridge size, or arm size (for example, according to the BOXING or DATUM standard). These distances can be expressed, in particular, by frame markings, typically located on the inside of the arms, usually referred to as "frame markings." The reference distance can then be paired with a pair of points on the eyeglasses model.

[0168] [Formula 10] TIFF0007879156000011.tif12170

[0169] Here, v j and w j Each of these represents an index of an individual 3D point in the parametric model Mg of the eyeglasses.

[0170] Therefore, the input data for the algorithm is - p images from n cameras of a person wearing glasses. - Characteristic 2D points of faces detected in the image, - Optionally, in the case of so-called sliding points (e.g., along a horizontal line), the 2D or 3D pairing of some points is evaluated at each iteration. - Mask of glasses in at least one image, - Calibration matrix and the orientation of each camera.

[0171] This algorithm produces the following output data, namely, - p poses of the avatar: R fl , T fl , - v modes of the parametric model of the face: α1, α2, ..., α v , - The posture of the model wearing glasses relative to the model with a face: R g , T g , - μ modes of the parametric model of eyeglasses: β1, β2, ..., β μ You will be able to calculate ,.

[0172] For this purpose, the algorithm follows the following procedure: TIFF0007879156000012.tif75170

[0173] [Formula 11] TIFF0007879156000013.tif77170

[0174] Here, γ1, γ2, γ3, γ4, γ5 are the weights between each constraint block, and `visi` is a function that indicates whether point p2D is visible in the image, i.e., whether it is hidden by the face model Mf or the glasses model Mg, and #(visi==1) corresponds to the number of visible points.

[0175] In this particular embodiment of the present invention, the camera's focal length forms part of the parameters to be optimized. This is because, if the image acquisition is performed by an unknown camera, some of the acquired images may have been previously reframed or resized. In this case, it is preferable to leave the camera's focal length as a degree of freedom during minimization.

[0176] In this particular embodiment of the present invention, variations in the variance matrix and covariance matrix, which represent the parameter axes and uncertainty / confidence values ​​of the contact constraint equation between the face model and the glasses model, are taken into consideration when solving the equation.

[0177] In this particular embodiment of the present invention, several parameters of the orientation of the glasses model relative to the face model are fixed. This can represent a hypothetical alignment between the glasses model and the face model. In this case, only rotation along the X-axis, i.e., the axis perpendicular to the sagittal plane, and translational movement along the y and z axes, i.e., within the sagittal plane, are calculated. The cost function expressed in [Equation 11] can be simplified, making it easier to converge toward the result. In this way, very satisfactory results can be obtained even with highly asymmetrical faces, where the glasses are positioned differently compared to a symmetrical face, for example, if one side of the face is slightly tilted.

[0178] Example of a parametric model for eyeglasses Each pair of glasses contains common elements such as lenses, bridges, and arms. Thus, as shown in Figure 7, the parametric model (3DMM) 700 of the glasses can be defined as a set of sections 710 connected to each other by previously defined triangular faces 715.

[0179] The triangular surface 715 forms a convex envelope 720, but a portion of it is not shown in Figure 7.

[0180] Each of the 710 sections, defined by the same number of points, is favorably positioned in the same location on all models of eyeglasses.

[0181] Furthermore, each section 710 intersects with its pair on a plane perpendicular to the framework 730.

[0182] Therefore, there are three types of partitions, namely, - A section around the lens 710, parameterized by an angle with respect to a reference plane perpendicular to the rim's skeleton, such that there is one section every n degrees.A , - Section 710 of the bridge parallel to the reference plane B , - Arm frame 730 B Section 730 of the arm, along with C It can be defined.

[0183] When a pair of lenses has no rim around the lens, it is usually called "rimless" in English, or when the rim only surrounds a portion of the lens, it is called "semi-rimless," and the rim surrounds the lens section 710 A All or part of one and the same section 710 A It is emphasized that there is only one point corresponding to all combinations of points.

[0184] Furthermore, the principal component analysis (PCA) used to align the eyeglass model 700 with the representation of the eyeglasses in the image requires many common points. For this purpose, points are selected that lie on the convex envelope 720 of the eyeglass model so that all pixels belonging to the aligned eyeglasses can be reliably found in the image.

[0185] For example, in the case of eyeglasses with a double bridge, a template of an eyeglass model with a double bridge can be pre-selected to fit the eyeglasses as closely as possible, in order to make it possible to find the opening for the eyeglasses.

[0186] Since the points in the parametric model referenced by a given index are sequentially placed at the same relative points on the eyeglass model, it becomes easy to define a known distance between two points. This known distance can be obtained from "frame markings" engraved on the eyeglasses, which define the overall width of the lenses, the width of the bridge, and the length of the arms.

[0187] This information can influence the solution of the eyeglass model 700 by selecting corresponding points, as shown in Figure 8. In Figure 8, only point 810 characterizing the contour of the front section 710 of the eyeglass is shown, where d corresponds to the lens width, specifically defined by the "frame marking". In the face-eyeglass alignment deformation, numerous faces and numerous eyeglasses are generated from the two respective parametric models of face and eyeglasses. Next, an automatic placement algorithm is used to place each eyeglass model on each face model. Advantageously, the generation of noise and different placement statistics, i.e., eyeglasses at the ends of the nose, pad recesses, loose placement at the temples, etc., are used to automatically place the eyeglasses on the face. Next, a new parametric model of eyeglasses and face is computed from all points of the face-eyeglass model. This new parametric model guarantees contact and perfect placement of the eyeglasses on the face and simplifies the solution. This is because one transformation is required that corresponds to the computation of 6 parameters instead of 12, and the contact equation is canceled out. However, since it is modes that encode these constraints, generally more modes are estimated in this case.

Claims

1. A tracking method (200) performed by a data processor (140) for tracking the face (125) of an individual (120) in a video stream acquired by an image acquisition device (130), wherein the face is wearing glasses (110), the video stream includes a plurality of sequentially acquired images, the tracking method includes the steps (220, 235) of calculating parameters of a representation of the face including a model of the glasses and a model of the face, and overlaying the representation of the face based on the calculated parameters of the representation of the face onto the image of the face in the video stream, wherein all or some of the parameters of the representation are calculated by considering at least one proximity constraint between one or more points of the model of the face and one or more points of the model of the glasses relating to the points of the model of the face. The parameters of the representation include external values ​​of the representation of the face and internal values ​​of the representation of the face, wherein the external values ​​include the three-dimensional position and three-dimensional orientation of the representation of the face relative to the image acquisition device, and the internal values ​​include the three-dimensional position and three-dimensional orientation of the model of the glasses relative to the model of the face, and the parameters are calculated with respect to a plurality of feature points of the representation of the face that have been previously detected in the image of the video stream called the first image, or in a set of images including the first image, which are acquired simultaneously by a plurality of image acquisition devices. All or part of the internal values ​​of the representation are updated with respect to the location of all or part of the feature points tracked or detected in a second image of the video stream or in a second set of images acquired simultaneously by multiple image acquisition devices, the set of the second images includes the second image. A tracking method (200) characterized by the following.

2. The tracking method according to claim 1, wherein all or some of the parameters of the representation are updated with respect to the locations of all or some of the feature points tracked or detected in a second image of the video stream or in a second set of images acquired simultaneously by a plurality of the image acquisition devices, the set of the second images comprising the second images.

3. The tracking method according to claim 1 or 2, wherein when calculating all or some of the parameters of the representation, at least one proximity constraint is also taken into consideration between a three-dimensional point of one of the models included in the representation of the face and at least one point included in at least one image of the video stream, or a horizon line.

4. The tracking method according to any one of claims 1 to 3, wherein when calculating all or some of the parameters of the representation, at least one dimensional constraint of one of the models included in the representation of the face is also taken into consideration.

5. A tracking method according to any one of claims 1 to 4, comprising the step of pairing two distinct points, each belonging to one of the two models included in the representation of the face, or to distinct models included in the representation of the face.

6. A tracking method according to any one of claims 1 to 5, comprising a preliminary step of pairing a point of one of the two models included in the representation of the face with at least one point of an image acquired by an image acquisition device.

7. The tracking method according to any one of claims 1 to 6, wherein during the calculation of the parameters of the representation, the alignment of the model of the glasses with the image of the glasses in the video stream is performed in succession with the alignment of the model of the face with the image of the face in the video stream.

8. The tracking method according to claim 7, wherein the alignment of the face to the model is performed by minimizing the distance between the feature points of the face detected in the image of the face and the feature points of the model of the face projected in the image.

9. The tracking method according to claim 7 or 8, wherein the alignment of the model of the eyeglasses is performed by minimizing the distance between at least a portion of the contour of the eyeglasses in the image and a similar contour portion of the model of the eyeglasses projected in the image.

10. The parameters of the above expression are the following list, namely, - The three-dimensional position of the aforementioned representation of the face, - The three-dimensional orientation of the aforementioned representation of the face, - The size of the aforementioned model of eyeglasses, - The size of the face of the aforementioned model, - The relative three-dimensional position between the model of the glasses and the model of the face, - The relative three-dimensional orientation between the model of the glasses and the model of the face, - One or more parameters of the configuration of the aforementioned model of eyeglasses, - One or more parameters of the configuration of the face model, A tracking method according to any one of claims 1 to 9, comprising all or some of one or more parameters of a camera.

11. - A step of detecting multiple points of the face in the first image of the video stream, - An initialization step of the set of parameters for the face model relating to the image of the face in the first initial image, - A detection step of detecting multiple points of glasses worn on the individual's face in a second image of the video stream, referred to as a second initial image, wherein the second initial image is after or before the first initial image in the video stream, or is identical to the first image in the video stream. The tracking method according to claim 10, further comprising the step of initializing a set of parameters for the model of the glasses for the image of the glasses in the second initial image.

12. The tracking method according to claim 11, wherein the initialization of the parameters of the model of the face is performed by a deep learning method that analyzes all or some of the detected points of the face.

13. The tracking method according to claim 12, wherein the deep learning method also determines the initial position of the model of the face in a three-dimensional reference frame corresponding to the reference frame of the image acquisition device.

14. A tracking method according to any one of claims 1 to 13, comprising the step of determining the scale of the image of the glasses worn on the face of the individual by the dimensions of the known size elements of the glasses in the image.

15. The tracking method according to claim 14, wherein the scale is determined by prior recognition of the glasses worn on the face of the individual.

16. The tracking method according to any one of claims 1 to 15, wherein the image acquired by the second image acquisition device is used to calculate the parameters of the representation.

17. The tracking method according to any one of claims 1 to 16, wherein the model of the glasses in the representation corresponds to the pre-modeling of the glasses, differing only in deformation.

18. - A step of acquiring at least one stream of images of an individual wearing glasses on their face using at least one image acquisition device, - The data processor tracks the individual's face using the tracking method described in any one of claims 1 to 17, and tracks the position and orientation of the facial representation. - A step of modifying all or part of the images in the image stream or one of the image streams, called the main video stream, acquired by the image acquisition device or by one of the image acquisition devices called the main image acquisition device, in the main video stream by the data processor with the representation of the face superimposed in real time on the face of the individual; An augmented reality method comprising the step of displaying the previously modified main video stream on a screen by the data processor.

19. An electronic device comprising: a computer memory storing instructions which are programs for causing a computer to perform a method according to any one of claims 1 to 18; and a data processor which executes the instructions.

Images