Method for obtaining a digital representation of a head of hair of a physical person

Abstract

The present invention relates to a method and device for obtaining a digital representation of a head of hair of a physical person. The method comprises: - obtaining (231) at least a first parameter of a parametric model of hair, output by a trained model for estimating parameters on the basis of image data representative of at least the head of the physical person according to different viewpoints; and - obtaining (232) the digital representation of the head of hair of the physical person on the basis of the at least a first parameter, the obtaining step (232) comprising: - obtaining (2322) a head-of-hair class identifier output by a trained head-of-hair classification model when the image data are input into the trained head-of-hair classification model; and - adjusting (2323) a head-of-hair model identified by the head-of-hair class identifier on the basis of the at least a first parameter.

Description

Method for obtaining a digital representation of a natural person's hair Technical field

[0001] The present invention relates to systems for virtual fitting of accessories by a digitally represented natural person. In particular, the present invention relates to obtaining a digital representation of a head of hair of a natural person, said digital representation being intended to be used to add a head of hair to a digital twin (avatar) of the natural person. Technological background

[0002] Applications or services are known that allow a person to try on an accessory virtually. These applications or services are based on augmented reality, i.e., for example, on a combination of two-dimensional (2D) digital images captured of the person that are combined with a three-dimensional (3D) rendering of an accessory. Note that augmented reality can also be achieved by combining a volumetric image and a rendering of a 3D scene.

[0003] The combination of 2D and 3D image data in augmented reality-based applications limits the display of the image resulting from the combination of these data to a single viewpoint or at best to different viewpoints whose view axes are very close to each other.

[0004] Current virtual fitting applications or services for accessories thus allow a natural person to form an opinion concerning the purchase of these accessories by viewing on a screen an image formed by the combination of 2D images of this natural person and these accessories.

[0005] These augmented reality-based applications or services are usually limited to rendering images resulting from a front view of a physical person. Indeed, a capture of image data representing a front view of a natural person is generally made because this natural person usually looks at the camera which performs this image data capture. The image resulting from the combination of this captured image data and data representing an accessory is then a 2D image of this natural person representing a front view of this natural person. Applications or services based on augmented reality are therefore suitable when the accessory in question is intended to be worn on the face of the natural person, for example makeup.On the other hand, when the accessory is intended to be worn on one side of the natural person, for example earrings, these applications or services are not well suited because they only offer 2D digital representations whereas the natural person may want to have front and side views, for example, before deciding on their accessory purchase.

[0006] In its French patent application No. FR2301334 filed on February 14, 2023, the applicant describes a system for virtual fitting of accessories by a digitally represented living being. This system makes it possible to create a three-dimensional digital twin JN of a part of a living being, of a set of parts of a living being or of a living being in its entirety, and to calculate a 3D view of this digital twin JN wearing a digitally represented accessory.

[0007] When the virtual fitting system is used to try on head accessories such as earrings, makeup, hats, etc., the digital twin JN represents at least the head of the natural person (and possibly other parts of this natural person) and it is necessary to define a digital representation of the hair of the natural person which will be placed on the digital representation of the head of the digital twin JN. This digital representation of the hair of a natural person must correspond to the haircut of the natural person so that the digital twin JN is as realistic as possible and the virtual fitting system for accessories is close to a real fitting of an accessory. Summary of the present invention

[0008] An object of the present invention is to solve at least one of the drawbacks of the technological background.

[0009] Another object of the present invention is to improve the realism of a digital twin representing at least the head of a physical person.

[0010] Another object of the present invention is to improve virtual accessory fitting systems.

[0011] According to a first aspect, the present invention relates to a method for obtaining a digital representation of a head of hair of a natural person, said digital representation being intended to be used for adding a head of hair to a digital twin of the natural person, said method comprising the following steps: - obtaining at least a first parameter of a parametric hair model as output from a trained parameter estimation model when image data representative of at least the head of the physical person from different points of view are presented as input to the trained parameter estimation model; and - obtaining the digital representation of the hair of the natural person based on said at least one first parameter, - obtaining the digital representation of the hair of the natural person as a function of said at least one first parameter comprises the following sub-steps: - obtaining a hair class identifier as output from a trained hair classification model from a set of hair classes when the image data is presented as input to the trained hair classification model, each hair class identifier referencing a parametric hair model; and - adjustment of the hair model identified by the hair class identifier according to said at least one first parameter.

[0012] According to a particular and non-limiting embodiment of the present invention, the trained parameter estimation model is implemented by a first deep convolutional neural network.

[0013] According to a particular and non-limiting embodiment of the present invention, said at least one first parameter can be modified as a function of at least one second parameter.

[0014] According to a particular and non-limiting embodiment of the present invention, said at least one second parameter is obtained from an action of a user on a human-machine interface or from image data.

[0015] According to a particular and non-limiting embodiment of the present invention, said at least one second parameter is obtained from image data representative of at least the head of the physical person.

[0016] According to a particular and non-limiting embodiment of the present invention, said at least one second parameter is a scene lighting parameter in which the digital twin operates or said at least one second parameter is defined as a function of data which defines a choice of scene.

[0017] According to a particular and non-limiting embodiment of the present invention, the digital representation of the hair of the physical person is a parametric curve obtained (2321) from image data representative of at least the head of the physical person and said at least one first parameter is a parameter of the parametric curve.

[0018] According to a particular and non-limiting embodiment of the present invention, the trained hair classification model can be implemented by a second deep convolutional neural network.

[0019] According to a particular and non-limiting embodiment of the present invention, the first and second deep convolutional neural networks each comprise a multiple instance learning type architecture.

[0020] According to a particular and non-limiting embodiment of the present invention, training of the first deep convolutional neural network and / or of the second deep convolutional neural network is carried out from image data obtained synthetically or from groups of three images taken from the front, right profile and left profile of the physical person.

[0021] According to a second aspect, the present invention relates to a device for obtaining a digital representation of a head of hair of a natural person comprising means for implementing the steps of the method according to the first aspect of the present invention.

[0022] According to a third aspect, the present invention relates to a system for virtual fitting of accessories by a physical person digitally represented by a digital twin comprising a device according to the second aspect of the present invention.

[0023] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0024] Such a computer program may use any programming language, and may be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0025] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0026] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may include a storage medium, such as a ROM memory, a RAM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording medium or a hard disk.

[0027] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or terrestrial radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from a network such as the Internet.

[0028] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0029] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 6, in which:

[0030] [Fig. 1] schematically illustrates a system for virtual fitting of accessories by a physical person digitally represented by a digital twin, according to a particular and non-limiting embodiment of the present invention;

[0031] [Fig. 2] illustrates a flowchart of the different steps of a method for obtaining and rendering the digital twin from image data according to a particular and non-limiting exemplary embodiment of the present invention;

[0032] [Fig. 3] illustrates a flowchart of the different steps of a method for obtaining a digital representation of the hair of a natural person from image data, according to a particular and non-limiting exemplary embodiment of the present invention;

[0033] [Fig. 4] illustrates a flowchart of the different steps of a method for obtaining at least a first parameter of a parametric hair model at the output of a trained parameter estimation model when image data are presented as input to the model, according to a particular and non-limiting exemplary embodiment of the present invention;

[0034] [Fig. 5] illustrates a flowchart of the different steps of a method for obtaining a hair class identifier at the output of a trained hair classification model when the image data are presented as input to the model, according to a particular and non-limiting exemplary embodiment of the present invention;

[0035] [Fig. 6] schematically illustrates the device configured to obtain a digital representation of a head of hair of a natural person digitally represented by a digital twin, according to a particular and non-limiting exemplary embodiment of the present invention. Description of examples of implementation

[0036] A method and a device for obtaining a digital representation of a natural person's hair and a system for virtual fitting of an accessory by a natural person digitally represented by a digital twin, will now be described in what follows with joint reference to figures 1 to 6. The same elements are identified with the same reference signs throughout the description which follows.

[0037] Figure 1 schematically illustrates a system 1 for virtual fitting of accessories by a physical person 10 represented digitally by a digital twin JN, according to a particular and non-limiting embodiment of the present invention.

[0038] The system 1 comprises a device 101 for obtaining DIG image data representative of at least the head of the natural person 10 from different points of view, a device 102 for obtaining a digital representation RNT of at least the head of the natural person 10 in a three-dimensional geometric space from the DIG image data, a device 103 for obtaining a digital representation RNC of the hair of the natural person 10 in the space three-dimensional geometric from the DIC image data, a device 104 for obtaining DAC accessory data representative of a digital representation of at least one accessory in the three-dimensional geometric space and a device 105 for rendering the digital twin JN representative of at least the head of the natural person 10 from the RNC and RNT representations and the DAC accessory data.

[0039] The three-dimensional geometric space is a space in which the part or set of parts of the natural person 10 or the natural person 10 as a whole is geometrically represented from the DIC image data and the camera(s) used to capture the images from which this DIC image data is obtained.

[0040] The digital twin JN, also called realistic avatar, is a set of three-dimensional digital data which represents in the metaverse at least the head of the physical person 10 which is represented by the DIC image data.

[0041] The geometry and texture of the JN digital twin are defined by the RNT representation. The DAC data can then be added to the JN digital twin to visualize the virtual wearing of accessories by the JN digital twin.

[0042] According to one variant, the system 1 may further comprise a human-machine interface (HMI).

[0043] For example, the HMI may be partly implemented by several of the devices 101 to 105.

[0044] For example, it may allow a user to enter parameters through graphical means such as a keyboard or touchscreen of the device 101 or 105.

[0045] Alternatively, devices 101, 104 and 105 may be implemented in separate devices.

[0046] According to a variant, the devices 101, 104 and / or 105 can be implemented in the same device.

[0047] According to an exemplary embodiment of this variant, the devices 101, 104 and / or 105 can be implemented in the same mobile communication device.

[0048] According to variants, the device 101, 104 and / or 105 may be one of the following devices: - a telephone; - a computer; - a tablet; - a pair of glasses equipped with at least one camera and one screen or in communication with at least one camera and one screen; - a pair of ocular contact lenses equipped with at least one camera and a screen then adaptive to a user's view.

[0049] The present invention is not limited to these exemplary devices but may extend to any device that would be configured to implement the devices 101 to 105.

[0050] The devices 102 and 103 are calculators (CPU for Computer Processing Units in English, in French digital calculation units, or GPU for Graphical Processing Unit in English, in French graphic calculation unit) which are associated with buffer memories to execute calculation tasks.

[0051] The devices 101 to 105 may be connected for example in communication via a wired network (for example according to Ethernet and / or via a fiber optic link) and / or via a wireless network of the Wifi® type (according to IEEE 802.11 or one of the variations of IEEE 802.11 or via a cellular network of the LTE (Long-Term Evolution), LTE-Advanced and / or 5G and / or 6G type. We speak of “cloud computing” in English (computing in the clouds in French) to refer to this type of architecture which uses memory and computing capacities of computers and servers distributed throughout the world and linked by a communication network. Such an architecture is illustrated in FIG. 1 by a cloud 100.

[0052] The devices 101 to 105 are thus configured to transmit data to the “cloud” 100 and / or to receive data from the “cloud” 100.

[0053] The mobile communication infrastructure enabling wireless communication of data between the devices 101 to 105 and the “cloud” 100 may comprise, for example, one or more communication devices (not shown in FIG. 1) of the relay antenna type (cellular network). In a communication mode using such a network architecture, the data intended for the devices 101 to 105 may be received, for example, from the “cloud” 100 and the data may be transmitted by the devices 101 to 105 via the “cloud” 100 through one or more relay antennas (each relay antenna being, for example, connected to the “cloud” 100 via a wired link).

[0054] The DIG image data, the data representative of the RNT and RNC representations and the DAC accessory data are exchanged by the devices 101 to 105 via the “cloud” 100.

[0055] The present invention is not limited to the communication architecture between the devices 101 to 105 illustrated in FIG. 1 but extends to any type of communication architecture.

[0056] For example, when the devices 101, 104 and 105 are the same mobile communication device, this mobile communication device can be configured to transmit the DIG image data to the devices 102 and 103 via the “cloud” 100 and to receive data representative of the RNT and RNC representations via the “cloud” 100.

[0057] According to another example, the devices 102 and 103 may be the same device which is then configured to receive DIC image data via the “cloud” 100 and to transmit the data representative of the RNT and RNC representations via the “cloud” 100.

[0058] Other variants of communication architecture between the devices 101 to 105 are obviously conceivable without departing from the scope of the present claimed invention.

[0059] Figure 2 illustrates a flowchart 2 of the different steps of a method for obtaining and rendering the digital twin JN from the DIC image data according to a particular and non-limiting exemplary embodiment of the present invention.

[0060] In a first step 21, the DIC image data representative of at least the head of the physical person 10 from different points of view are obtained from the device 101.

[0061] In a second step 22, the digital representation RNT is obtained from the DIC image data.

[0062] In a third step 23, the RNC digital representation is obtained from the DIC image data.

[0063] In a fourth step 24, the DAC accessory data is obtained.

[0064] In the fifth step 25, the digital twin JN defined from the digital representations RNT and RNC and the accessory data DAC is rendered, by calculating a view of this digital twin JN according to a viewpoint.

[0065] The device 101 comprises an image capturing means.

[0066] According to an exemplary embodiment of the present invention, the image capture means may comprise at least one camera corresponding for example to: - an infrared camera; - an RGB type image acquisition camera (from the English “Red, Green, Blue” or in French “Rouge, vert, bleu”); - a camera platform (“Rig” in English) (called Lightfield) or Volumetric camera Rig of RGB type; - a LIDAR type acquisition camera (“light detection and ranging” or “laser imaging detection and ranging” in English); or - an image acquisition camera associated with one or more devices (for example one or more LEDs (from the English “Light-Emitting Diode” or in French “Diode electroluminescent") emitting light in the infrared or near infrared band.

[0067] According to an exemplary embodiment of the present invention, the image capturing means may correspond to a camera of a mobile communication device comprising a camera.

[0068] According to variants, the captured images may correspond either to still images captured from different points of view of at least the head of the natural person 10 or to images from a video captured of the natural person 10 which represents at least the head of said natural person 10 from different points of view.

[0069] According to an exemplary embodiment of the present invention, the image capturing means may be a scanning system comprising a plurality of cameras synchronized with each other to capture several images at different viewpoints of at least the head of the physical person. Captured image data then represents these images captured at different viewpoints.

[0070] In the case of a camera platform (sometimes referred to as "Lightfield" in English and "volumetric image data" in French), the cameras can be positioned in a circular or planar manner on a physical capture device and can capture in a single iteration or several (if motion capture) at least the head of the physical person.

[0071] According to an exemplary embodiment of the present invention, the image capture means may comprise at least one volumetric sensor making it possible to obtain volumetric image data (in English "lightfield").

[0072] According to an exemplary embodiment of the present invention, the device 101 may be configured so that the DIG image data corresponds to image data of the images captured by the image capturing means.

[0073] Typically, three images are captured, one representing the head of the physical person 10 from the front, another representing it in a left profile and the other representing it in a right profile.

[0074] According to an exemplary embodiment of the present invention, the device 101 may be configured to apply processing to the images captured by the image capturing means and the DIG image data corresponds to the processed image data.

[0075] According to an exemplary embodiment of the present invention, if a video can be captured by the image capturing means, a processing can be the selection of images representative of at least the head of the natural person among the images of this video. Typically, three images are selected, one representing the head from the front, another representing it in a left profile and the other representing it in a right profile.

[0076] According to an exemplary embodiment of the present invention, the hair of the natural person depicted in the captured images may be segmented from the rest of the captured images, i.e., at least one spatial region that delimits the hair of the natural person's head is identified in each of the captured images. The DIG image data may then correspond to the image data of the pixels contained in the spatial regions defined in the captured images.

[0077] For example, segmentation of hair from the rest of a captured image can be implemented using a model based on a PSPNet-like architecture developed by Li et al. (Li, T., Bolkart, T., Black, M., Li, H., & Romero, J. (2017) or BiSeNet (https: / / arxiv.org / abs / 2004.02147) Learning a Model of Facial Shape and Expression from 4D Scans. ACM Trans. Graph., 36(6)).

[0078] According to a variant, the DIC image data may further comprise information representative of image contents and / or descriptive information of geometry of at least the head of the natural person in a three-dimensional geometric space.

[0079] The device 102 is configured to obtain the digital representation RNT of at least the head of the physical person 10 in the three-dimensional geometric space from the DIC image data.

[0080] According to a particular and non-limiting exemplary embodiment of the present invention, the device 102 can be configured to implement the method described in French patent application No. FR2301334 filed on February 14, 2023.

[0081] According to this patent application, obtaining the digital representation RNT of at least the head of the natural person 10 comprises obtaining DM mesh data (“mesh” in English) representative of a three-dimensional mesh structure represented in the three-dimensional geometric space by a set of vertices of polygonal shapes forming a structure, each polygonal shape, for example triangles or quadrangles, sharing at least one side with another polygonal shape.

[0082] The device 102 is configured to calculate mesh data DM of the digital twin JN by modifying mesh data DMI of an initial digital twin JNI according to morphological information extracted from the image data DIC.

[0083] The initial digital twin JNI is a digital representation of at least one head of an asexual human being. This digital representation includes DMI mesh data representative of a three-dimensional mesh structure represented in three-dimensional geometric space by a set of vertices of polygonal shapes forming a structure, each polygonal shape sharing at least one side with another polygonal shape.

[0084] According to a particular and non-limiting exemplary embodiment, the DMI mesh data can be modified by modifying the spatial positions of the vertices defined in the three-dimensional geometric space.

[0085] The three-dimensional mesh structure of the initial JNI twin is then deformed (stretched, pressed, etc.) so that the three-dimensional mesh structure modified from the initial twin JNI resembles in three-dimensional geometric space the geometry of at least the head of the physical person 10.

[0086] The DM mesh data of the JN digital twin is then equal to the modified DMI data.

[0087] The DM data of the JN digital twin are obtained by an iterative process of deformation of the DMI data of the JNI digital twin which incorporates all the knowledge of the morphology of a living being such as for example all the morphology of a Caucasian, Afro, Asian human being and / or an animal being, that is to say the position, a structure and a deformation of the eyes, mouth and folds of the skin, an underlying muscular system and a series of controllers allowing the deformations.

[0088] An iterative deformation process can be based on a neural network (Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv:1704.04861 , 2017) and / or on a deep learning method (Ayush Tewari, Michael Zollôfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian. MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In The IEEE International Conference on Computer Vision (ICCV), 2017.) and / or on a reinforcement learning method and / or using generative adversarial networks (GANs). “Generative Adversarial Network”).

[0089] In the case where a deep learning method is used for the reconstruction of the face of the physical person 10, a parametric modeling of the face called "3DMM" can be used. This is based on a statistical model of human faces which makes it possible to represent any face in a weighted sum of 3D mesh structures, 3D mesh structure which supposes the prior creation of an adapted 3D topology. Only the frontal part of the face - the only deformable one - is taken into account (we speak of "monkey mask" in English). The appearance geometric is enriched by the representation of deformations corresponding to facial expressions. This is accompanied by a set of several parameters, typically of the order of 150 parameters for face identity and 100 parameters for expressions. The estimation of these hundreds of parameters can be carried out by regression of these parameters using a deep learning method learned on a large set of face images (Ayush Tewari, Michael Zollôfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian. MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In The IEEE International Conference on Computer Vision (ICCV), 2017), and (Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15:2018, 2018).

[0090] Thus, during this iterative deformation process, a first three-dimensional (3D) mesh structure defined by the DNI data is positioned in the three-dimensional geometric space. This first three-dimensional mesh structure is then compared to three-dimensional mesh structures stored in a database, each stored 3D mesh structure representing at least the face of the natural person. The iterative process also compares each two-dimensional (2D) image calculated from the first 3D mesh structure to 2D images calculated from a 3D mesh structure stored in the database. The method of comparing these elements seeks to bring the 2D images calculated from the first 3D mesh structure closer to 2D images calculated from 3D mesh structures stored in the database.The method thus iteratively improves the distances between the vertices of the first 3D mesh structure and the vertices of a 3D mesh structure and by micro-displacements of the vertices of the first 3D mesh structure in the three-dimensional geometric space so as to optimize the resulting first 3D mesh structure. This spatial operation of deformation of the first 3D mesh structure therefore proceeds by successive iterations until a resemblance is obtained between the first 3D mesh structure and a 3D mesh structure stored in the database. geometrically representing at least the head of the physical person. The 3D mesh structure thus obtained corresponds in resemblance to the DIG image data.

[0091] According to this patent application, obtaining the digital representation RNT of at least the head of the natural person 10 may further comprise obtaining texture data DT from texture data extracted from the image data DIG.

[0092] According to a particular and non-limiting exemplary embodiment of the present invention, the texture data DT may be texture data extracted from a defined area on the face of the physical person 10 from the image data DIG.

[0093] According to an exemplary embodiment, the texture data of the area can be obtained by assembling the DIG image data, isolated, distributed and assembled to reconstruct a single texture.

[0094] For example, the stitching method may deform the texture of each of the images obtained from the DIG image data by modifying the color and contrast of the pixels of the texture images, corresponding to the texture of the images obtained from the DIG image data, and correcting the boundaries between the texture images to generate a single texture.

[0095] This unique texture can then be projected onto the surface of the 3D mesh structure corresponding to the DM data (projection known as UV mapping, the letters U and V designating the axes of the 2D texture). This then allows a reprojection of the said assembled texture onto the said 3D mesh structure.

[0096] For example, texture data can be obtained from a surface of a defined area between the forehead, ears and chin ("monkey mask" in English).

[0097] The device 105 is configured to obtain DAC accessory data representative of a three-dimensional representation of at least one accessory in the three-dimensional geometric space. This three-dimensional representation is intended to be added to the JN digital twin.

[0098] DAC accessory data is known to those skilled in the art. It represents all types of objects, makeup. We often talk about 2D / 3D layers which incorporate a material, transparency, relief and which are calculated by a shading and rendering process.

[0099] The device 103 is configured to obtain the RNC digital representation of the hair of the natural person 10 from the DIC image data. For this purpose, the device 103 is configured to implement the method of FIG. 3.

[0100] Figure 3 illustrates a flowchart of the different steps of a method for obtaining (step 23) the RNC digital representation of the hair of the natural person 10 from the DIC image data, according to a particular and non-limiting exemplary embodiment of the present invention.

[0101] In a step 231, the device 103 obtains at least a first parameter P1 of a parametric hair model at the output of a trained MP parameter estimation model when the DIC image data representative of at least the head of the physical person according to different points of view are presented at the input of the trained MP parameter estimation model.

[0102] According to a variant of step 231, said at least one first parameter P1 is modified as a function of at least one second parameter P2.

[0103] For example, said at least one parameter P1 may be replaced by said at least second parameter P2, for example by concatenating said at least one first and second parameters.

[0104] According to a particular and non-limiting exemplary embodiment, said at least second parameter P2 can be obtained from an action of a user on the HMI of the virtual fitting system of at least one accessory.

[0105] This variant can allow a user, for example a hairdresser, to change a hairstyle (haircut) by modifying at least a first parameter P1 of its digital representation according to at least a second parameter P2 that he chooses. This hairdresser can then present to his client different variations of haircut from the virtual accessory fitting system 1.

[0106] According to a particular and non-limiting exemplary embodiment, said at least second parameter P2 can be obtained from the DIC image data.

[0107] For example, said at least one second parameter P2 can be obtained by known image processing methods such as linear filtering, morphological operations for example to obtain color information.

[0108] According to another particular and non-limiting exemplary embodiment, said at least second parameter P2 may be a scene lighting parameter in which the digital twin JN evolves or is defined as a function of data which define a choice of scene.

[0109] In a step 232, the digital representation RNC is obtained as a function of said at least one first parameter P1.

[0110] According to a particular and non-limiting exemplary embodiment, the second step 232 may comprise a step 2321 of obtaining a representation of locks of hair by parametric curves from the DIC image data and said at least one first parameter P1 may be a parameter of said parametric curves. The RNC digital representation may then be defined by said parametric curves.

[0111] For example, the parametric curve can be an interpolation spline (sometimes called a cerce in French). An interpolation spline is a piecewise function consisting of a polynomial over each interval defined between two knots that can be likened to points in three-dimensional geometric space. The degree of the interpolation spline is defined by the highest-degree polynomial used: if we simply join the knots with straight lines, that is, if the interpolation spline is made up of first-degree polynomials, the interpolation spline is of degree 1, etc. If all the polynomials have the same degree, we speak of a uniform interpolation spline. If all the polynomials are of the third degree, we speak of a cubic interpolation spline. The strands of hair represented by interpolation splines can then vary in length, thickness and curvature according to coefficients of the polynomials defining the interpolation splines. These coefficients can then be said at least one first parameter P1 or be determined from said at least one first parameter P1.

[0112] According to another particular and non-limiting embodiment, the second step 232 may comprise a step 2322 of obtaining a hair class identifier IDC as output from a trained MC hair classification model from a set of hair classes when the DIC image data are presented as input to the trained MC hair classification model, each hair class identifier referencing a parametric hair model. The second step 232 may further comprise a step 2323 of adjusting the hair model identified by the hair class identifier IDC as a function of said at least one first parameter P1.

[0113] According to a variant of the method of Figure 3, the MP parameter estimation model is implemented by a first deep convolutional neural network and the MC hair classification model is implemented by a second deep convolutional neural network.

[0114] A deep convolutional neural network, also called a convolutional neural network and noted CNN or ConvNet (from the English "Convolutional Neural Networks") corresponds to an acyclic artificial neural network (from the English "feed-forward"). Such a convolutional neural network comprises a convolutional part implementing one or more convolution layers and a densely connected part implementing one or more densely connected (or fully connected) neural layers ensuring the classification of information according to an MLP type model (from the English "Multi Layers Perceptron" or in French "Multilayer Perceptrons") for example.

[0115] According to the present invention, the first and second convolutional neural networks may each comprise a MIL (multiinstance learning) type architecture which makes it possible to label all of the images present at the input of these convolutional neural networks which represent the same hair from different points of view. The first and second convolutional neural networks then learn how the hair of a natural person is represented from different points of view and which are the first parameters which best correspond to this representation of the hair of the natural person.

[0116] Figure 4 illustrates a flowchart of the different steps of a method for obtaining (step 231) at least a first parameter P1 of a parametric hair model at the output of the trained MP model for parameter estimation when the DIC image data are presented as input to the MP model, according to a particular and non-limiting exemplary embodiment of the present invention.

[0117] In a sub-step 2311 of step 231, the device 103, implementing the first convolutional neural network, provides the DIC image data as input to a so-called convolutional part of the first convolutional neural network to determine data D1 representative of characteristics of the hair of the natural person 10. The convolutional part advantageously implements one or more convolution layers whose objective is to detect the presence of these characteristic(s) which may be of an aesthetic and / or mechanical nature of the hair of the natural person or to detect patterns associated with said characteristics sought in the DIC image data introduced as input to the convolutional part.For this purpose, a set of at least one convolution filter is applied to the DIC image data, where the DIC image data may be provided, for example, in the form of a two-dimensional matrix (corresponding to the pixel grid of an image, for example). For example, 50, 100, 200 or more convolution filters may be applied, where each filter has a determined size and stride. For example, each filter may have a size of 1x7 and a stride of 1. In other words, the input matrix containing the DIC image data is passed. in a first convolution layer and convolution operations are applied to this input matrix based on the determined size and step filters. The stride corresponds to the number of pixels by which the window corresponding to the filter moves in the input tensor (input matrix for example). Each convolution filter can represent for example a determined characteristic sought in an image by sliding the window corresponding to the filter on the image, and by calculating the convolution product between this determined characteristic and each portion of the image scanned by the filter associated with this determined characteristic. The result of the convolution product makes it possible to determine the presence or absence of the determined characteristic in the input image.

[0118] Alternatively, one or more successive convolution operations may be applied to the DIC image data, with each convolution operation applying a set of convolution filters to the data obtained as output from the previous convolution operation.

[0119] According to a variant, one or more so-called "pooling" operations (for example, one or more "max pooling" operations and / or one or more "average pooling" operations) may be implemented and applied to the data obtained from the convolution operation(s). According to this variant, the "pooling" operation(s) may, for example, and optionally, be matched or associated with an operation of random deactivation of a portion of the neurons in the network, with a determined probability, for example, a probability of 0.2 (20%), 0.3 (30%) or 0.4 (40%), this technique being known as "dropout".

[0120] In a sub-step 2312 of step 231, the device 103 provides the data D1 as input to the so-called densely connected part of the first convolutional neural network to determine a set of at least one first parameter P1.

[0121] The densely connected layer implements a classification operation on the D1 data. For this purpose, the densely connected part of the first network of Convolutional neurons consist of one or more layers of densely connected or fully connected neurons. Each layer of neurons provides output data that is obtained by applying a linear combination and optionally an activation function to the data present at its input.

[0122] The densely connected layer outputs a vector of size N, where N is the number of sets of at least one first parameter P1, each set of at least one first parameter P1 corresponding to a parametric hair model. Each element of the output vector provides the probability for the data D1 to correspond to a set of at least one first parameter P1.

[0123] The input vector may pass for example through a layer of densely connected neurons, with for example 128, 256 or more neurons each connected to each of the neurons of a layer comprising as many neurons as there are sets of at least one first parameter P1, a set of at least one first parameter P1 of a parametric hair model being associated with each neuron of the output layer of the densely connected part.

[0124] Figure 5 illustrates a flowchart of the different steps of a method for obtaining (step 2322) a hair class identifier IDC at the output of the trained hair classification model MC when the image data DIC are presented as input to the model MC, according to a particular and non-limiting exemplary embodiment of the present invention.

[0125] In a sub-step 23221 of step 2322, the device 103 implementing the second convolutional neural network provides the DIC image data as input to a so-called convolutional part of the second convolutional neural network to determine data D1 representative of characteristics of the hair of the natural person 10. The convolutional part advantageously implements one or more convolution layers whose objective is to detect the presence of these characteristic(s) which may be of an aesthetic and / or mechanical nature of the hair of the natural person or to detect patterns associated with said characteristics. searched in the DIC image data introduced as input to the convolutional part. For this purpose, a set of at least one convolution filter is applied to the DIC image data, this DIC image data being able to be provided for example in the form of a two-dimensional matrix (corresponding to the pixel grid of an image for example). For example, 50, 100, 200 or more convolution filters can be applied, each filter having a determined size and stride. For example, each filter has a size equal to 1x7 and a stride of 1. In other words, the input matrix containing the DIC image data passes into a first convolution layer and convolution operations are applied to this input matrix on the basis of the determined size and stride filters.The stride is the number of pixels by which the window corresponding to the filter moves in the input tensor (input matrix for example). Each convolution filter can represent, for example, a specific characteristic sought in an image by sliding the window corresponding to the filter over the image, and by calculating the convolution product between this specific characteristic and each portion of the image scanned by the filter associated with this specific characteristic. The result of the convolution product makes it possible to determine the presence or absence of the specific characteristic in the input image.

[0126] Alternatively, one or more successive convolution operations may be applied to the DIC image data, with each convolution operation applying a set of convolution filters to the data obtained as output from the previous convolution operation.

[0127] According to a further variant, one or more so-called “pooling” operations (for example, one or more “max pooling” operations and / or one or more “average pooling” operations) can be implemented and applied to the data obtained from the convolution operation(s). According to this variant, the “pooling” operation(s) can be, for example, and optionally, combined or associated with a deactivation operation. random selection of a portion of the neurons in the network, with a specific probability, for example a probability of 0.2 (20%), 0.3 (30%) or 0.4 (40%), this technique being known as "dropout".

[0128] The convolutional part of the second convolutional neural network outputs a set of detected feature(s) from the DIC image data provided as input to the convolutional part. Information representative of the presence of determined feature(s) is provided as input to a so-called densely connected part of the second convolutional neural network.

[0129] In a sub-step 23222 of step 2322, the device 103 provides the data D1 as input to the so-called densely connected part of the second convolutional neural network to determine the hair class identifier IDC.

[0130] The densely connected layer implements a classification operation on the data D1 . For this purpose, the densely connected part of the second convolutional neural network comprises one or more layers of densely connected or fully connected neurons. Each layer of neurons provides output data that are obtained by applying a linear combination and then optionally an activation function to the data present at its input.

[0131] The densely connected layer outputs a vector of size N, where N is the number of sets of at least one first parameter P1 of hair classes, each hair class corresponding to a parametric hair model. Each element of the output vector provides the probability for the data D1 to correspond to a hair class.

[0132] The values ​​of the convolution filters of the first and second convolutional neural networks and the parameters of their densely connected layers are advantageously determined in a so-called learning phase, for example supervised learning, according to a method known to those skilled in the art. In a learning phase, a large number (for example hundreds, thousands, tens of thousands or more) of hair images whose associated features are used to learn the different values ​​or coefficients of the convolution filters. In such a learning phase, a method known as error gradient backpropagation can be implemented, for example. Similarly, in this learning phase, a large number (e.g., hundreds, thousands, tens of thousands or more) of data associations representative of hair features are used to learn the parameters of the densely connected part allowing the classification of the data in order to obtain a set of at least a first parameter P1 of a parametric hair model (first convolutional neural network) or a hair class identifier (second convolutional neural network).

[0133] For example, learning can be done from synthetically obtained image data.

[0134] According to a variant, a set of at least one first parameter P1 (first convolutional neural network) or a hair class (second convolutional neural network) can be determined for each image of a sequence of consecutive images, from the DIG image data of these images. A plurality of sets of at least one first parameter P1 (or hair classes) can thus be obtained for the sequence with as many sets of at least one first parameter P1 (or hair classes) as there are images in the sequence. A temporal filtering (for example an average) can advantageously be applied to this plurality of sets of at least one first parameter P1 (or hair classes) to determine a single set of at least one first parameter P1 (an identifier of a hair class).Such temporal filtering may for example be carried out or implemented by a digital filter known to those skilled in the art, for example defined by a mathematical equation, such as a difference equation, or by a neural network, for example recurrent.

[0135] Alternatively, the image data used for training the first and second convolutional neural networks may be data from groups of 3 images taken from the front, right profile and left profile of a same hair. Each group of images can be associated with annotations that indicate the expected hair class corresponding to the 3 images in the group of images and that indicate a value of at least a first parameter P1 of a parametric hair model.

[0136] Alternatively, the image data used for training the first and second convolutional neural networks may be synthetically obtained image data, which facilitates the generation of a large amount of image data required for training the first and second convolutional neural networks.

[0137] Synthetic image data used for training can be generated from parametric hair models and randomly chosen parameter sets of these models to obtain hair variations. Viewpoint variations can also be used to introduce viewpoint variations in hair images.

[0138] A problem of generalizing the training of the first and second convolutional neural networks arises when synthetically generating image data for training from images of hair variations when several of these images include the same face. To avoid this problem, the image data used for training can be segmented to identify at least one spatial region of these images that includes image data representative of the hair. The image data used for training can then correspond to created images that only include these spatial regions of the images and therefore exclude the spatial regions corresponding to the faces.

[0139] The device 104 is configured to render a digital avatar representative of the physical person from the digital twin JN (of the digital representations RNT and RNC) and the accessory data DAC. For this purpose, the device 104 calculates data representative of a view according to a point of view of the RNT representation combined with the accessory data DAC and the digital representation RNC.

[0140] The device 104 for rendering a digital avatar representing the physical person wearing at least one accessory may comprise a rendering engine which calculates data representative of an image of the digital twin JN wearing said at least one accessory according to a point of view. The image may then be displayed on a screen of the device 104.

[0141] According to an exemplary embodiment, the rendering engine may be a real-time rendering engine.

[0142] According to an exemplary embodiment, the rendering engine can be implemented on a GPU computing server.

[0143] For example, the real-time rendering engine can be a 3D simulation engine.

[0144] According to a particular and non-limiting exemplary embodiment, said at least one parameter P1 can be applied to a parametric hair model using software.

[0145] The methods of Figures 2 through 5 allow a person to virtually try on accessories and decide whether or not to purchase those accessories.

[0146] Through the HMI of the virtual accessory fitting system 1, a natural person can access a sales site and select accessories. They can also choose to virtually try on these accessories. The HMI can then be designed so that this person adds the selected accessories to a basket, which can be called a virtual fitting room. The method described above is then implemented so that a digital twin of this person is calculated and the person can visualize a view of their digital twin then wearing accessories that this person would have selected. Thus, the natural person can interact with their digital twin from this HMI which displays the images of this digital twin wearing these accessories according to points of view that can be selected by this person. The person can also choose the conditions of the scene (lighting, content of the scene) in which their digital twin evolves.The person can also choose the lighting conditions of the scene in which they are evolving. digital twin or even change the scene in which the digital twin is presented.

[0147] Figure 6 schematically illustrates the device 103 configured to obtain a digital RNC representation of a head of hair of a physical person digitally represented by a digital twin, according to a particular and non-limiting exemplary embodiment of the present invention.

[0148] According to a particular embodiment, the device 103 corresponds to a server or a calculator of the “cloud” 100.

[0149] The device 103 is for example configured for implementing the operations described with regard to figures 1 and / or the steps of the methods described with regard to figures 2 to 5. The elements of the device 103, individually or in combination, can be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. The device 103 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

[0150] The device 103 comprises one (or more) processor(s) 1030 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 103. The processor 1030 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 103 further comprises at least one memory 1031 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0151] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored in the memory 1031.

[0152] According to various particular and non-limiting exemplary embodiments, the device 103 is coupled in communication with other similar devices or systems and / or with communication devices.

[0153] According to a particular and non-limiting exemplary embodiment, the device 103 comprises a block 1032 of interface elements for communicating with external devices, for example a remote server or the “cloud” 100. The interface elements of the block 1032 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); - HDMI interface (from the English “High Definition Multimedia Interface”).

[0154] According to a particular and non-limiting exemplary embodiment, the device 103 can provide output signals to one or more external devices, such as a display screen 1033, touch-sensitive or not, one or more speakers 1034 and / or other peripherals 3601034 (projection system for example) via output interfaces 1036, 1037, 1038 respectively. According to a variant, one or other of the external devices is integrated into the device 103.

[0155] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for obtaining a digital representation of a head of hair of a natural person which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a system configured for the implementation of such a method.

Claims

CLAIMS 1. Method for obtaining a digital representation of a head of hair of a natural person, said digital representation being intended to be used for adding a head of hair to a digital twin of the natural person, said method comprising the following steps: - obtaining (231) at least a first parameter of a parametric hair model at the output of a trained parameter estimation model when image data representative of at least the head of the physical person from different points of view are presented at the input of the trained parameter estimation model; and - obtaining (232) the digital representation of the hair of the natural person as a function of said at least one first parameter, characterized in that obtaining (232) the digital representation of the hair of the natural person as a function of said at least one first parameter comprises the following sub-steps: - obtaining (2322) a hair class identifier as output from a trained hair classification model from a set of hair classes when the image data are presented as input to the trained hair classification model, each hair class identifier referencing a parametric hair model; and - adjustment (2323) of the hair model identified by the hair class identifier according to said at least one first parameter.

2. The method of claim 1, wherein the trained parameter estimation model is implemented by a first deep convolutional neural network.

3. Method according to claim 1 or 2, wherein said at least one first parameter is modified as a function of at least one second parameter.

4. Method according to claim 3, wherein said at least one second parameter is obtained from an action of a user on a human-machine interface or from image data.

5. Method according to claim 3, wherein said at least one second parameter is obtained from image data representative of at least the head of the natural person.

6. Method according to claim 3, in which said at least one second parameter is a scene lighting parameter in which the digital twin operates or said at least second parameter is defined as a function of data which defines a choice of scene.

7. Method according to one of the preceding claims, in which the digital representation of the hair of the natural person is a parametric curve obtained (2321) from image data representative of at least the head of the natural person and said at least one first parameter is a parameter of the parametric curve.

8. The method of claim 2, wherein the trained hair classification model is implemented by a second deep convolutional neural network.

9. The method of claim 8, wherein the first and second deep convolutional neural networks each comprise a multiple instance learning type architecture.

10. Method according to claim 9, in which training of the first deep convolutional neural network and / or of the second deep convolutional neural network is carried out from image data obtained synthetically or from groups of three images taken from the front, right profile and left profile of the physical person.

11. Device for obtaining a digital representation of a head of hair of a natural person comprising means for implementing the steps of the method according to one of claims 1 to 5.

12. System for virtual fitting of accessories by a natural person digitally represented by a digital twin comprising a device according to claim 6.

13. Computer program comprising instructions for implementing the method according to any one of claims 1 to 5, when these instructions are executed by a processor.

14. Computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to one of claims 1 to 5.

Images