Method for obtaining a digital representation of a natural person's hair

Abstract

The present invention relates to a method and device for obtaining a digital representation of a head of hair of a natural person, said digital representation being intended to be used for adding a head of hair to a digital twin of the natural person, said method comprising the following steps: - obtaining (231) at least a first parameter of a parametric hair model at the output of a trained parameter estimation model when image data representative of at least the head of the natural person according to different points of view are presented at the input of the trained parameter estimation model; and - obtaining (232) the digital representation of the hair of the natural person as a function of said at least a first parameter. Figure for abstract: Figure 3

Description

Title of the invention: Method for obtaining a digital representation of a natural person's hair Technical field

[0001] The present invention relates to systems for virtual fitting of accessories by a digitally represented natural person. In particular, the present invention relates to obtaining a digital representation of a head of hair of a natural person, said digital representation being intended to be used to add a head of hair to a digital twin (avatar) of the natural person. Technological background

[0002] Applications or services are known that allow a person to try on an accessory virtually. These applications or services are based on augmented reality, that is, for example, on a combination of two-dimensional (2D) digital images captured of the person that are combined with a three-dimensional (3D) rendering of an accessory. Note that augmented reality can also be obtained by a combination of a volumetric image and a rendering of a 3D scene.

[0003] The combination of 2D and 3D image data of augmented reality-based applications limits the display of the image resulting from the combination of this data to a single viewpoint or at best to different viewpoints whose view axes are very close to each other.

[0004] Current virtual accessory fitting applications or services thus allow a natural person to form an opinion concerning the purchase of these accessories by viewing on a screen an image formed by the combination of 2D images of this natural person and these accessories.

[0005] These applications or services based on augmented reality are usually limited to rendering images resulting from a front view of a natural person. Indeed, a capture of image data representing a front view of a natural person is generally made because this natural person usually looks at the camera which performs this image data capture. The image resulting from the combination of this captured image data and data representing an accessory is then a 2D image of this natural person representing a front view of this natural person. Applications or services based on augmented reality are therefore suitable when the accessory in question is intended to be worn on the face of the natural person, for example makeup. On the other hand, when the accessory is intended to be worn on one side of the natural person, for example earrings, these applications or services are not well suited because they only offer 2D digital representations while the physical person may want to have front and side views, for example, before deciding on their accessory purchase.

[0006] In its French patent application No. FR2301334 filed on February 14, 2023, the applicant describes a system for virtual fitting of accessories by a digitally represented living being. This system makes it possible to create a three-dimensional digital twin JN of a part of a living being, of a set of parts of a living being or of a living being in its entirety, and to calculate a 3D view of this digital twin JN wearing a digitally represented accessory.

[0007] When the virtual fitting system is used to try on head accessories such as, for example, earrings, makeup, hats, etc., the digital twin JN represents at least the head of the physical person (and possibly other parts of this physical person) and it is necessary to define a digital representation of the hair of the physical person which will be placed on the digital representation of the head of the digital twin JN. This digital representation of the hair of a physical person must correspond to the haircut of the physical person so that the digital twin JN is as realistic as possible and the virtual fitting system for accessories is close to a real fitting of an accessory. Summary of the present invention

[0008] An object of the present invention is to solve at least one of the drawbacks of the technological background.

[0009] Another object of the present invention is to improve the realism of a digital twin representing at least the head of a physical person.

[0010] Another object of the present invention is to improve virtual accessory fitting systems.

[0011] According to a first aspect, the present invention relates to a method for obtaining a digital representation of a head of hair of a natural person, said digital representation being intended to be used for adding a head of hair to a digital twin of the natural person, said method comprising the following steps: - obtaining at least a first parameter of a parametric hair model as output from a trained parameter estimation model when image data representative of at least the head of the physical person from different points of view are presented as input to the trained parameter estimation model; and - obtaining the digital representation of the hair of the natural person depending on said at least one first parameter.

[0012] According to a particular and non-limiting embodiment of the present invention, the trained parameter estimation model can be implemented by a first deep convolutional neural network.

[0013] According to a particular and non-limiting embodiment of the present invention, said at least one first parameter can be modified as a function of at least one second parameter.

[0014] According to a particular and non-limiting embodiment of the present invention, said at least second parameter can be obtained from an action of a user on a human-machine interface or from image data.

[0015] According to a particular and non-limiting embodiment of the present invention, obtaining the digital representation of the hair of the natural person as a function of said at least one first parameter may comprise the following sub-steps: - obtaining a hair class identifier as output from a trained hair classification model from a set of hair classes when the image data is presented as input to the trained hair classification model, each hair class identifier referencing a parametric hair model; and - adjustment of the hair model identified by the hair class identifier according to said at least one first parameter.

[0016] According to a particular and non-limiting embodiment of the present invention, the trained hair classification model can be implemented by a second deep convolutional neural network.

[0017] According to a second aspect, the present invention relates to a device for obtaining a digital representation of a head of hair of a natural person comprising means for implementing the steps of the method according to the first aspect of the present invention.

[0018] According to a third aspect, the present invention relates to a system for virtual fitting of accessories by a physical person digitally represented by a digital twin comprising a device according to the second aspect of the present invention.

[0019] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0020] Such a computer program may use any programming language, and be in the form of source code, object code, or int an intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0021] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0022] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a RAM memory, a CD-ROM or a microelectronic circuit type ROM memory, or even a magnetic recording means or a hard disk.

[0023] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.

[0024] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0025] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 6, in which:

[0026] [Fig-1] schematically illustrates a virtual accessory fitting system by a natural person digitally represented by a digital twin, according to a particular and non-limiting embodiment of the present invention;

[0027] [Fig.2] illustrates a flowchart of the different steps of a method for obtaining and rendering the digital twin from image data according to a particular and non-limiting exemplary embodiment of the present invention;

[0028] [Fig.3] illustrates a flowchart of the different steps of a method for obtaining a digital representation of the hair of the natural person from image data, according to a particular and non-limiting exemplary embodiment of the present invention;

[0029] [Fig.4] illustrates a flowchart of the different steps of a method for obtaining at least a first parameter of a parametric hair model at the output of a trained parameter estimation model when image data are presented as input to the model, according to a particular exemplary embodiment and not limiting of the present invention;

[0030] [Fig.5] illustrates a flowchart of the different stages of a process for obtaining of a hair class identifier at the output of a trained hair classification model when the image data is presented as input to the model, according to a particular and non-limiting exemplary embodiment of the present invention;

[0031] [Fig.6] schematically illustrates the device configured to obtain a representation digital representation of a head of hair of a natural person digitally represented by a digital twin, according to a particular and non-limiting exemplary embodiment of the present invention. Description of examples of implementation

[0032] A method and a device for obtaining a digital representation of a natural person's hair and a system for virtual fitting of an accessory by a natural person represented digitally by a digital twin, will now be described in what follows with joint reference to Figures 1 to 6. The same elements are identified with the same reference signs throughout the description which follows.

[0033] [Fig.l] schematically illustrates a system 1 for virtual fitting of accessories by a physical person 10 represented digitally by a digital twin JN, according to a particular and non-limiting embodiment of the present invention.

[0034] The system 1 comprises a device 101 for obtaining DIC image data representative of at least the head of the physical person 10 from different points of view, a device 102 for obtaining a digital representation RNT of at least the head of the physical person 10 in a three-dimensional geometric space from the DIC image data, a device 103 for obtaining a digital representation RNC of the hair of the physical person 10 in the three-dimensional geometric space from the DIC image data, a device 104 for obtaining DAC accessory data representative of a digital representation of at least one accessory in the three-dimensional geometric space and a device 105 for rendering the digital twin JN representative of at least the head of the physical person 10 from the RNC and RNT representations and the DAC accessory data.

[0035] The three-dimensional geometric space is a space in which the part or set of parts of the physical person 10 or the physical person 10 as a whole is geometrically represented from the DIC image data and the camera(s) used to capture the images from which this DIC image data is obtained.

[0036] The digital twin JN, also called realistic avatar, is a set of three-dimensional digital data which represents in the metaverse at least the head of the natural person 10 who is represented by the DIC image data.

[0037] The geometry and texture of the JN digital twin are defined by the RNT representation. The DAC data can then be added to the JN digital twin to visualize the virtual wearing of accessories by the JN digital twin.

[0038] According to a variant, the system 1 may further comprise a human-machine interface (HMI).

[0039] For example, the HMI may be partly implemented by several of the devices 101 to 105.

[0040] For example, it may allow a user to enter parameters through graphical means such as a keyboard or touchscreen of the device 101 or 105.

[0041] According to a variant, the devices 101, 104 and 105 can be implemented in separate devices.

[0042] According to a variant, the devices 101, 104 and / or 105 can be implemented in the same device.

[0043] According to an exemplary embodiment of this variant, the devices 101, 104 and / or 105 can be implemented in the same mobile communication device.

[0044] According to variants, the device 101, 104 and / or 105 may be one of the following devices: - a telephone; - a computer; - a tablet; - a pair of glasses equipped with at least one camera and one screen or in communication with at least one camera and one screen; - a pair of ocular contact lenses equipped with at least one camera and a screen then adaptive to a user's view.

[0045] The present invention is not limited to these exemplary devices but may extend to any device that would be configured to implement the devices 101 to 105.

[0046] The devices 102 and 103 are computers (CPU for Computer Processing Units in English, in French digital calculation units, or GPU for Graphical Processing Unit in English, in French graphic calculation unit) which are associated with buffer memories to execute calculation tasks.

[0047] The devices 101 to 105 can be connected for example in communication via a wired network (for example according to Ethernet and / or via a fiber optic link) and / or via a wireless network of the Wifi® type (according to IEEE 802.11 or one of the variations of IEEE 802.11 or via a cellular network of the LTE (Long-Term Evolution) type, LTE-Advanced (or in French LTE-advanced) and / or 5G and / or 6G. This is called “cloud computing” in English. (cloud computing in French) to refer to this type of architecture which uses memory and computing capacity of computers and servers distributed around the world and linked by a communication network. Such an architecture is illustrated in [Fig.l] by a cloud 100.

[0048] The devices 1011 105 are thus configured to transmit data to the “cloud” 100 and / or to receive data from the “cloud” 100.

[0049] The mobile communication infrastructure enabling wireless communication of data between the devices 101 to 105 and the “cloud” 100 may comprise, for example, one or more communication devices (not shown in [Fig.l]) of the relay antenna type (cellular network). In a communication mode using such a network architecture, the data intended for the devices 101 to 105 may be received, for example, from the “cloud” 100 and the data may be transmitted by the devices 101 to 105 via the “cloud” 100 through one or more relay antennas (each relay antenna being, for example, connected to the “cloud” 100 via a wired link).

[0050] The DIC image data, the data representative of the RNT and RNC representations and the DAC accessory data are exchanged by the devices 101 to 105 via the “cloud” 100.

[0051] The present invention is not limited to the communication architecture between the devices 101 to 105 illustrated in [Fig.l] but extends to any type of communication architecture.

[0052] For example, when the devices 101, 104 and 105 are the same mobile communication device, this mobile communication device can be configured to transmit the DIC image data to the devices 102 and 103 via the “cloud” 100 and to receive data representative of the RNT and RNC representations via the “cloud” 100.

[0053] According to another example, the devices 102 and 103 may be the same device which is then configured to receive DIC image data via the “cloud” 100 and to transmit the data representative of the RNT and RNC representations via the “cloud” 100.

[0054] Other variants of communication architecture between the devices 101 to 105 are obviously conceivable without departing from the scope of the present claimed invention.

[0055] [Fig.2] illustrates a flowchart 2 of the different steps of a method for obtaining and rendering the digital twin JN from the DIC image data according to a particular and non-limiting exemplary embodiment of the present invention.

[0056] In a first step 21, the DIC image data representative of at least the head of the physical person 10 according to different points of view, are obtained from the device 101.

[0057] In a second step 22, the digital representation RNT is obtained from the DIC image data.

[0058] In a third step 23, the RNC digital representation is obtained from the DIC image data.

[0059] In a fourth step 24, the DAC accessory data is obtained.

[0060] In the fifth step 25, the digital twin JN defined from the digital representations RNT and RNC and the accessory data DAC is rendered, by calculating a view of this digital twin JN according to a point of view.

[0061] The device 101 comprises an image capture means.

[0062] According to an exemplary embodiment of the present invention, the image capture means may comprise at least one camera corresponding for example to: - an infrared camera; - an RGB type image acquisition camera (from the English “Red, Green, Blue” or in French “Rouge, vert, bleu”); - a camera platform (“Rig” in English) (called Lightfield) or Volumetric camera Rig of RGB type; - a LIDAR type acquisition camera (“light detection and ranging” or “laser imaging detection and ranging” in English); or - an image acquisition camera associated with one or more devices (for example one or more LEDs (from the English “Light-Emitting Diode” or in French “Electroluminescent Diode”) emitting light in the infrared or in the near infrared band.

[0063] According to an exemplary embodiment of the present invention, the image capturing means may correspond to a camera of a mobile communication device comprising a camera.

[0064] According to variants, the captured images may correspond either to still images captured from different points of view of at least the head of the physical person 10 or to images from a video captured of the physical person 10 which represents at least the head of said physical person 10 from different points of view.

[0065] According to an exemplary embodiment of the present invention, the image capturing means may be a scanning system comprising a plurality of cameras synchronized with each other to capture several images at different viewpoints of at least the head of the physical person. Captured image data then represents these images captured at different viewpoints.

[0066] In the case of a camera platform (sometimes referred to as a “Lightfield” case in English and "volumetric image data" in French), the cameras can be positioned in a circular or planar manner on a physical capture device and can capture in a single iteration or several (if motion capture) at least the head of the physical person.

[0067] According to an exemplary embodiment of the present invention, the image capture means may comprise at least one volumetric sensor making it possible to obtain volumetric image data (in English “lightfield”).

[0068] According to an exemplary embodiment of the present invention, the device 101 may be configured so that the DIC image data corresponds to image data of the images captured by the image capturing means.

[0069] Typically, three images are captured, one representing the head of the physical person 10 from the front, another representing it in a left profile and the other representing it in a right profile.

[0070] According to an exemplary embodiment of the present invention, the device 101 may be configured to apply processing to the images captured by the image capturing means and the DIC image data corresponds to the processed image data.

[0071] According to an exemplary embodiment of the present invention, if a video can be captured by the image capture means, a processing operation can be the selection of images representative of at least the head of the physical person from among the images of this video. Typically, three images are selected, one representing the head from the front, another representing it in a left profile and the other representing it in a right profile.

[0072] According to an exemplary embodiment of the present invention, the hair of the natural person represented in the captured images can be segmented from the rest of the captured images, that is to say that at least one spatial zone which delimits the hair of the head of the natural person is identified in each of the captured images. The DIC image data can then correspond to the image data of the pixels contained in the spatial zones defined in the captured images.

[0073] For example, the segmentation of hair from the rest of a captured image can be implemented from a model based on a PSPNet type architecture developed by Li et al. (Li, T., Bolkart, T., Black, M., Li, H., & Romero, J. (2017) or BiSeNet (https: / / arxiv.org / abs / 2004.02147) Learning a Model of Facial Shape and Expression from 4D Scans. ACM Trans. Graph., 36(6)).

[0074] According to a variant, the DIC image data may further comprise information representative of image contents and / or descriptive information of geometry of at least the head of the physical person in a three-dimensional geometric space.

[0075] The device 102 is configured to obtain the digital representation RNT of at least the head of the physical person 10 in the three-dimensional geometric space from the DIC image data.

[0076] According to a particular and non-limiting exemplary embodiment of the present invention, the device 102 can be configured to implement the method described in French patent application No. FR2301334 filed on February 14, 2023.

[0077] According to this patent application, obtaining the digital representation RNT of at least the head of the physical person 10 comprises obtaining DM mesh data (“mesh” in English) representative of a three-dimensional mesh structure represented in the three-dimensional geometric space by a set of vertices of polygonal shapes forming a structure, each polygonal shape, for example triangles or quadrangles, sharing at least one side with another polygonal shape.

[0078] The device 102 is configured to calculate mesh data DM of the digital twin JN by modifying mesh data DMI of an initial digital twin JNI according to morphological information extracted from the image data DIC.

[0079] The initial digital twin JNI is a digital representation of at least one head of an asexual human being. This digital representation comprises the DMI mesh data representative of a three-dimensional mesh structure represented in the three-dimensional geometric space by a set of vertices of polygonal shapes forming a structure, each polygonal shape sharing at least one side with another polygonal shape.

[0080] According to a particular and non-limiting exemplary embodiment, the DMI mesh data can be modified by modifying the spatial positions of the vertices defined in the three-dimensional geometric space.

[0081] The three-dimensional mesh structure of the initial twin JNI is then deformed (stretched, pressed, etc.) so that the modified three-dimensional mesh structure of the initial twin JNI resembles in the three-dimensional geometric space the geometry of at least the head of the physical person 10.

[0082] The DM mesh data of the digital twin JN is then equal to the modified DMI data.

[0083] The DM data of the digital twin JN are obtained by an iterative process of deformation of the DMI data of the digital twin JNI which incorporates all the knowledge of the morphology of a living being such as for example all the morphology of a Caucasian, Afro, Asian human being and / or of an animal being, that is to say the position, a structure and a deformation of the eyes, mouth and folds of the skin, an underlying muscular system and a series of controllers allowing the deformations.

[0084] An iterative deformation method can be based on a neural network (Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv: 1704.04861, 2017) and / or on a deep learning method (Ayush Tewari, Michael Zollôfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian. MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In The IEEE International Conference on Computer Vision (ICCV), 2017.) and / or on a reinforcement learning method and / or using generative adversarial networks (GAN from the English “Generative Adversarial Network”).

[0085] In the case where a deep learning method is used for the reconstruction of the face of the physical person 10, a parametric modeling of the face called "3DMM" can be used. This is based on a statistical model of human faces which makes it possible to represent any face in a weighted sum of 3D mesh structure, 3D mesh structure which assumes the prior creation of an adapted 3D topology. Only the frontal part of the face - the only deformable one - is taken into account (we speak of "monkey mask" in English). The geometric aspect is enriched by the representation of the deformations corresponding to the expressions of the face. This is accompanied by a set of several parameters, typically of the order of 150 parameters for the identity of the face and 100 parameters for the expressions.The estimation of these hundreds of parameters can be achieved by regression of these parameters using a deep learning method learned on a large set of face images (Ayush Tewari, Michael Zollôfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian. MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In The IEEE International Conference on Computer Vision (ICCV), 2017), and (Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August,15:2018, 2018).

[0086] Thus, during this iterative deformation process, a first three-dimensional (3D) mesh structure defined by the DNI data is positioned in the three-dimensional geometric space. This first three-dimensional mesh structure is then compared to three-dimensional mesh structures stored in a database, each stored 3D mesh structure representing at least the face of the natural person. The iterative process also compares each two-dimensional (2D) image calculated from the first structure of 3D mesh to 2D images calculated from a 3D mesh structure stored in the database. The method of comparing these elements seeks to bring the 2D images calculated from the first 3D mesh structure closer to 2D images calculated from 3D mesh structures stored in the database. The method thus iteratively improves the distances between the vertices of the first 3D mesh structure and the vertices of a 3D mesh structure and by micro-displacements of the vertices of the first 3D mesh structure in the three-dimensional geometric space so as to optimize the resulting first 3D mesh structure.This spatial operation of deformation of the first 3D mesh structure therefore proceeds by successive iterations until a resemblance is obtained between the first 3D mesh structure and a 3D mesh structure stored in the database geometrically representing at least the head of the physical person. The 3D mesh structure thus obtained corresponds in resemblance to the DIC image data.

[0087] According to this patent application, obtaining the digital representation RNT of at least the head of the physical person 10 may further comprise obtaining texture data DT from texture data extracted from the image data DIC.

[0088] According to a particular and non-limiting exemplary embodiment of the present invention, the texture data DT may be texture data extracted from a defined area on the face of the physical person 10 from the image data DIC.

[0089] According to an exemplary embodiment, the texture data of the area can be obtained by assembling the DIC image data, isolated, distributed and assembled to reconstruct a single texture.

[0090] For example, the stitching method may deform the texture of each of the images obtained from the DIC image data by modifying the color and contrast of the pixels of the texture images, corresponding to the texture of the images obtained from the DIC image data, and correcting the boundaries between the texture images to generate a single texture.

[0091] This unique texture can then be projected onto the surface of the 3D mesh structure corresponding to the DM data (projection known as UV mapping, the letters U and V designating the axes of the 2D texture). This then allows a reprojection of said assembled texture onto said 3D mesh structure.

[0092] For example, texture data may be obtained from a surface of an area defined between the forehead, ears, and chin (“monkey mask” in English).

[0093] The device 105 is configured to obtain DAC accessory data representative of a three-dimensional representation of at least one accessory in the three-dimensional geometric space. This three-dimensional representation is intended to be added to the JN digital twin.

[0094] DAC accessory data is known to those skilled in the art. It represents any type of object, makeup. We often speak of a 2D / 3D layer which incorporates a material, a transparency, a relief and which is calculated by a shading and rendering process.

[0095] The device 103 is configured to obtain the digital representation RNC of the hair of the physical person 10 from the DIC image data. For this purpose, the device 103 is configured to implement the method of [Fig.3].

[0096] [Fig. 3] illustrates a flowchart of the different steps of a method for obtaining (step 23) the RNC digital representation of the hair of the natural person 10 from the DIC image data, according to a particular and non-limiting exemplary embodiment of the present invention.

[0097] In a step 231, the device 103 obtains at least a first parameter PI of a parametric hair model at the output of a trained MP parameter estimation model when the DIC image data representative of at least the head of the physical person according to different points of view are presented at the input of the trained MP parameter estimation model.

[0098] According to a variant of step 231, said at least one first parameter PI is modified as a function of at least one second parameter P2.

[0099] For example, said at least one parameter PI can be replaced by said at least second parameter P2, for example by concatenating said at least one first and second parameters.

[0100] According to a particular and non-limiting exemplary embodiment, said at least second parameter P2 can be obtained from an action of a user on the HMI of the virtual fitting system of at least one accessory.

[0101] This variant can allow a user, for example a hairdresser, to change a hairstyle (haircut) by modifying at least a first parameter PI of its digital representation as a function of at least a second parameter P2 that he chooses. This hairdresser can then present to his client different variations of haircut from the virtual accessory fitting system 1.

[0102] According to a particular and non-limiting exemplary embodiment, said at least second parameter P2 can be obtained from the DIC image data.

[0103] For example, said at least one second parameter P2 can be obtained by known image processing methods such as linear filtering, morphological operations for example to obtain color information.

[0104] According to another particular and non-limiting exemplary embodiment, said at least second parameter P2 can be a scene lighting parameter in which the digital twin JN operates or is defined based on data that defines a choice of scene.

[0105] In a step 232, the digital representation RNC is obtained as a function of said at least one first parameter PI.

[0106] According to a particular and non-limiting exemplary embodiment, the second step 232 may comprise a step 2321 of obtaining a representation of locks of hair by parametric curves from the DIC image data and said at least one first PI parameter may be a parameter of said parametric curves. The RNC digital representation may then be defined by said parametric curves.

[0107] For example, the parametric curve can be an interpolation spline (sometimes called a cerce in French). An interpolation spline is a piecewise function consisting of a polynomial over each interval defined between two knots that can be likened to points in three-dimensional geometric space. The degree of the interpolation spline is defined by the highest-degree polynomial used: if we simply join the knots by straight lines, that is, if the interpolation spline is made up of first-degree polynomials, the interpolation spline is of degree 1, etc. If all the polynomials have the same degree, we speak of a uniform interpolation spline. If all the polynomials are of third-degree, we speak of a cubic interpolation spline. The strands of hair represented by interpolation splines can then vary in length, thickness and curvature according to coefficients of the polynomials defining the interpolation splines.These coefficients can then be said at least one first PI parameter or be determined from said at least one first PL parameter.

[0108] According to another particular and non-limiting embodiment, the second step 232 may comprise a step 2322 of obtaining a hair class identifier IDC as output from a trained MC hair classification model among a set of hair classes when the DIC image data are presented as input to the trained MC hair classification model, each hair class identifier referencing a parametric hair model. The second step 232 may further comprise a step 2323 of adjusting the hair model identified by the hair class identifier IDC as a function of said at least one first parameter PL

[0109] According to a variant of the method of [Fig.3], the MP model for parameter estimation is implemented by a first deep convolutional neural network and the MC model for hair classification is implemented by a second deep convolutional neural network.

[0110] A deep convolutional neural network also called a neural network convolutional and noted CNN or ConvNet (from the English "Convolutional Neural Networks") corresponds to an acyclic artificial neural network (from the English "feed-forward"). Such a convolutional neural network comprises a convolutional part implementing one or more convolution layers and a densely connected part implementing one or more densely connected (or fully connected) neuron layers ensuring the classification of information according to an MLP type model (from the English "Multi Layers Perceptron" or in French "Perceptrons multicouches") for example.

[0111] According to the present invention, the first and second convolutional neural networks may each comprise a MIL (multi-instance learning) type architecture which makes it possible to label all of the images present at the input of these convolutional neural networks which represent the same hair from different points of view. The first and second convolutional neural networks then learn how the hair of a physical person is represented from different points of view and what are the first parameters which best correspond to this representation of the hair of the physical person.

[0112] [Fig.4] illustrates a flowchart of the different steps of a method for obtaining (step 231) at least a first PI parameter of a parametric hair model at the output of the trained MP model for parameter estimation when the DIC image data are presented at the input of the MP model, according to a particular and non-limiting exemplary embodiment of the present invention.

[0113] In a sub-step 2311 of step 231, the device 103, implementing the first convolutional neural network, provides the DIC image data as input to a so-called convolutional part of the first convolutional neural network to determine DI data representative of characteristics of the hair of the natural person 10. The convolutional part advantageously implements one or more convolution layers whose objective is to detect the presence of these characteristic(s) which may be of an aesthetic and / or mechanical nature of the hair of the natural person or to detect patterns associated with said characteristics sought in the DIC image data introduced as input to the convolutional part.For this purpose, a set of at least one convolution filter is applied to the DIC image data, where the DIC image data may be provided, for example, in the form of a two-dimensional matrix (corresponding to the pixel grid of an image, for example). For example, 50, 100, 200 or more convolution filters may be applied, where each filter has a determined size and stride. For example, each filter may have a size of 1x7 and a stride of 1. In other words, the input matrix containing the DIC image data is passed through a . first convolution layer and convolution operations are applied to this input matrix based on the determined size and step filters. The stride corresponds to the number of pixels by which the window corresponding to the filter moves in the input tensor (input matrix for example). Each convolution filter can represent for example a determined characteristic sought in an image by sliding the window corresponding to the filter on the image, and by calculating the convolution product between this determined characteristic and each portion of the image scanned by the filter associated with this determined characteristic. The result of the convolution product makes it possible to determine the presence or absence of the determined characteristic in the input image.

[0114] According to a variant, one or more successive convolution operations can be applied to the DIC image data, with for each convolution operation the application of a set of convolution filters to the data obtained at the output of the previous convolution operation.

[0115] According to a variant, one or more so-called "pooling" operations (for example one or more "max pooling" operations and / or one or more "average pooling" operations) can be implemented and applied to the data obtained from the convolution operation(s). According to this variant, the "pooling" operation(s) can be, for example, and optionally, matched or associated with an operation of random deactivation of a portion of the neurons of the network, with a determined probability, for example a probability of 0.2 (20%), 0.3 (30%) or 0.4 (40%), this technique being known as "dropout".

[0116] In a sub-step 2312 of step 231, the device 103 provides the data DI as input to the so-called densely connected part of the first convolutional neural network to determine a set of at least one first parameter PL

[0117] The densely connected layer implements a classification operation of the DI data. For this purpose, the densely connected part of the first convolutional neural network comprises one or more layers of densely connected or fully connected neurons. Each layer of neurons provides output data which are obtained by applying a linear combination and then optionally an activation function to the data present at its input.

[0118] The densely connected layer outputs a vector of size N, where N corresponds to the number of sets of at least one first PI parameter, each set of at least one first PI parameter corresponding to a parametric hair model. Each element of the output vector provides the probability for the DI data to correspond to a set of at least one first PL parameter.

[0119] The input vector may pass for example through a layer of densely connected neurons, with for example 128, 256 or more neurons each connected to each of the neurons of a layer comprising as many neurons as there are sets of at least one first PI parameter, a set of at least one first PI parameter of a parametric hair model being associated with each neuron of the output layer of the densely connected part.

[0120] [Fig.5] illustrates a flowchart of the different steps of a method for obtaining (step 2322) a hair class identifier IDC at the output of the trained hair classification model MC when the image data DIC are presented as input to the model MC, according to a particular and non-limiting example embodiment of the present invention.

[0121] In a sub-step 23221 of step 2322, the device 103 implementing the second convolutional neural network provides the DIC image data as input to a so-called convolutional part of the second convolutional neural network to determine DI data representative of characteristics of the hair of the natural person 10. The convolutional part advantageously implements one or more convolution layers whose objective is to detect the presence of these characteristic(s) which may be of an aesthetic and / or mechanical nature of the hair of the natural person or to detect patterns associated with said characteristics sought in the DIC image data introduced as input to the convolutional part.For this purpose, a set of at least one convolution filter is applied to the DIC image data, where the DIC image data may be provided, for example, in the form of a two-dimensional matrix (corresponding to the pixel grid of an image, for example). For example, 50, 100, 200 or more convolution filters may be applied, each filter having a determined size and stride. For example, each filter has a size equal to 1x7 and a stride of 1. In other words, the input matrix containing the DIC image data is passed through a first convolution layer and convolution operations are applied to this input matrix based on the determined size and stride filters. The stride corresponds to the number of pixels by which the window corresponding to the filter moves in the input tensor (input matrix, for example).Each convolution filter can represent, for example, a specific characteristic sought in an image by sliding the window corresponding to the filter over the image, and calculating the convolution product between this specific characteristic and each portion of the image scanned by the filter associated with this specific characteristic. The result of the convolution product makes it possible to determine the presence or absence of the specific characteristic in the input image.

[0122] According to a variant, one or more successive convolution operations can be applied to DIC image data, with for each convolution operation the application of a set of convolution filters to the data obtained at the output of the previous convolution operation.

[0123] According to a further variant, one or more so-called “pooling” operations (for example one or more “max pooling” operations and / or one or more “average pooling” operations) can be implemented and applied to the data obtained from the convolution operation(s). According to this variant, the “pooling” operation(s) can be, for example, and optionally, matched or associated with an operation of random deactivation of a portion of the neurons of the network, with a determined probability, for example a probability of 0.2 (20%), 0.3 (30%) or 0.4 (40%), this technique being known as “dropout”.

[0124] The convolutional part of the second convolutional neural network outputs a set of detected feature(s) from the DIC image data provided as input to the convolutional part. The information representative of the presence of determined feature(s) is provided as input to a so-called densely connected part of the second convolutional neural network.

[0125] In a sub-step 23222 of step 2322, the device 103 provides the data DI as input to the so-called densely connected part of the second convolutional neural network to determine the hair class identifier IDC.

[0126] The densely connected layer implements a classification operation of the DI data. For this purpose, the densely connected part of the second convolutional neural network comprises one or more layers of densely connected or fully connected neurons. Each layer of neurons provides output data which are obtained by applying a linear combination and then optionally an activation function to the data present at its input.

[0127] The densely connected layer outputs a vector of size N, where N corresponds to the number of sets of at least a first PI parameter of hair classes, each hair class corresponding to a parametric hair model. Each element of the output vector provides the probability for the DI data to correspond to a hair class.

[0128] The values ​​of the convolution filters of the first and second convolutional neural networks and the parameters of their densely connected layers are advantageously determined in a so-called learning phase, for example supervised learning, according to a method known to those skilled in the art. In a learning phase, a large number (for example hundreds, thousands, tens of thousands or more) of hair images whose associated characteristics are known are used to learn the different values ​​or coefficients of the convolution filters. In such a learning phase, a method known as gradient error backpropagation can be implemented, for example. Similarly, in this learning phase, a large number (e.g., hundreds, thousands, tens of thousands or more) of data associations representative of hair characteristics are used to learn the parameters of the densely connected part allowing the classification of the data in order to obtain a set of at least a first PI parameter of a parametric hair model (first convolutional neural network) or a hair class identifier (second convolutional neural network).

[0129] Learning can for example be carried out from image data obtained synthetically.

[0130] According to a variant, a set of at least one first PI parameter (first convolutional neural network) or one hair class (second convolutional neural network) can be determined for each image of a sequence of consecutive images, from the DIC image data of these images. A plurality of sets of at least one first PI parameter (or hair classes) can thus be obtained for the sequence with as many sets of at least one first PI parameter (or hair classes) as there are images in the sequence. A temporal filtering (for example an average) can advantageously be applied to this plurality of sets of at least one first PI parameter (or hair classes) to determine a single set of at least one first PI parameter (an identifier of a hair class).Such temporal filtering may for example be carried out or implemented by a digital filter known to those skilled in the art, for example defined by a mathematical equation, such as a difference equation, or by a neural network, for example recurrent.

[0131] According to a variant, the image data used for training the first and second convolutional neural networks may be data from groups of 3 images taken from the front, right profile and left profile of the same hair. Each group of images may be associated with annotations which indicate the expected class of hair corresponding to the 3 images of the group of images and which indicate a value of at least a first PI parameter of a parametric hair model.

[0132] According to a variant, the image data used for training the first and second convolutional neural networks may be synthetically obtained image data, which facilitates the generation of a large number of image data required for training the first and second convolutional neural networks. convolutional neurons.

[0133] Synthetic image data used for training may be generated from parametric hair models and parameter sets of these models randomly chosen to obtain hair variations. Viewpoint variations may also be used to introduce viewpoint variations of the hair images.

[0134] A problem of generalizing the learning of the first and second convolutional neural networks arises when synthetically generating image data for learning from images of hair variations when several of these images comprise the same face. To avoid this problem, the image data used for learning can be segmented to identify at least one spatial area of ​​these images which includes image data representative of the hair. The image data used for learning can then correspond to created images which only include these spatial areas of the images and which therefore exclude the spatial areas corresponding to the faces.

[0135] The device 104 is configured to render a digital avatar representative of the physical person from the digital twin JN (of the digital representations RNT and RNC) and the accessory data DAC. For this purpose, the device 104 calculates data representative of a view according to a point of view of the RNT representation combined with the accessory data DAC and the digital representation RNC.

[0136] The device 104 for rendering a digital avatar representing the physical person wearing at least one accessory may comprise a rendering engine which calculates data representative of an image of the digital twin JN wearing said at least one accessory according to a point of view. The image may then be displayed on a screen of the device 104.

[0137] According to an exemplary embodiment, the rendering engine may be a real-time rendering engine.

[0138] According to an exemplary embodiment, the rendering engine can be implemented on a GPU computing server.

[0139] For example, the real-time rendering engine may be a 3D simulation engine.

[0140] According to a particular and non-limiting exemplary embodiment, said at least one PI parameter can be applied to a parametric hair model using software.

[0141] The methods of Figures 2 to 5 allow a person to virtually try on accessories and decide whether or not to purchase those accessories.

[0142] Through 1THM of the virtual accessory fitting system 1, a natural person can access a sales site and select accessories. He can also choose to virtually try on these accessories. The HMI can then be designed so that this person adds the selected accessories to a basket, which can be called a virtual fitting room. The process described above is then implemented so that a digital twin of this person is calculated and the person can view a view of their digital twin then wearing accessories that this person would have selected. Thus, the physical person can interact with their digital twin from this HMI which displays the images of this digital twin wearing these accessories according to points of view that can be selected by this person. The person can also choose the scene conditions (lighting, scene content) in which their digital twin evolves.The person can also choose the lighting conditions of the scene in which their digital twin is evolving or even change the scene in which the digital twin is presented.

[0143] [Fig.6] schematically illustrates the device 103 configured to obtain a digital RNC representation of a head of hair of a physical person digitally represented by a digital twin, according to a particular and non-limiting exemplary embodiment of the present invention.

[0144] According to a particular embodiment, the device 103 corresponds to a server or a calculator of the “cloud” 100.

[0145] The device 103 is for example configured for implementing the operations described with regard to figures 1 and / or the steps of the methods described with regard to figures 2 to 5. The elements of the device 103, individually or in combination, can be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. The device 103 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

[0146] The device 103 comprises one (or more) processor(s) 1030 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 103. The processor 1030 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 103 further comprises at least one memory 1031 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0147] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the memory 1031.

[0148] According to various particular and non-limiting embodiments, the device 103 is coupled in communication with other similar devices or systems and / or with communication devices.

[0149] According to a particular and non-limiting exemplary embodiment, the device 103 comprises a block 1032 of interface elements for communicating with external devices, for example a remote server or the “cloud” 100. The interface elements of the block 1032 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (LTE) “Long-Term Evolution” or in French “Long-Term Evolution”), LTE-Advanced (or in French LTE-advanced); - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); - HDMI interface (from the English “High Definition Multimedia Interface”).

[0150] According to a particular and non-limiting exemplary embodiment, the device 103 can provide output signals to one or more external devices, such as a display screen 1033, touch-sensitive or not, one or more speakers 1034 and / or other peripherals 3601034 (projection system for example) via output interfaces 1036, 1037, 1038 respectively. According to a variant, one or other of the external devices is integrated into the device 103.

[0151] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for obtaining a digital representation of a head of hair of a natural person which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a system configured for the implementation of such a method.

Claims

Claims

1. Method for obtaining a digital representation of a head of hair of a natural person, said digital representation being intended to be used to add a head of hair to a digital twin of the natural person, said method comprising the following steps: - obtaining (231) at least a first parameter of a parametric hair model at the output of a trained parameter estimation model when image data representative of at least the head of the natural person according to different points of view are presented at the input of the trained parameter estimation model; and - obtaining (232) the digital representation of the hair of the natural person as a function of said at least a first parameter.

2. The method of claim 1, wherein the trained parameter estimation model is implemented by a first deep convolutional neural network.

3. A method according to claim 1 or 2, wherein said at least one first parameter is modified as a function of at least one second parameter.

4. Method according to claim 3, wherein said at least second parameter is obtained from an action of a user on a human-machine interface or from image data.

5. Method according to one of the preceding claims, wherein obtaining (232) the digital representation of the hair of the natural person as a function of said at least one first parameter comprises the following sub-steps: - obtaining (2322) a hair class identifier as an output of a trained hair classification model from a set of hair classes when the image data are presented as input to the trained hair classification model, each hair class identifier referencing a parametric hair model; and - adjusting (2323) the hair model identified by the hair class identifier as a function of said at least one first parameter.

6. The method of claim 5, wherein the trained hair classification model is implemented by a second deep convolutional neural network.

7. Device for obtaining a digital representation of hair

8.

9.

10. of a natural person comprising means for implementing the steps of the method according to one of claims 1 to 6. A system for virtual fitting of accessories by a natural person digitally represented by a digital twin comprising a device according to claim 7. Computer program comprising instructions for implementing the method according to any one of claims 1 to 6, when these instructions are executed by a processor. Computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to one of claims 1 to 6.

Images