Machine learning, detection and modeling methods, and computer program products and devices

Abstract

The invention relates to a learning method for a machine learning system. According to this method, the machine learning system performs the following steps: obtaining augmented reality images comprising real images and at least one virtual element representative of an object and / or a feature area; for each augmented reality image, obtaining learning information comprising, for at least one given virtual element of the augmented reality image: a model for segmenting the given virtual element, and a set of contour points corresponding to a parameterization of the given virtual element; and performing a learning based on the plurality of augmented reality images and the learning information, providing a set of parameters enabling the machine learning system to detect and model the object and / or the feature area in a given image.

Description

Technical Field

[0001] The field of this invention is image processing.

[0002] More specifically, the present invention relates to a method for detecting and modeling objects and / or feature regions (e.g., eyes, nose, etc.) detected in an image.

[0003] This invention has a variety of applications, particularly, but not exclusively, for virtual testing of a pair of glasses.

[0004] Existing technologies and their disadvantages

[0005] The remainder of this document specifically describes the existing problems faced by the inventors of this patent application in the field of virtual testing of a pair of glasses. Of course, the invention is not limited to this particular application area, but is interested in the detection and modeling of any type of object represented in an image and / or any type of feature region (i.e., the portion of interest) of such an image.

[0006] It is known from the prior art that feature points of some objects and / or some feature regions are used to detect the objects and / or feature regions under consideration. For example, the corner of the eye is often used as a feature point to allow the detection of the eyes of an individual in an image. Other feature points such as the nose or the corner of the mouth can also be considered to detect faces. Generally, the detection quality of faces depends on the number and location of the feature points used. Such techniques are specifically described in French Patent No. FR 2955409 and International Patent Application No. WO 2016 / 135078, filed by the company that filed this patent application.

[0007] For manufactured objects, edges or corners can be considered as feature points, for example.

[0008] However, using such feature points may lead to a lack of detection accuracy, and thus a lack of accuracy in modeling the objects and / or feature regions under consideration.

[0009] Alternatively, images are sometimes manually annotated to artificially generate feature points for the objects and / or feature regions under consideration. However, this paper again notes the lack of accuracy in detecting the objects and / or feature regions under consideration. Consequently, this inaccuracy can lead to problems in modeling the objects and / or feature regions thus detected.

[0010] Therefore, there is a need for a technique that allows for the accurate detection and modeling of one (or more) objects shown in an image and / or one (or more) feature regions present in the image under consideration. Summary of the Invention

[0011] In an embodiment of the present invention, a learning method for a machine learning system is provided, the machine learning system being used to detect and model at least one object represented in at least one given image and / or at least one feature region of said at least one given image. According to this method, the machine learning system performs the following operations: - Generate multiple augmented reality images, which include real images and at least one virtual element representing at least one object and / or at least one feature region;

[0012] - For each augmented reality image, learning information is obtained, which, for at least one given virtual element in the augmented reality image, includes:

[0013] - The segmentation model of the given virtual element obtained from the given virtual element, and

[0014] - A set of contour points corresponding to the parameterization of a given virtual element, or a parameterization obtained from a given virtual element; and

[0015] - Learning from multiple augmented reality images and the aforementioned learning information, a set of parameters is provided that enables the machine learning system to detect at least one object and / or at least one feature region in at least one given image and determine the corresponding modeling information, which includes:

[0016] - A segmentation model for at least one object and / or at least one feature region, and

[0017] - A set of contour points, or parameterizations, corresponding to the parameterization of at least one object and / or at least one feature region.

[0018] Therefore, the present invention provides a novel and inventive solution for learning machine learning systems (e.g., convolutional neural networks) to enable them to detect one (or more) objects (e.g., a pair of glasses) and / or one (or more) feature regions (e.g., the outline of an eye or the outline of the iris) in a given image (e.g., an image showing the face of a person wearing glasses) and to determine the corresponding modeling information.

[0019] More specifically, learning is performed from augmented reality images comprising virtual elements representing objects (or multiple objects) and / or feature regions (or multiple feature regions). Thus, segmentation (e.g., using binary masks of objects and / or feature regions simulated by (multiple) virtual elements) and the distribution of points on the contour are accurately performed. This allows, for example, resolving the ambiguities inherent in manually annotating these points. For instance, the distribution of points corresponds to 2D or 3D parameterization of the objects (e.g., 3DMM models or "3D deformable models") and / or feature regions simulated by (multiple) virtual elements. In this way, annotation is accurate and can be easily retrieved from contour points to parameterization.

[0020] Furthermore, the use of such virtual elements allows for the resolution of occlusion issues, thus avoiding incomplete annotations, such as when the temple of a pair of glasses is obscured by an ear or the iris is obscured by an eyelid. The blending of real data (images) with virtual objects allows machine learning systems to be specifically tailored for real-world application cases. Therefore, augmented reality images offer the optimal trade-off between image realism and the ease of generating sufficiently variable images and annotations.

[0021] In this implementation, for each augmented reality image, the learning process of the machine learning system includes joint learning from a segmentation model of a given virtual element and a set of contour points corresponding to the parameterization of the given virtual element.

[0022] Therefore, the learning of the machine learning system, the learning of the segmentation model, and the learning of a set of contour points reinforce each other, resulting in a synergistic effect. The segmentation model allows for improved accuracy by maximizing the number of pixels correctly detected as belonging to the virtual object (i.e., minimizing the number of pixels incorrectly detected as belonging to that object). Furthermore, the set of points thus detected has a consistent shape with the object. In the current case, the fact that these points originate from a parametric model reinforces this consistency. Thus, a consistent shape of the object is obtained regardless of the position of the camera capturing the real image and regardless of the settings of the object in the augmented reality image.

[0023] In the implementation, joint learning implements a cost function that depends on a linear combination between the cross-entropy associated with the segmentation model of a given virtual element and the Euclidean distance associated with a parameterized set of contour points corresponding to the given virtual element.

[0024] For example, a machine learning system includes a branch for learning a segmentation model and a branch for learning a set of contour points. Therefore, cross-entropy is associated with the branch for learning the segmentation model, and Euclidean distance is associated with the branch for learning the set of contour points.

[0025] In some implementations, the real image includes an illustration of a face. For at least one contour point in a set of contour points corresponding to the parameterization of a given virtual element, the learned information includes visibility information indicating whether the contour point is visible or whether it is occluded by a face.

[0026] Therefore, the visibility of the contour points was taken into consideration.

[0027] In some implementations, the cost function further depends on the binary cross-entropy associated with the visibility of the contour points.

[0028] In some implementations, the learning information includes the parameterization of a given virtual element.

[0029] Therefore, machine learning systems can directly provide the parameterizations under consideration.

[0030] The present invention also relates to a method for detecting and modeling at least one object represented in at least one image and / or at least one feature region of said at least one image. This detection and modeling method is implemented by a machine learning system trained by performing the learning method described above (according to any of the foregoing embodiments). According to this detection and modeling method, the machine learning system performs the detection of said at least one object and / or said at least one feature region in said at least one image, and performs the determination of modeling information for said at least one object and / or said at least one feature region.

[0031] Therefore, learning has been completed from augmented reality images that include virtual elements representing objects (or multiple objects) and / or feature regions (or multiple feature regions), ensuring the consistency of modeling information with the modeling of objects (or multiple objects) and / or feature regions (or multiple feature regions). Furthermore, when simultaneously detecting and modeling one (or more) objects and one (or more) feature regions (e.g., eyes, irises, nose), a synergistic effect is achieved compared to detecting and modeling only one of these two elements, improving performance for both objects and feature regions.

[0032] In some of the above embodiments, the learning of the machine learning system includes joint learning, on the one hand, from a segmentation model of a given virtual element, and on the other hand, from a set of contour points corresponding to the parameterization of the given virtual element. In some of these embodiments, the determination implemented in the detection and modeling methods includes joint determination of the following:

[0033] - A segmentation model for at least one object and / or at least one feature region; and

[0034] - A set of contour points corresponding to the parameterization of at least one object and / or at least one feature region.

[0035] In the above embodiments, the joint learning of the machine learning system implements a cost function that depends on a linear combination of cross-entropy associated with a segmentation model of a given virtual element on one hand, and Euclidean distance associated with a parameterized set of contour points corresponding to the given virtual element on the other hand. In some embodiments of these embodiments, the joint determination implemented in the detection and modeling methods implements a given cost function that depends on a linear combination of cross-entropy associated with a segmentation model of the at least one object and / or the at least one feature region on one hand, and Euclidean distance associated with a parameterized set of contour points corresponding to the at least one object and / or the at least one feature region on the other hand.

[0036] In the above embodiments, the learning implementation of the machine learning system includes augmented reality images of real images, which include illustrations of faces. In some of these embodiments, at least one image includes a representation of a given face, and for at least one given contour point from a set of contour points corresponding to parameterization of at least one object and / or at least one feature region, the machine learning system further determines visibility information indicating whether the given contour point is visible or whether the given contour point is occluded by the given face.

[0037] Therefore, machine learning systems also determine the visibility of contour points.

[0038] In the above embodiments, the cost function implemented during joint learning in the machine learning system further depends on the binary cross-entropy associated with the visibility of contour points. In some of these embodiments, a given cost function is implemented in the detection and modeling methods. This given cost function further depends on the binary cross-entropy associated with the visibility of a given contour point.

[0039] In some embodiments of the above implementations, the learning information includes parameterization of a given virtual element. In some embodiments of these implementations, the modeling information includes parameterization of at least one object and / or at least one feature region.

[0040] In some implementations, at least one image comprises multiple images, each representing a different view of at least one object and / or at least one feature region. Detection and determination are performed jointly on each of the multiple images.

[0041] Therefore, it improves the performance of detecting and determining modeling information of objects (or multiple objects) and / or feature regions (or multiple feature regions).

[0042] The present invention also relates to a computer program comprising program code instructions for implementing the methods described above according to any of the different embodiments when the computer program is executed on a computer.

[0043] The present invention also relates to an apparatus for detecting and modeling at least one object shown in at least one image and / or at least one feature region of said at least one image. Such an apparatus includes at least one processor and / or at least one dedicated computing machine configured to implement the steps of the learning method according to the invention (according to any of the different embodiments described above). Therefore, the features and advantages of this apparatus are the same as those of the corresponding steps of the aforementioned learning method. Therefore, further detailed description is not required.

[0044] In some embodiments, at least one processor and / or at least one dedicated computing machine is further configured to implement the steps of the detection and modeling method according to the invention (according to any of the different embodiments described above). Therefore, the features and advantages of this device are the same as those of the corresponding steps of the aforementioned detection and modeling method. Therefore, they will not be described in further detail.

[0045] In some implementations, the device described above includes the machine learning system described above.

[0046] In some implementations, the device described above is the machine learning system described above. Attached Figure Description

[0047] Other objects, features, and advantages of the invention will become clearer after reading the following description, which is given by way of illustrative and non-limiting example only, with reference to the accompanying drawings:

[0048] [ Figure 1 The following describes the steps of a learning method for a machine learning system according to an embodiment of the present invention, the machine learning system being used to detect and model one (or more) objects shown in at least one image and / or one (or more) feature regions of at least one image under consideration;

[0049] [ Figure 2a The illustration shows a real image including a face.

[0050] [ Figure 2b The diagram illustrates the inclusion of [ Figure 2a A real image and an augmented reality image of a pair of glasses;

[0051] [ Figure 2c The illustration shows [ Figure 2b A segmentation model of a pair of glasses representing an augmented reality image;

[0052] [ Figure 2d The diagram illustrates the relationship between [[] and [[]] Figure 2b The parameterization of a pair of glasses corresponding to a set of contour points in an augmented reality image;

[0053] [ Figure 3 The diagram illustrates the steps of a method for detecting and modeling one (or more) objects shown in at least one image and / or one (or more) feature regions of at least one image under consideration, according to an embodiment of the present invention.

[0054] [ Figure 4a The illustration shows a face and an image of a pair of glasses;

[0055] [ Figure 4b The illustration shows [ Figure 4a A segmentation model of a pair of glasses in an image, and with [ Figure 4a The image shows a pair of glasses with a parameterized set of contour points corresponding to them;

[0056] [ Figure 5 The illustration shows a set of contour points corresponding to the parameterization of the eye in the image and the iris of the eye under consideration.

[0057] [ Figure 6 An example of the structure of a device according to an embodiment of the present invention is shown, which is capable of implementing […]. Figure 1 Learning methods and / or [ Figure 3 Some steps of the detection and modeling methods. Detailed Implementation

[0058] The general principle of this invention is based on using augmented reality images to train a machine learning system (e.g., a convolutional neural network) so that the machine learning system can detect one (or more) objects (e.g., a pair of glasses) and / or one (or more) feature regions (e.g., the outline of an eye or the iris of an eye, the outline of a nose) in a given image (e.g., an image depicting the face of a person wearing glasses) and determine the corresponding modeling information.

[0059] More specifically, such augmented reality image includes a real image and at least one virtual element representing the object (or objects) and / or feature regions (or feature regions) under consideration.

[0060] In fact, learning convolutional neural networks requires a large amount of labeled data. Acquiring and labeling this data is extremely costly. Furthermore, the accuracy of the labels cannot be guaranteed, which limits the robustness and accuracy of the resulting inference models. The use of synthetic object images from parametric 2D or 3D models allows for a large amount of learning data and guarantees the localization and visibility of 2D or 3D labeled points. These virtual objects are illuminated by realistic environment maps (“environment maps”), which can be set or estimated from real images. Moreover, the use of such virtual elements allows for the resolution of occlusion problems and thus avoids incomplete or inconsistent labeling due to arbitrary choices by the labeling operator.

[0061] Furthermore, it is suggested that the learned information be used in a complementary manner with the augmented reality image, including a segmentation model associated with the corresponding virtual element and a set of contour points corresponding to the parameterization of the virtual element under consideration.

[0062] Therefore, segmentation (e.g., binary masks of objects and / or feature regions simulated by (multiple) virtual elements) and the distribution of points on the contour are accurately performed without the need to annotate the real image.

[0063] In the remainder of this application, "machine learning system" should be understood as a system configured to perform training of a learning model and use the model under consideration.

[0064] refer to[ Figure 1 According to an embodiment of the present invention, the steps of a learning method PA100 for a machine learning system (e.g., a convolutional neural network) for detecting and modeling one (or more) objects represented in at least one image and / or one (or more) feature regions of the at least one image under consideration. Also refer to [ Figure 2a ]、[ Figure 2b ]、[ Figure 2c ]and[ Figure 2d The paper discusses an implementation example of the steps of the considered method PA100. More specifically, according to [ Figure 2a ]、[ Figure 2b ]、[ Figure 2c ]and[ Figure 2d For example, real image 200 includes an illustration of face 220, and virtual element 210 is a pair of glasses. Accordingly, [ Figure 2c The illustration shows [ Figure 2b A segmentation model of a pair of virtual glasses took 210ms, and [ Figure 2d The diagram illustrates the relationship between [[] and [[]] Figure 2b The parameterization of a pair of virtual glasses corresponds to a set of contour points of 210pt. For clarity, this will be mentioned below. Figure 2a ]、[ Figure 2b]、[ Figure 2c ]and[ Figure 2d The elements are used to describe the characteristics of method PA100 in a non-restrictive manner.

[0065] Return to [ Figure 1 During step E110, the machine learning system acquires multiple augmented reality images 200ra, which include real images 200 and at least one virtual element 210 representing an object (or multiple objects) and / or a feature region (or multiple feature regions).

[0066] For example, each augmented reality image 200ra is generated by a tool specifically designed to insert virtual elements 210 into the real image 200. In some variations, the generation of the augmented reality image 200ra includes adding (e.g., via adding Gaussian noise, blurring) at least one virtual element 210 before inserting it into the real image 200. This addition may include, for example, illuminating the virtual object using a realistic environment map, which may be pre-defined or estimated from the real image. Therefore, the realism of the virtual element 210 and / or its integration with the real image 200 is improved. For example, this improved realism allows for improved learning and enhanced detection performance on real images.

[0067] For example, the resulting augmented reality image 200ra is stored in a database, which the machine learning system accesses to obtain the augmented reality image 200ra under consideration.

[0068] Return to [ Figure 1 During step E120, the machine learning system obtains learning information for each augmented reality image. For at least one given virtual element 210 of the considered augmented reality image 200ra, the learning information includes:

[0069] - A segmentation model 210ms given virtual element 210. For example, this segmentation model is a binary mask of an object (or multiple objects) and / or feature regions (or multiple feature regions) simulated by the given virtual element 210; and

[0070] - A set of contour points 210pt corresponding to the parameterization of a given virtual element 210. For example, the distribution of points corresponds to a 2D or 3D parameterization of an object (or multiple objects) and / or feature regions (or multiple feature regions) simulated by the given virtual element 210. For example, such parameterization (also called a parameterized model) is a 3DMM model (representing "3D Deformable Model"). For example, in the context of 3D parameterization, the contour points of a 3D object can be referenced in 2D by projecting and parameterizing a geodesic curve on the surface of the 3D object, which represents the contour from the perspective of a camera capturing a real image.

[0071] Therefore, segmentation and the distribution of points on the contour 210pt can be performed accurately. The segmentation model 210ms and a set of contour points 210pt are obtained directly from the corresponding virtual elements 210, rather than by post-processing the augmented reality image including the virtual elements 210 under consideration. For example, this allows for resolving the ambiguity inherent in manually annotating these points and allows for easy regression from the contour points 210pt to parameterization.

[0072] In some implementations, the learned information includes the parameterization of the segmentation model 210ms and the contour points 210pt (rather than the coordinates of these contour points 210pt). This parameterization can be derived from the modeling of the virtual element 210, or it can be a posterior-specific modeling of the virtual element 210. For example, the machine learning system learns the control points of one or more spline curves and then finds the contour points 210 from them. In this case, the output of the machine learning system consists of these modeling parameters (e.g., control points), while the cost function (e.g., the Euclidean distance between the contour points and the ground truth) remains unchanged. To achieve this, the transformation that allows switching from the modeling parameters to the contour points 210pt should be differentiable, such that gradients can be backpropagated through the learning algorithm of the machine learning system.

[0073] In some implementations, the learned information further includes an additional consistency term between the segmentation model 210ms and the contour points 210pt. This consistency is measured by the intersection between the segmentation model 210ms and the surface defined by the contour points 210pt. For this purpose, a mesh is defined on the surface (e.g., by the Delaunay algorithm well-known in the art), which is then used by a differential rendering engine that colors ("fills") the surface with a uniform value. Subsequently, a consistency term (e.g., cross-entropy) can be defined, which measures the proximity of pixels on the segmented and rendered surfaces.

[0074] For example, the learning information is stored in the database in association with the corresponding augmented reality image 200ra.

[0075] Return to [ Figure 1 During step E130, the machine learning system performs a learning phase based on multiple augmented reality images 200ra and learning information. This learning allows the generation of a set of parameters (or learning models) that enable the machine learning system to detect at least one considered object (or multiple objects) and / or feature regions (or multiple feature regions) in a given image and determine the corresponding modeling information.

[0076] For example, during a given iteration of this learning process, the input to the learning system is an augmented reality image 200ra containing the virtual element 210. The learning also implements learning information associated with the augmented reality image 200ra. For example, the learning information is a segmentation model 210ms of the virtual object 210 and its contour points 210pt. For instance, understanding the virtual element 210 through its parameterized 3D model allows this learning information to be generated in a perfect manner by projecting the 3D model onto the image to obtain the segmentation model 210ms and by sampling the points of the model to obtain the contour points 210pt. The output of the learning system is the segmentation model of the virtual element and its contour points, or its parameterization. Learning is performed by comparing the output of the learning system with the learning information until convergence. If the output of the learning system is parameterized (e.g., 3DMM, spline curve, Bézier curve, etc.), contour points are determined based on these parameters, and these contour points are compared with real contour points.

[0077] In some implementations, for each augmented reality image 200ra, the learning by the machine learning system includes joint learning from the segmentation model 210ms and a set of contour points 210pt. Therefore, the learning by the machine learning system, the learning by the segmentation model 210ms, and the learning by the set of contour points 210pt mutually reinforce each other, thus achieving a synergistic effect.

[0078] For example, the machine learning system includes a branch for learning a segmentation model 210ms and a branch for learning the set of contour points 210pt. Cross-entropy is associated with the branch for learning the segmentation model, and Euclidean distance is associated with the branch for learning the set of contour points. Joint learning implements a cost function that depends on a linear combination of the cross-entropy associated with the segmentation model 210ms and the Euclidean distance associated with the set of contour points 210pt.

[0079] In some implementations, the machine learning system is a convolutional semantic segmentation network. For example, the machine learning system described in the 2015 paper by Ronneberger, Fischer, and Brox is... U-Net: Convolutional Networks for Biomedical Image SegmentationThe “Unet” type network described in “[U-Net: A Convolutional Network for Biomedical Image Segmentation]”, or as described in the 2018 paper by Chen, Zhu, Papandreou, Schroff, and Adam, is a network of the “Unet” type. Encoder- Decoder with Atrous Separable Convolution for Semantic Image Segmentation The “Deeplabv3+” type described in “[Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation]”.

[0080] In the case of "U-Net," the network structure can be modified to jointly learn a segmentation model and a set of contour points. For example, splitting the network into two branches (one for learning the segmentation model and one for learning the set of contour points) occurs in the last convolutional layer of the decoder section. This ensures consistency between learning the segmentation model and learning the set of contour points. Furthermore, pooling layers followed by fully connected layers allow for a reduction in the dimensionality of the branch dedicated to learning the set of contour points.

[0081] In the case of "Deeplabv3+", the step of concatenating low-level features and encoder features is performed at 4x resolution. For example, it is at this level that the splitting into two branches is completed (one branch for learning the segmentation model and one branch for learning a set of contour points). In some implementations, convolutional layers, pooling layers with max pooling (or "max pooling"), and finally fully connected layers can be added to learn the set of contour points.

[0082] according to[ Figure 2a ]、[ Figure 2b ]、[ Figure 2c ]and[ Figure 2d The example shown in [] Figure 2d The set of contour points 210pt specifically includes contour points 210pt that are occluded by the face 220 once a pair of virtual glasses is inserted into the real image 200 to generate the augmented reality image 200ra. Therefore, the use of this virtual element allows the occlusion problem to be solved and thus avoids obtaining incomplete annotations, such as the case in this example where the temple of the glasses is occluded by the ear.

[0083] Therefore, in an implementation of the method PA100 that includes a representation of a face in a real image 200, for at least one contour point 210pt in a set of points, the learning information includes information about the visibility of the contour point 210pt, indicating whether the contour point is visible or whether it is occluded by the face 220. Thus, the visibility of the contour point is taken into consideration.

[0084] For example, in the above implementation where the machine learning system is a convolutional semantic segmentation network (e.g., "Unet" type or "Deeplabv3+" type), the cost function further depends on the binary cross-entropy associated with the visibility of the contour point 210pt.

[0085] In some implementations, the learning information includes the parameterization of a given virtual element 210, and therefore indirectly includes the parameterization of the object (or objects) and / or feature regions (or feature regions) simulated by the given virtual element 210. Thus, the machine learning system is able to directly provide the parameterization under consideration.

[0086] In some implementations, method PA100 includes a refinement learning step, in which the machine learning system refines a set of parameters provided during the implementation of step E310 based on real data including labeled real images. This labeling can be done manually or automatically (e.g., by implementing a facial parsing algorithm).

[0087] refer to[ Figure 3 The steps of a method for detecting and modeling one (or more) objects represented in at least one image and / or one (or more) feature regions of the at least one image, according to embodiments of the present invention, are now described.

[0088] More specifically, the detection and modeling method according to this technology is implemented by the machine learning system described above, which is trained by implementing the learning method PA100 described above (according to any of the embodiments described above).

[0089] Therefore, during step E310, the machine learning system performs the detection of one (or more) objects and / or one (or more) feature regions in at least one image (real or enhanced image) and the determination of modeling information for that object (or more objects) and / or that feature region (or more feature regions).

[0090] Therefore, learning has been completed from the augmented reality image 200ra, which includes virtual elements 210 representing the object (or multiple objects) and / or feature regions (or multiple feature regions) under consideration, and the consistency of the modeling information with the modeling of the object (or multiple objects) and / or feature regions (or multiple feature regions) is guaranteed.

[0091] Furthermore, in embodiments that simultaneously detect and model at least one object (e.g., a pair of glasses) and at least one feature region (e.g., eyes, iris, nose) of the at least one image, a synergistic effect is achieved compared to detecting and modeling only one of these two elements, thus improving the performance obtained for detecting and modeling the at least one object and the at least one feature region.

[0092] In some implementations, the modeling information includes:

[0093] - Segmentation models for objects (or multiple objects) and / or feature regions (or multiple feature regions); and

[0094] - A set of contour points corresponding to the parameterization of an object (or multiple objects) and / or a feature region (or multiple feature regions).

[0095] Therefore, it is easy to return to the parameterization of the model from the contour points.

[0096] References above Figure 1 In some embodiments described, the learning of the machine learning system includes joint learning from a segmentation model 210ms and a set of contour points 210pt. In some of these embodiments, step E310 includes jointly determining a segmentation model of an object (or multiple objects) and / or a feature region (or multiple feature regions) and a set of contour points corresponding to the parameterization of the object (or multiple objects) and / or the feature region (or multiple feature regions).

[0097] References above Figure 1 In some embodiments described, joint learning implements a cost function based on a linear combination of the cross-entropy associated with the segmentation model 210ms and the Euclidean distance associated with the set of contour points 210pt. In some implementations of these embodiments, the detection and modeling in step E310 implements the aforementioned cost function, which depends on a linear combination of the cross-entropy associated with the segmentation model and the Euclidean distance associated with the set of points.

[0098] References above Figure 1 In some embodiments described, the learning information includes visibility information indicating whether the contour point 210pt is visible or whether the contour point is occluded by the face 220. In some of these embodiments, the detection and modeling in step E310 further determines visibility information for at least one contour point, indicating whether the contour point is visible or whether the contour point is occluded by a face in the image analyzed during step E310. Therefore, the machine learning system also determines the visibility of the contour point. Some implementations of these embodiments depend on a cost function of the binary cross-entropy loss associated with the visibility of the contour point, as referenced above. Figure 1 The implementation described herein is carried out in the corresponding implementation method for learning.

[0099] References above Figure 1In some embodiments described, the learning information includes parameterization of virtual elements used in the augmented reality image, and therefore indirectly includes parameterization of objects (or objects) and / or feature regions (or feature regions) simulated by the virtual elements under consideration. In some of these embodiments, the detection and modeling in step E310 thus determine the parameterization of objects (or objects) and / or feature regions (or feature regions) detected in the image analyzed during step E310.

[0100] In some implementations, the object (or multiple objects) and / or feature regions (or multiple feature regions) are represented in multiple images, each image representing a different view of the object (or multiple objects) and / or the feature regions (or multiple feature regions) under consideration. In this way, jointly performing detection and modeling in different images across multiple images during step E310 allows for the determination of modeling information for the object (or multiple objects) and / or feature regions (or multiple feature regions) under consideration in an improved manner.

[0101] In some implementations, the image to be analyzed by the machine learning system during step E310 is standardized in location. For example, when the image represents a face, for predetermined sub-parts of the face (e.g., eyes), the size of the region surrounding the considered sub-part is adjusted, for example, using facial markers (“landmarks”). These markers can be obtained using any known marker detection or face recognition method. Therefore, step E310 is performed for each resized region to detect and determine modeling information. In this way, it helps to detect objects (or multiple objects) and / or feature regions (or multiple feature regions) and determine the corresponding modeling information.

[0102] In some implementations, markers (e.g., facial markers) are added to the image to be analyzed by the machine learning system during step E310 to indicate feature points (e.g., the location of the nose, the location of the temples). These markers can be obtained, for example, through facial analysis algorithms. In this way, it helps to detect objects (or multiple objects) and / or feature regions (or multiple feature regions) and determine the corresponding modeling information.

[0103] Now, first refer to [[ Figure 4a ]and[ Figure 4b This section discusses an implementation example of the steps involved in the detection and modeling methods. According to this example, image 400 includes an illustration of face 420. Furthermore, it is assumed that the machine learning system has been trained, for example, according to the above reference […]. Figure 2a ]、[ Figure 2b ]、[ Figure 2c ]and[ Figure 2dThe augmented reality image described above is used to detect and model a pair of glasses. Therefore, the object 410 to be detected and modeled in image 400 is a pair of glasses. By implementing step E310, the machine learning system determines modeling information, which includes a segmentation model 410ms of the considered pair of glasses and a set of contour points 410pt. Specifically, [ Figure 4b The set of contour points 410pt in the image 400 includes contour points 410pt that are occluded by the face 420 in the image 400. Therefore, the method described herein allows for the resolution of the occlusion problem and thus avoids obtaining incomplete annotations, such as the case in this example where the temple of the glasses is occluded by the ear.

[0104] Now refer to [ Figure 5 Let's discuss another example of the steps involved in implementing the detection and modeling method. More specifically, according to this example, image 500 includes a partial illustration of a face. The feature region 510zc to be detected and modeled is the eye represented in image 500. Furthermore, it is assumed that a machine learning system has been trained to detect and model the eye based on augmented reality images, which include one or more virtual elements placed at eye level for modeling the eye. In this way, by implementing step E310, the machine learning system determines the modeling information for the feature region 510zc, which in this example specifically includes the eye under consideration and a set of contour points 510pt of the iris.

[0105] refer to[ Figure 6 Now, an example of the structure of a device 600 according to an embodiment of the present invention will be described, which allows for the implementation of […]. Figure 1 The learning methods PA100 and / or [ Figure 3 Some steps of the detection and modeling methods.

[0106] Device 600 includes random access memory 603 (e.g., RAM) and a processing unit 602, which is equipped with one or more processors and controlled by a computer program stored in read-only memory 601 (e.g., ROM or hard disk). During initialization, the code instructions of the computer program are loaded into real-time memory 603, for example, before the processor of processing unit 602 executes them.

[0107] [ Figure 6 The illustration only shows the manufacturing equipment 600 for performing [the task]. Figure 1 The learning methods PA100 and / or [ Figure 3 The detection and modeling methods for ] (based on the above reference [ Figure 1 ]and[ Figure 3This refers to a particular manner in which some steps of the described implementation and / or variations are performed in one of several possible ways. In fact, these steps can be performed indiscriminately on a reprogrammable computing machine (PC, one or more DSP processors, or one or more microcontrollers) that executes a program including a sequence of instructions, or on a dedicated computing machine (e.g., a set of logic gates, such as one or more FPGAs or one or more ASICs, or any other hardware module).

[0108] In the case where the device 600 is at least partially made of a reprogrammable computing machine, the corresponding program (i.e., the sequence of instructions) may or may not be stored on a removable storage medium (such as a CD-ROM, DVD-ROM, or flash drive) that can be partially or fully read by a computer or processor.

[0109] In some implementations, device 600 includes a machine learning system.

[0110] In some implementations, device 600 is a machine learning system.

Claims

1. A learning method for a machine learning system, the machine learning system being used to detect and model at least one object represented in at least one given image and / or at least one feature region of the at least one given image. Its features are, The machine learning system performs the following operations: - Generate multiple augmented reality images, the multiple augmented reality images including real images and at least one virtual element representing the at least one object and / or the at least one feature region; - For each augmented reality image, learning information is obtained, wherein for at least one given virtual element of the augmented reality image, the learning information includes: - A segmentation model of the given virtual element obtained from the given virtual element, and - A set of contour points corresponding to the parameterization of the given virtual element, or the parameterization obtained from the given virtual element; and - Learning from the plurality of augmented reality images and the learning information, providing a set of parameters that enable the machine learning system to detect at least one object and / or at least one feature region in at least one given image and determine corresponding modeling information, the modeling information including: - The segmentation model of the at least one object and / or the at least one feature region, and - A set of contour points corresponding to the parameterization of the at least one object and / or the at least one feature region, or the parameterization itself.

2. The learning method according to claim 1, wherein, For each augmented reality image, the learning by the machine learning system includes joint learning from the segmentation model of the given virtual element on the one hand and from the set of contour points corresponding to the parameterization of the given virtual element on the other hand.

3. The learning method according to claim 2, wherein, The joint learning implementation cost function depends on a linear combination of, on the one hand, the cross-entropy associated with the segmentation model of the given virtual element and on the other hand, the Euclidean distance associated with the parameterized set of contour points corresponding to the given virtual element.

4. The learning method according to claim 1, wherein, The real image includes an illustration of a face, and wherein, for at least one contour point in the set of contour points corresponding to the parameterization of the given virtual element, the learning information includes visibility information indicating whether the contour point is visible or whether the contour point is occluded by the face.

5. The learning method according to claim 4, wherein, For each augmented reality image, the learning by the machine learning system includes joint learning from the segmentation model of the given virtual element on the one hand and from the set of contour points corresponding to the parameterization of the given virtual element on the other hand. The joint learning implementation cost function depends on a linear combination of the cross-entropy associated with the segmentation model of the given virtual element on the one hand and the Euclidean distance associated with the parameterized set of contour points corresponding to the given virtual element on the other hand. The cost function further depends on the binary cross-entropy associated with the visibility of the contour points.

6. A method for detecting and modeling at least one object represented in at least one image and / or at least one feature region of said at least one image. Its features are, A machine learning system trained by implementing the learning method described in claim 1 performs the detection of at least one object and / or at least one feature region in the at least one image, and performs the determination of modeling information for the at least one object and / or the at least one feature region.

7. The detection and modeling method according to claim 6, wherein, For each augmented reality image, the learning by the machine learning system includes, on the one hand, learning from the segmentation model of the given virtual element, and on the other hand, learning from the set of contour points corresponding to the parameterization of the given virtual element. Furthermore, the determination includes a joint determination of the following: - The segmentation model of the at least one object and / or the at least one feature region; and - The set of contour points corresponding to the parameterization of the at least one object and / or the at least one feature region.

8. The detection and modeling method according to claim 7, wherein, The real image includes an illustration of a face, and wherein, for at least one contour point in the set of contour points corresponding to the parameterization of the given virtual element, the learned information includes visibility information indicating whether the contour point is visible or whether the contour point is occluded by the face. The at least one image includes a representation of a given face, and wherein, for at least one given contour point from the set of contour points corresponding to the parameterization of the at least one object and / or the at least one feature region, the machine learning system further determines visibility information indicating whether the given contour point is visible or whether the given contour point is occluded by the given face.

9. The detection and modeling method according to claim 6, wherein, The at least one image comprises multiple images, each image representing a different view of the at least one object and / or the at least one feature region. Furthermore, the detection and determination are jointly performed for each of the plurality of images.

10. A computer program product comprising program code instructions for implementing the method according to any one of claims 1 to 9 when the program is executed on a computer.

11. An apparatus for detecting and modeling at least one object represented in at least one image and / or at least one feature region of said at least one image. Its features are, The device includes at least one processor and / or at least one dedicated computing machine, the at least one processor and / or the at least one dedicated computing machine being configured to perform the following operations: - Generate multiple augmented reality images, the multiple augmented reality images including real images and at least one virtual element representing the at least one object and / or the at least one feature region; - For each augmented reality image, learning information is obtained, wherein for at least one given virtual element of the augmented reality image, the learning information includes: - A segmentation model of the given virtual element obtained from the given virtual element, and - A set of contour points corresponding to the parameterization of the given virtual element, or the parameterization obtained from the given virtual element; and - Learning from the plurality of augmented reality images and the learning information, a set of parameters is provided that enables the machine learning system to detect at least one object and / or at least one feature region in at least one given image and determine corresponding modeling information, the modeling information including: - The segmentation model of the at least one object and / or the at least one feature region, and - A set of contour points corresponding to the parameterization of the at least one object and / or the at least one feature region, or the parameterization itself.

Images