Lightweight, real-time face alignment via one-shot neural architecture search

Abstract

Convolutional Neural Networks (CNNs) Face Alignment Networks (FANs) have achieved significant accuracy on a wide range of public datasets, but this comes at the expense of large model sizes and expensive computational costs, making it impossible to adapt them to real-time applications on edge devices. To overcome this issue while maintaining performance criteria, a model compression approach on FANs using one-shot neural architecture search is presented. The method and device provide efficient training and search (on one GPU), and the resulting models can be deployed to run in real-time on browser-based apps on edge devices. The compressed models have a 30x smaller model size while providing comparable state-of-the-art accuracy, and run in 40.7 ms per frame in a common browser on a common smartphone and OS.

Description

[Technical field]

[0001] 《Cross reference》 This application claims the benefit of priority to U.S. Provisional Application No. 63 / 155,839, filed March 3, 2021, the entire contents of which are incorporated herein by reference. This application also claims the benefit of priority to French Patent Application No. 2201813, filed March 2, 2022, the entire contents of which are incorporated herein by reference.

[0002] This application relates to improvements in computer processing, image processing and neural networks for image processing, and more particularly to systems, methods and techniques for lightweight, real-time face alignment using one-shot neural architecture search (NAS). [Background technology]

[0003] Face alignment is a common computer vision task that localizes facial landmarks on a human face. This technique is widely used as a building block in various kinds of face-related industrial applications, such as live face tracking, three-dimensional (3D) face shape estimation, and facial feature extraction. When applying designed artificial intelligence (AI) models to real-world applications, computation speed and accuracy are the main bottlenecks due to limited resources on edge devices (e.g., personal computing devices including tablets and smartphones). Cloud computing offers a solution that can increase the inference time of AI models while maintaining good accuracy, but cloud-based solutions suffer from privacy and stability issues. Other solutions use manually designed mobile target models (Howard et al., 2017) (Sandler et al., 2019) to reduce computation time, but such models do not provide sufficient results compared to carefully designed models that target only face alignment.

[0004] Thus, while Convolutional Neural Networks (CNNs) facial alignment networks (FANs) have achieved reasonable accuracy on a wide range of public datasets, this leads to larger model sizes and expensive computational costs, making them unfeasible for real-time applications on edge devices. Summary of the Invention

[0005] In CNN, face alignment networks have achieved significant accuracy on a wide range of public datasets, but this leads to larger model sizes and expensive computational costs, making it impossible to apply them to real-time applications on edge devices. To overcome this problem while maintaining performance criteria, a model compression approach on FANs using One-Shot Neural Architecture Search is provided. The method and device provide efficient training and search (on a single GPU), and the resulting models can be deployed to run in real-time in browser-based applications on edge devices, including tablets and smartphones. The compressed model provides comparable cutting-edge accuracy while having a 30 times smaller model size and can run in 40.7 milliseconds per frame in a common browser on a common smartphone and OS.

[0006] In one embodiment, a method for defining a face alignment network (FAN) model is provided, the method including: i) generating a set of candidate sub-models from a network structure by training the network structure using a training dataset; and ii) an expansion and contraction training (EST) operation, where the EST operation defines and maintains instances of the candidate sub-models for various configuration parameters that define the set of candidate sub-models; and performing an evolutionary search of the candidate sub-models using speed and accuracy evaluation criteria to select a FAN model from the candidate sub-models.

[0007] In one embodiment, a FAN model is defined for run-time execution by an edge device to process face images.

[0008] In one embodiment, the EST operation starts training from a baseline model previously defined according to the network structure. In one embodiment, the EST operation utilizes pre-trained parameters from the baseline model as partially trained parameters in each of the candidate sub-models for further training with the EST operation. In one embodiment, the candidate sub-models are further trained for any one of between 2 and 60 epochs, between 2 and 25 epochs, and 2 epochs.

[0009] In one embodiment, the EST operation activates a random subnet at each training iteration and trains the partial weights of the subnet as integrals, effectively training sharing weights between the sub-models.

[0010] In one embodiment, the EST operation defines a search space for generating candidate sub-models according to a number of search dimensions including a depth size, a kernel size, a channel ratio, and an expansion ratio. In one embodiment, the EST operation starts with a small depth size, a small kernel size, a large channel ratio, and a large expansion ratio, and then the operation gradually expands the depth size and kernel size to their respective maximums and gradually reduces the channel ratio and expansion ratio to their respective minimums in a stepwise manner and in the following order: In one embodiment, the EST operation comprises a kernel training operation in an early phase, training a first size kernel and subsequently training a second size kernel with the first size kernel, where the first size kernel is centered on the second size kernel.

[0011] In one embodiment, the EST operation comprises a deep training operation in a middle phase that trains fewer layers and then gradually adds larger layer options to the training.

[0012] In one embodiment, the EST operation comprises a channel ratio and expansion ratio training operation, which reduces the channel ratio and expansion ratio, respectively, in a termination phase.

[0013] In one embodiment, the evolutionary search evaluates a baseline model to determine speed and accuracy criteria, hi one embodiment, the evolutionary search evolves from the baseline model to define a population of candidate sub-models for subsequent sampling and evaluation.

[0014] In one embodiment, the evolutionary search performs multiple cycles, the multiple cycles including sampling from the population to select a quantity of candidate sub-models, selecting a fastest sub-model from the quantity, evolving the fastest sub-model to determine a related sub-model, removing the oldest sub-model from the population, adding the related sub-model to the population for later sampling if the related sub-model satisfies an accuracy threshold test, adding the related sub-model to a final set of fastest candidate sub-models, and selecting the fastest sub-model from the final set of fastest candidate sub-models as the FAN model according to performance over the multiple cycles.

[0015] In one embodiment, the evolutionary search performs multiple cycles, the multiple cycles including sampling from the population to select a quantity of candidate sub-models, selecting a most accurate sub-model from the quantity, evolving the most accurate sub-model to determine a related sub-model, removing the oldest sub-model from the population, adding the related sub-model to the population for later sampling if the related sub-model satisfies a speed threshold test, adding the related sub-model to a set of final most accurate candidate sub-models, and selecting the most accurate sub-model from the set of final most accurate candidate sub-models as the FAN model according to performance of the multiple cycles.

[0016] In one embodiment, the network structure is a two-stage structure that outputs to a Region of Interest (RoI) alignment layer to refine the extracted features for prediction. In one embodiment, the first stage includes four blocks, the second stage includes one block, each block includes up to four layers, and each layer includes an inverted residual layer.

[0017] In one embodiment, a method includes defining an application including the FAN model for execution by an edge device, in each embodiment, the application includes any one of a makeup simulation application for simulating trying on a makeup product on the face, a skin analysis application for analyzing and scoring an area or areas of the face for at least one skin sign, an effects simulation application for simulating a graphical effect associated with an area of ​​the face, and a face tracking application for tracking and indicating a position of the face.

[0018] In one embodiment, the application is configured to process the video according to the FAN model. In one embodiment, the edge device is a smartphone or tablet terminal, and the FAN model is configured for execution by a graphics processing unit (GPU). In one embodiment, the application is a makeup simulation application or a skin analysis application, and the application is configured to provide an interface to an e-commerce purchasing service to purchase products or services associated with makeup and skin treatments, respectively.

[0019] In one embodiment, the application is a make-up simulation application or a skin analysis application, the application being configured to provide an interface to an e-commerce recommendation service that recommends products or services related to make-up and skin treatments, respectively.

[0020] In one embodiment, there is provided a computing device comprising a processing unit coupled to a storage device storing instructions, the instructions, when executed on the processing unit, configuring the computing device to perform a method according to any of the methods described in the embodiment.

[0021] In one embodiment, a computer program product is provided comprising a non-transitory storage device storing instructions and data defining a FAN model for execution by a processing unit of a computing device, the FAN model being defined in a manner according to any of the methods described in the embodiment.

[0022] In one embodiment, a computing device is provided that includes a processing unit coupled to a storage device that stores instructions, the instructions, when executed by the processing unit, configure the computing device to provide a FAN model defined in a manner according to any of the methods described in the embodiment. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a simplified diagram of a two-stage CNN structure according to one embodiment. [Figure 2A] FIG. 1 is a block diagram illustrating a comparison between OFA training operation and expand-and-shrink training (EST) operation according to one embodiment. [Figure 2B] FIG. 1 is a block diagram illustrating a comparison between OFA training operation and expand-and-shrink training (EST) operation according to one embodiment. [Figure 2C] FIG. 1 is a block diagram illustrating a comparison between OFA training operation and expand-and-shrink training (EST) operation according to one embodiment. [Figure 2D] FIG. 1 is a block diagram illustrating a comparison between OFA training operation and expand-and-shrink training (EST) operation according to one embodiment. [Figure 2E] FIG. 1 illustrates an EST extension and channel selection operation according to one embodiment. [Diagram 3] 1 is a listing of pseudocode for an evolutionary search process according to one embodiment. [Figure 4A] 1 is a graph showing experimental results according to one embodiment. [Figure 4B] 1 is a graph showing experimental results according to one embodiment. [Figure 4C] 1 is a graph showing experimental results according to one embodiment. [Figure 4D] 1 is a graph showing experimental results according to one embodiment. [Diagram 5] 1A and 1B are graphs showing the results of an experiment according to one embodiment. [Figure 6A] 1 is an image showing annotations of a dataset. [Figure 6B] 1 is an image showing annotations of a dataset. [Figure 7] FIG. 1 is a computer network diagram illustrating a development computing device, a website computing device, and an edge computing device according to one embodiment. [Figure 8]FIG. 2 is a diagram of a computing device according to one embodiment. [Figure 9] 4 is a flowchart of an operation according to one embodiment.

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] <Related Achievements> "Facial landmark alignment" The long history of face alignment begins with classical computer vision, where regression tree ensemble algorithms (Kazemi and Sullivan 2014) provide fast, acceptable and accurate solutions for early-stage real-time face tracking. These methods achieved reasonable results with very large model sizes.

[0033] Recent CNN solutions for face alignment have achieved good performance in terms of accuracy. Most of the work uses stacks of hourglass structures (Newell, Yang, and Deng 2016) (Liu et al. 2019) (Kowalski, Naruniec, and Trzcinski 2017) along with heat maps. These kinds of methods significantly improve the accuracy of landmarks, but these cascaded structures are still cumbersome.

[0034] Efficient models targeting mobile and embedded vision applications, such as MobileNet (Howard et al. 2017) and MobilenetV2 (Sandler et al. 2019), run smoothly and perform well. The accuracy and speed of FAN still have room for improvement. Neural Architecture Search

[0035] Early NAS work (Zoph and Le 2017) significantly improved the results of classification tasks, but the search required more than 1000 GPU days. Since the search cost comes from the repeated training of sampled models, one-shot NAS approaches (Liu, Simonyan, and Yang 2019) (Cai et al. 2020) propose to train a supermodel that covers all parameters of the sampled models. This approach reduces the search time to less than 7 GPU days, making it practical to run NAS in a deployment scenario.

[0036] Along with efficient search for classification tasks, NAS methods have also been applied to object detection (Chen et al. 2019), segmentation (Nekrasov et al. 2019), and pose estimation (Yang, Yang, and Cui 2020). Although the cross-field of NAS and FAN has not been studied to date, it shares similarities with detection work searching backbones and bottlenecks (Chen et al. 2019). <Embodiment> CNN model architecture

[0037] According to one embodiment, a two-stage CNN structure is used similar to that proposed (Li et al., Lightweight Real-time Makeup Try-on in Mobile Browsers with Tiny CNN Models for Facial Tracking, 2019 URL: arxiv.org / abs / 1906.02260, filed April 22, 2020 as U.S. Patent Application No. 16 / 854,993 entitled "CONVOLUTION NEURAL NETWORK BASED LANDMARK TRACKER", all of which are incorporated herein by reference).

[0038] FIG. 1 is a simplified diagram of a two-stage CNN structure 100 according to one embodiment. The CNN structure 100 shows stage 1 (102) and stage 2 (104). Stage 1 is fed by a backbone 106 and includes four blocks: block 1 (108A), block 2 (108B), block 3 (108C) and block 4 (108D). Stage 2 receives the output from block 1. The CNN structure 100 extracts features from the output of stage 1 using a region of interest (RoI) alignment layer 110 to refine the features for prediction at 112. The CNN structure 100 also represents the output of the network with a heatmap. According to an embodiment, pixel-wise sigmoid cross entropy and the usual L2 loss are used (during model training) as suggested in the literature.

[0039] According to an embodiment, the layers (e.g., backbone, blocks 1-4, and lone blocks of stage 2) are inverted residual layers. The layers use stride=1 and keep the same number of channels in one block, with the following exceptions: i) the backbone layer (106) is a convolution with s=2 in the first layer and c=16 in the second layer; ii) the first layer of block 1 (108A) has s=2 and c=24; the first layer of block 2 (108B) has s=2 and c=32; the first layer of block 3 (108C) has s=2 and c=64; the first layer of block 4 (108D) has s=1 and c=96; and stage 2 (104) has s=1 and c=32 (where "c" stands for "channel" and "s" stands for "stride"). Search Space

[0040] According to one embodiment, the model developed by the CNN structure 100 uses a part of MobileNetV2 (Sandler et al. 2019) as its first stage. This developed model is called the baseline model in the following section. The goal is to compress the model size of this baseline model by searching for similar structures. Therefore, we used the inverted residual layer as the basic structure of the model, as described with reference to Figure 1.

[0041] According to an embodiment, the layer depth per block can be selected from [2,3,4], [3,5,6], the layer expansion rate from [3,5], and the blockwise channel ratio from [0.7,0.8,0.9,1.0]. This channel ratio is multiplied with the number of base channels to add more variation to the search space. Overall, the search space is

number

[0042] The maximal submodel obtained by selecting the maximal option from each search dimension is the supermodel. Each model sampled from this search space, including the baseline model, can be considered a subset of the supermodel. One-shot NAS with Expand & Shrink Training

[0043] Choosing the maximum value of each search dimension (as described in the previous section), according to one embodiment, the supermodel has 1.14M parameters. Under the one-shot NAS approach to training, the supermodel is trained only once (e.g., the supermodel is not trained again during the search phase) and its pre-trained weights are provided to all sampled models instead of training them from scratch. Here, sampled models refer to sub-models generated during an earlier training phase that are sampled during the search phase according to the supermodel.

[0044] Thus, while the supermodel takes longer to train (e.g., compared to a model of a similar structure under a more typical approach to training), once it is complete, the search time for each sampled model is O(1) instead of O(N).

[0045] Training the supermodel as a whole is a naive approach. Although it is simple, the shared weights are not optimized for the different submodels and are therefore not promising (see below in relation to Fig. 4C). The weights are trained integrally so that the supermodel works as a whole. To simultaneously optimize all subnets, a technique from Once-For-All (OFA, (Cai et al. 2020)) is adapted and, according to one embodiment, a new Expand-and-Shrink training (EST) strategy is proposed. Both OFA and EST (presented here) work to solve the same problem, but allow the weights in the supermodel to work when they are only partially activated - EST shows improved results as demonstrated.

[0046] The following example illustrates a general simplification of the problem. Assume there is a supermodel with layers A, B, and C. It is desirable to run not only the entire supermodel (i.e., ABC), but also submodels AB, BC (partially). OFA and EST each jointly train partial weights by activating random subnets at each training iteration. Each focuses on how to effectively train the sharing weights between submodels using different approaches. Referring again to the exemplary supermodel of layer ABC, a typical training trains ABC as an integral, which updates all parameters of ABC simultaneously and in the same direction based on the output of A→B→C→. Thus, jointly training partial weights is for a submodel defined only by the AB layer in the supermodel, such a layer can be considered as an independent model to train. Thus, when training this submodel, only the weights of AB are updated, and the direction of the update is based on the output of A→B→. The weights in C are still present (e.g., in the supermodel) but are ignored (they are not activated) and the focus of training is limited to only A and B. A similar approach is taken for submodel BC, where only these layers are trained jointly, ignoring layer A.

[0047] The approaches under OFA and EST differ in the details of solving the aforementioned problems. Figures 2A, 2B, 2C, and 2D are block diagrams for making a comparison between OFA and EST, and Figure 2A shows an OFA kernel training operation 200 according to an example. The operation 200 first shows training 202A of a 5x5 kernel, and then applies a linear transformation to its center 3x3 to run as a 3x3 kernel 202B. Figure 2B shows an EST kernel training operation 202 according to an example, which is the inverse of operation 200: in 202A, the operation first trains a 3x3 kernel, and then takes the center (202B) of the 5x5 kernel to train it. Figure 2C shows an OFA depth training operation 204 according to an example, which first trains all layers (204A), and then gradually reduces the minimum layer option (204B, 204C) to allow more selection of smaller models. FIG. 2D illustrates an example EST depth training operation 206. Operation 206 trains fewer layers (206A) and then gradually adds larger layer options (206B) to the search space. And FIG. 2E illustrates an example EST expansion and channel selection operation 208. In OFA, once the expansion ratio is determined and C channels are selected, the top C most significant (by sum of L1 norm of per-channel weights) channels are selected. EST follows this approach for the expansion ratios (208A, 208B) and then applies it to the block-wise channel ratios (208C).

[0048] In summary, OFA tends to train the largest layer weights first, then share those weights to gradually train smaller layers. In contrast, EST gradually reduces the size of layers for expansion ratios and channel ratios, while expanding search options for depth and kernel size. (The advantages of EST over OFA are shown with reference to graphs 500 and 502 in FIGS. 5A and 5B, and are described below.)

[0049] Table 1 shows the five phases of EST according to one embodiment. The search dimensions include depth (D), kernel (K), channel ratio (C), and expansion ratio (E). Each of these dimensions expands or contracts its search range accordingly in one of the five phases. The training order shown has been demonstrated to achieve better results than different orders.

[0050] EST starts with a small depth and kernel size, and large channels and expansion ratios. Then, EST gradually expands the depth and kernel to the maximum. In the ending phases, EST gradually reduces the channels and expansion ratios to the minimum. Notably, during the expansion and contraction phases, the search options from the previous phases are retained (e.g., expanding the search space of the submodels for use in later searches). Thus, the activated search space (of the submodels) grows with each step as submodels are added according to the EST operation, and the larger and smaller submodels created using the EST operation have the opportunity to be trained together. A further example is useful. In this example, we have a supermodel with four layers ABCD, and for simplicity, the search space has only one search option, which is the number of layers. If there is only one choice of layer number to choose from, e.g. "4", then the submodel will have exactly 4 layers, which is the supermodel itself with ABCD, and when the search space is expanded by allowing "2,3,4" layers in the search options, the submodels can be, e.g., AB (2 layers), ABC (3 layers), ABCD (4 layers), etc. Thus, by allowing more search options ("2,3,4" vs. "4"), it becomes more likely (more likely) that a larger model (e.g., ABCD) can be trained with a smaller model (AB, ABC). [Table 1] Performance evaluation and estimation

[0051] According to one embodiment, model compression is at least partially driven by the deployment scenario, so that both accuracy and speed of the models are evaluated. Full evaluation by training from scratch and running on edge devices is costly. To mitigate the evaluation cost, according to one embodiment, estimators are implemented to predict their performance. These should give consistent performance rankings as well as full evaluation. According to one embodiment, speed (time) estimation uses estimators developed to use FLOPS, Params and Memory Read&Write. (The accuracy is illustrated with reference to graphs 400, 402, 404 and 406 in Figs. 4A, 4B, 4C and 4D, and explained below.)

[0052] Instead of training a fully sampled model from scratch for N epochs for accuracy estimation, according to one embodiment, significant benefits are obtained from the EST operation. In one embodiment, EST is performed by loading partial weights of the supermodel into a submodel, which is trained to converge the supermodel quickly. Also, as described below in Figures 4A-4D, for a child sampled from the search space, loading the corresponding weights from the supermodel and training it for two epochs has proven to be sufficient to give an accurate estimate. This ensures that the evaluation cost is O(1) instead of O(N). Evolutionary Search

[0053] After expansion and contraction training, an evolutionary search (ES) is performed on the search space. According to one embodiment, the details of the evolutionary search are shown in FIG. 3, which provides pseudocode for Algorithm 1 (300). Unlike the basic ES (Real et al. 2019), the child acc and child time is used for evaluation and search identification.

[0054] Algorithm 1 (300) shows the case where an accuracy threshold is used to search for speed to sample models and sort the fastest candidates; if a speed threshold is used instead, the size of the sampled models can be compressed and the most accurate models can be found. That is, in the loop from lines 13 to 23, the parent in line 15 is the most accurate model in the candidates, line 18 compares the speed values, and finally in line 24, the most accurate model is returned from the history set containing the most accurate model candidates generated from the loop from lines 13 to 23.

[0055] Another difference is the generation of the initial population, in one embodiment, where the search begins by evolving from a baseline (e.g., via the EVOLVE function in line 16) instead of random sampling. Subsequent evolution is performed on the (in one embodiment) fastest sub-model evolved from the baseline.

[0056] In the EVOLVE function, there is a randomness used to select different submodels from the search space. (Just like the same mother gives birth to different children, evolving the baseline model also results in different child models. In fact, the chances that two child models will evolve the same are roughly the same as a mother giving birth to twins.) For deployment purposes, the search searches for models that have both better efficiency and accuracy. With random sampling, it is difficult to control the model size and compare the accuracy fairly with the baseline. Dataset

[0057] The in-house dataset (referred to as the "Modi dataset") has a total of 3681 images. The Modi dataset has 62 inner points that define regions within the face and 3 contour points for each face. In the application scenario, the poses range from -30° to 30°. In one embodiment, the ESTNet model is trained separately on the 300W dataset (Sagonas et al. 2013) to perform a fair comparison with cutting-edge methods. The difference in annotations between the Modi and 300W datasets is shown in images 600 and 602 in Figures 6A and 6B. Results and Ablation Studies Training and Searching

[0058] Discussed above are the differences between the original OFA and the EST technique disclosed herein. The results below show that for depths and kernels in the FAN search space, EST augmentation works much better than OFA shrinkage. In FIG. 5A (graph 500), training from [2,3] depth to [2,3,4] drops the error to 0.031, while when training first from the full [4] layer, the error remains high. In FIG. 5B (graph 502), training starts from kernel size 3 or 5, and the learning rate is adjusted for transformation matrices.

[0059] The shrinkage experiments cluster with a much higher error. As discussed, experiments were conducted to validate the performance estimator. 17 randomly sampled models were evaluated for their actual performance and their performance rank compared to the estimated values. The sample Pearson correlation coefficient (Freedman, Pisani, and Purves 2007) in Equation 1 was used to evaluate the consistency between the estimated and actual evaluations.

number

[0060] In Fig. 4A, graph 400 shows the actual inference time, and in Fig. 4B, graph 402 shows the time estimation done by weighted average of FLOPS, Params and Memory Read & Write. Using Equation 1, we get a strong correlation of 0.677, which validates the correctness of the estimator.

[0061] Graphs 404 and 406 in Figures 4C and 4D show a comparison of two supermodel training strategies: naive training and expand-and-shrink training. In Figure 4(C), the model is simply loaded with pre-trained weights and trained for 2 and 25 epochs.

[0062] The correlation between the accuracy rankings after adjusting for different epochs is only 0.151. Thus, the naive approach does not provide a good accuracy estimate.

[0063] In contrast, Figure 4D uses a pre-trained supermodel from the expand-and-shrink training strategy described above. The same model is trained for 2, 25 and 60 epochs, and the consistency of the model accuracy rank is very strong, with correlation coefficients of (2,25), (25,60), and (2,60) being 0.667, 0.916, and 0.819, respectively. This proves the effectiveness of EST, and two epochs of tuning the model are sufficient to estimate the accuracy performance.

[0064] Compared to the naive approach, EST provides a much better accuracy estimate without spending much more time: on one Tesla® V-100 (a tensor core GPU from NVIDIA Corporation, Santa Clara, CA), dilation and contraction training takes 40 hours to finish, where naive training also takes 36 hours.

[0065] In summary, in this section we prove the effectiveness of both the time estimator and the accuracy estimator. It also shows the advantages of EST over naive approaches. Search results

[0066] Tables 2 and 3 show the search results according to each embodiment. α is a parameter that is multiplied with the channel number to customize the dataset with different complexity. The Modi dataset uses α=0:5, and the 300W dataset uses α=1:0 because it has more data. All the calculated errors in this section are calculated with the average error with Inter-pupil Normalization. In one embodiment, the operation searches the Modi dataset, and the one called ESTNet in Table 2 obtains the best result. Its inner error and contour error on the test set are lower than the baseline model, and its model size is only 75% of the baseline. On different edge devices, ESTNet outperforms in both accuracy and speed. [Table 2]

[0067] In one embodiment, the operation trains ESTNet on the 300W dataset separately to evaluate its robustness and generality. The results are compared with the baseline and state-of-the-art work, Look-at-Boundary (LAB, (Wu et al. 2018)), as shown in Table 3. Params and FLOPS of 8-stack LAB are taken from (Zhu et al. 2020). The results of 4-stack are estimated by halving. ESTNet outperforms the baseline on all common, challenging, and full validation sets while reducing 25% of model params. It also achieves comparable results to LAB. ESTNet runs in real-time at 40.7 ms per frame in Chrome on an iPhone 11 (a trademark of Apple Inc. Cupertino, CA). Given that LAB has 29.2x more parameters and 50.2x more FLOPS than ESTNet, LAB cannot run in real time, which is an advantage of the compressed model ESTNet. [Table 3]

[0068] Thus, as illustrated in the embodiments, EST techniques, devices and methods are proposed to develop FAN models that execute in real time, particularly via a website or the like (e.g., a web browser-based application) on a device. Web browser-based applications are typically more constrained than native (non-browser-based) applications. Web browser-based applications are typically provided because the web browsing delivery approach is user-friendly and may be preferred by users compared to obtaining and installing a native application from an application delivery service or the like (e.g., an application store).

[0069] Edge devices on which the FAN model can run include smartphones and tablets running common browsers and operating systems (e.g., devices, browsers and operating systems from Apple Inc. Cupertino, Calif.). One embodiment model generated using the EST techniques herein has a 30x smaller model size and achieves state-of-the-art accuracy while running in 40.7 milliseconds in Chrome on an iPhone 11.

[0070] 7 is a diagram of a computer network 700 showing a development computing device 702, a website computing device 704, an application delivery computing device 706, and respective edge computing devices, namely a smartphone 708 and a tablet terminal 710, according to one embodiment. The computing devices are coupled via a communication network 712. The computer network 700 is simplified. For example, the website computing device 704 and the application delivery computing device 706 are examples of devices of the respective website and application delivery systems. The communication network 712 may include multiple wired and / or wireless networks, which may include private and pobrik networks.

[0071] In this embodiment, the development computing device 702 is coupled to a data store 714 (which may include a database) that stores one or more data sets for training and testing the network model, etc. The data store 714 can store software, other components, tools, etc. to aid in development and implementation. In another embodiment, not shown, the data sets are stored in a storage device of the development computing device 702. The development computing device 702 is configured to define a FAN model 716 according to one or more of the embodiments described herein. For example, the development computing device 702 is configured to generate a set of candidate models from the network structure by training the network structure using i) a training data set and ii) an expand-and-shrink training (EST) operation, where the EST operation defines and maintains candidate model instances of various configuration parameters to define the set of candidate models. The development computing device also performs an evolutionary search of the candidate models using speed and accuracy evaluation criteria to select a FAN model from the candidate models.

[0072] A development computing device 702 defines a FAN model to run on edge devices such as a smartphone 708 and a tablet 710. The development computing device 702 (or another, not shown) incorporates the FAN model 716 into an application 720A, such as for a website (for serving on the website computing device 704) and / or an application 720B for an application distribution computing device (e.g., 706) for distribution to the respective edge devices. In one embodiment, the application distribution computing device 706 provides an application store (e-commerce) service to distribute applications for execution on target devices running a supported operating system (OS). An example of an application distribution computing device is Apple's App Store® for iPhone® or iPAD® devices running iOS® or iPADOS (registered trademarks of Apple Inc. Cupertino CA). Another example is Google Play (registered trademark of Google LLC, Mountain View, CA) for smartphone and tablet devices from various sources running Android OS (registered trademark of Google LLC, Mountain View, CA). In this embodiment, a smartphone 708 receives an application 720A having a FAN model 716 from a website computing device 704 and a tablet terminal 710 receives an application 720B having a FAN model 716 from an application delivery computing device 706 .

[0073] FIG. 8 is a block diagram of a representative computing device 800. The computing devices of FIG. 7 are similarly configured according to their respective needs and capabilities. The computing device 800 comprises a processing unit 802 (e.g., a CPU and / or GPU, or other processor, in one embodiment comprising at least one processor) and a storage device 804 (in one embodiment, at least one storage device) for storing instructions (and data) that, when executed by the processing unit (e.g., processor), configure the operation of the computing device, e.g., execute a method. The storage device 804 may include any of a memory device (e.g., RAM, ROM, EEPROM, etc.), a solid-state drive (e.g., comprising a semiconductor storage device / IC that may define a flash memory), a hard disk drive or other type of drive and storage medium such as tape, disk (e.g., CD-ROM, etc.), and additional components include a communication unit 806 for coupling the device to a communication network via wired or wireless means, an input device 808, and an output device 810 that may comprise a display device 812. In some examples, the display device is a touch screen device providing an input / output device. The components of the computing device 800 are coupled through an internal communications system 814, which may have external ports for coupling to additional devices.

[0074] In some examples, output devices include a speaker, a bell, a light, an audio output jack, a fingerprint reader, etc. In some examples, input devices include a keyboard, buttons, a microphone, a camera, a mouse or pointing device, etc. Other devices (not shown) may include a position determining device (e.g., GPS).

[0075] Storage device 804 stores, in one example, an operating system 816, user applications 818 (which may be one of applications 720A or 720B), a browser 820 (a type of user application) for browsing websites and executing executables such as application 720A using FAN models 716 received from the websites, and data 822 in which, for example, images and / or video frames from a camera or otherwise are stored.

[0076] Continuing to refer to FIG. 7, in one example, a user of a smartphone 708 visits a website provided by the website computing device 704 using a browser. The smartphone 708 receives an application 720A having a FAN model 716. In this example, the application is an effects simulation application, such as a make-up try-on application, that provides a virtual reality experience. A user uses a camera to capture a still or video image (e.g., a selfie), and this source image is processed in the application (e.g., using the FAN model 716). The user selects various effects to be applied, such as via a graphical user interface. The effects are applied to the image in real time using facial regions (e.g., trained facial landmarks) identified through the FAN model 716. Make-up effects may be applied using overlays to define an updated image from the source image. The updated image may be displayed via a display device, stored in a storage device, shared (communicated) via social media, text messages, email, and the like. In one embodiment, the website computing device 704 is capable of e-commerce services and facilitates the purchase of virtually tried-on makeup via the application 720A. In one embodiment, the website computing device 704 provides a recommendation service for recommending products.

[0077] In one embodiment, the application 720A simulates the effects of a skin or other facial treatment product. In one embodiment, the application 720A simulates the effects of a skin or other facial treatment service. In one embodiment, the website computing device 704 provides a recommendation service to recommend products. In one embodiment, the website computing device 704 is capable of e-commerce services to facilitate the purchase of such products and / or such services. In one embodiment, the application 720A performs a skin analysis and provides a scoring of at least one skin sign (e.g., acne). The website computing device 704 can provide product recommendations. The application 720A can simulate results or product usage that show a reduction in skin sign effects.

[0078] Continuing to refer to FIG. 7, in one example, a user of a tablet device 710 uses a browser to visit a website provided by the application delivery computing device 706. The tablet device 710 receives an application 720B having a FAN model 716. In this example, the application is an effects simulation application that simulates face-related unrealistic effects (e.g., graphical effects) to provide a virtual reality experience. In one example, the effect is a filter that is applied to a selfie image. In one example, the effect uses graphical effects to simulate a face as a cartoon character. The user uses a camera to capture a still or video image (e.g., a selfie image), and this source image is processed by the application (e.g., using the FAN model 716). The user can select the character to be applied, for example, via a graphical user interface. The graphic effect or effects are applied to the image in real time using the facial regions identified by the FAN model 716. The effects can be applied using an overlay to define an updated image from the source image. The updated image may be displayed via a display device, stored in a storage device, shared (communicated) via social media, text message, email, etc.

[0079] In one embodiment, application 720B provides a treatment product simulation or treatment service simulation (a virtual or augmented reality experience, respectively) for makeup try-on within the application. Application 720B, in one embodiment, is configured to point the device to an e-commerce service for recommendations and / or purchase facilitation.

[0080] In one example, the application is a video calling or conferencing application. Effects are applied in real time to images from the participants' cameras using the results of the FAN model 716. The images with the effects applied are communicated via the video calling or conferencing service for communication to other callers / conference participants. In one embodiment, the effects include makeup effects.

[0081] In examples, developed applications including the FAN model may include a makeup simulation application for simulating the trying on of makeup products on the face, a skin analysis application for analyzing and scoring an area or areas of the face for at least one skin indication, an effects simulation application for simulating effects associated with areas of the face, a face tracking application for tracking and indicating the position of the face, and the like.

[0082] 9 is a flow chart of operations 900 for defining a face alignment network (FAN) model. To define the FAN model, an operation 902 generates a set of candidate sub-models from the network structure by training the network structure using i) a training dataset and ii) an expansion and shrinkage training (EST) operation that defines and maintains instances of the candidate sub-models for various configuration parameters to define the set of candidate sub-models.

[0083] To select a FAN model from the candidate sub-models, operation 904 performs an evolutionary search of the candidate sub-models using speed and accuracy criteria.

[0084] In one embodiment, the FAN model is defined for runtime execution by an edge device, for example a smartphone or tablet, to process face images.

[0085] In one embodiment, the EST operation starts training from a baseline model previously defined according to the network structure. In one embodiment, the EST operation utilizes pre-trained parameters from the baseline model as partially trained parameters in each of the candidate sub-models for further training with the EST operation. In one embodiment, the candidate sub-models are further trained for any one of between 2 and 60 epochs, between 2 and 25 epochs, and 2 epochs.

[0086] In one embodiment, the EST operation activates a random subnet in each training iteration and trains the partial weights of the subnet as integrals, effectively training shared weights among the sub-models.

[0087] In one embodiment, the EST operation defines a search space for generating candidate sub-models according to multiple search dimensions including depth size, kernel size, channel ratio and expansion ratio. In one embodiment, the EST operation starts with a small depth size, a small kernel size, a large channel ratio and a large expansion ratio, and then the operation gradually expands the depth size and kernel size to their respective maximums and gradually reduces the channel ratio and expansion ratio to their respective minimums in phases and in the following order in a termination phase. In one embodiment, the EST operation comprises a kernel training operation in an initial phase, training a first size kernel and subsequently training a second size kernel with the first size kernel, where the first size kernel is centered on the second size kernel.

[0088] In one embodiment, the EST operation comprises a deep training operation in a middle phase that trains fewer layers and then gradually adds larger layer options to the training.

[0089] In one embodiment, the EST operations comprise channel ratio and expansion ratio training operations that reduce the channel ratio and expansion ratio, respectively, in a termination phase.

[0090] In one embodiment, an evolutionary search evaluates baseline models to determine speed and accuracy criteria.

[0091] In one embodiment, an evolutionary search evolves from a baseline model to define a population of candidate sub-models for subsequent sampling and evaluation. The evolutionary search may perform multiple cycles, including: sampling from the population to select a quantity of candidate sub-models, selecting the fastest sub-model from the quantity, evolving the fastest sub-model to determine a related sub-model, removing the oldest sub-model from the population, adding the related sub-model to the population for later sampling if the related sub-model meets an accuracy threshold test, adding the related sub-model to a final set of fastest candidate sub-models, and selecting the fastest sub-model from the final set of fastest candidate sub-models as the FAN model according to performance over the multiple cycles. The evolutionary search can perform multiple cycles, including sampling from the population to select a quantity of candidate sub-models, selecting a most accurate sub-model from the quantity, evolving the most accurate sub-model to determine a related sub-model, removing the oldest sub-model from the population, and if the related sub-model satisfies a rate threshold test, adding the related sub-model to the population for later sampling, adding the related sub-model to a set of final most accurate candidate sub-models, and selecting the most accurate sub-model from the set of final most accurate candidate sub-models as the FAN model according to performance of the multiple cycles.

[0092] In one embodiment, the network structure is a two-stage structure that outputs to a region of interest (RoI) alignment layer to refine the extracted features for prediction. In one embodiment, the first stage includes four blocks, the second stage includes one block, each block has up to four layers, and each layer includes an inverted residual layer.

[0093] Thus, in one embodiment, a computing device comprises a processor coupled to a storage device that stores instructions that, when executed by the processor, configure the computing device to perform a method according to any one of the method-related embodiments.

[0094] Thus, in one embodiment, a computer program product comprises a non-transitory storage device storing instructions and data defining a FAN model that are executed on a processor of a computing device, the FAN model being defined according to any one of the methods of the related method embodiments. Similar embodiments relate to computer program products for defining such a FAN model (e.g., through training).

[0095] Thus, in one embodiment, a computing device includes a processor coupled to a storage device that stores instructions that, when executed by the processor, configure the computing device to provide a FAN model defined according to any one of the method related embodiments. Similar embodiments relate to a computing device for defining such a FAN model (e.g., through training).

[0096] A practical implementation may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, and other ways of combining the features described herein. Some embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided or steps may be eliminated from the described processes, and other components may be added or removed from the described systems. Accordingly, other aspects are within the scope of the claims.

[0097] Throughout the description and claims of this specification, the words "comprise" and "contain" and variations thereof mean "including but not limited to" and are not intended to exclude other elements, integers or steps. Throughout this specification, the singular encompasses the plural unless the context requires otherwise. In particular, when the indefinite article is used, it is to be understood that the specification contemplates the plural as well as the singular, unless the context requires otherwise.

[0098] It should be understood that features, integers, properties or groups described in connection with a particular aspect, embodiment or example of the invention are applicable to any other aspect, embodiment or example, except where incompatible therewith. All of the features disclosed herein (including any accompanying claims, abstract and drawings) and / or all of the steps of any method or process so disclosed may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any of the foregoing examples or embodiments. The invention extends to any novel one or any novel combination of features disclosed herein (including any accompanying claims, abstract and drawings) or any novel one or any novel combination of steps of any method or process disclosed. References - incorporated herein by reference in their entirety Cai, H.; Gan, C.; Wang, T.; Zhang, Z.; and Han, S. 2020. Once-for-All: Train One Network and Specialize it for Efficient Deployment. Chen, Y.; Yang, T.; Zhang, X.; Meng, G.; Xiao, X.; and Sun, J. 2019. DetNAS: Backbone Search for Object Detection. Freedman, D.; Pisani, R.; and Purves, R. 2007. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York. He, K.; Gkioxari, G.; Doll´ ar, P.; and Girshick, R. 2018. Mask R-CNN. Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Kazemi, V.; and Sullivan, J. 2014. One millisecond face alignment with an ensemble of regression trees. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, 1867?1874. doi:10.1109 / CVPR.2014.241. Kowalski, M.; Naruniec, J.; and Trzcinski, T. 2017. Deep Alignment Network: A convolutional neural network for robust face alignment. Li, T.; Yu, Z.; Phung, E.; Duke, B.; Kezele, I.; and Aarabi, P. 2019. Lightweight Real-time Makeup Try-on in Mobile Browsers with Tiny CNN Models for Facial Tracking. Liu, H.; Simonyan, K.; and Yang, Y. 2019. DARTS: Differentiable Architecture Search. Liu, Z.; Zhu, X.; Hu, G.; Guo, H.; Tang, M.; Lei, Z.; Robertson, N. M.; and Wang, J. 2019. Semantic Alignment: Finding Semantically Consistent Ground-truth for Facial Landmark Detection. Nekrasov, V.; Chen, H.; Shen, C.; and Reid, I. 2019. Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells. Newell, A.; Yang, K.; and Deng, J. 2016. Stacked Hourglass Networks for Human Pose Estimation. Real, E.; Aggarwal, A.; Huang, Y.; and Le, Q. V. 2019. Regularized Evolution for Image Classifier Architecture Search. Sagonas, C.; Tzimiropoulos, G.; Zafeiriou, S.; and Pantic, M. 2013. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In the IEEE International Conference on Computer Vision Workshops. Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C. 2019. MobileNetV2: Inverted Residuals and Linear Bottlenecks. Wu, W.; Qian, C.; Yang, S.; Wang, Q.; Cai, Y.; and Zhou, Q. 2018. Look at boundary: A boundary-aware face alignment algorithm. In the IEEE Conference on Computer Vision and Pattern Recognition. Yang, S.; Yang, W.; and Cui, Z. 2020. Pose Neural Fabrics Search. Zhu, B.; Lin, C.; Wang, Q.; Liao, R.; and Qian, C. 2020. Fast and Accurate: Structure Coherence Component for Face Alignment. Zoph, B.; and Le, Q. V. 2017. Neural Architecture Search with Reinforcement Learning. <Other> <Means> The method of technical idea 1 includes: defining a face alignment network (FAN) model includes: i) generating a set of candidate sub-models from a network structure by training the network structure using a training dataset; and ii) an expansion and contraction training (EST) operation, where the EST operation defines and maintains instances of candidate sub-models of various configuration parameters for defining the set of candidate sub-models; and performing an evolutionary search of the candidate sub-models using speed and accuracy evaluation criteria to select the FAN model from the candidate sub-models. A method of technical idea 2 is the method according to technical idea 1, in which the FAN model is defined for runtime execution by an edge device for processing facial images. A method according to technical idea 3 is the method according to technical idea 1 or 2, in which the EST operation starts training from a baseline model previously defined according to the network structure. The method of technical idea 4 is the method described in technical idea 3, wherein the EST operation utilizes pre-trained parameters from the baseline model as partially trained parameters in each of the candidate sub-models for further training using the EST operation. The method of technical idea 5 is characterized in that, in the method described in the technical idea, the candidate sub-model is further trained for any one of between 2 epochs and 60 epochs, between 2 epochs and 25 epochs, and 2 epochs. A method of technical idea 6 is a method according to any one of technical ideas 3 to 5, in which the EST operation activates a random subnet in each training iteration and trains partial weights of the subnet as integrals, effectively training shared weights between sub-models. A method of technical idea 7 is a method according to any one of technical ideas 3 to 6, in which the EST operation defines a search space for generating candidate sub-models according to a number of search dimensions including a depth size, a kernel size, a channel ratio, and an expansion ratio. The method of technical idea 8 is the method described in technical idea 7, in which the EST operation starts with a small depth size, a small kernel size, a large channel ratio and a large expansion ratio, and then the operation gradually expands the depth size and kernel size to their respective maximums in a stepwise and following order, and in an end phase, gradually reduces the channel ratio and expansion ratio to their respective minimums. A method of technical idea 9 is a method according to technical idea 7 or 8, in which the EST operation includes a kernel training operation in an initial phase of training a first size kernel and then using the first size kernel to train a second size kernel, where the first size kernel is centered around the second size kernel. A method of technical idea 10 is a method according to any one of technical ideas 6 to 9, in which the EST operation includes a depth training operation in an intermediate phase that trains fewer layers and then gradually adds larger layer options to the training. A method of technical idea 11 is a method according to any one of technical ideas 6 to 10, wherein the EST operation includes a channel ratio and expansion ratio training operation, which reduces the channel ratio and the expansion ratio, respectively, in a termination phase. The method of technical idea 12 is a method according to any one of technical ideas 2 to 11, wherein the evolutionary search evaluates the baseline model to determine speed and accuracy evaluation criteria, as in technical ideas 3 to 10 which are subordinate to technical idea 2. The method of technical idea 13 is a method according to any one of technical ideas 2 to 12, wherein the evolutionary search evolves from the baseline model to define a population of candidate sub-models for subsequent sampling and evaluation, as in technical ideas 3 to 12 which are subordinate to technical idea 2. The method of technical idea 14 is the method of technical idea 13, wherein the evolutionary search performs multiple cycles including sampling from the population to select a quantity of candidate submodels, selecting a fastest submodel from the quantity, evolving the fastest submodel to determine related submodels, removing the oldest submodel from the population, and if the related submodel satisfies an accuracy threshold test, adding the related submodel to the population so that it can be sampled later, and adding the related submodel to a final set of fastest candidate submodels, and selecting the fastest submodel from the final set of fastest candidate submodels as the FAN model according to the performance of the multiple cycles. The method of technical idea 15 is the method of technical idea 13, wherein the evolutionary search performs multiple cycles including sampling from the population to select a quantity of candidate submodels, selecting a selection of a most accurate submodel from the quantity, evolving the most accurate submodel to determine related submodels, removing the oldest submodel from the population, and if the related submodel satisfies a speed threshold test, adding the related submodel to the population so that it can be sampled later, sampling the related submodel and adding it to a set of final most accurate candidate submodels, and selecting the most accurate submodel from the set of final most accurate candidate submodels as the FAN model according to the performance of the multiple cycles. The method of technical idea 16 is a two-stage structure in the method of any one of technical ideas 1 to 15, in which the network structure outputs to a region of interest (RoI) alignment layer to refine the extracted features for prediction. The method of technical idea 17 is the method described in technical idea 16, wherein the first stage includes four blocks, the second stage includes one block, each block includes up to four layers, and each layer includes an inverted residual layer. The method of technical idea 18 is a method according to any one of technical ideas 1 to 17, further comprising defining an application including the FAN model for execution by an edge device. The method of technical idea 19 is the method described in technical idea 18, wherein the application is one of a makeup simulation application for simulating the trying on of a makeup product on the face, a skin analysis application for analyzing and scoring an area or areas of the face for at least one skin indication, an effects simulation application for simulating a graphical effect associated with the area of ​​the face, and a face tracking application for tracking and indicating the position of the face. The method of technical idea 20 is the method of technical idea 18 or 19, wherein the application is configured to process video according to the FAN model. A method of technical idea 21 is a method according to any one of technical ideas 18 to 20, in which the edge device is a smartphone or a tablet terminal, and the FAN model is configured for execution by a graphics processing unit (GPU). A method of technical idea 22 is a method according to any one of technical ideas 19 to 21, wherein the application is the makeup simulation application or the skin analysis application, and the application is configured to provide an interface to an e-commerce purchasing service for purchasing products or services related to makeup and skin treatment, respectively. A method of technical idea 23 is a method according to any one of technical ideas 19 to 22, wherein the application is the makeup simulation application or the skin analysis application, and the application is configured to provide an interface to an e-commerce recommendation service that recommends products or services related to makeup and skin treatments, respectively. The computing device of technical concept 24 comprises a processor coupled to a storage device that stores instructions, which, when executed by the processor, configure the computing device to perform a method according to any of the methods described in the technical concept. The computer program product of technical idea 25 comprises a non-transitory storage device for storing instructions and data defining a FAN model for execution by a processing unit of a computing device, the FAN model being defined in a manner according to a method according to any of the above technical ideas. The computing device of technical idea 26 comprises a processing unit coupled to a storage device that stores instructions, which, when executed by the processing unit, configure the computing device to provide a FAN model defined according to the method of any of the above technical ideas.

Claims

1. 1. A method comprising: Defining a Face Alignment Network (FAN) model generating a set of candidate sub-models from the network structure by training the network structure using i) a training dataset and ii) an expansion and shrinkage training (EST) operation, where the EST operation defines and maintains instances of the candidate sub-models for various configuration parameters to define the set of candidate sub-models; and performing an evolutionary search of the candidate sub-models using speed and accuracy criteria to select the FAN model from the candidate sub-models.

2. The method of claim 1 , wherein the FAN model is defined for run-time execution by an edge device for processing facial images.

3. 3. The method of claim 1 or 2, wherein the EST operation starts training from a baseline model previously defined according to the network structure.

4. 4. The method of claim 3, wherein the EST operation utilizes pre-trained parameters from the baseline model as partially trained parameters in each of the candidate sub-models for further training using the EST operation.

5. 2. The method of claim 1, wherein the candidate sub-model is further trained for any one of between 2 and 60 epochs, between 2 and 25 epochs, and 2 epochs.

6. 6. The method of claim 3, wherein the EST operation activates a random subnetwork at each training iteration to train partial weights of the subnetwork as integrals, effectively training shared weights between sub-models.

7. 7. The method of claim 3, wherein the EST operation defines a search space for generating candidate sub-models according to a number of search dimensions including a depth size, a kernel size, a channel ratio and an expansion ratio.

8. 8. The method of claim 7, wherein the EST operation starts with a small depth size, a small kernel size, a large channel ratio and a large expansion ratio, and then the operation proceeds stepwise and in the following order: gradually expanding the depth size and kernel size to their respective maximums, and gradually reducing the channel ratio and expansion ratio to their respective minimums in an end phase.

9. 9. The method of claim 7 or 8, wherein the EST operation comprises a kernel training operation in an initial phase of training a kernel of a first size and then training a kernel of a second size using the kernel of the first size, the kernel of the first size being the center of the kernel of the second size.

10. 10. The method of claim 6, wherein the EST operation includes a depth training operation in an intermediate phase, training fewer layers and then gradually adding larger layer options to the training.

11. 11. The method according to claim 6, wherein the EST operation comprises a channel ratio and expansion ratio training operation, which reduces the channel ratio and the expansion ratio, respectively, in a termination phase.

12. 12. The method of claim 2, wherein the evolutionary search evaluates the baseline models to determine speed and accuracy criteria, as claimed in claim 3 to 10 depending on claim 2.

13. 13. The method of claim 2, wherein the evolutionary search evolves from the baseline model to define a population of candidate sub-models for subsequent sampling and evaluation.

14. The evolutionary search comprises: selecting a quantity of candidate sub-models by sampling from said population; Selecting the fastest sub-model from said quantity; evolving the fastest sub-model to determine relevant sub-models; removing the oldest sub-model from said population; performing a number of cycles including adding the associated sub-model to the population for subsequent sampling if the associated sub-model satisfies an accuracy threshold test, and adding the associated sub-model to a set of final fastest candidate sub-models; 14. The method of claim 13, further comprising: selecting the fastest submodel from the final set of fastest candidate submodels as the FAN model according to the performance over the multiple cycles.

15. The evolutionary search comprises: selecting a quantity of candidate sub-models by sampling from said population; Selecting a most accurate sub-model selection from said quantities; evolving the most accurate sub-model to determine a relevant sub-model; removing the oldest sub-model from said population; performing multiple cycles including adding the associated sub-model to a population for later sampling if the associated sub-model satisfies a speed threshold test, sampling the associated sub-model and adding it to a final set of most accurate candidate sub-models; 14. The method of claim 13, further comprising: selecting the most accurate submodel from the final set of most accurate candidate submodels as the FAN model according to the performance of the plurality of cycles.

16. 16. The method according to any of claims 1 to 15, characterized in that the network structure is a two-stage structure that outputs to a Region of Interest (RoI) alignment layer to refine the extracted features for prediction.

17. 17. The method of claim 16, wherein the first stage includes four blocks and the second stage includes one block, each block including up to four layers, each layer including an inverted residual layer.

18. 18. The method of claim 1, further comprising: defining an application including the FAN model for execution by an edge device.

19. The application may include a makeup simulation application for simulating the trying on of makeup products on the face; a skin analysis application for analyzing and scoring an area or areas of said face for at least one skin indication; an effects simulation application for simulating graphical effects associated with said facial regions; 20. The method of claim 18, wherein the method further comprises one of: and a face tracking application for tracking and indicating the position of the face.

20. 20. The method according to claim 18 or 19, wherein the application is configured to process video according to the FAN model.

21. 21. The method of claim 18, wherein the edge device is a smartphone or tablet, and the FAN model is configured for execution by a graphics processing unit (GPU).

22. 22. A method according to any one of claims 19 to 21, wherein the application is the makeup simulation application or the skin analysis application, the application being configured to provide an interface to an e-commerce purchasing service for purchasing products or services related to makeup and skin treatment, respectively.

23. 23. A method according to any one of claims 19 to 22, wherein the application is the makeup simulation application or the skin analysis application, the application being configured to provide an interface to an e-commerce recommendation service which recommends products or services related to makeup and skin treatments, respectively.

24. A computing device comprising a processor coupled to a storage device storing instructions, the instructions being configured, when executed by the processor, to perform a method according to any preceding claim.

25. 13. A computer program product comprising a non-transitory storage device for storing instructions and data defining a FAN model for execution by a processing unit of a computing device, said FAN model being defined in a manner according to the method of any preceding claim.

26. A computing device comprising a processing unit coupled to a storage device storing instructions, the instructions, when executed on the processing unit, configuring the computing device to provide a FAN model defined according to a method according to any preceding claim.

Images