System and method for age estimation

Abstract

Systems, methods, and computer-readable storage media for age estimation / classification, and more particularly, for estimating / classifying the age of a person appearing within a video. A system configured as disclosed herein can receive a video and then identify a plurality of facial images of each person captured within the video. The system can create embedding of facial images and then cluster those images together based on the distance between the corresponding embedding. The system can also perform a matching degree algorithm on those facial images to determine which image provides the clearest image of the person (s), and then can estimate the age of the person (s) and / or send the best matching degree image of each person to a third party for analysis using the best matching degree image.

Description

[0001] priority

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 419,605 filed on October 26, 2022 and U.S. Non-Provisional Patent Application No. 18 / 383,292 filed on October 24, 2023, the entire contents of the referred applications are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to age estimation, and more particularly to estimating the age of a person appearing in a video. Background Art

[0004] Estimating the age of persons appearing in video content is important for preventing minors from uploading content, as well as preventing the uploading and / or trafficking of minor video content. Summary of the invention

[0005] Additional features and advantages of the present disclosure will be set forth in the following description, will be understood in part from the description, or may be learned by practicing the principles disclosed herein. The features and advantages of the present disclosure may be realized and obtained by means of the means and combinations specifically indicated in the appended claims. These and other features of the present disclosure will become more apparent from the following description and appended claims, or may be learned by practicing the principles set forth herein.

[0006] A system, method, and non-transitory computer-readable storage medium that provide a technical solution to the described technical problem are disclosed. A method for executing the concepts disclosed herein may include: receiving a video at a computer system; extracting periodic frames from the video via at least one processor of the computer system, thereby generating a plurality of frames; performing face detection on the plurality of frames via at least one processor, thereby generating a plurality of faces; performing an embedding algorithm on the plurality of faces via at least one processor, thereby generating a plurality of embeddings corresponding to the plurality of faces; performing a matching algorithm on the plurality of faces via at least one processor, thereby generating a plurality of matching scores for the plurality of faces; clustering the plurality of faces via at least one processor based on similarity scores of the plurality of embeddings, thereby generating at least one cluster of faces; performing an age classifier algorithm on at least one cluster of faces via at least one processor, thereby generating an age classification; identifying a best facial image within at least one cluster of younger classified faces based on the plurality of matching scores; and transmitting an age estimation request from the computer system to a third party, the age estimation request including the best facial image and a request to estimate the age of an individual corresponding to the best younger classified facial image.

[0007] Another exemplary method may include: receiving a video at a computer system; and performing an age estimation on an individual captured within the video.

[0008] Yet another exemplary method may include receiving a video at a computer system, the video having a plurality of frames; performing face detection on the plurality of frames via at least one processor of the computer system, thereby generating a plurality of facial images; performing a matching algorithm on the plurality of facial images via the at least one processor, thereby generating a plurality of matching scores, each of the plurality of matching scores corresponding to an image within the plurality of facial images; performing an embedding algorithm on the plurality of facial images via the at least one processor, thereby generating a plurality of embeddings, each of the plurality of embeddings corresponding to an image within the plurality of facial images; calculating distances between the plurality of embeddings via the at least one processor, thereby generating a Euclidean distance matrix; clustering the plurality of embeddings using the Euclidean distance matrix via the at least one processor, thereby generating at least one cluster of embeddings; identifying within the plurality of facial images a best facial image in each of the at least one cluster of embeddings based on the plurality of matching scores; and transmitting from the computer system to a third party an age estimation request, the age estimation request including the best facial image of an individual and a request to estimate the age of the individual corresponding to the best facial image.

[0009] A system configured to perform the concepts disclosed herein may include: at least one processor; and a non-transitory computer-readable storage medium storing instructions, which when executed by the at least one processor cause the at least one processor to perform operations, the operations including: receiving a video, the video having a plurality of frames; performing face detection on the plurality of frames, thereby generating a plurality of facial images; performing a matching algorithm on the plurality of facial images, thereby generating a plurality of matching scores, each of the plurality of matching scores corresponding to an image within the plurality of facial images; performing an embedding algorithm on the plurality of facial images, thereby generating a plurality of embeddings, each of the plurality of embeddings corresponding to an image within the plurality of facial images; calculating distances between the plurality of embeddings, thereby generating a Euclidean distance matrix; clustering the plurality of embeddings using the Euclidean distance matrix, thereby generating at least one cluster of embeddings; identifying within the plurality of facial images a best facial image in each of the at least one cluster of embeddings based on the plurality of matching scores; and transmitting an age estimation request to a third party, the age estimation request including a best facial image of an individual and a request to estimate an age of the individual corresponding to the best facial image.

[0010] A non-transitory computer-readable storage medium configured as disclosed herein may store instructions that, when executed by at least one processor, cause the at least one processor to perform operations, the operations comprising: receiving a video having a plurality of frames; performing face detection on the plurality of frames, thereby generating a plurality of facial images; performing a matching algorithm on the plurality of facial images, thereby generating a plurality of matching scores, each of the plurality of matching scores corresponding to an image within the plurality of facial images; performing an embedding algorithm on the plurality of facial images, thereby generating a plurality of embeddings, each of the plurality of embeddings corresponding to an image within the plurality of facial images; calculating distances between the plurality of embeddings, thereby generating a Euclidean distance matrix; clustering the plurality of embeddings using the Euclidean distance matrix, thereby generating at least one cluster of embeddings; identifying within the plurality of facial images a best facial image in each of the at least one cluster of embeddings based on the plurality of matching scores; and transmitting an age estimation request to a third party, the age estimation request comprising a best facial image of an individual and a request to estimate the age of the individual corresponding to the best facial image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An example of a first portion of an age estimation process is illustrated;

[0012] Figure 2 illustrates an example of the second part of the age estimation process;

[0013] Figure 3 illustrates an alternative example of an age estimation process;

[0014] Figure 4 An example system is illustrated;

[0015] Figure 5 illustrates an example method embodiment; and

[0016] Figure 6 An example computer system is illustrated. DETAILED DESCRIPTION

[0017] Various embodiments of the present disclosure are described in detail below. Although specific implementations are described, this is for illustrative purposes only. Other components and configurations may be used without departing from the spirit and scope of the present disclosure.

[0018] Video sharing websites (such as but not limited to YOUTUBE, RUMBLE, and VIMEO) allow users to upload videos to a centralized platform while also allowing users to view and / or download the uploaded content. Reviewing content on such video distribution websites is a difficult technical task. At some point, the volume of uploaded content becomes too large for manual review - some kind of automated detection of prohibited content is required.

[0019] Banning content from certain platforms based on the age of the individuals captured within the video or image is another technical challenge. Some video distribution systems may allow videos containing people of all ages, but then remove those containing prohibited content (e.g., allowing videos of children's birthday parties, but removing any videos containing child pornography). Other video distribution systems, such as the one described herein, may suppress or ban all videos of people under a certain age.

[0020] Systems configured as described herein can perform age estimation on individuals detected within videos, animations, and / or images and prohibit uploading, distributing, and / or otherwise sharing the content if the age estimate is below a predetermined threshold age (e.g., 18 or 25). In some configurations, if the age estimate is below the threshold, the system can flag the content for additional manual review.

[0021] Exemplary, non-limiting steps taken by the system to perform age estimation may include, upon receiving a video, animation, animated image, or image requesting age estimation, or upon receiving a message indicating that age estimation is required, may include:

[0022] 1. Download and extract the frame

[0023] 2. Detect faces within the extracted frames to generate facial images

[0024] 3. Calculate the detection match of the facial image (i.e., the likelihood that a matching image will be found)

[0025] 4. Embed the detected facial image

[0026] 5. Cluster the facial image embeddings so that each individual’s facial image detected can be identified as illustrating a common individual

[0027] 6. Each individual's in-house age classification

[0028] 7. Third-party age estimation based on the match score using the best image of a given individual

[0029] This application considers each of these steps in turn.

[0030] 1) Download and extract the frame

[0031] Content is downloaded to the system and the file type can be identified (e.g., image, animation, video). For animation, all unique frames can be extracted (e.g., determined by a file integrity algorithm such as the MD5 algorithm (Message Digest Algorithm 5) for each frame) and saved to a file. For video, frames can be periodically extracted (e.g., one frame per second, one frame per half second, etc.) and saved to a file. After extraction, the system has a list of all saved frames or (if the content is images rather than video) a list of individual image files.

[0032] 2) Detecting faces

[0033] In this step, the list of frames (or a single image) can be passed through the MTCNN (Multi-Task Cascaded Convolutional Neural Network) face detection model. If a face is detected in the image, the MTCNN model can return:

[0034] -Bounding Box

[0035] -Detection score

[0036] -Facial landmarks

[0037] The bounding box can give the location of the face in the image to be cropped, as well as the dimensions of the bounding box, which can give the size (i.e., area) of the detected face. A detection score (e.g., 0-1) can indicate the model prediction as to whether the detected object is a face (e.g., 1 indicates that the detected object is almost certain to be a face, while 0 indicates that the detected object is almost certain not to be a face). In order to reduce false positive face detections, only detection scores above a threshold (e.g., 0.7 or another threshold) can be considered good face detections. This determination of detection can be based on the detection of facial landmarks, their relative sizes, the distance / angle between those facial landmarks, etc. Non-limiting examples of facial landmarks can include the location of eyes, noses, mouths, ears, etc. By identifying these facial landmark points, a bounding box around the face can be generated, thereby producing a facial image, and the angle of the face (from side portrait to front facing) can be calculated. The angle of the face can be output for future use. Facial landmark points can also be used to align any rotated face.

[0038] Consider if only a single person is captured in the video. Each frame of the video may contain a different image of the face of that person. The system may capture all of those facial images. Similarly, if multiple individuals are captured in the video, each frame may contain multiple facial images (e.g., a first facial image corresponds to the face of the first person captured in a first bounding box, and a second facial image corresponds to the face of the second person captured in a second bounding box). In this way, all facial images from each frame may be captured. Alternatively, in some configurations, the system may capture facial images from a subset of possible frames (e.g., every other frame, every five seconds, etc.).

[0039] 3) Calculate the detection matching degree

[0040] A score can be predicted based on the quality of the captured facial image, which is called the "match". While MTCNN can return a score indicating the probability that the image within the frame / image is a face, MTCNN can also detect how useful the image is for age estimation / classification. That is, because some well-detected faces are not practical for age estimation / classification, the system can score the detected faces to determine their usefulness in age estimation and / or classification. An ideal face for age estimation / classification resembles a passport, where the face faces forward, is larger, clearer, and has a neutral expression. To score the match, the system first generates additional information for each cropped face. The machine learning model can then use this additional information to score the match for a given face.

[0041] For example, the system can generate an image quality score and / or an emotion score (one or more emotion scores) using the detected faces. The image quality score can be based on, for example, image clarity (or blurriness) of a face crop, where image clarity is measured using an image quality assessment (IQ) model such as (but not limited to) the KonIQ-10k quality model. The emotion score can be generated, for example, by gray-scaling images of face crops and then feeding those gray-scaled face crops into a facial emotion recognition (FER) model (such as, but not limited to, the FER+ emotion model). The result of the FER model can be, for example, a vector of scores for eight emotions: neutral, happy, surprised, sad, angry, disgusted, fearful, and contemptuous. Vectors can be used to represent facial expressions, but other types of scores from other models are also possible.

[0042] With this additional information, a face detected by the system described in this article can have the following characteristics:

[0043] -Bounding box area (i.e., the geometric boundaries of the box)

[0044] - Detection score (i.e., how certain the system is that a face has been detected based on the landmarks and the relative positioning of those landmarks)

[0045] - Facial angle (i.e., frontal to portrait)

[0046] - Image quality score

[0047] - Sentiment score

[0048] A machine learning model can then be executed by the system to score the "match" of the face using one or more of these features (i.e., one or more of bounding box area, detection score, facial angle, image quality score, and emotion score(s), where the match score classifies how well the cropped facial features match the face. To train the machine learning model, the system can match a dataset of faces of known actors to a database of faces. If the best match is the same actor, then the detected features of the queried face can be considered good. Otherwise, if the best match is a different actor, then the detected features of the queried face can be considered poor.

[0049] 4) Embedded Faces

[0050] Face cropping can be processed with an embedding model (such as, but not limited to, FaceNet). The output of the embedding model can be a 512-dimensional (or other dimension count) embedding for each detected face, where more similar faces have more similar embeddings. That is, if a given person is captured in more than one frame, then multiple facial images of that person (taken from multiple frames) will each be processed by the embedding algorithm. While the resulting facial embeddings will not be exactly the same (e.g., due to the person moving, changing facial expressions, etc.), they will be similar to each other.

[0051] 5) Clustering the embeddings

[0052] As discussed above, for animations and videos, an individual may appear in many frames, resulting in many facial images corresponding to the same individual. As will be discussed below, age estimation / classification based on the individual may be performed by the system and / or outsourced to a third-party system. Preferably, for each captured individual, internal and external age estimation / classification of the captured individual is performed only once. To achieve this, and to reduce the load on those computing systems (e.g., by not processing the same face multiple times), the system calculates which faces are likely to belong to the same individual. This calculation is done by measuring the distances between facial embeddings, which can then be clustered based on the measured distances. For example, if the distance between embeddings A and B is small, but the distance between embeddings A and C is large, then there are likely two different individuals, which will result in two clusters of embeddings and associated facial images.

[0053] First, a Euclidean distance matrix between embeddings can be calculated. If the distance between faces is within a predetermined threshold, then the faces can be connected in the adjacency matrix (meaning that the Euclidean distance between the embeddings is within a predetermined threshold), indicating that the faces may belong to the same individual. In this configuration, the distance threshold cannot be too high, otherwise all faces will be considered to belong to the same person, resulting in all faces being connected in the adjacency matrix. To be conservative, a lower threshold can be applied to create fewer connections so that two different individuals are not inadvertently treated as a single individual. The connected faces in the adjacency matrix can be grouped into clusters so that each individual captured in the video will have a corresponding cluster of their facial images.

[0054] 6) Age classification

[0055] Each embedding can be passed to its own age classifier, which determines whether the face is more likely to be 25 years old (or another age) and assigns a value (e.g., between 0 and 1) based on that determination. For example, the system can provide a numerical prediction indicating how likely it is that the person being analyzed is under 25 or over 25 years old (or another age). For example, if the system is 70% confident that the person is under 25 years old, then the system can assign a classification of 0.7 (or 0.3, depending on the configuration). This classifier can be trained on a dataset that splits faces into two classes, with the threshold determined based on known / labeled data.

[0056] For each cluster, the age classification scores can be averaged across all grouped faces within that cluster. For example, if a cluster has multiple embeddings for a person (each embedding corresponding to a facial image of that person), then each of those embeddings can be classified as described above. Thus, if there are three embeddings with age classification scores of 0.6, 0.7, and 0.8, then the average for the corresponding person would be 0.7.

[0057] 7) Third-party age estimation

[0058] If the cluster appears young (i.e., the average classification score is above a threshold, or below a threshold, depending on the configuration), then the best face in the cluster, as determined by the match score, can be sent to a third-party provider for secondary confirmation. While in some configurations it is possible to send all faces that appear young in their own age classification, there can be multiple images, which can result in excessive bandwidth and processing costs. By clustering and sending the best faces in the cluster, third-party processing time and costs can be reduced.

[0059] The present application then turns to the specific examples shown in the figures. Although many of the examples in the illustrated examples use video, the video can be an animated video or a live-action video. Similarly, the examples shown and disclosed in the figures can also be applied to images, the difference being the amount of data processed by the system (e.g., a single image rather than an image captured in multiple frames or images).

[0060] Figure 1 An example of a first portion of an age estimation process is illustrated. In this example, a message 102 is received by the system. Based on the message 102, the system downloads 104 content 106 (e.g., an image, animation, or video) tagged or otherwise identified by the message 102. The system then identifies 108 the type of content 106 being downloaded. If the content 106 is an animation 112 or a video 114, the system may periodically sample the content, extracting frames in the form of extracted images 116, 118 (or snapshots or individual frames) at each period (e.g., every second, every 0.5 seconds, every other frame, etc.). If the content 106 is an image 110, it is identified that only a single image has been downloaded. The system may list the images 120, whether in the form of extracted images 116, 118 captured from the animation 112 or video 114, or in the form of extracted images 116, 118 captured from the downloaded image 110. Note that the systems, methods, and processes disclosed herein are equally applicable to animations 112, videos 114, and downloaded images 110. In some configurations, this list of images 120 may take the form of an array 122 .

[0061] From the resulting list of images 120, the system may perform face detection 124, resulting in a set of cropped faces 126. As shown, the set of cropped faces 126 may be converted into an embedding 128 (e.g., a numerical or vector representation), resulting in a facial embedding 130 corresponding to the set of cropped faces 126. The set of cropped faces 126 may also be used to detect one or more emotions 132 of the user within each cropped facial image, as well as an image quality score 136 for each cropped facial image. Non-limiting examples of detectable emotions 134 may include anger, neutral, sadness, happiness, excitement, boredom, etc. The image quality score 136 may quantify, for example, the angle of the cropped facial image relative to the camera's face, the number of pixels, the lighting, etc. Using the detected emotions 132, the image quality score 136, and / or other features, the system may calculate a match score 138 for the images of the individuals within the set of cropped faces.

[0062] Figure 2 An example of the second part of the age estimation process is shown. Figure 1, the system can calculate the similarity of the individuals captured in the images, preferably using a Euclidean distance matrix 202, resulting in a similarity distance matrix 204, and then calculate whether they are within a threshold 206 distance of each other, resulting in an adjacency matrix 208. As shown, the adjacency matrix 208 indicates that if the images are within the threshold distance 206 based on the similarity in their facial embeddings 130, then using a binary representation, the facial images 136 corresponding to those embeddings 130 should be considered to be the same individual. In other words, the system converts the detected faces 126 into embeddings 130, and then measures how similar those embeddings are to each other using a Euclidean distance measure 202 (in other configurations, other mechanisms for calculating distances may be used). If the distance between the embeddings is small enough (i.e., below a threshold), then the system considers those embeddings to represent facial images of a single individual and clusters those closely related images together. As shown, the system creates edges 210 between similar images, resulting in clusters 212, 214, each of which represents images associated with a unique individual based on the similarity distance 204.

[0063] The system may then perform a first age estimation and / or classification (on each image within each cluster). The first age classification may be, for example, an own age classifier. In the illustrated example, if the captured individual is above or below an age threshold (e.g., twenty-five years old), then the system seeks an age classification 216. In some configurations, the system may average the estimated ages for a given cluster, resulting in a classification 218 of the average age of each group on a scale between 0 and 1 (the scale indicating the system's confidence level that they are above or below the threshold). As shown, group one 220 has an average age of 0.8 (i.e., most of the estimated ages are below twenty-five years old), while group two 222 has an average age of 0.0 (i.e., no one is estimated to be below twenty-five years old).

[0064] If the system detects images (or cluster averages) where the estimated age of an individual within the cluster is below a predetermined threshold, the system can add / combine 224 the cropped facial images of the cluster with the corresponding match scores 138, thereby producing a combination 226 of the match scores 138 and the cluster's age estimates 220. Based on the cluster's age estimates 220, 222 and the match scores 138, the system can identify a subset of images from the cluster (e.g., the best images 228 or images from a group that appear to violate the age threshold), and then send the subset of images (i.e., the best images of the individuals in question) to a third-party age estimator 230.

[0065] If the third party age estimator 230 identifies that the face is above a predetermined threshold age, the system can allow the content to be uploaded to the system for wider distribution. If the third party age estimator identifies that the face is below a predetermined threshold age, or there is doubt, the system can mark the content as requiring additional manual review.

[0066] Figure 3 An alternative example of an age estimation process is illustrated. In this exemplary process, the matching and clustering of cropped faces occurs sequentially, rather than through parallel processing. As shown, a video 302 is uploaded 304 to a video sharing platform and then sampled 306 (e.g., every other frame; every second; every I, P, and / or B frame; etc.), thereby generating sample screenshots 308 or frames. Using the sample screenshots 308, the system can execute one or more face detection algorithms 310, thereby generating cropped faces 312 corresponding to the faces identified within the sample screenshots 308. The system can vectorize (embed) the cropped faces 314, thereby generating embeddings 316 of the detected faces. The system can then identify similar faces 318 within those embeddings 316 using a distance measure (such as cosine distance), thereby generating clusters 320 of faces, each cluster corresponding to a single individual captured within the video 302. From these clusters 320 of faces, the system attempts to identify matching faces 322 (i.e., faces that are most likely to be identified or generate a correct age estimate). The best sample face 324 from each cluster 320 of faces is then sent to an in-house age classifier 326, which estimates 328 whether the best sample face 324 belongs to an individual above a threshold (e.g., if the individual is over an age limit). If so, the system allows the video 302 to be uploaded 330 to a video sharing platform. If the individual does not exceed the threshold, or if the age is in question, the best sample face 324 can be sent to a third party 332, which can perform a similar age classification 334 and / or age estimation. If the third party indicates that the individual is over the threshold, the video can be uploaded 336, while if the third party indicates that the individual does not exceed the threshold, or the analysis is in question, the video 302 can be directed to manual review 338.

[0067] Figure 4An example system 406 is illustrated. In this example, a message 402 comes in and is placed in a message queue 404. When the system 406 is able to process the message 402 from the message queue 404, the system 406 reads the message using a message handler 408, thereby issuing a get message 412 command (thereby pulling the message 402 from the queue in response to a download initiative 420), reading 410 the message 402 by passing 414 the message 402 to a Celery chain 418 (also referred to as a Celery workflow) within a pipeline handler 416. Using the Celery chain 418, the pipeline handler 416 can issue a command 424 (such as a load 424 command caused by a save initiative 422) to the message handler 408 to save 426 the message. Again using the Celery chain 418, the pipeline handler 416 can issue commands to various pipeline classes 428, including other commands 436 not explicitly identified in the example illustration. Non-limiting examples of these commands may include download 430, face 432 (as in recognize face), action 434 (e.g., take a specific action), or other commands 438 not explicitly identified in the example illustration. A YAML (Yet Another Markup Language) file 440 may be used as a configuration file, providing parameters to instantiate 442, 444 the message handler 408 and / or the pipe handler 416. The YAML file 440 may also be used to provide parameters to instantiate 446 a shutdown handler 448, which may be called if there are no messages for the system 406 to process.

[0068] In this example, message 402 indicates "Age estimation is performed on specific video content". System 406 then performs age estimation as described above and further breaks down the specific video content into different categories (e.g., face, action, etc.) and is able to download the content, save the content, or perform other actions on the content. The results are then sent to result queue 450 as a response to message 402.

[0069] Figure 5 An exemplary method embodiment is illustrated. As shown, the method may include receiving a video at a computer system, the video having a plurality of frames (502), and performing face detection on the plurality of frames via at least one processor of the computer system to generate a plurality of face images (504). Although the illustrated method describes a video, the video may be an animated video or a live-action video. Similarly, with respect to Figure 5The processing shown and disclosed in can also be applied to images, with the difference being the amount of data processed by the system (e.g., a single image rather than images captured in multiple frames or images). The system can perform a matching algorithm on the multiple facial images via at least one processor, thereby generating a plurality of matching scores, each of the plurality of matching scores corresponding to an image in the multiple facial images (506), and perform an embedding algorithm on the multiple facial images via at least one processor, thereby generating a plurality of embeddings, each of the plurality of embeddings corresponding to an image within the multiple facial images (508). The system can then calculate distances between the multiple embeddings via at least one processor, thereby generating a Euclidean distance matrix (510), and cluster the multiple embeddings using the Euclidean distance matrix via at least one processor, thereby generating at least one cluster of embeddings (512). Generally speaking, each cluster within the at least one cluster of embeddings will have an embedding associated with a single individual captured in the video, but in some cases (such as twins or people with similar faces), multiple individuals may have their faces clustered together. The system may then identify, within the plurality of facial images, a best facial image for each of the at least one cluster embedded within the plurality of facial images based on the plurality of match scores (514), and transmit, from the computer system to a third party, an age estimation request including the best facial image of the individual and a request to estimate the age of the individual corresponding to the best facial image.

[0070] In some configurations, identifying the best facial image for each individual captured in the video may also include: extracting at least two frames from the video via at least one processor of the computer system, thereby generating a plurality of frames; and performing facial detection on each of the plurality of frames via the at least one processor, thereby generating a plurality of facial images.

[0071] In some configurations, execution of the matching algorithm may include generating an emotion score for each of the plurality of facial images, wherein the emotion score is used to calculate the plurality of matching scores. In such configurations, the emotion score may include a vector of scores for emotions for each of the plurality of facial images, the emotions including: happiness, surprise, sadness, anger, disgust, fear, and / or contempt.

[0072] In some configurations, execution of the matching algorithm may include generating a facial angle for each facial image in the plurality of facial images, wherein the facial angle may be used to calculate a plurality of matching scores.

[0073] In some configurations, the method may further include: prior to transmitting the age estimation request, executing, via the at least one processor, an internal age classification algorithm on at least one cluster of the embeddings to produce an age classification for each individual appearing in the video, wherein the transmission of the age estimation request may be modified based on the age classification for each individual appearing in the video. In such configurations, the internal age classification algorithm may use the plurality of embeddings to estimate the age of each individual appearing in the video.

[0074] refer to Figure 6 , an exemplary system includes a computing device 600, including a processing unit (CPU or processor) 620 and a system bus 610, which couples various system components including system memory 650 (such as read-only memory (ROM) 640 and random access memory (RAM) 650) to the processor 620. The computing device 600 may include a cache of high-speed memory that is directly connected to the processor 620, close to the processor 620, or integrated as part of the processor 620. The computing device 600 copies data from the RAM 650 and / or storage device 660 to the cache for rapid access by the processor 620. In this way, the cache provides a performance improvement by avoiding delays in the processor 620 while waiting for data. These and other modules can control or be configured to control the processor 620 to perform various actions. Other system memory 630 is also available. The RAM 650 may include a variety of different types of memory with different performance characteristics. It can be appreciated that the present disclosure can operate on a computing device 600 having more than one processor 620, or on a group or cluster of computing devices networked together to provide greater processing power. The processor 620 can include any general purpose processor and hardware modules or software modules (such as module 1 662, module 2 664, and module 3 666 stored in the storage device 660 and configured to control the processor 620), as well as a dedicated processor in which the software instructions are incorporated into the actual processor design. The processor 620 can essentially be a completely independent computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0075] The system bus 610 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. A basic input / output (BIOS) stored in a ROM 640 or the like can provide basic routines that help transfer information between elements within the computing device 600, such as during startup. The computing device 600 also includes a storage device 660, such as a hard disk drive, a disk drive, an optical drive, a tape drive, etc. The storage device 660 may include software modules 662, 664, 666 for controlling the processor 620. Other hardware or software modules are contemplated. The storage device 660 is connected to the system bus 610 via a drive interface. The drive and associated computer-readable storage media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device 600. In one aspect, a hardware module that performs a specific function includes a software component stored in a tangible computer-readable storage medium that is connected to a hardware component necessary to perform the function (such as a processor 620, a system bus 610, an output device 670 (such as a display), etc.). In another aspect, the system may use a processor and a computer-readable storage medium to store instructions that, when executed by a processor (e.g., one or more processors), cause the processor to perform a method or other specific actions. Basic components and appropriate variations are contemplated depending on the type of device (such as whether device 600 is a small handheld computing device, a desktop computer, or a computer server).

[0076] Although the exemplary embodiments described herein employ a storage device 660 such as a hard disk, other types of computer-readable media that can store computer-accessible data may also be used in the exemplary operating environment, such as magnetic tapes, flash memory cards, digital versatile disks, cassettes, random access memory (RAM) 650, and read-only memory (ROM) 640. Tangible computer-readable storage media, computer-readable storage devices, or computer-readable memory devices expressly exclude media such as transient waves, energy, carrier signals, electromagnetic waves, and signals themselves.

[0077] In order to enable the user to interact with the computing device 600, the input device 690 represents any number of input mechanisms, such as a microphone for speaking, a touch screen for gesture or graphic input, a keyboard, a mouse, motion input, speech, etc. The output device 670 can also be one or more of a variety of output mechanisms known to those skilled in the art. In some cases, a multi-mode system enables a user to provide multiple types of input to communicate with the computing device 600. The communication interface 680 generally controls and manages user input and system output. There is no restriction on operating on any particular hardware arrangement, so the basic features here can be easily replaced with improved hardware or firmware arrangements after the improved hardware or firmware arrangements are developed.

[0078] The technology discussed herein refers to computer-based systems and the actions taken by computer-based systems and the information sent to and from computer-based systems. Those of ordinary skill in the art will recognize that the inherent flexibility of computer-based systems allows for various possible configurations, combinations, and partitioning of tasks and functionality between components. For example, the processes discussed herein can be implemented using a single computing device or a combination of computing devices. Databases, memories, instructions, and applications can be implemented on a single system, or can be distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0079] The use of language such as “at least one of X, Y, and Z,” “at least one of X, Y, or Z,” “at least one or more of X, Y, and Z,” “at least one or more of X, Y, or Z,” “at least one or more of X, Y, and / or Z,” or “at least one of X, Y, and / or Z” is intended to include both single items (e.g., only X, or only Y, or only Z) and multiple items (e.g., {X and Y}, {X and Z}, {Y and Z}, or {X, Y, and Z}). The phrase “at least one of” and similar phrases are not intended to convey a requirement that every possible item must be present, but that every possible item may be present.

[0080] The various embodiments described above are provided by way of example only and should not be construed as limiting the scope of the present disclosure. Various modifications and changes may be made to the principles described herein without departing from the spirit and scope of the present disclosure without following the example embodiments and applications shown and described herein. For example, unless otherwise expressly stated, the steps of a process or method may be performed in an order different from the example embodiments discussed above. Likewise, various components may be omitted, replaced, or arranged in a configuration different from the example embodiments discussed above, unless otherwise indicated.

[0081] Further aspects of the disclosure are provided by the subject matter of the following clauses.

[0082] A method includes: receiving a video at a computer system; extracting periodic frames from the video via at least one processor of the computer system to produce a plurality of frames; performing face detection on the plurality of frames via the at least one processor to produce a plurality of faces; performing an embedding algorithm on the plurality of faces via the at least one processor to produce a plurality of embeddings corresponding to the plurality of faces; performing a matching algorithm on the plurality of faces via the at least one processor to produce a plurality of matching scores for the plurality of faces; clustering the plurality of faces based on similarity scores of the plurality of embeddings via the at least one processor to produce at least one cluster of faces; performing an age classifier algorithm on at least one cluster of faces via the at least one processor to produce at least one age classification; identifying a best facial image within at least one younger cluster of faces based on the plurality of matching scores; and transmitting an age estimation request from the computer system to a third party, the age estimation request including the best facial image and a request to estimate the age of an individual corresponding to the best facial image.

[0083] A method includes: receiving a video at a computer system; and performing age estimation on an individual captured within the video.

[0084] A method includes: receiving a video at a computer system, the video having a plurality of frames; performing face detection on the plurality of frames via at least one processor of the computer system, thereby generating a plurality of facial images; performing a matching algorithm on the plurality of facial images via the at least one processor, thereby generating a plurality of matching scores, each of the plurality of matching scores corresponding to an image within the plurality of facial images; performing an embedding algorithm on the plurality of facial images via the at least one processor, thereby generating a plurality of embeddings, each of the plurality of embeddings corresponding to an image within the plurality of facial images; calculating distances between the plurality of embeddings via the at least one processor, thereby generating a Euclidean distance matrix; clustering the plurality of embeddings via the at least one processor using the Euclidean distance matrix, thereby generating at least one cluster of embeddings; identifying within the plurality of facial images a best facial image in each of the at least one cluster of embeddings based on the plurality of matching scores; and transmitting from the computer system to a third party an age estimation request, the age estimation request including a best facial image of an individual and a request to estimate an age of the individual corresponding to the best facial image.

[0085] A method according to any of the preceding clauses, wherein identifying the best facial image for each individual captured in the video further comprises: extracting at least two frames from the video via the at least one processor of the computer system, thereby generating a plurality of frames; and performing facial detection on each of the plurality of frames via the at least one processor, thereby generating the plurality of facial images.

[0086] A method according to any preceding clause, wherein executing the matching algorithm comprises generating a mood score for each facial image in the plurality of facial images, wherein the mood scores are used to calculate the plurality of matching scores.

[0087] A method according to any preceding clause, wherein for each facial image of the plurality of facial images, the emotion score comprises a vector of scores for emotions, the emotions comprising: happiness, surprise, sadness, anger, disgust, fear and contempt.

[0088] A method according to any preceding clause, wherein executing the matching algorithm comprises generating a facial angle for each facial image in the plurality of facial images, wherein the facial angle is used to calculate the plurality of matching scores.

[0089] The method according to any of the preceding clauses further comprises: prior to transmitting the age estimation request, executing, via the at least one processor, an internal age classification algorithm on the embedded at least one cluster to produce an age classification for each individual appearing in the video, wherein the transmission of the age estimation request is modified based on the age classification for each individual appearing in the video.

[0090] A method according to any preceding clause, wherein an internal age classification algorithm uses the plurality of embeddings to estimate the age of each individual appearing in the video.

[0091] A system includes: at least one processor; and a non-transitory computer-readable storage medium storing instructions, the instructions, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: receiving a video, the video having a plurality of frames; performing face detection on the plurality of frames, thereby generating a plurality of facial images; performing a matching algorithm on the plurality of facial images, thereby generating a plurality of matching scores, each of the plurality of matching scores corresponding to an image within the plurality of facial images; performing an embedding algorithm on the plurality of facial images, thereby generating a plurality of embeddings, each of the plurality of embeddings corresponding to an image within the plurality of facial images; calculating distances between the plurality of embeddings, thereby generating a Euclidean distance matrix; clustering the plurality of embeddings using the Euclidean distance matrix, thereby generating at least one cluster of embeddings; identifying within the plurality of facial images a best facial image in each of the at least one cluster of embeddings based on the plurality of matching scores; and transmitting an age estimation request to a third party, the age estimation request including a best facial image of an individual and a request to estimate an age of the individual corresponding to the best facial image.

[0092] A system according to any of the preceding clauses, wherein identifying the best facial image for each individual captured in the video further comprises: extracting at least two frames from the video, thereby producing a plurality of frames; and performing facial detection on each of the plurality of frames via the at least one processor, thereby producing the plurality of facial images.

[0093] The system of any preceding clause, wherein executing the matching algorithm comprises generating a mood score for each facial image in the plurality of facial images, wherein the mood scores are used to calculate the plurality of matching scores.

[0094] The system of any preceding clause, wherein for each facial image of the plurality of facial images, the emotion score comprises a vector of scores for emotions, the emotions comprising: happiness, surprise, sadness, anger, disgust, fear, and contempt.

[0095] The system of any preceding clause, wherein executing the matching algorithm comprises generating a facial angle for each facial image in the plurality of facial images, wherein the facial angle is used to calculate the plurality of matching scores.

[0096] A system according to any of the preceding clauses, the non-transitory computer-readable storage medium storing additional instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: prior to transmitting an age estimation request, executing an internal age classification algorithm on at least one embedded cluster to generate an age classification for each individual appearing in the video, wherein the transmission of the age estimation request is modified based on the age classification for each individual appearing in the video.

[0097] A system according to any preceding clause, wherein an internal age classification algorithm uses the plurality of embeddings to estimate the age of each individual appearing in the video.

[0098] A non-transitory computer-readable storage medium storing instructions, the instructions, when executed by at least one processor, cause the at least one processor to perform operations, the operations comprising: receiving a video, the video having a plurality of frames; performing face detection on the plurality of frames, thereby generating a plurality of facial images; performing a matching algorithm on the plurality of facial images, thereby generating a plurality of matching scores, each of the plurality of matching scores corresponding to an image within the plurality of facial images; performing an embedding algorithm on the plurality of facial images, thereby generating a plurality of embeddings, each of the plurality of embeddings corresponding to an image within the plurality of facial images; calculating distances between the plurality of embeddings, thereby generating a Euclidean distance matrix; clustering the plurality of embeddings using the Euclidean distance matrix, thereby generating at least one cluster of embeddings; identifying within the plurality of facial images a best facial image in each of the at least one cluster of embeddings based on the plurality of matching scores; and transmitting an age estimation request to a third party, the age estimation request comprising a best facial image of an individual and a request to estimate the age of the individual corresponding to the best facial image.

[0099] A non-transitory computer-readable storage medium according to any of the preceding clauses, wherein identifying the best facial image for each individual captured in the video further comprises: extracting at least two frames from the video, thereby producing a plurality of frames; and performing facial detection on each of the plurality of frames, thereby producing the plurality of facial images.

[0100] The non-transitory computer-readable storage medium of any preceding clause, wherein executing the matching algorithm comprises generating an emotion score for each facial image in the plurality of facial images, wherein the emotion scores are used to calculate the plurality of matching scores.

[0101] The non-transitory computer-readable storage medium of any preceding clause, wherein for each facial image of the plurality of facial images, the emotion score comprises a vector of scores for emotions, the emotions comprising: happiness, surprise, sadness, anger, disgust, fear, and contempt.

[0102] The non-transitory computer-readable storage medium of any preceding clause, wherein executing the matching algorithm comprises generating a facial angle for each facial image in the plurality of facial images, wherein the facial angle is used to calculate the plurality of matching scores.

[0103] A non-transitory computer-readable storage medium according to any preceding clause, having additional instructions stored thereon, which when executed by the at least one processor cause the at least one processor to perform operations comprising: prior to transmitting an age estimation request, executing an internal age classification algorithm on at least one embedded cluster to produce an age classification for each individual appearing in the video, wherein the transmission of the age estimation request is modified based on the age classification for each individual appearing in the video.

Claims

1. A method, include: receiving a video at a computer system, the video having a plurality of frames; performing, via at least one processor of the computer system, facial detection on the plurality of frames, thereby generating a plurality of facial images; executing, via the at least one processor, a matching algorithm on the plurality of facial images to generate a plurality of matching scores, each matching score in the plurality of matching scores corresponding to an image within the plurality of facial images; executing, via the at least one processor, an embedding algorithm on the plurality of facial images to thereby generate a plurality of embeddings, each embedding of the plurality of embeddings corresponding to an image within the plurality of facial images; calculating, via the at least one processor, distances between the plurality of embeddings, thereby generating a Euclidean distance matrix; clustering, via the at least one processor, the plurality of embeddings using a Euclidean distance matrix to generate at least one cluster of embeddings; identifying, within the plurality of facial images, a best facial image for each of the at least one embedded cluster based on the plurality of matching scores; as well as An age estimation request is transmitted from the computer system to a third party, the age estimation request including a best facial image of an individual and a request to estimate an age of the individual corresponding to the best facial image.

2. The method of claim 1, wherein identifying the best facial image for each individual captured in the video further comprises: include: extracting, via the at least one processor of the computer system, at least two frames from the video to generate a plurality of frames; as well as Face detection is performed on each of the plurality of frames via the at least one processor, thereby generating the plurality of facial images. 3 . The method of claim 1 , wherein executing the matching algorithm comprises generating an emotion score for each of the plurality of facial images, wherein the emotion score is used to calculate the plurality of matching scores.

4. The method of claim 3, wherein for each facial image in the plurality of facial images, the emotion score comprises a vector of scores for emotions, the emotion include: Joy, surprise, sadness, anger, disgust, fear and contempt.

5. The method of claim 1, wherein executing a matching algorithm comprises generating a facial angle for each facial image in the plurality of facial images, wherein the facial angle is used to calculate the plurality of matching scores.

6. The method according to claim 1, further comprising: include: Prior to transmitting the age estimation request, executing, via the at least one processor, an internal age classification algorithm on the embedded at least one cluster to produce an age classification for each individual appearing in the video, wherein the transmission of an age estimation request is modified based on the age classification of each individual appearing in the video.

7. The method of claim 6, wherein an internal age classification algorithm uses the plurality of embeddings to estimate the age of each individual appearing in the video.

8. A system, include: at least one processor; as well as A non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: receiving a video, the video having a plurality of frames; performing face detection on the plurality of frames, thereby generating a plurality of face images; executing a matching algorithm on the plurality of facial images to generate a plurality of matching scores, each matching score in the plurality of matching scores corresponding to an image within the plurality of facial images; performing an embedding algorithm on the plurality of facial images to generate a plurality of embeddings, each embedding in the plurality of embeddings corresponding to an image within the plurality of facial images; calculating distances between the plurality of embeddings, thereby generating a Euclidean distance matrix; clustering the plurality of embeddings using a Euclidean distance matrix to produce at least one cluster of embeddings; identifying, within the plurality of facial images, a best facial image for each of the at least one embedded cluster based on the plurality of matching scores; and An age estimation request is transmitted to a third party, the age estimation request including the best facial image of the individual and a request to estimate the age of the individual corresponding to the best facial image.

9. The system of claim 8, wherein identifying the best facial image for each individual captured in the video further comprises: include: extracting at least two frames from the video to generate a plurality of frames; as well as Face detection is performed on each of the plurality of frames via the at least one processor, thereby generating the plurality of facial images.

10. The system of claim 8, wherein executing the matching algorithm comprises generating a mood score for each facial image in the plurality of facial images, wherein the mood score is used to calculate the plurality of matching scores.

11. The system of claim 10, wherein for each facial image in the plurality of facial images, the emotion score comprises a vector of scores for emotions, the emotion include: Joy, surprise, sadness, anger, disgust, fear and contempt.

12. The system of claim 8, wherein executing the matching algorithm comprises generating a facial angle for each facial image in the plurality of facial images, wherein the facial angle is used to calculate the plurality of matching scores.

13. The system of claim 8, wherein the non-transitory computer readable storage medium stores additional instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: Prior to transmitting an age estimation request, an internal age classification algorithm is performed on at least one of the embedded clusters, thereby producing an age classification for each individual appearing in the video, wherein the transmission of an age estimation request is modified based on the age classification of each individual appearing in the video.

14. The system of claim 13, wherein an internal age classification algorithm uses the plurality of embeddings to estimate the age of each individual appearing in the video.

15. A non-transitory computer-readable storage medium storing instructions, which, when executed by at least one processor, cause the at least one processor to perform operations, the operations include: receiving a video, the video having a plurality of frames; performing face detection on the plurality of frames, thereby generating a plurality of face images; executing a matching algorithm on the plurality of facial images to generate a plurality of matching scores, each matching score in the plurality of matching scores corresponding to an image within the plurality of facial images; performing an embedding algorithm on the plurality of facial images to generate a plurality of embeddings, each embedding in the plurality of embeddings corresponding to an image within the plurality of facial images; calculating distances between the plurality of embeddings, thereby generating a Euclidean distance matrix; clustering the plurality of embeddings using a Euclidean distance matrix to produce at least one cluster of embeddings; identifying, within the plurality of facial images, a best facial image for each of the at least one embedded cluster based on the plurality of matching scores; as well as An age estimation request is transmitted to a third party, the age estimation request including the best facial image of the individual and a request to estimate the age of the individual corresponding to the best facial image.

16. The non-transitory computer-readable storage medium of claim 15, wherein identifying the best facial image for each individual captured in the video further comprises: include: extracting at least two frames from the video to generate a plurality of frames; as well as Face detection is performed on each of the plurality of frames, thereby generating the plurality of facial images.

17. The non-transitory computer-readable storage medium of claim 15, wherein executing the matching algorithm comprises generating a mood score for each facial image in the plurality of facial images, wherein the mood score is used to calculate the plurality of matching scores.

18. The non-transitory computer-readable storage medium of claim 17, wherein for each facial image in the plurality of facial images, the emotion score comprises a vector of scores for emotions, the emotion include: Joy, surprise, sadness, anger, disgust, fear and contempt.

19. The non-transitory computer-readable storage medium of claim 15, wherein executing the matching algorithm comprises generating a facial angle for each facial image in the plurality of facial images, wherein the facial angle is used to calculate the plurality of matching scores.

20. The non-transitory computer readable storage medium of claim 15 having additional instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: Prior to transmitting an age estimation request, an internal age classification algorithm is performed on at least one of the embedded clusters, thereby producing an age classification for each individual appearing in the video, wherein the transmission of an age estimation request is modified based on the age classification of each individual appearing in the video.

Images