Human-centered ethical image dataset

Abstract

By collecting a plurality of images from a plurality of diverse persons, a diverse dataset of human images can be created. The first graphical user interface requires a user to provide subject data, instrument data, and environmental data as metadata for each of the plurality of images. The second graphical user interface requires the user to form a bounding box around the face of the subject in each of the plurality of images. A third graphical user interface requires the annotator to provide annotations for each of the plurality of images. The datasets may be used to train or evaluate machine learning or artificial intelligence systems, such as systems for body and face detection, body and face feature point detection, body and face analysis, face alignment, face recognition, face verification, image editing, and image synthesis.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to systems and methods for collecting datasets to provide a universal, consistent, human-centric image dataset of the human body, which must include the head. The obtained dataset can be used to perform various human-centric tasks, for example, for training and evaluating commercial machine learning (ML) and artificial intelligence (AI) systems in an unconstrained environment. Background Art

[0002] The following background information may present examples of specific aspects of the prior art (such as, but not limited to, schemes, facts, or common knowledge), and while these examples are expected to help further educate the reader about other aspects of the prior art, they should not be construed as limiting the present invention or any embodiment thereof to anything stated or implied or inferred therefrom.

[0003] In an unconstrained setting, factors related to the subjects (e.g., demographic information), the instrument (e.g., camera hardware and software), and the environment (e.g., lighting, camera distance) cannot be controlled.

[0004] Therefore, there is a need to collect a dataset of human-centric images that is as diverse as possible with respect to these three factors, where all annotated persons have given written informed consent. Summary of the invention

[0005] Aspects of the present invention provide apparatus and methods for implementing techniques for image datasets and collection. In one embodiment, a dataset is constructed using a plurality of images (including images of humans). The collection, processing, storage, and configuration of the dataset are managed to provide desired characteristics that reflect targeted norms, guidelines, and metrics, such as to represent ethical goals.

[0006] Features provided in embodiments may include, but are not limited to, one or more of the following: (1) collecting images and image data including human bodies; (2) collecting and creating data related to image data, such as information about demographics, physical characteristics, movements, postures, environments, and instruments; (3) managing data collection based on data specifications; (4) accepting and managing annotations of data provided by users, suppliers, and automatically generated; and (5) collecting and managing consent information of people shown in images.

[0007] As used herein, the term "AI" refers to any functionality or enabling technology that performs information processing that people consider to be intelligent for various purposes and is implemented through rules or knowledge extracted from data based on ML or through some method.

[0008] An embodiment of the present invention provides a computer-implemented method for constructing a dataset of human images, comprising: collecting multiple images from multiple diverse people; providing a first graphical user interface, requiring a user to provide subject data, instrument data, and environmental data as metadata for each of the multiple images; and storing the multiple images as a dataset, wherein the subject data includes demographic information, physical characteristics, movements, and head postures.

[0009] An embodiment of the present invention provides a computer-implemented method for training or evaluating a commercial machine learning or artificial intelligence system in an unconstrained setting, comprising: creating a diverse dataset of human images by: collecting multiple images from multiple diverse people; providing a first graphical user interface that requires a user to provide subject data, instrument data, and environmental data as metadata for each of the multiple images; providing a second graphical user interface that requires a user to form a bounding box around the face of the subject in each of the multiple images; providing a third graphical user interface that requires an annotator to provide annotations for each of the multiple images; storing the multiple images as a dataset; and training or evaluating the machine learning or artificial intelligence system by using the diverse dataset in the machine learning or artificial intelligence system.

[0010] An embodiment of the present invention provides a computer-implemented method for constructing a dataset of human images, comprising: collecting multiple images from multiple diverse people; providing a first graphical user interface, requiring a user to provide subject data, instrument data, and environmental data as metadata for each of the multiple images; providing a second graphical user interface, requiring a user to form a bounding box around the face of the subject in each of the multiple images; providing a third graphical user interface, requiring an annotator to provide annotations for each of the multiple images; and storing the multiple images as a dataset, wherein the subject data includes demographic information, physical characteristics, movements, and head postures.

[0011] These and other features, aspects and advantages of the present invention will become better understood with reference to the following drawings, description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Some embodiments of the present invention are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0013] Figure 1 illustrates a graphical representation of diversity provided in an image dataset by systems and methods according to exemplary embodiments of the present invention; and

[0014] Figure 2 A functional block diagram of a computer hardware platform that may be used to implement a specifically configured computing device that may create and use a diverse human image dataset is provided.

[0015] Now, by turning to the following detailed description describing the illustrated embodiments, the present invention and its various embodiments can be better understood. It should be clearly understood that the illustrated embodiments are set forth only as examples and are not limiting of the invention ultimately defined in the claims. DETAILED DESCRIPTION

[0016] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, the term "and / or" includes any and all combinations of one or more associated listed items. As used herein, the singular forms "a", "an" and "the" are intended to include plural forms as well as singular forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprises" and / or "comprising" are used in this specification, they specify the presence of the features, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components and / or groups thereof.

[0017] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention belongs. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as an idealized or overly formal meaning unless so explicitly defined herein.

[0018] In describing the present invention, it will be appreciated that many techniques and steps are disclosed. Each of these techniques and steps has its own benefits, and each may also be used in combination with one or more, or in some cases, all of the other disclosed techniques. Therefore, for clarity, this description will avoid repeating each possible combination of the various steps in an unnecessary manner. However, this specification and claims should be read with the understanding that such combinations are fully within the scope of the present invention and claims.

[0019] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it is apparent to one skilled in the art that the present invention can be practiced without these specific details.

[0020] The present disclosure is considered to be illustrative of the invention and is not intended to limit the invention to the specific embodiments illustrated in the following figures or description.

[0021] A "computer" or "computing device" may refer to one or more devices and / or one or more systems that are capable of accepting structured input, processing the structured input according to specified rules, and generating processed results as output. Examples of computers or computing devices may include: computers; fixed and / or portable computers; computers with a single processor, multiple processors, or multi-core processors that can operate in parallel and / or non-parallel; supercomputers; mainframes; supermicrocomputers; minicomputers; workstations; microcomputers; servers; clients; interactive televisions; web appliances; telecommunications equipment with Internet access; hybrid combinations of computers and interactive televisions; portable computers; tablet personal computers (PCs); personal digital assistants (PDAs); portable telephones; special-purpose hardware for emulating computers and / or software, such as, for example, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), one chip, multiple chips, systems on a chip, or chipsets; data acquisition equipment; optical computers; quantum computers; biometric computers; and generally devices that can accept data, process data according to one or more stored software programs, generate results, and generally include input, output, storage, arithmetic, logic, and control units.

[0022] "Software" or "application" may refer to specified rules for operating a computer. Examples of software or applications may include code segments in one or more computer-readable languages; graphical and / or textual instructions; applets; precompiled code; interpreted code; compiled code; and computer programs.

[0023] These computer program instructions may also be stored in a computer-readable medium, which may direct a computer, other programmable data processing apparatus, or other device to operate in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture, which includes instructions for implementing the functions / actions specified in the flowchart and / or one or more block diagrams.

[0024] In addition, although the processing steps, method steps, algorithms, etc. may be described in a sequential order, such processes, methods, and algorithms may be configured to work in an alternate order. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement to perform the steps in that order. The steps of the processes described herein may be performed in any practical order. In addition, some steps may be performed simultaneously.

[0025] It is apparent that the various methods and algorithms described herein can be implemented by, for example, appropriately programmed general-purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) will receive instructions from a memory or similar device and execute those instructions, thereby performing the processing defined by those instructions. In addition, a variety of known media can be used to store and transmit programs that implement these methods and algorithms.

[0026] As used herein, the term "computer-readable medium" refers to any medium that participates in providing data (e.g., instructions) that can be read by a computer, a processor, or a similar device. Such a medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other permanent memories. Volatile media include dynamic random access memory (DRAM), which usually constitutes a main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including wires that constitute a system bus coupled to a processor. Transmission media can include or transmit sound waves, light waves, and electromagnetic radiation, such as sound waves, light waves, and electromagnetic radiation generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAMs, PROMs, EPROMs, FLASH EEPROMs, or any other memory chips or boxes, carriers as described below, or any other media from which a computer can read.

[0027] Various forms of computer-readable media may be involved in carrying the sequence of instructions to the processor. For example, the sequence of instructions (i) may be transferred from a RAM to the processor, (ii) may be carried over a wireless transmission medium, and / or (iii) may be formatted according to a variety of formats, standards, or protocols, such as Bluetooth, TDMA, CDMA, 3G, 4G, 5G, etc.

[0028] Embodiments of the present invention may include apparatus for performing the operations disclosed herein. The apparatus may be specially constructed for the desired purposes, or the apparatus may include a device selectively activated or reconfigured by a program stored in the device.

[0029] Unless otherwise specifically noted, and as will be apparent from the following description and claims, it should be recognized that throughout the specification description, terms such as "processing," "computing," "calculating," "determining," etc. are utilized to refer to the actions and / or processes of a computer computing system or similar electronic computing device that manipulate and / or transform data represented as physical (e.g., electronic) quantities within registers and / or memories of the computing system into other data similarly represented as physical quantities within the computing system's memories, registers, or other such information storage devices, transmission or display devices.

[0030] In a similar manner, the term "processor" may refer to any device or portion of a device that processes electronic data in registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory or can be transmitted to an external device to cause a physical change or actuation of the external device.

[0031] As is well known to those skilled in the art, many careful considerations and compromises must generally be made when designing the optimal configuration for a commercial implementation of any method or system, particularly embodiments of the present invention. Commercial implementations in accordance with the spirit and teachings of the present invention may be configured as needed for a particular application, whereby those skilled in the art may use their average skills and known techniques to appropriately omit, include, adjust, mix and match, or improve and / or optimize any aspect(s), feature(s), function(s), result(s), component(s), scheme(s), or step(s) of the teachings associated with any described embodiment of the present invention to achieve a desired implementation that addresses the needs of a particular application.

[0032] In a broad sense, embodiments of the present invention provide a dataset of diverse human images that can be created by collecting multiple images from multiple diverse people. A first graphical user interface requires a user to provide subject data, instrument data, and environmental data as metadata for each of the multiple images. A second graphical user interface requires a user to form a bounding box around the face of the subject in each of the multiple images. A third graphical user interface requires an annotator to provide annotations for each of the multiple images. The dataset can be used to train or evaluate machine learning or artificial intelligence systems, such as systems for body and face detection, body and face landmark detection, body and face parsing, face alignment, face recognition, face verification, image editing, and image synthesis.

[0033] Figure 1The figure illustrates how image diversity is obtained through subject data, instrument data, and environmental data. Subject data can include demographic data, body characteristics, movements, and head pose. Environmental data can include lighting, camera distance, scene, and camera position. Details of these image diversity factors are provided below.

[0034] In one embodiment, a computer system collects, stores and manages an image data set including data representing multiple images. In one embodiment, the image is an image of a person. The data set includes or indicates various metadata and data associated with the image data. When collecting data and accessing data, the computer system receives or assigns metadata values ​​to the image data. In some cases, the data is collected or derived directly from the user and from the subject in the image by the system or a part thereof or a related system. In other cases, some or all of the data is collected and provided to the system by other systems and parties (e.g., suppliers). Some embodiments are provided as computer systems, including components and subsystems for data processing, storage, access, exchange, input and output to provide the operations and functionality described herein, while other embodiments may include a combination of local and distributed systems, components, hardware and software. Various embodiments and examples are described here, which illustrate various aspects of these image data set collections and collection and management systems. Not every embodiment necessarily includes every specification, feature, operation or part thereof as described herein.

[0035] In one embodiment, the image dataset provides a generic, consistent, human-centric image dataset of the human body. In one such embodiment, the images must include the head. The obtained dataset can be used to perform various human-centric tasks in unconstrained settings, for example, for training and evaluating commercial machine learning and artificial intelligence systems.

[0036] In an unconstrained setting, it is impossible to control factors related to the subject (e.g., demographic information), the instrument (e.g., camera hardware and software), and the environment (e.g., lighting, camera distance). Therefore, one goal is to collect a dataset of human-centric images that is highly diverse with respect to these three factors, where all annotated people have given written informed consent. This consent can be managed as data associated with the dataset or the images.

[0037] agree

[0038] In one embodiment, consent data for each person ("subject") depicted in or otherwise identifiable by the dataset is collected and stored. The consent data may represent the person's explicit informed consent. In one example, each subject consents to have their biometric information used for research and development of commercial ML / AI systems. In another example, an electronic signature may be used to record informed consent. One way electronic signatures may be allowed in some jurisdictions is to use a secure system for electronic or digital signing that provides an encrypted, recognizable "signature". When obtained correctly, an electronic signature may be considered an "original" for record keeping. An electronic signature may include a "digital" signature on a consent document, such as using a commercial tool such as "DocuSign" or "Adobe Sign", or using an "I consent to the processing of my data" button or checkbox in an online form. In one embodiment, consent is separate from any general terms and conditions for additional user information or agreements. All subjects are provided with a version of the consent form that they can keep for their records, either in hard copy or electronic form.

[0039] In one embodiment, subject characteristics (such as age) are factors used when collecting and managing data. In some embodiments, additional consent is used when collecting images of any subject under the age of 18 ("child"). In other embodiments, data is not collected from any subject who has not reached the legal age of majority in their country of residence. In another example, additional consent is used for any subject 18 years of age or older whose ability to protect themselves from neglect, abuse, or violence is severely impaired due to disability, illness, or other reasons ("vulnerable adult"). The additional consent requirement means that parents or guardians need to provide written informed consent so that the child or vulnerable adult under their legal guardianship appears in the data set and agree that the biometric information of the child or vulnerable adult is used for research and development of commercial ML / AI systems. These consent conditions and consent details are used in the collection of images and associated data.

[0040] In some embodiments, in addition to any steps the data provider would normally take to verify the data subject's English proficiency, the data provider may randomly select three multiple-choice questions to present to each data subject before they begin working on the project. To reduce the chance that correct answers are shared among data subjects, the provider should randomly select three questions from a pool of questions to ask each data subject. In some embodiments, in order for a data subject to qualify for the project, the data subject must answer at least two of the three questions correctly.

[0041] In some embodiments, the data provider must verify the English proficiency of all data subjects (including but not limited to image subjects and data annotators).

[0042] Example Use Case

[0043] Embodiments may use image datasets to provide a variety of functionalities, including, but not limited to, one or more of the following use cases or applications: (1) body and face detection, including the task of locating a human body and / or face within a bounding box in an image; (2) body and face landmark detection, including identifying the geometry of a human body and face, i.e., locating several key points (e.g., inside right eye, nose, right hip, index of left foot); (3) body and face parsing, including the process of partitioning a human body and face into multiple segments (sets of pixels called image objects), i.e., assigning a label to each pixel in an image such that pixels with the same label share certain characteristics (e.g., left eye, left arm, right hip); (4) face alignment, including the task of obtaining a normalized translation, scale, and rotation representation of a face whose geometry has been identified; (5) face recognition, including one-to-many matching of a query image with the most similar images in a given repository; (6) face verification, including one-to-one confirmation of a query image with a given image; (7) image editing, including semantic editing of image properties, e.g., manipulation of viewpoint and lighting, as well as higher-level abstractions such as, but not limited to, manipulation of the face of a depicted subject such that its identity, ancestry, and / or age is changed; and (8) image synthesis, including the synthesis of photo-realistic person-centric portraits (e.g., generating novel facial images of non-existent subjects).

[0044] Data Collection

[0045] Implementations can use various data collection specifications and guidelines to control how data is collected, stored, and managed. In one example, the system adjusts the data or adjusts the collection to meet or approach the selected specifications. The following is an example of an implementation for managing data collection and several variations.

[0046] In this example, the "primary subject" refers to the person who is the "focus" of the image. The primary subject can be designated as anyone in the image who has provided informed consent. All other "consenting" people in the image (i.e., anyone else who has provided informed consent) are designated as "secondary subjects."

[0047] Suppliers may not collect data from residents of designated locations, including, for example, California, Illinois, Washington, Texas, and China to the extent Supplier's business operations / employees / subcontractors are located or established in China and data is collected, accessed, or otherwise processed on servers located in China. In some embodiments, data submitted from China and Russia is required to be marked in the delivery document.

[0048] Of the total number of images to be delivered, no more than 20% of the images shall contain inter-subject interaction annotations, i.e., where the primary subject is annotated as interacting with a secondary subject. In particular, no more than 20% of the images shall contain annotations relating to more than a single subject, where the maximum number of subjects that can be annotated in an image is two. Thus, for any one image, there will be informed consent from at most two subjects.

[0049] In one embodiment, the dataset is a collection of 1,000,000 images of 100,000-250,000 unique principal subjects (with exactly 4-10 images per principal subject) and all agreed-upon annotations. In another example, the dataset is a collection of 10,000 images of 1,000-2,500 unique principal subjects (with exactly 4-10 images per principal subject) and all agreed-upon annotations.

[0050] In one embodiment, the consent data reflects each principal's written consent not to submit any images containing themselves to multiple entities or systems. That is, they may only contribute to one data collection effort for a single vendor.

[0051] In one embodiment, the collection and data of the dataset reflect wage requirements and information. In one example, the collection provider pays at least the legal minimum wage per hour worked to the primary image subject and annotator. The minimum wage should be determined based on the country / region where the image subject / annotator is located. In countries / regions without a legal minimum wage, the provider must propose a rate in advance for approval.

[0052] The data in the dataset reflects compliance with the requirements, such as the required salary and the salary paid. In some embodiments, the vendor must submit a method report as part of the deliverables, documenting recruitment, compensation, and task details. Recruitment details include, for example, how image subjects / annotators are recruited (e.g., social media advertising, on-site recruitment). For social media / online / print media recruitment advertising, the text and images that will be used in the advertisement are required, as well as the name of the platform / publication on / in which it will be published. Compensation details include, for example, how much image subjects / annotators located in each country are paid per hour, how many hours they work to get paid, how these numbers are calculated, when and how image subjects / annotators are informed of the hourly rate or total compensation, how they are paid (e.g., bank transfer, electronic payment), and any fees incurred by the vendor to image subjects / annotators as part of the payment process (e.g., currency conversion fees, bank transfer fees). Task details include, for example, how quality assurance (QA) is performed, for example, for collection, review X images for each person; for annotation, review the first Y images annotated by a unique individual, and then, assuming that Y images meet a given quality threshold, review Z% of all future images.

[0053] Additional specifications and objectives are discussed below for each category.

[0054] Imaging

[0055] In an unconstrained setting, factors related to the subject, the instrument, and the environment are not controlled. Therefore, when collecting data, it is valuable to maximize diversity along these dimensions to obtain a dataset that represents the variability that exists in real-world environments. Various conditions and combinations of conditions can be used to control the dataset. In one example, images taken of the main subject are collected over a wide time span, or the images are collected over a time span as wide as possible in the configuration. In one example, at least 50% of the images (per main subject) should be captured at least seven days apart. In another example, each image (per main subject) must be captured at least one day apart. In one example, when there are less than seven days between images, the main subject must wear different clothes in each image, and each image must be taken in a different location and at a different time of day. To help facilitate the collection process, in one example, submissions of previously captured images are also used, that is, images captured before the start of the data collection process initiated by the vendor. In one example, images will only be used for collection if they meet all requirements outlined for the system and collection. For example, but not limited to, the requirement that the capture device used is digital and released in 2011 or later is met.

[0056] Furthermore, the 4-10 images per main subject must be as diverse as possible in terms of head and body pose, subject-object interaction, subject-subject interaction, time of capture, geographic location, scene, lighting and weather conditions, camera position, distance to the camera, and feature point occlusion.

[0057] In one example, the image of the primary subject must be the actual primary subject. For example, but not limited to, images of drawings, paintings, and / or reflections of the primary subject will be rejected. In addition, while, for example, glass constitutes an occlusion, imaging of the subject through glass (e.g., a window) is allowed if other requirements are met, such as, but not limited to, requirements related to feature point annotability. It should be noted that these requirements also extend to requiring images of at least two subjects (primary and secondary).

[0058] In another example, images should not contain any objectionable content, such as, but not limited to, graphic violence, nudity, explicit sexual acts and / or themes, cruelty, and / or obscene gestures.

[0059] Images should avoid containing trademarks and / or features as much as possible (e.g., images of a subject wearing a logo). It is especially important to try to avoid collecting images where any trademarks and / or features are very prominent.

[0060] Human annotators and annotations

[0061] In one embodiment, human annotators are used to estimate the characteristics of the subject and / or image. In one embodiment, the annotators must be demographically diverse. In addition, each annotator must be associated with a unique identification number ("annotator identifier") so that the system can match the tasks they complete with demographic information. The identifier should be different from the identifier used internally by the vendor.

[0062] Human annotator pre-screening survey and consent

[0063] In one embodiment, the system uses characteristics and information about human annotators to manage data and collection. Various combinations of information can be used to control the selection and participation of annotators and the review of annotator data. The following are a few examples.

[0064] In some embodiments, all human annotators complete a pre-screening survey to determine their demographic information (i.e., age, pronouns, nationality, and ancestry). Furthermore, as part of the pre-screening survey, each annotator will be asked to provide explicit written informed consent for the processing of their personal information (i.e., age, pronouns, nationality, and ancestry). All demographic questions are mandatory and must be self-reported by each annotator.

[0065] In some embodiments, the exact age of each annotator is required and must be self-reported. The annotator's self-reported age must correspond to their age at the time they provided explicit informed consent.

[0066] In some embodiments, each annotator's pronouns are required and must be self-reported. The annotator's self-reported pronouns must correspond to their pronouns when providing explicit informed consent. If an annotator selects "None of the Above" or "Prefer Not to Say," they are not allowed to select another option.

[0067] In some embodiments, the nationality of each annotator is required and must be self-reported. The annotator's self-reported nationality must correspond to their nationality when providing explicit informed consent. Multiple nationality selections are allowed.

[0068] In some embodiments, each annotator's country of residence is required and must be self-reported. An annotator's self-reported country of residence must correspond to their country of residence when providing explicit informed consent. Multiple country of residence selections are allowed.

[0069] In some embodiments, the ancestry of each annotator is required and must be self-reported. The annotator's self-reported ancestry must correspond to their ancestry when providing explicit informed consent. To obtain this information, for example, the system asks the annotator the following question: "Where are your ancestors (e.g., great-grandparents) from?". In addition, the system can also provide examples for the annotator to highlight the potential differences between their nationality and ancestry, such as: "For example, your nationality may be 'American', but your ancestors (e.g., great-grandparents) may be from 'Europe' ('Southern Europe') and 'Africa' ('West Africa' and 'Central Africa')". That is, the system ensures that the annotator understands that ancestry does not necessarily correspond to the region and sub-region of their nationality. It should be noted that multiple region and sub-region selections are allowed. The annotator must first select the region(s) and sub-region(s) that best describe their ancestry. For example, "Africa ("North Africa") and "Oceania" ("Polynesia" and "Melanesia"). Also, it should be noted that only region-level ancestry is required, while sub-region-level ancestry is optional. This is to allow for annotators who, for some reason, do not know their sub-region-level ancestry. Nonetheless, annotators should be encouraged to provide this information if they are able to do so. If an annotator selects a region but does not select any sub-regions within that region, the annotator will contribute to the counts for each sub-region within that region. For example, consider an annotator who selects regions "Americas" and "Asia" and sub-regions "Central Asia" and "East Asia". In this scenario, the annotator contributes to the four sub-regions in "Americas" and the selected two sub-regions in "Asia". Therefore, 1 / 6 is added to each sub-region count in "Caribbean", "Central America", "South America", "North America", "Central Asia", and "East Asia".

[0070] Human annotator diversity

[0071] In one embodiment, the system uses criteria for annotator demographics. Various combinations of information can be used to control the selection and participation of annotators, as well as the review of annotator data. The following are a few examples.

[0072] The vendor uses a demographically diverse collection of annotators. Demographic diversity only concerns age, pronoun, and ancestry categories, defined as follows: (1) Age group: 18-30, 31-45, 46+; (2) Pronouns: she / her / hers, he / him / his; and (3) Ancestry: African, American, Asian, European, Oceanian.

[0073] There are a total of 30 demographic groups based on the age, pronoun, and ancestry categories defined above. For any particular annotation (e.g., bounding box annotation), each of the 30 demographic groups must perform at least 0.5% of the total number of annotations.

[0074] If a vendor has multiple annotation stages, including but not limited to multiple quality assurance stages, then the annotators at each stage are required to be demographically diverse, where diversity relates to the definition given above.

[0075] "Raw" Note

[0076] In one embodiment, for any annotation task requiring more than one annotator, the system uses "raw" annotations (i.e., individual annotations of each annotator) rather than aggregated annotations to model uncertainty. In one example, all annotations are stored as JSON files along with the images to which they belong.

[0077] It should be noted that the “raw” comments include comments performed before quality assurance as well as comments from each stage of quality assurance.

[0078] Percentage distribution range

[0079] In one embodiment, unless otherwise noted, the percentage distribution ranges (such as the percentage distribution ranges used throughout this document) are related to the primary subject of the image. In addition, all self-reported annotations from voluntary subjects (per image) are obtained directly from the subject themselves. The system avoids or limits annotating any person in the image who has not provided explicit written informed consent, such as image-based consent data. In one example, any image found to have any annotations related to a person who has not provided explicit written informed consent will be rejected by the system.

[0080] Submission Date

[0081] In some embodiments, when the image subject submits data to the data provider, the data provider must record the date and time that the image subject submits the data. In this case, the submission date and time of the delivery must be associated with the data submitted by the image subject.

[0082] When the image subject provides their consent to the data provider, the data provider must record the date and time the image subject provided the consent. In this case, the consent submission date and time delivered must be associated with each data point submitted by the image subject.

[0083] When an annotator provides their consent to a data provider, the data provider must record the date and time the image subject provided the consent. In this case, the consent submission date and time delivered must be associated with each annotation performed by the annotator.

[0084] main body

[0085] Various embodiments use various operations and data to collect, store and manage images, image data, and data corresponding to the images. For example, in one embodiment, each volunteer subject is associated with a unique identification number or subject identifier. The following are various additional embodiments and examples regarding subject data.

[0086] Involuntary Personnel Notes

[0087] In one embodiment, if at least one non-consensual person is visible in the scene of the image, then the system annotates the image as containing a non-consensual person, regardless of its spatial dimensions. This annotation can be initially performed by the primary subject or image creator, but must be verified as correct by the vendor. The system avoids annotating any non-consensual person unless explicitly requested and authorized, such as by an approval code associated with the image.

[0088] In some embodiments, in order for an image containing an involuntary subject to be accepted, the image must meet three requirements: (1) the involuntary subject must not be in front of (or obscure) the primary and / or secondary subject; (2) the involuntary subjects should not collectively cover more than 20% of the image; and (3) for each primary / secondary subject, the total overlap of the involuntary subject with the primary / secondary subject should not be greater than 15% of the primary / secondary subject outline. It should be noted that in some embodiments, all three requirements must be met simultaneously.

[0089] Face Bounding Box Annotation

[0090] In one embodiment, the system uses bounding boxes in the images. In one example, all volunteer subjects (per image) must have a bounding box around their faces (provided as a set of coordinates defining a rectangle), where the bounding box must tightly contain the forehead, chin, and cheeks. Each bounding box must be annotated (drawn) by a human annotator and associated with a unique identifier for each subject. In one example, the use of automated tools as part of this process is documented by the data and may require additional authorization data to be used or included. The system may provide a graphical user interface for the annotator to provide bounding boxes to the faces of volunteer subjects in the images.

[0091] Segmentation Mask Annotation

[0092] In one embodiment, the system uses segmentation to manage or label images, such as identifying body parts. In one example, the entire body (per image) of each volunteer subject must be segmented based on the following 28 segmentation categories: facial skin (including ears), upper body (from above the neck to the hip line), left arm skin (including the skin of the left hand), right arm skin (including the skin of the right hand), left leg skin (including the left foot), right leg skin (including the right foot), hair, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, inside of the mouth, left shoe, right shoe, headwear (e.g., headphones, hat, helmet, scarf, bandana, hairnet, sports / headband, turban, hijab, hood, veil, wig, crown, kippah, headnet, earmuffs), mask (e.g., medical mask), masks, face shields, military masks, respirators, costume / theatrical masks, sports masks, ceremonial masks, balaclavas), eyewear (e.g., sunglasses, glasses, goggles, VR headsets), upper body clothing (e.g., tops, t-shirts, sweatshirts, shirts, blouses, sweaters, cardigans, vests, capes, jackets, coats), lower body clothing (e.g., pants / trousers, shorts, skirts, leggings, stockings), full body clothing (e.g., dresses, jumpsuits), socks or leggings, neckwear (e.g., scarves, ties), bags, gloves, and jewelry or timepieces (e.g., watches, earrings, nose rings, eyebrow rings, lip rings, rings, bracelets, rings (finger), necklaces, watches).

[0093] In one example, annotations must be estimated (drawn) by a human annotator, and the relevant segmentation of each subject (each image) must be associated with its unique identifier. In some embodiments, a graphical user interface is provided to the annotator to add appropriate annotations on the subject, including the associated unique identifier. In one example, the use of automated tools as part of this process is recorded by the data, and additional authorization data may be required to be used or included. Each segmentation category must be provided as a set of coordinates defining a polygon. If the subject is occluded by movable non-environmental objects (e.g., cups, newspapers, food, hockey sticks, balls) or immovable environmental objects / structures (e.g., doors, trees, fences, cars, water), then segmentation masks should be drawn around these objects, which may result in disconnected segmentation mask areas and polygons.

[0094] Pose feature point annotation

[0095] In one embodiment, the system manages images using landmarks. In one example, facial and body landmarks of each volunteer subject (per image) must be annotated based on the following landmarks: nose, right eye inside, right eye, right eye outside, left eye inside, left eye, left eye outside, right ear, left ear, mouth right, mouth left, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist, right little finger joint, left little finger joint, right index finger joint, left index finger joint, right thumb joint, left thumb joint, right hip, left hip, right knee, left knee, right ankle, left ankle, right heel, left heel, right index finger, and left index finger. The annotations must be estimated by a human annotator and the landmarks of each subject (per image) must be associated with their unique identifier. It should be noted that each type of landmark must be annotable in at least 10% of the delivered images.

[0096] In some embodiments, if a feature point is occluded by an extraneous accessory, structure, object, or person, then it should not be annotated. Extraneous refers to accessories, structures, objects, and people that are not part of or worn by the annotated subject. However, if a feature point is occluded by the subject's own body clothing or accessories (e.g., dress, T-shirt, pants), then it can be annotated as long as the location of the feature point is not ambiguous.

[0097] In cases where the thumb, pinky, and index finger of the left and right hands are not visible (e.g., because the subject is wearing gloves), the annotator can assume and annotate the locations of these feature points. In cases where the index finger and heel of the left and right feet are not visible (e.g., because the subject is wearing shoes), the annotator can assume and annotate the locations of these feature points.

[0098] Subject-Demographic Information

[0099] In various embodiments, the system uses demographic information for subjects. In one example, the system collects data from various voluntary subjects, specifically by focusing on their demographics (sensitive attributes). Subjects always share one or more characteristics (e.g., age, pronouns, ancestry, nationality), thereby forming demographic groups. For example, using a combination of factors for age, pronouns, and ancestry categories, a large number of demographic groups can be formed.

[0100] In some embodiments, there are a total of 264 cross-sectional demographic subgroups based on age, pronoun, and ancestry categories, defined as follows: Age Group: 18-29, 30-39, 40-49, 50-59, 60-69, 70+; Pronouns: she / her / hers, he / him / his; and Ancestry: North Africa, East Africa, Central Africa, South Africa, West Africa, Caribbean, Central America, South America, North America, Central Asia, East Asia, Southeast Asia, South Asia, Western Asia, Eastern Europe (including North Asia), Northern Europe (including Channel Islands), Southern Europe, Western Europe, Australia and New Zealand, Melanesia, Micronesia, Polynesia.

[0101] The main subject in each demographic group appears in at least 0.1% of the images, relative to the total number of images to be delivered. While this represents a minimum representation, for example, vendors should aim to collect images from subjects such that each group is roughly equally represented in the final dataset.

[0102] Additionally, to avoid spurious correlations, the system avoids collecting images of any one group under the same conditions, such as, but not limited to, collecting images of one demographic group only indoors.

[0103] Subject-Demographics-Age

[0104] In one embodiment, the system uses age as demographic information. In one example, the system collects data from voluntary subjects of varying ages. The system can use the following rough age group (in years) categories and the desired percentage distribution range (in square brackets, e.g., [min-max %]): 18-29 [2%-], 30-39 [2%-], 40-49 [2%-], 50-59 [2%-], 60-69 [2%-], and 70-[2%-].

[0105] In one example, the exact age (in years) of each imaged subject (per image) is required and must be self-reported. The subject's self-reported age must correspond to the age at the time of image capture. Additionally, each subject's facial bounding box (per image) must be associated with the age at the time of image capture.

[0106] Subject - Demographic Information - Pronouns

[0107] In some embodiments, data can be collected from voluntary subjects whose self-identified pronouns vary. The system defines the following pronoun categories and the required percentage distribution ranges (in square brackets, e.g., [min-max %]): she / her / hers [45%-], he / him / his [45%-], they / them / theirs [0%-], gender neutral (Ze / zir / zirs) [0%-], none of the above [0%-], and prefer not to say [0%-].

[0108] In some embodiments, the pronouns of the subject of each image are required and must be self-reported. The subject's self-reported pronouns must correspond to the pronouns at the time of image capture (per image). In addition, each subject's facial bounding box (per image) must be associated with their pronouns. Multiple pronoun selections are allowed per image, unless the subject selects "None of the Above" or "Prefer Not to Say." If the subject selects "None of the Above" or "Prefer Not to Say," they are not allowed to select another option.

[0109] Subject-Demographics-Nationality

[0110] In one embodiment, the system uses nationality as demographic information. In one example, the system collects data from voluntary subjects of varying nationalities.

[0111] In one example, each subject's nationality is required and must be self-reported. The subject's self-reported nationality must correspond to their nationality at the time of image capture (per image). Multiple nationality selections are allowed. Additionally, each subject's facial bounding box (per image) must be associated with their nationality.

[0112] In some embodiments, the supplier may provide the subject with a list of nationalities from which to choose. In some embodiments, the list of nationalities may be selected from a government list, such as a list provided by the UK government. An "unlisted" option should also be listed, which allows free text entry of an alternative nationality.

[0113] Subject - Demographic Information - Country of Residence

[0114] In one embodiment, the system uses country of residence as demographic information. In one example, the system collects data from voluntary subjects with different countries of residence.

[0115] In one example, each subject's country of residence is required and must be self-reported. The subject's self-reported country of residence must correspond to the country of residence at the time of image capture (for each image). Multiple country of residence selections are allowed. In some embodiments, country of residence does not need to be collected for each image.

[0116] In some embodiments, the supplier may provide the subject with a list of countries to choose from. In some embodiments, the list of countries may be selected from a government list (e.g., a list provided by the UK government). An "unlisted" option should also be listed, which allows for free text entry of optional countries.

[0117] Subject-Demographics-Ancestry

[0118] In one embodiment, the system uses ancestry as demographic information. In one example, the system collects data from voluntary subjects of varying ancestry.

[0119] In some embodiments, the ancestry can be based on the United Nations Geographical Scheme system. The Geographical Scheme was designed by the United Nations Statistics Division based on the M49 coding classification. It should be noted that Antarctica is excluded because it does not have any sub-regions or country-level regions.

[0120] In some embodiments, the system defines the following ancestry categories and subcategories based on a geographic scheme and the required percentage distribution ranges (in square brackets, e.g., [minimum-maximum %]): Africa, with subcategories of North Africa [1%-], East Africa [1%-], Central Africa [1%-], South Africa [1%-], and West Africa [1%-]; Americas, with subcategories of Caribbean [1%-], Central America [1%-], South America [1%-], and North America [1%-]; Asia, with subcategories of The subcategories are Central Asia [1%-], East Asia [1%-], Southeast Asia [1%-], South Asia [1%-] and West Asia [1%-]; Europe, whose subcategories include Eastern Europe (including North Asia) [1%-], Northern Europe (including Channel Islands) [1%-], Southern Europe [1%-] and Western Europe [1%-]; Oceania, whose subcategories include Australia and New Zealand [1%-], Melanesia [1%-], Micronesia [1%-] and Polynesia [1%-].

[0121] In one example, each subject's ancestry is required and must be self-reported. For example, the system asks the subject the following question: "Where are your ancestors (e.g., great-grandparents) from?". In addition, the system provides the subject with an example that highlights the potential difference between their nationality and ancestry, such as: "For example, your nationality may be 'American', but your ancestors (e.g., great-grandparents) may be from 'Europe' ('Southern Europe') and 'Africa' ('West Africa' and 'Central Africa')". That is, the system ensures that the subject understands that ancestry does not necessarily correspond to the region and sub-region of their nationality. In addition, each subject's facial bounding box (per image) must be associated with their ancestry. It should be noted that multiple region and sub-region selections are allowed. The subject must first select the region (one or more) and sub-region (one or more) that best describes their ancestry, such as "Africa" ​​("North Africa") and "Oceania" ("Polynesia" and "Melanesia"). Moreover, it should be noted that only region-level ancestry is required, while sub-region-level ancestry is optional. This is to allow the subject to not know their sub-region-level ancestry for some reason. Nonetheless, if subjects are able to provide this information, they should be encouraged to do so. If an annotator selects a region but does not select any subregions within that region, then the annotator contributes to the counts for each subregion within that region. For example, consider an annotator who selects the regions "Americas" and "Asia", and the subregions "Central Asia" and "East Asia". In this case, the annotator contributes to the four subregions in "Americas" and the two selected subregions in "Asia". Therefore, 1 / 6 is added to each of the subregion counts in "Caribbean", "Central America", "South America", "North America", "Central Asia", and "East Asia".

[0122] Subject - Demographic Information - Disability

[0123] In one embodiment, the system uses disability information as demographic information. In one example, the system collects data from voluntary subjects who vary in the types of difficulties they face. This is to ensure that subjects with disabilities are included. Based on the American Community Survey, the system defines the following disability categories and subcategories: Hearing difficulties - deafness or severe hearing difficulties; Vision difficulties - blindness or severe vision difficulties, even with glasses; Cognitive difficulties - difficulty remembering, concentrating, or making decisions due to physical, mental, or emotional problems; Walking difficulties - severe difficulty walking or climbing stairs; Self-care difficulties - difficulty bathing or dressing; Independent living difficulties - difficulty doing things on your own, such as going to the doctor or shopping, due to physical, mental, or emotional problems; or unwillingness to speak.

[0124] In one example, each subject's disability category must be self-reported; however, this is optional, i.e., the subject can choose whether to disclose this information. Multiple "disability" category selections are allowed. The subject's self-reported disability must correspond to the disability at the time of image capture (per image). Additionally, each subject's facial bounding box (per image) must be associated with their disability category.

[0125] In some embodiments, to obtain a response from the participant, the system may ask them: "Do you have any disabilities / difficulties?" It should be noted that if the subject selects "None of the above" or "Prefer not to say", they are not allowed to select another option.

[0126] Subject-Physical Characteristics

[0127] In various embodiments, the system uses information about the subject's physical characteristics. In one example, the system collects data from voluntary subjects with various physical characteristics. Population diversity results in a variety of subjects with different physical characteristics.

[0128] Subject - Physical Features - Skin Color

[0129] In one embodiment, the system uses skin color as physical characteristic information. In one example, the system collects data from voluntary subjects with various skin colors. In some embodiments, the system defines the following six skin color categories based on six red-green-blue [R, G, B] colors, such as [102, 78, 65], [136, 105, 81], [164, 131, 103], [175, 148, 120], [189, 163, 137], and [198, 180, 157].

[0130] The system can obtain from each subject (once) their self-reported "natural" skin color and associate this information with their unique subject identifier. The subject must select one of the six categories that best matches their natural skin color. In addition, if the system detects that the subject's skin color differs from their natural skin color when the image was captured, the subject can self-report their apparent skin color (per image) and associate it with a facial bounding box.

[0131] Of course, while six exemplary skin tones are described herein, additional skin tones may be provided to provide additional choices for a subject's self-reported skin tone.

[0132] Subject - Physical Characteristics - Eye Color

[0133] In one embodiment, the system uses eye color as physical characteristic information. In one example, the system collects data from volunteer subjects with various eye colors. The system defines the following eye color categories: none (no eyes), blue, gray, green, hazel, brown, red and purple, and unlisted (any hair color that does not belong to one of the above categories).

[0134] In one example, the system obtains from each subject (once) their self-reported "natural" left and right eye color and associates this information with their unique subject identifier. For clarity, subjects must report their left and right eye color separately. Multiple left and right eye color selections are allowed (e.g., "blue" and "gray" for the left eye). If the subject selects "not listed", the system offers the subject the opportunity to report their left eye color in free-form text. The free-form text box must be marked as optional.

[0135] In addition, if the subject's eye color at the time the image was captured differs from their natural eye color due to, for example, contact lenses, the system lets the subject self-report their "unnatural" left and right eye colors (per image) and associates them with the face bounding box. However, note that if one eye in the image is closed (or not visible, for example, due to sunglasses), then its color should be classified as "none". Again, the subject must report their left and right eye colors separately. For clarity, the subject's self-reported eye color must correspond to the eye color at the time of image capture (per image). Multiple left and right eye color choices are allowed (e.g., "blue" and "gray" for the left eye). If the subject selects "not listed", the system offers the subject the opportunity to report their left eye color in free-form text. The free-form text box must be marked as optional.

[0136] Subject - Body Features - Hair Type

[0137] In one embodiment, the system uses hair type as physical characteristic information. In one example, the system collects data from volunteer subjects with various hair types. The system defines the following hair type categories: none (no hair), straight (fine, medium, or thick), wavy (loose, clear, or wide waves), curly (loose curls, tight curls, or spirals), kinky (clear curls, Z curls, or tight curls), or unlisted (any hair type that does not belong to the above categories).

[0138] In one example, the system obtains from each subject (once) their self-reported "natural" hair type and associates this information with their unique subject identifier. "Natural" refers to the subject's hair type if the subject allowed their hair to grow naturally without applying any processes to it.

[0139] The system can provide a visual aid to help the subject decide which category their hair type belongs to.

[0140] Additionally, if the subject's hair type was different from their natural hair type when the image was captured, the system lets the subject self-report their "unnatural" hair type (per image) and associates it with their facial bounding box.

[0141] It should be noted that if the subject's hair is not visible at all in the image, then the hair type should be classified as "None". If the subject selects "Not Listed", the system provides the subject with the opportunity to report their hair type in free-form text. The free-form text box must be marked as optional.

[0142] Subject - Body Features - Head Hair Style

[0143] In one embodiment, the system uses head hairstyle as physical characteristic information. In one example, the system collects data from voluntary subjects with various head hairstyles. The system defines the following list of head hairstyle categories: none (no hair); crew cut; short hair, including up, half up, down, and unlisted (any head hairstyle that does not belong to one of the above subcategories) subcategories; medium length, including up, half up, down, and unlisted (any head hairstyle that does not belong to one of the above subcategories) subcategories; long hair, including up, half up, down, and unlisted (any head hairstyle that does not belong to one of the above subcategories) subcategories; and unlisted (any head hairstyle that does not belong to one of the above categories).

[0144] In one example, each subject's head hairstyle (per image) must be self-reported and associated with their facial bounding box. The subject's self-reported head hairstyle must correspond to their head hairstyle at the time of image capture (per image). If the subject selects "Not Listed", the system provides the subject with the opportunity to report their head hairstyle in free-form text. The free-form text box must be marked as optional. It should be noted that if the subject selects "Short", "Medium", or "Long", then they do not need to select a subcategory (i.e., the subcategory selection is optional).

[0145] It should also be noted that if the subject's hair is completely invisible in the image, then the head hairstyle should be classified as "None".

[0146] Subject - Physical Characteristics - Hair Color

[0147] In one embodiment, the system uses hair color as physical characteristic information. In one example, the system collects data from volunteer subjects with various hair colors. The system defines the following list of hair color categories: none (no hair), very light blonde, light blonde, blonde, dark blonde, light brown to medium brown ("chestnut"), dark brown / black ("dark brown"), red, red blonde, gray, white, and unlisted (any hair color that does not belong to one of the above categories).

[0148] In one example, the system obtains from each subject (once) their self-reported "natural" hair color and associates this information with their facial bounding box. "Natural" refers to the subject's hair color if the subject allowed their hair to grow naturally without applying any processes to it or exposing it to sunlight. So, for example, someone who is "going" gray should report their natural hair color as gray, because that color is due to naturally occurring processes (e.g., loss of color-producing melanocytes).

[0149] Allow multiple hair color selections. If the subject selects "Not Listed", the system provides the subject with the opportunity to report their hair color in free-form text. The free-form text box must be marked as optional.

[0150] Additionally, if the subject's hair color at the time the image was captured differs from their natural hair color, due to, for example but not limited to, hair dye, then the system has the subject self-report their "unnatural" hair color (per image) and associates it with their face bounding box. For clarity, the subject's self-reported hair color must correspond to their hair color at the time of image capture. Multiple hair color choices are allowed (e.g., "white" and "gray"). If the subject selects "not listed," then the system offers the subject the opportunity to report their hair color in free-form text. The free-form text box must be marked as optional.

[0151] It should be noted that if the subject's hair is completely invisible in the image, then the hair color should be classified as "None".

[0152] Subject - Physical Features - Facial Hairstyle

[0153] In one embodiment, the system uses facial hairstyle as physical characteristic information. In one example, the system collects data from volunteer subjects with various facial hairstyles. The system defines the following list of facial hairstyle categories and subcategories: none (including shaved), beard, mustache, and goatee.

[0154] In one example, each subject's facial hairstyle (per image) must be self-reported and associated with their facial bounding box. Multiple facial style selections are allowed, unless the subject selects "None (including shaved)". It should be noted that if the subject's facial hair is completely invisible in the image, then the facial hairstyle should be classified as "None".

[0155] Subject - Physical Features - Facial Hair Color

[0156] In one embodiment, the system uses facial hair color as physical characteristic information. In one example, the system collects data from voluntary subjects with various facial hair colors. The system defines the following list of facial hair color categories: none (no facial hair, including but not limited to shaved), very light blonde, light blonde, blonde, dark blonde, light brown to medium brown ("chestnut"), dark brown / black ("dark brown"), red, red blonde, gray, white, and unlisted (any facial hair color that does not belong to one of the above categories).

[0157] In one example, the system obtains from each subject (once) their self-reported "natural" facial hair color and associates this information with their facial bounding box. "Natural" refers to the subject's facial hair color if the subject allowed their facial hair to grow naturally without applying any processes to it or exposing it to sunlight. So, for example, someone whose facial hair color "turned" gray should report their natural facial hair color as gray, because the color is due to naturally occurring processes (e.g., loss of color-producing melanocytes). Multiple facial hair color selections are allowed. If the subject selects "Not Listed," the system offers the subject the opportunity to report their facial hair color in free-form text. The free-form text box must be marked as optional.

[0158] Additionally, if the subject's facial hair color at the time the image was captured was different from their natural facial hair color, due to, for example, but not limited to, hair dye or sun exposure, the system has the subject self-report their "unnatural" facial hair color (per image) and associate it with their facial bounding box. For clarity, the subject's self-reported facial hair color must correspond to the facial hair color at the time of image capture. Multiple facial hair color choices are allowed (e.g., "white" and "gray"). If the subject selects "not listed," the system offers the subject the opportunity to report their facial hair color in free-form text. The free-form text box must be marked as optional.

[0159] It should be noted that if the subject's facial hair is completely invisible in the image, then the facial hair color should be classified as "None".

[0160] Subject - Physical Characteristics - Height

[0161] In one embodiment, the system uses height as the physical characteristic information. In one example, the system collects data from volunteer subjects of different heights.

[0162] In one example, each subject's height (per image) must be self-reported and associated with their unique identifier. The subject's self-reported height must correspond to their height at the time of image capture (per image). The system allows each subject to self-report their height using their preferred measurement system (e.g., feet (inches) or meters (centimeter)), and then converts all values ​​to the SI base unit of length, i.e., meters (centimeter).

[0163] Subject - Physical Characteristics - Weight

[0164] In one embodiment, the system uses body weight as physical characteristic information.In one example, the system collects data from voluntary subjects of varying weights.

[0165] In one example, each subject's weight (per image) must be self-reported and associated with their unique identifier. The subject's self-reported weight must correspond to their weight at the time of image capture. The system allows each subject to self-report their weight using their preferred measurement system (e.g., feet (pounds) or kilograms (grams)), and then converts all values ​​to the SI base unit of mass, i.e., kilograms (grams).

[0166] Subject - Physical Features - Facial Markings

[0167] In one embodiment, the system uses facial markers as body feature information. In one example, the system collects data from subjects with varying facial markers. The system defines a list of the following facial marker categories: None (no visible facial markers), Tattoos, Birthmarks, Scars, Burns, Tumors, Makeup, Face Paint, Acne, or Unlisted (any facial marker that does not fall into the above categories).

[0168] In one example, each subject's visible facial markers (per image) must be self-reported and associated with their facial bounding box. Multiple facial marker category selections are allowed. If the subject selects "Not Listed", the system provides the subject with an opportunity to report their facial marker(s) in free-form text. The free-form text box must be marked as optional. It should be noted that the subject should be made aware that the facial markers must be visible in the image, and if not visible in the image, they should not be annotated.

[0169] Subject - Physical Characteristics - Biologically Related Subjects

[0170] In one embodiment, the system uses biological relationships as physical characteristic information. In one example, the system defines a list of the following relationship categories: biologically related (e.g., brother, sister, mother, father, uncle, grandmother, cousin) and unrelated.

[0171] In one example, each volunteer subject (each image) must be associated with identifiers of other volunteer subjects in the image who are biologically related. For clarity, a husband and wife are typically not biologically related and therefore "unrelated" in the above classification scheme.

[0172] Subject - Physical Characteristics - Pregnancy Status

[0173] In one embodiment, the system uses pregnancy status as physical characteristic information. In one example, the system collects data from voluntary subjects with different pregnancy statuses. The system defines the following pregnancy status categories: not pregnant; pregnant, including subcategories of obviously pregnant and not obviously pregnant; and unwilling to say.

[0174] In one example, each subject's pregnancy status category must be self-reported; however, this is optional, i.e., the subject can choose whether to disclose this information. Multiple category and subcategory selections are not allowed. It should be noted that if a subject selects "Pregnant," then they do not need to select any subcategories (i.e., subcategory selection is optional). The subject's self-reported pregnancy status category must correspond to their pregnancy status at the time of image capture (per image). Additionally, each subject's facial bounding box (per image) must be associated with their pregnancy status category. This question must be asked of all subjects, regardless of their demographic attributes (e.g., pronouns).

[0175] Subject-Action

[0176] In various embodiments, the system uses the subject's action information. In one example, the system collects data from subjects performing various actions, specifically, by encouraging the subject to move freely and interact with their environment (objects and other subjects) as usual.

[0177] The system considers subject body pose, subject-object interaction and subject-subject interaction and their combination. Maximizing the diversity in terms of subject body pose, subject-object interaction and subject-subject interaction will inevitably lead to, for example, subject variability, arbitrary face scales, non-uniform lighting conditions, ambient occlusion, arbitrary camera positions and orientations, wide variability in head pose and motion blur.

[0178] Subject-Action-Body Posture

[0179] In one embodiment, the system uses body posture as action information. In one example, the system collects data from voluntary subjects in various body postures. The system defines the following list of body posture categories and the required percentage distribution range (in square brackets, e.g., [minimum-maximum %]): standing [5%-], sitting [5%-], walking [5%-], bending / bow (at the waist) [5%-], lying down / sleeping [5%-], performing martial arts / combat techniques (including wrestling, boxing, etc.) [5%-], dancing [5%-], running / jogging [5%-], squatting / kneeling [5%-], standing up [5%-], jumping [5%-], falling [0%-], crawling [5%-], swimming [5%-], and not listed (any action that does not belong to one of the above categories).

[0180] In one example, each subject's body pose (per image) must be self-reported and associated with their unique identifier. If the subject selects "Not Listed", the system provides the subject with the opportunity to report their body pose in free-form text. The free-form text box must be marked as optional.

[0181] Subject-Action-Subject-Object Interaction

[0182] In one embodiment, the system uses subject-object interactions as action information. In one example, the system collects data from voluntary subjects interacting with various objects. It should be noted that another person does not constitute an object.The system defines the following list of subject-object interaction categories along with the desired percentage distribution ranges (in square brackets, e.g., [min-max %]): (subject not interacting with any object) [0.1-50%], riding (objects: e.g., bicycle, motorcycle, skateboard, scooter, horse) [0.1%-], driving (objects: e.g., car, truck) [0.1%-], watching (objects: e.g., television, sports game, theater) or reading (objects: e.g., book, magazine, newspaper, e-reader, leaflet) [0.1%-], smoking (objects: e.g., hookah, cigarette, cigar) [0%-], eating (objects: e.g., food) [0.1%-], drinking (objects: e.g., alcohol) [0.1%-], and drinking (objects: e.g., alcohol) [0.1%-]. to lift / pick up or put down (objects: e.g., chair, phone, bag, food) [0.1-], write / draw or paint (objects: e.g., letter, email) [0.1%-], catch or throw (objects: e.g., ball, keys) [0.1-], push (objects: e.g., shopping cart, car), pull (objects: e.g., bag, box, toy, duvet) or pluck (i.e., remove / take out an object from another, especially with force or effort, such as teeth, root vegetables, etc.) [0.1-], put on or take off clothes (objects: e.g., hat, pants, etc.) [0.1-], , shoes, socks, shirts, sweaters) [0.1%-], entering or exiting (objects: e.g., doors, elevators) [0.1%-], climbing (objects: e.g., stairs, hills, rock walls, ropes) [0.1%-], pointing (objects) [0.1%-], shooting (objects) [0%-], digging / shoveling with (objects: e.g., shovel, spade, fork, trowel, pickaxe) [0.1%-], playing with pets / animals [0.1%-], playing with musical instruments (e.g., violin, piano, saxophone, clarinet, drums) [0.1%-], playing with (objects: e.g., tabletop games, board games, sports) [0.1%-], using electronic devices (objects: e.g., mobile phones, cameras, video recorders) [0.1%-], computer, laptop, tablet, video game controller, headset) [0.1%-], cutting or chopping (object) [0.1%-], cooking (excluding cutting or chopping) [0.1%-], fishing [0.1%-], boating (object: e.g., boat) [0.1%-], sailing (object: e.g., boat) [0.1%-], brushing teeth [0.1%-], hitting with another object or with the hand (object: e.g., ball, wall) [0.1%-], kicking (object: e.g., ball) [0.1%-], turning (object: e.g., screwdriver, door handle) [0.1%-], and not listed (any subject-object interaction that does not fall into one of the categories defined above).

[0183] In one example, each subject's interaction with one or more objects (per image) must be self-reported and associated with its unique identifier. Multiple subject-object interaction selections are allowed. For clarity, the subject's self-reported subject-object interactions must correspond to content that is visibly visible in the image. For example, if the subject selects "cut or chop", then the subject must "cut or chop" something in the image that a third party should be able to see when first viewing the image. If the subject selects "not listed", then the system provides the subject with the opportunity to report their interactions with one or more objects in free-form text. The free-form text box must be marked as optional.

[0184] In some implementations, if the subject selects N categories, the system may add 1 / N to each category count.

[0185] Agent-Action-Agent-Agent Interaction

[0186] In one embodiment, the system uses subject-subject interactions as action information. In one example, the system collects data from voluntary subjects interacting with other voluntary subjects. The system may use restrictions on these images with respect to the total number of images to be delivered. In one example, the system requires that no more than 20% of the images contain subject-subject interaction annotations, i.e., where the primary subject is annotated as interacting with a secondary subject. In particular, no more than 20% of the images should contain annotations related to more than a single subject, where the maximum number of subjects that can be annotated in an image is two. For any one image, the system requires explicit informed consent from a maximum of two subjects. The system defines a list of subject-subject interaction categories below and the required percentage distribution range (in square brackets, e.g., [min-max %]). It should be noted that the distribution range is relative to the total number of images in which the primary subject interacts with at least one secondary subject.

[0187] The subject-subject interaction categories include the following: talking / listening / singing to (person or group) [1%-], watching / looking at (person) [1%-], grabbing (person) (e.g., martial arts, wrestling, contact sports, etc.) [1%-], hitting (person) (e.g., martial arts, wrestling, contact sports, etc.) [1%-], kicking (person) (e.g., martial arts, wrestling, contact sports, etc.) [1%-], pushing (person) (e.g., martial arts, wrestling, contact sports, etc.) [1%-], hugging / hugging (person) [1%-], giving / offering (object) to (person) or taking / receiving (object) from (person) [1%-], kissing (person) [1%-], lifting (person) [1%-], shaking hands [1%-], playing with (person or group) [1%-], and not listed (any subject-subject interaction that does not fall into one of the above defined categories).

[0188] In one example, each subject's interactions with another subject or subjects (per image) must be self-reported and associated with their unique identifier. Multiple subject-subject interaction selections are allowed. For clarity, the subject's self-reported subject-subject interactions must correspond to interactions that are clearly shown in the image. For example, if the subject selects "Handshake", then the subject must shake hands with the other subject, and this should be obvious to a third party when first viewing the image. If the subject selects "Not Listed", then the system provides the subject with the opportunity to report their interactions with the other subject or subjects in free-form text. The free-form text box must be marked as optional.

[0189] Subject-Head Pose

[0190] In one embodiment, the system uses head pose information. In one example, the system collects data from voluntary subjects with various head poses, where poses refer to turning, pitching, and shaking of the head.

[0191] The system defines the following list of coarse head posture categories: Typical [10%-], where absolute pitch is less than 30° and absolute head turn is less than 30°; Atypical [10%-], where absolute pitch is greater than 30° and / or absolute head turn is greater than 30°.

[0192] In one example, each subject's head pose (per image) must be annotated by a human annotator and labeled as belonging to one of the two categories mentioned above. The system associates the labeled head pose with the subject's unique identifier.

[0193] instrument

[0194] In one embodiment, the system uses instrument information. Instrument refers to the camera device used to capture data. In one example, the system collects data from volunteer subjects using a variety of different digital camera brands and models. Changing instruments will result in huge differences in quality, resolution, and color.

[0195] In one embodiment, any images captured using a device that does not meet the following requirements will be rejected by the system: (1) the device must be digital, such as a smartphone, DSLR camera, or compact camera; (2) the device must be capable of recording Exif data when capturing images; (3) the device must capture images using a camera with at least 8 megapixels; and (4) the device must have been released in 2011 or later.

[0196] In one embodiment, any image that does not meet the following image capture requirements will be rejected by the system: (1) Each captured image must be stored as a JPEG or TIFF file, regardless of the device's default viewable output file format. Using a different file format to store images is acceptable if and only if the device utilizes a different default viewable output file format / container (e.g., HEIC); (2) The Exif data for each image must be intact, with the exception of the device serial number and GPS coordinate meta tags, which must be removed before delivery to the system; (3) Images must not have been post-processed, including any additional compression; (4) Images must not be panoramas; (5) Images must have an aspect ratio less than 2:1; (6) Images must not be captured using a fisheye lens or any other lens that causes spherical distortion; (7) Images should not be captured using digital zoom; however, optical zoom is permitted; (8) Images must not be captured while using filters; (9) Images that are excessively blurred due to motion blur or other reasons will be rejected by the system. It should be noted that while images inevitably exhibit some degree of motion blur due to the action the subject is performing (e.g., "running / jogging"), the system will accept these images as long as the feature point annotatability requirements (if defined) are met; and (10) images must not be captured with a "bokeh" (shallow depth of field) effect. That is, the majority of the image, particularly the background and the primary subject (and secondary subjects if relevant), must be in sharp focus.

[0197] Taking the above requirements into account, a non-exhaustive list of instrument-specific properties and settings that should be changed where appropriate is as follows: lens, image sensor, image stabilization, high dynamic range mode, aperture, shutter speed, ISO, and flash mode.

[0198] In one example, Exif metadata is required to record the diversity of camera brands and models used for capture, the date and time of capture (allowing for an approximation of the weather at the time of capture), and the settings used during capture.

[0199] To account for miscalibrated device settings (i.e., original datetime), the creator (or subject) is asked to report the date of capture (e.g., February 4, 2021) and the approximate one-hour capture time window (e.g., 00:00-00.59, 10:00-10:59, 14:00-14:59) along with the captured image. The system does not edit the datetime of the original Exif metadata tag and provides the reported datetime as a separate annotation.

[0200] To account for miscalibrated device settings (i.e., GPS coordinates), the creator (or subject) is required to report the captured city, state / province / county, and country (e.g., New York City, NY, USA) along with the captured image. Prior to removing the Exif GPS metatag, the system (or another system prior to delivery to the system) extracts and includes the captured city, state / province / county, and country as part of the annotation based on the Exif GPS metatag. For clarity, the vendor must provide the captured city, state / province / county, and country as reported by the image creator, as well as the captured city, state / province / county, and country based on the Exif GPS metatag.

[0201] environment

[0202] In some implementations, the system uses context information for a data set.Context refers to the scene in which the data was captured.

[0203] Environment - Lighting

[0204] In one embodiment, the system uses lighting as environmental information. In one example, the system collects data from voluntary subjects under various lighting conditions, with a particular focus on capturing data under different ambient lighting conditions. A non-exhaustive list of factors that may affect lighting includes time of day, season, weather conditions, geographic location, scene (indoor vs. outdoor).

[0205] In some embodiments, the system defines the following coarse time categories and required percentage distribution ranges (in square brackets, e.g., [minimum - maximum %]): 00:00-05:59 [5% -], 06:00-11:59 [15% -], 12:00-17:59 [15% -], and 18:00-23:59 [15% -].

[0206] The system also defines the following coarse weather categories and the required percentage distribution ranges (in square brackets, e.g., [minimum - maximum %]): Fog [1%-], Haze [1%-], Snow / Hail [1%-], Rain [1%-], Humid [1%-], Clouds [5%-], and Clear [5%-].

[0207] The system also defines the following coarse facial lighting categories and the required percentage distribution ranges (in square brackets, e.g., [min - max %]): lighting from above the head / face [0% -], lighting from below the head / face [0% -], lighting from in front of the head / face [0% -], lighting from behind the head / face [0% -], lighting from the left side of the head / face [0% -], and lighting from the right side of the head / face [0% -].

[0208] In one example, the temporal category associated with each image should be derived from a self-reported one-hour image capture time window.

[0209] Each image must be associated with a weather category. This must be self-reported by the subject (per image). Multiple weather category selections are allowed.

[0210] Additionally, each subject's facial illumination category must be self-reported (per image). Multiple illumination category selections are allowed. Each subject's facial bounding box (per image) must be associated with its facial illumination category.

[0211] Environment-Scene

[0212] In one embodiment, the system uses scenes as environmental information. In one example, the system collects data from volunteer subjects in a variety of different scenes in different geographical locations. Maximizing diversity relative to scenes will inevitably lead to wide variations in, for example, subject variations, non-uniform lighting conditions, ambient occlusion, and background clutter.

[0213] The system defines the following rough scene categories and the required percentage distribution range (in square brackets, e.g., [minimum - maximum %]): indoor scenes, including shopping and dining [1%-], workplaces (e.g., office buildings, factories, laboratories) [1%-], homes or hotels [1%-], transportation (e.g., vehicle interiors, stations) [1%-], sports and leisure [1%-], and culture (e.g., art, education, religion, military, law, politics) [1%-]; outdoor natural and man-made scenes, including These include water, ice, snow [2%-], mountains, hills, deserts, sky [2%-], forests, fields, jungles [2%-], man-made elements [2%-], transportation (e.g., roads, parking lots, bridges, ships, airports) [2%-], cultural or historical buildings / sites (e.g., military, religious) [2%-], sports fields, parks, recreational spaces [2%-], industries and buildings [2%-], houses, cottages, gardens and farms [2%-], and commercial buildings, shops, markets, cities and towns [2%-].

[0214] In one example, each image must be associated with a scene category and subcategory, such as "Indoor Scene (Sports and Leisure)". This must be self-reported by the subject (per image).

[0215] Environment - Camera Position

[0216] In one embodiment, the system uses camera positions as environmental information. In one example, the system collects data from volunteer subjects using a variety of different camera positions.

[0217] The system defines the following coarse camera position categories and the required percentage distribution ranges (in square brackets, e.g., [min-max %]): Typical Position [5%-] (camera positioned at eye or shoulder level of the primary subject), Atypical High Position [5%-] (camera positioned above eye level of the primary subject; or at an elevated vantage point relative to the primary subject's head), and Atypical Low Position [5%-] (camera positioned below the primary subject's eye level; at hip, knee, or ground level; or at a point below the primary subject's feet).

[0218] In one example, each image must be associated with a camera position category relative to a subject designated as the primary subject. This must be self-reported (per image).

[0219] Environment - Camera distance and feature point annotability

[0220] In one implementation, the system uses camera distance and feature point annotability as context information. In one example, the system collects data from volunteer subjects by varying the distance between the camera and the primary subject.

[0221] The camera distance is defined in terms of the length (in pixels) of the main subject's facial bounding box (tightly containing its forehead, chin, and cheeks), i.e., the distance in pixels from the top of its forehead to its chin. It should be noted that in the following, when the term "camera frame" is used, it refers to the subject being in the frame of the image, but does not mention the annotability of its feature points.

[0222] The system defines the following camera distance (“CD”) categories along with the required percentage distribution ranges (in square brackets, e.g., [minimum - maximum %]) and their associated landmark annotability requirements: CD I [10%-], face height in the range of 10-49 pixels, CD II [10%-], face height in the range of 50-299 pixels, CD III [10%-], face height 300-899 pixels, CD IV [10%-], face height 900-1499 pixels, CD V [10%-], face height 1500+ pixels.

[0223] In one embodiment, each type of feature point must be annotable in at least 10% of the delivered images. In another embodiment, at least 70% of the collected images must contain (within the camera frame) but not limited to the entire body of the main subject, including the head, feet, and hands, as well as all accessories and / or clothing worn on the head, face, hair, body, feet, and hands. In these images, at least 5 of the 22 body feature points (numbered 11-32) of the main subject must be annotated in each image.

[0224] The remaining images must contain (within the camera frame) at least the subject's entire head, and all accessories and / or clothing worn on the head, face, and hair.

[0225] For each image in the dataset, at least 3 of the 11 facial landmarks (numbered 0-10) of the main subject must be annotable.

[0226] In one example, each image must be associated with a camera distance class, a number of annotable facial landmarks, and a number of annotable body landmarks. This can be inferred from the annotated landmarks and / or the annotated segmentation masks.

[0227] Involuntary Person Bounding Box Annotation Review

[0228] In one embodiment, for each involuntary subject appearing in the image, the vendor will provide a segmentation mask of the entire involuntary subject or a bounding box around the involuntary subject. It should be noted that the segmentation mask or bounding box should encapsulate any objects held or worn by the involuntary subject. In general, all annotations of involuntary subjects must be delivered to the system via a JSON file accompanying the corresponding image being studied.

[0229] In some embodiments, the vendor should provide a segmentation mask of an involuntary subject as long as the following criteria apply: (1) if the involuntary subject is "large", then the vendor should provide a segmentation mask of the involuntary subject; (2) if the involuntary subject or its assumed bounding box overlaps with a voluntary (primary / secondary) subject, then the vendor should provide a segmentation mask of the involuntary subject regardless of whether the involuntary subject is in the background or part of a crowd.

[0230] It should be noted that the segmentation mask: (1) is not required to be a segmentation class; (2) should be limited to the visible area of ​​the involuntary subject in question; and (3) can be composed of disjoint polygons if necessary.

[0231] In some embodiments, the vendor should provide a bounding box that encapsulates the entire involuntary subject, as long as the following guidelines apply: (1) in general, a segmentation mask is always preferred, but if the involuntary subject is "small", located in the background, and does not overlap with the primary / secondary subjects, then the vendor may provide a bounding box of the involuntary subject instead of a segmentation mask; and (2) if a group of involuntary subjects forms a "crowd", and if the crowd is small, located in the background, and does not overlap with the primary / secondary subjects, then the vendor may provide a bounding box that encapsulates all involuntary subjects belonging to the crowd.

[0232] Various implementations, examples, data management, and specifications have been discussed. These are examples of how implementations of the system operate. They are not intended to be exhaustive or indicative of requirements for all implementations. For example, many of the examples focus on human images, but many of the techniques can be applied to other biological and non-biological subjects, such as animals, plants, structures, or purely generated images (such as from animations) without departing from the nature of the technology.

[0233] Exemplary Systems

[0234] Figure 2 A functional block diagram illustration of a computer hardware platform 200 is provided that may be used to implement a specifically configured computing device that may host an image dataset creation, evaluation, and training engine 250. As discussed above, the image dataset creation, evaluation, and training engine 250 may include an annotation application user interface 252, a bounding box application interface 254, a consent database 256, and a self-reported data application interface 258 that may allow a user to submit self-reported information such as subject data, instrument data, and environmental data.

[0235] Computer platform 200 may include a central processing unit (CPU) 202 , a hard disk drive (HDD) 204 , random access memory (RAM) and / or read only memory (ROM) 206 , a keyboard 208 , a mouse 210 , a display 212 , and a communication interface 214 , which are connected to a system bus 216 .

[0236] In one embodiment, HDD 204 has the capability to include stored programs that can perform various processes in a manner that performs the methods described herein, such as data set creation, evaluation, and training engine 250 .

[0237] All features disclosed in this specification, including any accompanying abstract and drawings, may be replaced by alternative features having the same, equivalent or similar purpose, unless expressly stated otherwise. Therefore, unless expressly stated otherwise, each feature disclosed is merely one example of a common, equivalent or similar feature.

[0238] The claim elements and steps herein may be numbered and / or lettered only to aid readability and understanding. Any such numbering and lettering itself is not intended to and should not be considered to indicate the order of elements and / or steps in the claims.

[0239] Many changes and modifications may be made by those skilled in the art without departing from the spirit and scope of the present invention. Therefore, it must be understood that the illustrated embodiments are set forth for illustrative purposes only and should not be viewed as limitations on the invention as defined by the following claims. For example, although the elements of the claims are listed below in certain combinations, it must be clearly understood that the present invention includes other combinations of fewer, more, or different disclosed elements.

[0240] The words used in this specification to describe the present invention and its various embodiments should be understood not only in their generally defined meanings, but also should be understood to include the general structures, materials, or acts they represent through special definitions in this specification.

[0241] Therefore, the words or elements of the following claims defined in this specification include not only the combinations of elements literally listed. Therefore, in this sense, any one element of the following claims can be replaced by two or more elements, or a single element can be substituted for two or more elements in the claims. Although the above elements may be described as working in certain combinations and even initially claimed in this way, it should be clearly understood that in some cases, one or more elements in the claimed combination can be cut out from the combination, and the claimed combination can be directed to sub-combinations or variations of sub-combinations.

[0242] Changes now known or later devised by one of ordinary skill in the art that are not related to the claimed subject matter are expressly contemplated to be equivalently within the scope of the claims. Therefore, obvious substitutions now or later known to one of ordinary skill in the art are defined to be within the scope of the defined elements.

[0243] Therefore, it should be understood that the claims include the contents specifically illustrated and described above, contents that are conceptually equivalent, contents that can be obviously substituted, and contents that incorporate the basic idea of ​​the present invention.

Claims

1. A computer-implemented method for constructing a dataset of human images, comprising: Collect multiple images from multiple and diverse people; providing a first graphical user interface that requests a user to provide subject data, instrument data, and environmental data as metadata for each of the plurality of images; as well as storing the plurality of images as a dataset, The subject data includes demographic information, physical characteristics, movements and head posture.

2. The computer-implemented method of claim 1 , further comprising providing a second graphical user interface that allows a user to form a bounding box around the face of the subject in each of the plurality of images. 3 . The computer-implemented method of claim 1 , further comprising providing a third graphical user interface that allows an annotator to provide annotations for each of the plurality of images. 4 . The computer-implemented method of claim 1 , wherein the plurality of images comprises 4 to 10 images of each person from the plurality of diverse persons.

5. The computer-implemented method of claim 1, wherein each of the plurality of images of each of the plurality of diverse persons were captured at least one day apart.

6. The computer-implemented method of claim 1, further comprising obtaining explicit informed consent from each of the plurality of diverse persons, wherein the explicit informed consent is provided as metadata for each of the plurality of images.

7. The computer-implemented method of claim 3, wherein the annotators are demographically diverse in terms of age, pronouns, and ancestry.

8. The computer-implemented method of claim 1, wherein the annotations include segmentation labels for each part of the subject's body in each image of the plurality of images.

9. The computer-implemented method of claim 1, wherein the physical characteristics include age, pronouns, nationality, place of residence, ancestry, and disability; and the physical characteristics include skin color, eye color, hair type, hair style, hair color, facial hair style, facial hair color, height, weight, and facial markings.

10. The computer-implemented method of claim 1, wherein actions include body gestures, subject-object interactions, and subject-subject interactions.

11. The computer-implemented method of claim 1, wherein the environmental data includes lighting, scene, camera position, and camera distance.

12. The computer-implemented method of claim 1, further comprising providing an output illustrating a diversity of the data set relative to each of the subject data, the instrument data, and the environmental data.

13. A computer-implemented method for training or evaluating a commercial machine learning or artificial intelligence system in an unconstrained setting, the method comprising: Create a diverse dataset of human images by doing the following: Collect multiple images from multiple and diverse people; providing a first graphical user interface that requests a user to provide subject data, instrument data, and environmental data as metadata for each of the plurality of images; providing a second graphical user interface that requires a user to form a bounding box around a face of a subject in each of the plurality of images; providing a third graphical user interface requesting an annotator to provide annotations for each of the plurality of images; as well as storing the plurality of images as a dataset; as well as Training or evaluating a machine learning or artificial intelligence system by using the diverse dataset in the machine learning or artificial intelligence system.

14. The computer-implemented method of claim 13, wherein the machine learning or artificial intelligence system is operable for one or more of body and face detection, body and face landmark detection, body and face parsing, face alignment, face recognition, face verification, image editing, and image synthesis.

15. The computer-implemented method of claim 13, wherein: The plurality of images includes 4 to 10 images of each person from the plurality of diverse persons; and Each image in the plurality of images of each person in the plurality of diverse persons was captured at least one day apart.

16. The computer-implemented method of claim 13, further comprising obtaining explicit informed consent from each of the plurality of diverse persons, wherein the explicit informed consent is provided as metadata for each of the plurality of images.

17. The computer-implemented method of claim 13, wherein the annotators are demographically diverse in terms of age, pronouns, and ancestry.

18. The computer-implemented method of claim 13, wherein: Subject data included demographic information, physical characteristics, movements, and head posture; Physical characteristics include age, pronouns, nationality, place of residence, ancestry, and disability; Physical characteristics include skin color, eye color, hair type, hair pattern, hair color, facial hair pattern, facial hair color, height, weight, and facial markings; Actions include body postures, subject-object interactions, and subject-subject interactions; as well as Environmental data includes lighting, scene, camera position, and camera distance.

19. A computer-implemented method for constructing a dataset of human images, comprising: Collect multiple images from multiple and diverse people; providing a first graphical user interface that requires a user to provide subject data, instrument data, and environmental data as metadata for each of the plurality of images; providing a second graphical user interface that requires a user to form a bounding box around a face of a subject in each of the plurality of images; providing a third graphical user interface requesting an annotator to provide annotations for each of the plurality of images; as well as storing the plurality of images as a dataset, The subject data includes demographic information, physical characteristics, movements and head posture.

20. The computer-implemented method of claim 19, wherein: The plurality of images includes 4 to 10 images of each person from the plurality of diverse persons; each of the plurality of images of each of the plurality of diverse persons is captured at least one day apart; Physical characteristics include age, pronouns, nationality, place of residence, ancestry, and disability; Physical characteristics include skin color, eye color, hair type, hair pattern, hair color, facial hair pattern, facial hair color, height, weight, and facial markings; Actions include body postures, subject-object interactions, and subject-subject interactions; as well as Environmental data includes lighting, scene, camera position, and camera distance.

Images