Keypoint-guided human attribute recognition using statistical correlation models

Abstract

Techniques are provided for neural network-based, anatomy keypoint and statistical correlation model guided human attribute recognition. Attributes include characteristics that can be visibly identified or inferred from an image, such as gender, hairstyle, style of dress, etc. Methods implementing these techniques according to embodiments include applying an attribute feature extraction (AFE) convolutional neural network (CNN) to a human image to generate an attribute feature map based on the image. The methods further include applying a keypoint guided proposal (KPG) CNN to the human image to generate proposed hierarchical regions of the image based on associated anatomical keypoints. The methods further include generating a recognition probability for a human attribute using a CNN combination layer that merges the attribute feature map, the proposed hierarchical regions, and a statistical correlation model (SCM) that provides correlations between features in the attribute feature map and the proposed hierarchical regions.

Description

Background Technology

[0001] Recognizing human attributes such as gender, hairstyle, and clothing style (wearing a hat or not, jeans or not) can be a crucial component of many applications, including personnel identification, supervision, security, intelligent home robots, self-driving cars, and drones (to name just a few). Developing automated, accurate, and efficient vision systems capable of recognizing human attributes is increasingly important in academia, business, and industry. However, this task remains challenging in unconstrained scenarios. Attached Figure Description

[0002] The features and advantages of the claimed subject matter will become apparent as the following detailed description is carried out and when the accompanying drawings are taken in, wherein the same numerals depict the same parts.

[0003] Figure 1 This is a block diagram of a human attribute recognition system configured according to certain embodiments of the present disclosure.

[0004] Figure 2 It is a block diagram of a proposal generation network configured according to certain embodiments of this disclosure, guided by key points.

[0005] Figure 3 The figures illustrate key anatomical points according to certain embodiments of this disclosure.

[0006] Figure 4A The figure illustrates a proposed layered region according to certain embodiments of this disclosure.

[0007] Figure 4B The figure illustrates a hierarchical structure of proposed regions according to certain embodiments of this disclosure.

[0008] Figure 5 This is a block diagram of a statistical correlation model generation circuit configured according to certain embodiments of the present disclosure.

[0009] Figure 6 This is another block diagram of a human attribute recognition system configured according to certain embodiments of the present disclosure.

[0010] Figure 7 This is a flowchart illustrating a method for identifying human attributes according to certain embodiments of the present disclosure.

[0011] Figure 8 This is a schematic diagram illustrating a computing platform configured to perform human attribute recognition according to certain embodiments of the present disclosure.

[0012] Although the following detailed description will be made with reference to illustrative embodiments, many alternatives, modifications and variations will be apparent from the present disclosure. Detailed Implementation

[0013] In general, this disclosure provides techniques for neural network-based human attribute recognition guided by anatomical keypoints and statistical correlation models. Attributes to be identified include characteristics that can be visually identified or inferred from an image, such as gender, hairstyle, clothing style, hat wearing, eyeglass wearing, footwear, etc. Keypoints are associated with joints or connection points between different regions of the body (such as, for example, the neck, shoulder, elbow, wrist, hip, knee, and ankle). Different parts of the human body can be defined by such joint locations. For example, the forearm can be located based on the positions of the elbow and wrist. Therefore, according to embodiments, keypoints are used to generate a hierarchical structure of proposed body regions that are more likely to be useful for consideration by the neural network in attribute recognition. Furthermore, as will be described in more detail below, statistical correlations between attribute features extracted from the image and the proposed hierarchical regions are employed by the neural network to improve recognition performance and accelerate the training process with reduced processing time and memory requirements. For example, since hats are typically located on a person's head and shoes are typically located on a person's feet, there can be a high correlation between key points, proposed regions, and attributes, which can be used for attribute recognition.

[0014] The disclosed techniques can be implemented in, for example, computing systems or software products that can be executed by or otherwise controlled by such systems, but other embodiments will also be apparent. In some embodiments, the system or product is a neural network comprising several sub-networks configured to identify human attributes in a provided image while reducing any constraints imposed on the activities or poses depicted in the image. According to an embodiment, a method for implementing these techniques includes: applying an Attribute Feature Extraction (AFE) convolutional neural network (CNN) to a human image to generate an attribute feature map based on the image. The method further includes: applying a Keypoint Guided Proposal (KPG) CNN to the human image to generate proposed hierarchical regions of the image based on associated anatomical keypoints, which can reduce the number of regions to be considered by several orders of magnitude. The method further includes: generating recognition probabilities of human attributes using a CNN ensemble layer that combines the attribute feature map, the proposed hierarchical regions, and a statistical correlation model (SCM) that provides the correlation between features of the attribute feature map and the proposed hierarchical regions.

[0015] As will be appreciated, the techniques described herein allow for more efficient and accurate attribute recognition by limiting the number of regions to be considered by the network to a subset of all possible regions, where proposed regions within this subset are more likely to be useful for accurate recognition. This approach provides improved performance compared to existing methods that require consideration of the entire body image and may blindly generate thousands of proposed regions for consideration, thus consuming more processing and memory resources. The disclosed techniques can be implemented on a wide range of platforms, including laptops, tablets, smartphones, workstations, cameras, and embedded systems or devices. These techniques can be further implemented in hardware or software or a combination thereof. In some embodiments, the efficiency improvements provided by the disclosed techniques enable real-time execution, such as attribute recognition processed at a rate of 15 image frames per second or more.

[0016] Figure 1 This is a block diagram of a human attribute recognition system 100 configured according to certain embodiments of the present disclosure. In this figure, system 100 is shown in a training configuration and includes an attribute recognition CNN 110 and an SCM generation circuit 162. The attribute recognition CNN 110 is further shown as including an AFE CNN 112, a KPG proposal generation CNN 114, and a CNN combination layer 116.

[0017] The AFE CNN 112 is configured to generate attribute feature maps 130 based on provided human images. The feature maps are activation maps output from the layers of the AFE CNN, and are generated using known neural network techniques, as per this disclosure. The AFE CNN is trained on training images 120, which depict humans in a wide variety of activities and exhibit significant variations in viewpoint, pose, lighting, occlusion, and scale. As per this disclosure, using known neural network training techniques, the training images include attribute labels as factual data, and the network loss can be evaluated relative to this factual data to compute network weights during backpropagation training iterations.

[0018] The KPG proposal generation CNN 114 is configured to generate proposed hierarchical regions 140 (sometimes simply referred to as proposals) of human images based on associated anatomical keypoints. According to certain embodiments of this disclosure, such as Figure 2 As illustrated in the figure, the KPG proposal generation CNN 114 is shown as including a keypoint generation CNN 210 and a proposal generation circuit 212.

[0019] KPG CNN 210 is configured to identify keypoints in provided human images, and KPG CNN 210 is trained on training images 120, which also include labeled keypoints as fact data. Additionally, in some embodiments, a keypoint training dataset 214 may be provided to enhance the training of KPG CNN 210. Keypoint training dataset 214 may include anatomical data, such as anatomical data from biological studies and reference sources.

[0020] exist Figure 3 The figure shows example key points 302 according to certain embodiments of the present disclosure. These key points include the neck (K1), left shoulder (K2), right shoulder (K5), left elbow (K3), right elbow (K6), left wrist (K4), right wrist (K7), left hip (K8), right hip (K9), left knee (K10), right knee (K11), left ankle (K12), and right ankle (K13), but other key points are possible.

[0021] The proposal generation circuit 212 is configured to generate a hierarchical structure of proposed regions based on keypoints provided by the KPG CNN 210. For example, the neck keypoint K1 defines a proposal region containing the head. Coarse-level proposals for the rest of the body can be defined by the positions of the left shoulder, right shoulder, left ankle, and right ankle. The body can then be divided into upper and lower halves at lower levels of the hierarchical structure. Coarse-level upper body proposals can be defined by the left shoulder, right shoulder, left wrist, and right wrist. The upper body can then be subdivided into more refined proposal regions, such as the torso, arms, upper arms, and forearms. The generation of proposal regions guided by keypoints that include human structural information reduces or eliminates consideration of less relevant regions and improves computational efficiency.

[0022] Figure 4AThe figure illustrates a proposed layered region 402 according to certain embodiments of the present disclosure, shown as a rectangular area overlaid on keypoint 302. In this example, the proposed layered region includes: the entire body (P1) at the highest level, followed by the head (P2), upper torso (P3), and lower torso (P4) at the next lower level. At the next lower level, the upper torso (P3) is shown including the chest (P8), left arm (P9), and right arm (P10). The lower torso (P4) includes the left leg (P11) and right leg (P5). At an even lower level, the left arm (P9) is shown including the left upper arm (P12) and left lower arm (P13), while the lower right leg P5 is shown including the right thigh (P6) and right calf (P7). It should be understood that this is an example, and in general, any number of suitable keypoints can be used to define other types of layered structures and associated layered regions.

[0023] Figure 4B A clearer map illustrates the hierarchical structure of the proposed regions according to certain embodiments of this disclosure. As shown, the body region is at the highest level 404. Subsequent lower levels 406, 408, 410 of the hierarchical structure contain increasingly refined regions. Each lower level can be considered a child of the immediately preceding higher parent level.

[0024] Figure 5 This is a block diagram of an SCM generation circuit 162 configured according to certain embodiments of the present disclosure. The SCM generation circuit 162 is configured to generate an SCM model 164 including an initial SCM correlation set 170. Since keypoints reflect the context of attributes in the human body, three types of statistical correlations can be considered: relationships between attributes, relationships between keypoints, and relationships between attributes and keypoints.

[0025] There are semantic correlations between attributes. For example, if a person is female, the probability of them having long hair is higher than that of a man. Similarly, if a person is male, the probability of them wearing a skirt is lower than that of a woman wearing a skirt.

[0026] Attributes can be represented as a1, a2, ..., a N The entire set of N attributes is represented as A = {a1, a2, ..., a...}. N For each attribute 'a' in the attributes... t The semantic relevance probability can then be expressed as:

[0027] a t =∑i ∈月，i≠t δ it a i

[0028]

[0029] Among them, according to the semantic relevance model, δ it For attribute a i With a t The correlation value between them.

[0030] Structural correlations exist between keypoints (e.g., the left shoulder connects to the left elbow). These structural correlations can be discovered, for example, from biological studies and anatomical reference sources. Furthermore, as previously described, since proposed regions can be derived from keypoints, hierarchical proposed regions can be considered a variation of the keypoint structural correlation model.

[0031] The proposal can be represented as p1, p2, ..., p M The entire set of M attributes can be represented as P = {p1, p2, ..., p...} M Since the proposals have a hierarchical structure derived from the skeleton of human keypoints, the probability of the parent proposal is derived from the child proposal. Let p d Represented as sub-proposal p t and p k The father suggested, p d The probability can be expressed as:

[0032] p d =θ dt p t +θ dk p k

[0033] θ dt +θ dk =1, 0≤θ dt ≤1, 0≤θ dk ≤1

[0034] Where, θ dt It is a hierarchical proposal based on the structural correlation model p d With p t The correlation value between them.

[0035] Finally, regarding the relationship between attributes and keypoints (and, by extension, the proposed regions), the correlation between the proposed regions and the attribute recognition rate / probability 150 obtained from training the attribute recognition CNN 110 can be calculated to process the proposals together or separately. An example of such calculation is illustrated in Table 1 below, which shows the relationship between the attribute recognition rate and the proposals.

[0036] Table 1: mAP between attributes and proposals

[0037]

[0038] In this example, nine attributes are selected: male, hair, glasses, hat, T-shirt, sleeved shirt, shorts, jeans, and trousers. Additionally, ten proposals P1-P10 are selected, where P10 is the highest-level full-body proposal, P1 is the head proposal, P5 is the lower body proposal, P8 is the left lower body proposal, and P9 is the right lower body proposal. Note that these proposals are numbered differently. Figure 4A and Figure 4B The example given is shown in the text.

[0039] As can be seen from the table, P10 is associated with most attributes (bold numbers) because it covers the entire body. In contrast, P1, which covers the head area, is more closely associated with hair, glasses, and hats (bold numbers) than T-shirts and shorts. Continuing the example, P5, P8, and P9, which cover the lower body, are more closely associated with shorts, jeans, and trousers. The bottom row of the table provides the attribute recognition rate as a percentage of the mean of the proposals (mAP) (i.e., the average of each column). Column M shows the attribute recognition rate using all proposals (P1-P10).

[0040] SCM model 164 includes a semantic relevance model generated by circuit 502 to provide the relevance between each attribute in the attributes, a structural relevance model generated by circuit 506 to provide the relevance between each proposed hierarchical region in the proposed hierarchical regions (based on keypoints), and a positional relevance model generated by circuit 504 to provide the relevance between each attribute in the attributes and each proposed hierarchical region in the proposed hierarchical regions. An initial SCM set is generated based on: training images 120 including attribute labels and keypoint labels; training data 160 including anatomical models; and attribute recognition probabilities 150 generated by the system in response to the training images.

[0041] These three types of relations—semantic relations, positional relations, and structural relations—provide relatively simple SCM models164, which need to be computed at least once as part of the training process, thus improving processing efficiency. SCM models can also be used during CNN training to update network relation weights, thereby improving the accuracy of relation description.

[0042] Now return to Figure 1 The CNN combination layer 116 is configured to generate the recognition probability of human attributes based on the combination of attribute feature map 130, proposed hierarchical region 140, and SCM 170 which provides the correlation between the features of the attribute feature map and the proposed hierarchical region.

[0043] In some embodiments, the output of the CNN combination layer 116 can be expressed by the following equation:

[0044]

[0045] Among them, S f Let the output attribute recognition probability be 150, γ be the normalization factor, A be the set of all attributes of interest, and P be the set of all proposals. j The child proposes the suggestion as suggested by the father. j The set of δ it For attribute a i With a t The correlation between them based on the semantic relevance model, β jk For attribute a k With proposal p j The correlation between them according to the location correlation model (an example of it is shown in Table 1), and θ kj For the proposal p k With p j The correlation between them is based on the structural correlation model.

[0046] Figure 6 This is another block diagram 600 of a human attribute recognition system 100 configured according to certain embodiments of the present disclosure. In this figure, the system is shown in an operational configuration, that is, after the attribute recognition CNN 110 (including subnetworks 112, 114, and 116) has been trained as previously described. In operational mode, an input image 610 is provided to the attribute recognition CNN 110, and an attribute recognition probability 150 is generated as output, the input image 610 depicting a human with unknown combinations of attributes.

[0047] method

[0048] Figure 7 This is a flowchart illustrating an example method 700 for identifying human attributes according to certain embodiments of the present disclosure. As can be seen, the example method includes several stages and sub-processes, the order of which may vary from one embodiment to another. However, when taken together, these stages and sub-processes form a process for attribute identification according to certain embodiments disclosed herein. These embodiments may, for example, use the methods described above. Figure 1 , Figure 2 , Figure 5 and Figure 6 The system architecture illustrated herein is used for implementation. However, as will be apparent from this disclosure, other system architectures may be used in other embodiments. Therefore, Figure 7The relevance of the various functions shown to the specific components illustrated in other figures is not intended to imply any structural and / or usage limitations. However, other embodiments may include, for example, different degrees of integration, where multiple functions are efficiently performed by a single system. For example, in an alternative embodiment, a single module with decoupled submodules may be used to perform all the functions of method 700. Thus, depending on the granularity of the implementation, other embodiments may have fewer or more modules and / or submodules. In yet another embodiment, the depicted method may be implemented as a computer program article comprising one or more non-transient machine-readable media that, when executed by one or more processors, causes the method to be performed. Numerous variations and alternative configurations will be apparent in light of this disclosure.

[0049] like Figure 7 As illustrated in the figure, in an embodiment, the method 700 for identifying human attributes begins at operation 710 by applying an attribute feature extraction (AFE) convolutional neural network (CNN) to a human image to generate an attribute feature map based on the image. Attributes may include characteristics that can be visually identified or inferred from the image, such as, but not limited to, gender, hairstyle, clothing style (e.g., jeans versus non-jeans), wearing of glasses, wearing of hats, wearing of shoes, and type of shoes and socks.

[0050] Next, at operation 720, a Keypoint Guided Proposal (KPG) CNN is applied to the human image to generate proposed hierarchical regions of the image based on associated anatomical keypoints. Keypoints are associated with joints or other types of connections between different regions of the body (such as, for example, the neck, shoulder, elbow, wrist, hip (or waist), knee, and ankle). Associated hierarchical regions may include, for example, the entire body at the top of the hierarchy, followed by the head, upper torso, and lower torso at the next level, then the chest, arms, and legs at a lower level, and then the upper arm, forearm, thigh, and calf at an even lower level.

[0051] At operation 730, recognition probabilities of human attributes are generated based on a CNN ensemble layer that combines attribute feature maps, proposed hierarchical regions, and a statistical correlation model (SCM). The SCM provides the correlation between features in the attribute feature maps and the proposed hierarchical regions. In some embodiments, the SCM includes a semantic correlation model for providing the correlation between each attribute, a structural correlation model for providing the correlation between each proposed hierarchical region, and a positional correlation model for providing the correlation between each attribute and each proposed hierarchical region.

[0052] Of course, as previously described in conjunction with the system, additional operations may be performed in some embodiments. For example, an initial SCM set may be generated based on: training images including attribute labels and keypoint labels; training data including an anatomical model; and attribute recognition probabilities generated by the system in response to the training images. Additionally, the AFE CNN may be trained on training images including attribute labels, and the KPG CNN may be trained on training images including attribute labels and on a keypoint training dataset.

[0053] Example System

[0054] Figure 8 The figure illustrates an example system 800 configured to perform human attribute recognition according to certain embodiments of the present disclosure. In some embodiments, system 800 includes a computing platform 810 that may control or otherwise incorporate into: personal computers, workstations, server systems, laptop computers, ultra-laptop computers, tablets, touchpads, portable computers, handheld computers, PDAs, personal digital assistants (PDAs), cellular phones, combined cellular phones and PDAs, smart devices (e.g., smartphones or smart tablets), mobile internet devices (MIDs), messaging devices, data communication devices, imaging devices, wearable devices, embedded systems, and so on. Any combination of different devices may be used in some embodiments.

[0055] In some embodiments, platform 810 may include any combination of the following: processor 820, memory 830, attribute recognition systems 100, 600, network interface 840, input / output (I / O) system 850, user interface 860, image capture device 862, and storage system 870. As can be further seen, a bus and / or interconnect 892 is also provided to allow communication between the components listed above and / or other components not shown. Platform 810 may be coupled to network 894 via network interface 840 to allow communication with other computing devices, platforms, or resources. Figure 8 Other components and functions reflected in the block diagram will be apparent from the present disclosure, and it will be appreciated that other embodiments are not limited to any particular hardware configuration.

[0056] Processor 820 can be any suitable processor and may include one or more coprocessors or controllers, such as an audio processor, graphics processing unit, or hardware accelerator, to assist in controlling and processing operations associated with system 800. In some embodiments, processor 820 may be implemented as any number of processor cores. A processor (or processor core) can be any type of processor, such as, for example, a microprocessor, embedded processor, digital signal processor (DSP), graphics processing unit (GPU), network processor, field-programmable gate array, or other device configured to execute code. Processors may be multi-threaded cores, as each core may include more than one hardware thread context (or “logical processor”). Processor 820 may be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processor. In some embodiments, processor 820 may be configured as an x86-compliant processor.

[0057] Memory 830 can be implemented using any suitable type of digital storage, including, for example, flash memory and / or random access memory (RAM). In some embodiments, memory 830 may include layers of a memory hierarchy known to those skilled in the art and / or a memory cache. Memory 830 may be implemented as a volatile memory device, such as, but not limited to, RAM, dynamic RAM (DRAM), or static RAM (SRAM) devices. Storage system 870 may be implemented as a non-volatile storage device, such as, but not limited to, one or more of the following: hard disk drive (HDD), solid-state drive (SSD), universal serial bus (USB) drive, optical disk drive, tape drive, internal storage device, attached storage device, flash memory, battery-backed synchronous DRAM (SDRAM), and / or network-accessible storage device. In some embodiments, storage 870 may include techniques for enhancing storage performance protection for valuable digital media when including multiple hard disk drives.

[0058] Processor 820 may be configured to execute operating system (OS) 880, which may include any suitable operating system, such as Google Android (Google Inc., Mountain View, California), Microsoft Windows (Microsoft Corporation, Redmond, Washington), Apple OS X (Apple Inc., Cupertino, California), Linux, or a real-time operating system (RTOS). As will be appreciated from this disclosure, the techniques provided herein can be implemented without regard to a specific operating system provided by system 800, and therefore can also be implemented using any suitable existing or subsequently developed platform.

[0059] The network interface circuit 840 can be any suitable network chip or chipset that allows wired and / or wireless connectivity between other components of the computer system 800 and / or the network 894, thereby enabling the system 800 to communicate with other local and / or remote computing systems, servers, cloud-based servers, and / or resources. Wired communication may conform to existing (or undeveloped) standards, such as, for example, Ethernet. Wireless communication may conform to existing (or undeveloped) standards, such as, for example, cellular communication including LTE (Long Term Evolution), Wi-Fi, Bluetooth, and / or Near Field Communication (NFC). Exemplary wireless networks include, but are not limited to, wireless LANs, wireless personal area networks, wireless metropolitan area networks, cellular networks, and satellite networks.

[0060] I / O system 850 may be configured to interface between various I / O devices and other components of computer system 800. I / O devices may include, but are not limited to, user interface 860 and image capture device 862 (e.g., camera). User interface 860 may include devices (not shown) such as display elements, touchpads, keyboards, mice, and speakers. I / O system 850 may include a graphics subsystem configured to perform image processing for rendering on display elements. The graphics subsystem may be, for example, a graphics processing unit or a vision processing unit (VPU). Analog or digital interfaces may be used to communicatively couple the graphics subsystem and the display elements. For example, the interface may be any of the following: High Definition Multimedia Interface (HDMI), DisplayPort, Wireless HDMI, and / or any other suitable interface using wireless high-definition compatible technology. In some embodiments, the graphics subsystem may be integrated into processor 820 of platform 810 or any chipset.

[0061] It will be appreciated that, in some embodiments, the components of system 800 may be combined or integrated in a system-on-a-chip (SoC) architecture. In some embodiments, the components may be hardware components, firmware components, software components, or any suitable combination of hardware, firmware, or software.

[0062] As previously described, in training mode 100 and operating mode 600, the attribute recognition system is configured to: identify selected human attributes from images guided by anatomical keypoints and a statistical correlation model. As described above, the attribute recognition system may include... Figure 1 , Figure 2 , Figure 5 and Figure 6Any or all of the circuits / components illustrated herein. These components may be implemented or otherwise used in conjunction with various suitable software and / or hardware coupled to or otherwise forming part of platform 810. These components may be implemented or otherwise used in conjunction with user I / O devices capable of providing information to and receiving information and commands from the user.

[0063] In some embodiments, such as in Figure 8 As shown in the example embodiments, the attribute identification system and circuitry may be installed locally on system 800. Alternatively, system 800 may be implemented in a client-server configuration, wherein at least some functionality associated with these systems is provided to platform 810 using applets such as JavaScript applets or other sets of downloadable modules or submodules. Such remotely accessible modules or submodules may be provisioned in real time in response to requests from client computing systems for access to resources of interest to users of the client computing systems on a given server. In such embodiments, the server may be locally coupled to network 894 or remotely coupled to network 894 via one or more other network and / or communication channels. In some cases, access to resources on a given network or computing system may require credentials, such as usernames, passwords, and / or credentials compatible with any other suitable security mechanisms.

[0064] In various embodiments, system 800 may be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, system 800 may include components and interfaces suitable for communication via a wireless shared medium, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc. Examples of a wireless shared medium may include portions of a wireless spectrum, such as radio frequency spectrum, etc. When implemented as a wired system, system 800 may include components and interfaces suitable for communication via a wired communication medium, such as input / output adapters, physical connectors connecting the input / output adapters to the corresponding wired communication medium, network interface cards (NICs), disk controllers, video controllers, audio controllers, etc. Examples of a wired communication medium may include wires, cables, metal leads, printed circuit boards (PCBs), backplanes, switching structures, semiconductor materials, twisted pairs, coaxial cables, optical fibers, etc.

[0065] The embodiments can be implemented using hardware components, software components, or a combination of both. Examples of hardware components may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, ASICs, programmable logic devices, digital signal processors, FPGAs, logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application programming interfaces, instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. The determination of whether to use hardware components and / or software components to implement the embodiments can vary depending on any number of factors, such as desired computational speed, power level, thermal tolerance, processing cycle budget, input data rate, output data rate, memory resources, data bus speed, and other design or performance constraints.

[0066] The terms “coupled” and “connected” and their derivatives may be used to describe some embodiments. These terms are not intended to be synonyms with each other. For example, some embodiments may be described using the terms “connected” and / or “coupled” to indicate direct physical or electrical contact between two or more elements. However, the term “coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0067] The embodiments disclosed herein can be implemented in various forms of hardware, software, firmware, and / or dedicated processors. For example, in one embodiment, at least one non-transient computer-readable storage medium has instructions encoded thereon that, when executed by one or more processors, cause one or more of the human attribute recognition methods disclosed herein to be implemented. The instructions can be encoded using suitable programming languages ​​or alternatively using custom or proprietary instruction sets, such as C, C++, object-oriented C, Java, JavaScript, Visual Basic, .NET, and Beginner's General Symbolic Instruction Code (BASIC). The instructions can be provided in the form of one or more computer software applications and / or applets that are tangibly embodied on a memory device and can be executed by a computer with any suitable architecture. In one embodiment, the system can be hosted on a given website and implemented, for example, using JavaScript or other suitable browser-based technologies. For example, in some embodiments, the system can utilize processing resources provided by a remote computer system accessible via a network 894. In other embodiments, the functionality disclosed herein can be incorporated into other software applications related to image processing, such as, for example, security and surveillance applications, smart home management, entertainment, and robotics applications. The computer software applications disclosed herein may include any number of different modules, submodules, or other components with different functions, and may provide information to or receive information from other components. These modules may be used, for example, to communicate with input and / or output devices such as displays, touch-sensitive surfaces, printers, and / or any other suitable devices. Other components and functions not illustrated in the figures will be apparent from this disclosure and will be appreciated that other embodiments are not limited to any particular hardware or software configuration. Therefore, in other embodiments, system 800 may include, compared to Figure 8 The example embodiments include additional, fewer, or alternative sub-components.

[0068] The aforementioned non-transient computer-readable medium can be any suitable medium for storing digital information, such as a hard drive, server, flash memory and / or random access memory (RAM), or a combination of memories. In alternative embodiments, the components and / or modules disclosed herein can be implemented using hardware, including gate-level logic such as a field-programmable gate array (FPGA), or alternatively, application-specific semiconductors such as application-specific integrated circuits (ASICs). Further embodiments may utilize a microcontroller with several input / output ports for receiving and outputting data, and multiple embedded routines for performing the various functions disclosed herein. It will be apparent that any suitable combination of hardware, software, and firmware can be used, and other embodiments are not limited to any particular system architecture.

[0069] For example, some embodiments may be implemented using a machine-readable medium or article of writing that stores instructions or instruction sets that, when executed by a machine, cause the machine to perform the methods and / or operations described in the embodiments. Such a machine may include, for example, any suitable processing platform, computing platform, computing device, processing device, computing system, processing system, computer, processing, etc., and may be implemented using any suitable combination of hardware and / or software. Machine-readable media or articles of writing may include, for example, any suitable type of memory cell, memory device, memory article, memory medium, storage device, storage article, storage medium and / or storage cell, such as memory, removable or non-removable media, erasable or non-erasable media, writable or rewritable media, digital or analog media, hard disk, floppy disk, compact disc read-only memory (CD-ROM), compact disc recordable (CD-R) memory, compact disc rewritable (CD-RW) memory, optical disc, magnetic media, magneto-optical media, removable memory cards or discs, various types of digital multifunction discs (DVDs), magnetic tape, cassette tape, etc. Instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, encrypted code, etc., implemented using any suitable high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming language.

[0070] Unless otherwise stated, it will be understood that terms such as “processing,” “computing,” “operation,” and “determining” refer to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as (e.g., electronically) physical quantities within the registers and / or memory of the computing system into other data similarly represented as physical quantities within the registers, memory units, or other such information transmission or display devices of the computing system. Embodiments are not limited to this context.

[0071] As used herein in any embodiment, the terms "circuit" or "circuit system" are functional and may include, for example, hardwired circuit systems alone or in any combination; programmable circuit systems, such as a computer processor including one or more individual instruction processing cores; state machine circuit systems; and / or firmware storing instructions executed by the programmable circuit system. A circuit system may include a processor and / or controller configured to execute one or more instructions to perform one or more operations described herein. Instructions may be embodied as, for example, an application, software, firmware, etc., configured to cause the circuit system to perform any of the foregoing operations. Software may be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a computer-readable storage device. Software may be embodied or implemented as including any number of processes, and processes may further be embodied or implemented as including any number of threads, etc., in a hierarchical manner. Firmware may be embodied as hard-coded (e.g., non-volatile) code, instructions, or instruction sets, and / or data in a memory device. A circuit system can be embodied collectively or individually as a part of a larger system, such as an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, etc. Other embodiments may be implemented as software executed by a programmable control device. In such cases, the term "circuit" or "circuit system" is intended to include a combination of software and hardware, such as a programmable control device or processor capable of executing software. As described herein, embodiments may be implemented using hardware elements, software elements, or any combination thereof. Examples of hardware elements may include: processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc.

[0072] Numerous specific details have been set forth herein to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that these embodiments may be practiced without these specific details. In other instances, well-known operations, components, and circuits have not been described in detail so as not to obscure the invention. It will be appreciated that the specific structural and functional details disclosed herein may be representative and are not intended to limit the scope of the embodiments. Furthermore, although the subject matter has been described using language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described herein. Rather, the specific features and actions described herein are disclosed as examples of implementing the claims.

[0073] Further example embodiments

[0074] The following examples relate to further embodiments, according to which numerous substitutions and configurations will be apparent.

[0075] Example 1 is a system for identifying human attributes, comprising: an Attribute Feature Extraction (AFE) convolutional neural network (CNN) for generating attribute feature maps based on human images; a Keypoint Guided Proposal (KPG) CNN for generating proposed hierarchical regions of human images based on associated anatomical keypoints; and a CNN combination layer for generating recognition probabilities of human attributes based on the attribute feature maps, the proposed hierarchical regions, and a statistical correlation model (SCM), whereby the SCM provides the correlation between features in the attribute feature maps and the proposed hierarchical regions.

[0076] Example 2 includes the subject matter as described in Example 1, and further includes a statistical correlation model generation circuit for generating an initial SCM set, wherein the SCM includes a semantic correlation model for providing correlations between each of the attributes, a structural correlation model for providing correlations between each of the proposed hierarchical regions, and a locational correlation model for providing correlations between each of the attributes and each of the proposed hierarchical regions.

[0077] Example 3 includes the subject matter as described in Example 1 or 2, wherein the generation of the initial SCM set is based on training images including attribute labels and keypoint labels, training data including anatomical models, and attribute recognition probabilities generated by the system in response to the training images.

[0078] Example 4 includes a subject as described in any one of Examples 1-3, wherein the AFE CNN is trained on training images including attribute labels.

[0079] Example 5 includes a subject as described in any one of Examples 1-4, wherein the KPG CNN is trained on a keypoint training dataset and training images including attribute labels.

[0080] Example 6 includes a subject as described in any one of Examples 1-5, wherein human attributes include at least one of the following: gender, hairstyle, clothing style, wearing a hat, wearing glasses, and type of shoes and socks.

[0081] Example 7 includes a subject as described in any one of Examples 1-6, wherein the key points include at least one of the following: neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0082] Example 8 includes a subject as described in any one of Examples 1-7, wherein the proposed layered regions include at least one of the following: body, head, upper torso, lower torso, chest, left arm, right arm, left leg, right leg, left upper arm, left lower arm, right upper arm, right lower arm, left thigh, left calf, right thigh, and right calf.

[0083] Example 9 is a computer-implemented method for recognizing human attributes, comprising: applying an attribute feature extraction (AFE) convolutional neural network (CNN) to a human image by a computer processor to generate an attribute feature map based on the image; applying a keypoint-guided proposal (KPG) CNN to the human image by the computer processor to generate proposed hierarchical regions of the image based on associated anatomical keypoints; and generating recognition probabilities of human attributes by the computer processor based on a CNN ensemble layer that combines the attribute feature map, the proposed hierarchical regions, and a statistical correlation model (SCM) that provides the correlation between features in the attribute feature map and the proposed hierarchical regions.

[0084] Example 10 includes the topic as described in Example 9, and further includes generating an initial set of SCMs, the SCMs including a semantic relevance model for providing the relevance between each attribute in the attributes, a structural relevance model for providing the relevance between each proposed hierarchical region in the proposed hierarchical regions, and a locational relevance model for providing the relevance between each attribute in the attributes and each proposed hierarchical region in the proposed hierarchical regions.

[0085] Example 11 includes the subject matter as described in Example 9 or 10, wherein the generation of the initial SCM set is based on training images including attribute labels and keypoint labels, training data including an anatomical model, and attribute recognition probabilities generated by a CNN ensemble layer in response to the training images.

[0086] Example 12 includes a subject as described in any one of Examples 9-11, and further includes training an AFE CNN on training images including attribute labels.

[0087] Example 13 includes a subject as described in any one of Examples 9-12, and further includes training a KPG CNN on a keypoint training dataset and training images including attribute labels.

[0088] Example 14 includes a subject as described in any one of Examples 9-13, wherein human attributes include at least one of the following: gender, hairstyle, clothing style, wearing a hat, wearing glasses, and type of shoes and socks.

[0089] Example 15 includes a subject as described in any one of Examples 9-14, wherein the key points include at least one of the following: neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0090] Example 16 includes a subject as described in any one of Examples 9-15, wherein the proposed layered regions include at least one of the following: body, head, upper torso, lower torso, chest, left arm, right arm, left leg, right leg, left upper arm, left lower arm, right upper arm, right lower arm, left thigh, left calf, right thigh, and right calf.

[0091] Example 17 is at least one non-transient computer-readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the following operations for recognizing human attributes, including: applying an attribute feature extraction (AFE) convolutional neural network (CNN) to a human image to generate an attribute feature map based on the image; applying a keypoint guided proposal (KPG) CNN to the human image to generate proposed hierarchical regions of the image based on associated anatomical keypoints; and generating recognition probabilities of human attributes based on a CNN ensemble layer that combines the attribute feature map, the proposed hierarchical regions, and a statistical correlation model (SCM) that provides the correlation between features in the attribute feature map and the proposed hierarchical regions.

[0092] Example 18 includes the subject matter as described in Example 17, and the operation further includes generating an initial set of SCMs, the SCMs including a semantic relevance model for providing the relevance between each attribute in the attributes, a structural relevance model for providing the relevance between each proposed hierarchical region in the proposed hierarchical regions, and a locational relevance model for providing the relevance between each attribute in the attributes and each proposed hierarchical region in the proposed hierarchical regions.

[0093] Example 19 includes the subject matter as described in Example 17 or 18, wherein the generation of the initial SCM set is based on training images including attribute labels and keypoint labels, training data including an anatomical model, and attribute recognition probabilities generated by a CNN ensemble layer in response to the training images.

[0094] Example 20 includes a subject as described in any one of Examples 17-19, and the operation further includes training an AFE CNN on training images including attribute labels.

[0095] Example 21 includes a subject as described in any one of Examples 17-20, and the operation further includes training a KPG CNN on a keypoint training dataset and training images including attribute labels.

[0096] Example 22 includes a subject as described in any one of Examples 17-21, wherein human attributes include at least one of the following: gender, hairstyle, clothing style, wearing a hat, wearing glasses, and type of shoes and socks.

[0097] Example 23 includes a subject as described in any one of Examples 17-22, wherein the key points include at least one of the following: neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0098] Example 24 includes a subject as described in any one of Examples 17-23, wherein the proposed layered regions include at least one of the following: body, head, upper torso, lower torso, chest, left arm, right arm, left leg, right leg, left upper arm, left lower arm, right upper arm, right lower arm, left thigh, left calf, right thigh, and right calf.

[0099] Example 25 is a system for recognizing human attributes, comprising: means for applying an attribute feature extraction (AFE) convolutional neural network (CNN) to a human image to generate an attribute feature map based on the image; means for applying a keypoint guided proposal (KPG) CNN to the human image to generate proposed hierarchical regions of the image based on associated anatomical keypoints; and means for generating recognition probabilities of human attributes based on a combined CNN layer that merges the attribute feature map, the proposed hierarchical regions, and a statistical correlation model (SCM), the SCM providing the correlation between features in the attribute feature map and the proposed hierarchical regions.

[0100] Example 26 includes the subject matter as described in Example 25, and further includes means for generating an initial set of SCMs, the SCMs including a semantic relevance model for providing the relevance between each of the attributes, a structural relevance model for providing the relevance between each of the proposed hierarchical regions, and a positional relevance model for providing the relevance between each of the attributes and each of the proposed hierarchical regions.

[0101] Example 27 includes the subject matter as described in Example 25 or 26, wherein the generation of the initial SCM set is based on training images including attribute labels and keypoint labels, training data including anatomical models, and attribute recognition probabilities generated by the system in response to the training images.

[0102] Example 28 includes a subject as described in any one of Examples 25-27, and further includes means for training an AFE CNN on training images including attribute labels.

[0103] Example 29 includes a subject as described in any one of Examples 25-28, and further includes means for training a KPG CNN on a keypoint training dataset and training images including attribute labels.

[0104] Example 30 includes a subject as described in any one of Examples 25-29, wherein human attributes include at least one of the following: gender, hairstyle, clothing style, wearing a hat, wearing glasses, and type of shoes and socks.

[0105] Example 31 includes a subject as described in any one of Examples 25-30, wherein the key points include at least one of the following: neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0106] Example 32 includes a subject as described in any one of Examples 25-31, wherein the proposed layered regions include at least one of the following: body, head, upper torso, lower torso, chest, left arm, right arm, left leg, right leg, left upper arm, left lower arm, right upper arm, right lower arm, left thigh, left calf, right thigh, and right calf.

[0107] The terms and expressions used herein are descriptive and not restrictive, and their use is not intended to exclude any equivalents of the shown and described features (or portions thereof), and it should be understood that various modifications are possible within the scope of the claims. Accordingly, the claims are intended to cover all such equivalents. Various features, aspects, and embodiments have been described herein. As will be understood by those skilled in the art, these features, aspects, and embodiments are readily combined with each other and varied and modified. This disclosure should therefore be considered to include such combinations, variations, and modifications. The scope of this disclosure is not limited by this detailed description but by the appended claims. Future applications claiming priority to this application may claim the disclosed subject matter in different ways and may generally include any set of one or more elements as disclosed or otherwise shown herein.

Claims

1. A system for recognizing human attributes, the system comprising: an attribute feature extraction convolutional neural network (CNN) for generating attribute feature maps based on a human image; a keypoint-guided proposal CNN for generating proposed hierarchical regions of the human image based on associated anatomical keypoints, wherein the keypoints are associated with joints or connection points between different regions of a human body and the proposed hierarchical regions are associated with a hierarchy of a human body; a CNN combination layer for generating recognition probabilities of the human attributes based on the attribute feature maps, the proposed hierarchical regions, and a statistical correlation model that provides correlations between features of the attribute feature maps and the proposed hierarchical regions.

2. The system of claim 1, further comprising statistical correlation model generation circuitry for generating an initial set of statistical correlation models, the statistical correlation models including semantic correlation models for providing correlations between each of the attributes, structural correlation models for providing correlations between each of the proposed hierarchical regions, and location correlation models for providing correlations between each of the attributes and each of the proposed hierarchical regions.

3. The system of claim 2, wherein, Generation of the initial set of statistical correlation models is based on training images including attribute labels and keypoint labels, training data including an anatomical model, and attribute recognition probabilities generated by the system in response to the training images.

4. The system of claim 1, wherein, The attribute feature extraction CNN is trained on training images including attribute labels.

5. The system of claim 1, wherein, The keypoint-guided proposal CNN is trained on a keypoint training dataset and training images including attribute labels.

6. The system of any one of claims 1-5, wherein, The human attributes include at least one of gender, hairstyle, dress style, hat wearing, glasses wearing, and type of footwear.

7. The system of any one of claims 1-5, wherein, The keypoints include at least one of a neck, a left shoulder, a right shoulder, a left elbow, a right elbow, a left wrist, a right wrist, a left hip, a right hip, a left knee, a right knee, a left ankle, and a right ankle.

8. The system of any one of claims 1-5, wherein, The proposed hierarchical regions include at least one of a body, a head, an upper torso, a lower torso, a chest, a left arm, a right arm, a left leg, a right leg, a left upper arm, a left lower arm, a right upper arm, a right lower arm, a left thigh, a left calf, a right thigh, and a right calf.

9. A computer-implemented method for recognizing human attributes, the method comprising: applying, by a computer processor, an attribute feature extraction convolutional neural network (CNN) to a human image to generate attribute feature maps based on the image; applying, by the computer processor, a keypoint-guided proposal CNN to the human image to generate proposed hierarchical regions of the image based on associated anatomical keypoints, wherein the keypoints are associated with joints or connection points between different regions of a human body and the proposed hierarchical regions are associated with a hierarchy of a human body; and generating, by the computer processor, recognition probabilities for the human attributes based on a CNN combination layer that merges the attribute feature maps, the proposed hierarchical regions, and a statistical correlation model that provides correlations between features of the attribute feature maps and the proposed hierarchical regions.

10. The method of claim 9, further comprising generating an initial set of statistical correlation models, the statistical correlation models including a semantic correlation model to provide correlations between each of the attributes, a structural correlation model to provide correlations between each of the proposed hierarchical regions, and a location correlation model to provide correlations between each of the attributes and each of the proposed hierarchical regions.

11. The method of claim 10, wherein, The generation of the initial set of statistical correlation models is based on training images including attribute labels and keypoint labels, training data including an anatomical model, and attribute recognition probabilities generated by the CNN combination layer in response to the training images.

12. The method of claim 9, further comprising training the attribute feature extraction CNN on training images including attribute labels.

13. The method of claim 9, further comprising training the keypoint-guided proposal CNN on a keypoint training dataset and training images including attribute labels.

14. The method of any one of claims 9-13, wherein, The human attributes include at least one of: gender, hairstyle, dress style, hat wearing, glasses wearing, and type of footwear.

15. The method of any one of claims 9-13, wherein, The key points include at least one of: neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

16. The method of any one of claims 9-13, wherein, The proposed hierarchical regions include at least one of: body, head, upper torso, lower torso, chest, left arm, right arm, left leg, right leg, left upper arm, left lower arm, right upper arm, right lower arm, left thigh, left calf, right thigh, and right calf.

17. A system for recognizing human attributes, the system comprising: means for applying an attribute feature extraction convolutional neural network (CNN) to a human image to generate attribute feature maps based on the image; means for applying a keypoint-guided proposal CNN to the human image to generate proposed hierarchical regions of the image based on associated anatomical key points, wherein the key points are associated with joints or points of connection between different regions of a human body and the proposed hierarchical regions are associated with a hierarchical structure of a human body; and means for generating recognition probabilities for the human attributes based on a CNN combination layer that merges the attribute feature maps, the proposed hierarchical regions, and a statistical correlation model that provides correlations between features of the attribute feature maps and the proposed hierarchical regions.

18. The system of claim 17, further comprising means for generating an initial set of statistical correlation models, the statistical correlation models comprising semantic correlation models for providing correlations between each of the attributes, structural correlation models for providing correlations between each of the proposed hierarchical regions, and location correlation models for providing correlations between each of the attributes and each of the proposed hierarchical regions.

19. The system of claim 18, wherein, Generation of the initial set of statistical correlation models is based on training images comprising attribute labels and keypoint labels, training data comprising anatomical models, and attribute recognition probabilities generated by the system in response to the training images.

20. The system of claim 17, further comprising means for training the attribute feature extraction CNN on training images comprising attribute labels.

21. The system of claim 17, further comprising means for training the keypoint-guided proposal CNN on a keypoint training dataset and training images comprising attribute labels.

22. The system of any one of claims 17-21, wherein, The human attributes comprise at least one of: gender, hairstyle, style of dress, hat, eyeglasses, and type of footwear.

23. The system of any one of claims 17-21, wherein, The key points comprise at least one of: neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

24. The system of any one of claims 17-21, wherein, The proposed hierarchical regions comprise at least one of: body, head, upper torso, lower torso, chest, left arm, right arm, left leg, right leg, left upper arm, left lower arm, right upper arm, right lower arm, left upper thigh, left lower thigh, right upper thigh, and right lower thigh.

Images