Robust kinship identification methods and devices that adapt to age and gender differences

Abstract

This application proposes a robust kinship recognition method that adapts to age and gender differences, comprising: acquiring image pairs to be identified; outputting feature maps of each image in the image pair through a backbone network and calculating the local correlation of the images through an attention mechanism; predicting the age and gender features of each image by passing the feature maps of each image through corresponding age prediction branches and gender prediction branches; determining a modulation factor based on face location encoding, age features, and gender features, and dynamically adjusting the local correlation of the images according to the modulation factor; fusing the feature maps according to the adjusted local correlation of the images, and performing kinship recognition based on the fusion result to obtain the recognition result. The present invention, employing the above scheme, effectively improves the accuracy of kinship recognition under different age and gender differences.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a robust kinship identification method and apparatus that adapts to age and gender differences. Background Technology

[0002] With the development of deep learning, more and more neural networks are being applied in social fields. Among them, the use of deep learning for kinship identification to assist in identity verification or crime solving is becoming increasingly important in the identification of missing persons and criminal investigations. Existing kinship identification models mainly extract features from two input images using a backbone network with shared weights, and then calculate the relationship between local or global features to predict whether the two images belong to relatives. However, in the process of determining kinship, the accuracy of existing kinship identification models drops significantly when there are obvious differences in gender and age between the two individuals. This is because when there are different degrees of gender and age differences between relatives, different facial features will have different degrees of feature shifts unrelated to identity, resulting in erroneous local correlations and causing incorrect kinship judgments. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, the first objective of this application is to propose a robust kinship identification method that adapts to age and gender differences, which solves the technical problem that the accuracy of kinship identification drops significantly when there are obvious age and gender differences in existing methods, and effectively improves the accuracy of kinship identification in the face of different age and gender differences.

[0005] The second objective of this application is to propose a robust kinship identification device that adapts to age and gender differences.

[0006] The third objective of this application is to propose a computer device.

[0007] The fourth objective of this application is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above objectives, the first aspect of this application proposes a robust kinship identification method that adapts to age and gender differences, comprising: acquiring image pairs to be identified; outputting feature maps of each image in the image pair through a backbone network and calculating the local correlation of the images through an attention mechanism; predicting the age and gender features of each image by passing the feature maps of each image through corresponding age prediction branches and gender prediction branches; determining the age and gender differences between the images based on the age and gender features; determining a modulation factor based on face location encoding, age features, and gender features, and dynamically adjusting the local correlation of the images based on the modulation factor; fusing the feature maps based on the adjusted local correlation of the images, and performing kinship identification based on the fusion result to obtain the identification result.

[0009] Optionally, in one embodiment of this application, the method further includes: learning age differences and gender differences between images;

[0010] The learning includes studying age and gender differences between images, including:

[0011] The age difference is determined by the cosine similarity of age features between images based on age features, and the gender difference is determined by the cosine similarity of gender features between images based on gender features.

[0012] Age difference and gender difference labels were designed based on kinship labels, and the design labels were trained using the age cosine similarity and gender cosine similarity of image pairs.

[0013] Optionally, in one embodiment of this application, determining a modulation factor based on face location encoding, age features, and gender features, and dynamically adjusting the local correlation of the image based on the modulation factor, includes:

[0014] The difference between the age and gender features of the image pairs is calculated to obtain the explicit age and gender differences.

[0015] Modulation factors are determined based on facial location coding and explicit age and gender differences;

[0016] The local correlation of the image calculated by the attention mechanism is linearly transformed based on the modulation factor to obtain the adjusted local correlation of the image.

[0017] To achieve the above objectives, a second aspect of the present invention provides a robust kinship identification device that adapts to age and gender differences, comprising:

[0018] Optionally, in one embodiment of this application, face location encoding learns regional information of facial features. Face location encoding includes age location encoding and gender location encoding. Modulation factors include age modulation factors and gender modulation factors. The modulation factors are determined based on the face location encoding and explicit age and gender differences, including:

[0019] The age modulation factor is determined based on age location coding and explicit age differences;

[0020] The gender modulation factor is determined based on gender position coding and explicit gender differences.

[0021] The data acquisition module is used to acquire image pairs to be identified;

[0022] The local relation determination module is used to output the feature map of each image in the image pair through the backbone network and calculate the local correlation of the images through the attention mechanism.

[0023] The age and gender feature prediction module is used to predict the age and gender features of each image by passing the feature map of each image through the corresponding age prediction branch and gender prediction branch.

[0024] A local relation modulation module that adapts to age and gender differences is used to determine modulation factors based on face location encoding, age features, and gender features, and to dynamically adjust the local correlation of the image based on the modulation factors.

[0025] The kinship recognition module is used to fuse feature maps based on the local correlation of the adjusted images, and to perform kinship recognition based on the fusion results to obtain the recognition results.

[0026] Optionally, in one embodiment of this application, the device further includes an age and gender difference learning module for learning age and gender differences between images, wherein learning age and gender differences between images includes:

[0027] The age difference is determined by the cosine similarity of age features between images based on age features, and the gender difference is determined by the cosine similarity of gender features between images based on gender features.

[0028] Age difference and gender difference labels were designed based on kinship labels, and the design labels were trained using the age cosine similarity and gender cosine similarity of image pairs.

[0029] Optionally, in one embodiment of this application, the local relationship modulation module that adapts to age and gender differences is specifically used for:

[0030] The difference between the age and gender features of the image pairs is calculated to obtain the explicit age and gender differences.

[0031] Modulation factors are determined based on facial location coding and explicit age and gender differences;

[0032] The local correlation of the image calculated by the attention mechanism is linearly transformed based on the modulation factor to obtain the adjusted local correlation of the image.

[0033] Optionally, in one embodiment of this application, face location encoding learns regional information of facial features. Face location encoding includes age location encoding and gender location encoding. Modulation factors include age modulation factors and gender modulation factors. The modulation factors are determined based on the face location encoding and explicit age and gender differences, including:

[0034] The age modulation factor is determined based on age location coding and explicit age differences;

[0035] The gender modulation factor is determined based on gender position coding and explicit gender differences.

[0036] To achieve the above objectives, a third aspect of the present invention provides another apparatus, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the above-described robust kinship identification method that adapts to age and gender differences.

[0037] To achieve the above objectives, a fourth aspect of the present invention provides a non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the execution of the robust kinship identification method that adapts to age and gender differences.

[0038] The robust kinship recognition method and apparatus that adapts to age and gender differences in this application learns the relative differences in age and gender between images only through kinship tags, greatly reducing the learning difficulty. At the same time, in order to prevent the feature differences of different facial parts caused by age and gender differences from affecting the judgment of kinship, a local relationship modulation module that adapts to age and gender differences is designed. It dynamically corrects the correlation of different local features based on the learned relative differences in age and gender, so as to avoid incorrect local correlation judgments caused by the relative differences in age and gender.

[0039] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0040] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0041] Figure 1This is a flowchart illustrating a robust kinship identification method that adapts to age and gender differences, as provided in Embodiment 1 of this application.

[0042] Figure 2 This is a schematic diagram illustrating the process of age and gender difference perception learning in an embodiment of this application;

[0043] Figure 3 This is a flowchart illustrating a local relation modulation method for attention mechanisms that adapts to age and gender differences, based on an embodiment of this application.

[0044] Figure 4 This is a schematic diagram of the structure of a robust kinship identification device that adapts to age and gender differences, as provided in an embodiment of this application. Detailed Implementation

[0045] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0046] The robust kinship identification method and apparatus that adapts to age and gender differences according to embodiments of this application are described below with reference to the accompanying drawings.

[0047] Figure 1 This is a flowchart illustrating a robust kinship identification method that adapts to age and gender differences, as provided in Embodiment 1 of this application. This kinship identification method can be embedded into any attention-based kinship identification method to improve its accuracy in addressing age and gender differences.

[0048] like Figure 1 As shown, this robust kinship identification method that adapts to age and gender differences includes the following steps:

[0049] Step 101: Obtain the image pair to be identified;

[0050] Step 102: Output feature maps of each image in the image pair through the backbone network, and calculate the local correlation of the images through the attention mechanism;

[0051] Step 103: Use the feature map of each image to predict the age and gender features of each image through the corresponding age prediction branch and gender prediction branch.

[0052] In this embodiment, before predicting age and gender features through the age and gender prediction branch, the model is trained to learn the perception of age and gender differences, and the model is trained to extract effective high-dimensional age and gender features.

[0053] Figure 2This is a schematic diagram of the learning process for age and gender difference perception in this embodiment.

[0054] like Figure 2 As shown, feature maps f1 and f2 of the input image pair are obtained using any depth vision module that processes images (such as ResNet50, ResNet101, etc.).

[0055] like Figure 2 As shown, an age prediction branch and a gender prediction branch are designed. Based on the feature maps f1 and f2 output by the backbone network, age and gender features are predicted. Since there are no accurate age and gender labels, this embodiment designs an age and gender difference perception learning method. Relative age and gender difference labels (1 or 0.2) are designed based on kinship labels, and the cosine similarity of age and gender features between image pairs is monitored. In this way, the age and gender information of a large amount of kinship data can be learned without real age and gender labels, which greatly reduces the difficulty of learning.

[0056] Step 105: Determine the modulation factor based on face location encoding, age features, and gender features, and dynamically adjust the local correlation of the image based on the modulation factor;

[0057] In this embodiment, to prevent feature shifts in different facial regions caused by age and gender differences, which could lead to erroneous local correlations, a local relation modulation method adapted to age and gender differences based on attention mechanisms is proposed. This method dynamically adjusts the local correlations of different facial regions by outputting modulation factors based on the positional encoding of different local facial regions and the learned age and gender differences.

[0058] by Figure 2 For example, based on Figure 2 The image pair shown has feature maps f1 and f2. The attention mechanism calculates the local correlation between each pixel in f1 and f2 for subsequent feature fusion.

[0059] Figure 3 This is a flowchart illustrating a local relation modulation method that adapts to age and gender differences in attention mechanisms.

[0060] like Figure 3 As shown, a learnable location code is first defined, which learns regional information of facial features and can effectively distinguish different facial features. Then, the difference between the age and gender features of the image pairs is performed to represent explicit age and gender differences. The relation modulation module then outputs a modulation factor based on the location code and age and gender differences, and performs a linear transformation on the local correlations calculated by the attention mechanism to correct erroneous local correlations caused by age and gender differences.

[0061] like Figure 3 As shown, by using the difference operation, when the model can effectively extract high-dimensional features of age and gender, the feature difference operation is used to capture more dimensional and richer difference information to generate the modulation factor.

[0062] like Figure 3 As shown, the location encoding includes age encoding and gender encoding. Both age location encoding and gender location encoding are randomly initialized C*H*W matrices, where C is the feature dimension, and H and W are the dimensions of the local correlation map of the face. Therefore, each element represents the correlation between a certain face part and other locations. The age and gender differences are both C*1 vectors.

[0063] For the age modulation factor, the age difference of C*1 is copied H*W times to obtain a matrix of C*H*W. Then, it is multiplied element-wise with the age position code to obtain a matrix of C*H*W. Finally, it is passed through multiple convolutional layers to obtain a multiplicative modulation factor of 1*H*W.

[0064] The process for gender modulation factor is the same as that for age modulation factor, obtaining an additive modulation factor of 1*H*W through multiple convolutional layers.

[0065] Still with Figure 2 For example, based on Figure 2 The feature maps f1 and f2 of the image pair shown are processed by correct local correlation to obtain corresponding feature output 1 and feature output 2. Feature fusion is performed based on feature output 1 and feature output 2, and kinship is determined based on the fused features. By modulating local correlation, the influence of age and gender differences is effectively avoided, and the accuracy of the kinship recognition model in the face of different age and gender differences is improved.

[0066] Step 106: The feature maps are fused according to the local correlation of the adjusted images, and kinship identification is performed based on the fusion results to obtain the identification results.

[0067] The robust kinship recognition method adapted to age and gender differences in this application improves the accuracy of kinship recognition models when faced with different age and gender differences. Specifically, since accurate age and gender labels are difficult to obtain directly, the model cannot directly and accurately output the age and gender of a person's image to obtain age and gender differences. To address this, this embodiment designs an age and gender difference perception learning method, defining relative age and relative gender difference labels only through kinship labels to supervise the cosine similarity between the age and gender features output by the model. This method can easily learn the age and gender differences of input image pairs, reducing the difficulty of model learning. Furthermore, this embodiment designs a local relationship modulation module adapted to age and gender differences for the attention mechanism. Specifically, the attention mechanism calculates the correlation between each pixel. To utilize the learned age and gender differences to reduce erroneous local correlations, a learnable positional encoding is first defined to identify different local facial regions. Then, the local relationship modulation module outputs a modulation factor based on the positional encoding and age and gender differences, performing a linear transformation on the correlations calculated by the attention mechanism to obtain more accurate local correlations for determining kinship.

[0068] To achieve the above embodiments, this application also proposes a robust kinship identification device that adapts to age and gender differences.

[0069] Figure 4 This is a schematic diagram of the structure of a robust kinship identification device that adapts to age and gender differences, as provided in an embodiment of this application.

[0070] like Figure 4 As shown, this robust kinship identification device that adapts to age and gender differences includes:

[0071] The data acquisition module is used to acquire image pairs to be identified;

[0072] The local relation determination module is used to output the feature map of each image in the image pair through the backbone network and calculate the local correlation of the images through the attention mechanism.

[0073] The age and gender feature prediction module is used to predict the age and gender features of each image by passing the feature map of each image through the corresponding age prediction branch and gender prediction branch.

[0074] A local relation modulation module that adapts to age and gender differences is used to determine modulation factors based on face location encoding, age features, and gender features, and to dynamically adjust the local correlation of the image based on the modulation factors.

[0075] The kinship recognition module is used to fuse feature maps based on the local correlation of the adjusted images, and to perform kinship recognition based on the fusion results to obtain the recognition results.

[0076] Optionally, in one embodiment of this application, the device further includes an age and gender difference learning module for learning age and gender differences between images, wherein learning age and gender differences between images includes:

[0077] The age difference is determined by the cosine similarity of age features between images based on age features, and the gender difference is determined by the cosine similarity of gender features between images based on gender features.

[0078] Age difference and gender difference labels were designed based on kinship labels, and the design labels were trained using the age cosine similarity and gender cosine similarity of image pairs.

[0079] Optionally, in one embodiment of this application, the local relationship modulation module that adapts to age and gender differences is specifically used for:

[0080] The difference between the age and gender features of the image pairs is calculated to obtain the explicit age and gender differences.

[0081] Modulation factors are determined based on facial location coding and explicit age and gender differences;

[0082] The local correlation of the image calculated by the attention mechanism is linearly transformed based on the modulation factor to obtain the adjusted local correlation of the image.

[0083] Optionally, in one embodiment of this application, face location encoding learns regional information of facial features. Face location encoding includes age location encoding and gender location encoding. Modulation factors include age modulation factors and gender modulation factors. The modulation factors are determined based on the face location encoding and explicit age and gender differences, including:

[0084] The age modulation factor is determined based on age location coding and explicit age differences;

[0085] The gender modulation factor is determined based on gender position coding and explicit gender differences.

[0086] It should be noted that the foregoing explanation of the embodiment of the robustly age-gender-adaptive kinship identification method also applies to the robustly age-gender-adaptive kinship identification device of this embodiment, and will not be repeated here.

[0087] To implement the above embodiments, the present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the methods described in the above embodiments.

[0088] To implement the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method of the above embodiments.

[0089] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0090] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0091] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0092] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0093] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0094] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0096] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A robust kinship identification method that adapts to age and gender differences, characterized in that, include: Obtain the image pairs to be identified; The backbone network outputs feature maps for each image in the image pair, and the local correlation of the images is calculated through an attention mechanism. The feature map of each image is used to predict the age and gender features of each image through the corresponding age prediction branch and gender prediction branch. The modulation factor is determined based on face location encoding, age features, and gender features, and the local correlation of the image is dynamically adjusted based on the modulation factor. The feature maps are fused based on the local correlation of the adjusted images, and kinship identification is performed based on the fusion results to obtain the identification results; The step of determining the modulation factor based on face location encoding, age features, and gender features, and dynamically adjusting the local correlation of the image based on the modulation factor, includes: The difference between the age and gender features of the image pairs is calculated to obtain the explicit age and gender differences. Modulation factors are determined based on facial location coding and explicit age and gender differences; The local correlation of the image calculated by the attention mechanism based on the modulation factor is linearly transformed to obtain the adjusted local correlation of the image. The face location encoding learns regional information of the face, including age location encoding and gender location encoding. The modulation factor includes an age modulation factor and a gender modulation factor. The modulation factor is determined based on the face location encoding and explicit age and gender differences, including: The age modulation factor is determined based on age location coding and explicit age differences; The gender modulation factor is determined based on gender position coding and explicit gender differences.

2. The method as described in claim 1, characterized in that, The method also includes: learning age and gender differences between images; The age and gender differences between the learned images include: The age difference is determined by the cosine similarity of age features between images based on age features, and the gender difference is determined by the cosine similarity of gender features between images based on gender features. Age difference and gender difference labels were designed based on kinship labels, and the design labels were trained using the age cosine similarity and gender cosine similarity of image pairs.

3. A robust kinship identification device that adapts to age and gender differences, characterized in that, include: The data acquisition module is used to acquire image pairs to be identified; The local relation determination module is used to output the feature map of each image in the image pair through the backbone network and calculate the local correlation of the images through the attention mechanism. The age and gender feature prediction module is used to predict the age and gender features of each image by passing the feature map of each image through the corresponding age prediction branch and gender prediction branch. A local relation modulation module that adapts to age and gender differences is used to determine modulation factors based on face location encoding, age features, and gender features, and to dynamically adjust the local correlation of the image based on the modulation factors. The kinship recognition module is used to fuse feature maps based on the local correlation of the adjusted image, and to perform kinship recognition based on the fusion result to obtain the recognition result; Specifically, the local relationship modulation module that adapts to age and gender differences is used for: The difference between the age and gender features of the image pairs is calculated to obtain the explicit age and gender differences. Modulation factors are determined based on facial location coding and explicit age and gender differences; The local correlation of the image calculated by the attention mechanism based on the modulation factor is linearly transformed to obtain the adjusted local correlation of the image. The face location encoding learns regional information of the face, including age location encoding and gender location encoding. The modulation factor includes an age modulation factor and a gender modulation factor. The modulation factor is determined based on the face location encoding and explicit age and gender differences, including: The age modulation factor is determined based on age location coding and explicit age differences; The gender modulation factor is determined based on gender position coding and explicit gender differences.

4. The apparatus as described in claim 3, characterized in that, The device further includes an age and gender difference learning module for learning age and gender differences between images, wherein the learning of age and gender differences between images includes: The age difference is determined by the cosine similarity of age features between images based on age features, and the gender difference is determined by the cosine similarity of gender features between images based on gender features. Age difference and gender difference labels were designed based on kinship labels, and the design labels were trained using the age cosine similarity and gender cosine similarity of image pairs.

5. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-2.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-2.

Images