Methods, apparatus, computing devices and storage media for verifying kinship

Abstract

This application provides a method, apparatus, computing device, and storage medium for kinship verification, comprising: acquiring at least two images; inputting each image into a feature extraction model to obtain output features of multiple specified residual blocks for each image and global features of individuals for each image; for any image, combining the output features of the multiple specified residual blocks to obtain a first combined feature; inputting the first combined feature into a local attention model to obtain local features of individuals; combining the local features of individuals and the global features of individuals for each image to obtain a second combined feature; inputting the second combined feature into a kinship verification model to obtain kinship verification results between individuals. By extracting significantly different local features of individuals from each image, combining them with the global features of individuals, and then inputting them into the kinship verification model, accurate kinship verification results are obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for verifying a kinship relationship. The present application also relates to a device for verifying a kinship relationship, a computing device, and a computer-readable storage medium. BACKGROUND

[0002] Verifying a kinship relationship is an important image recognition technology. The method for verifying a kinship relationship refers to using image recognition technology to verify whether the persons in a to-be-tested image have a kinship relationship. Verifying a kinship relationship can help people find missing children, handle case disputes, and the like. Verifying a kinship relationship is a research field that is currently extremely challenging and has attracted widespread attention from researchers. Research results can be widely applied in various fields such as national economy and people's livelihood. How to efficiently and accurately obtain a kinship relationship between different persons has become a research focus in the above-mentioned fields. SUMMARY

[0003] In view of this, the embodiments of the present application provide a method for verifying a kinship relationship to solve the technical defects in the prior art. The embodiments of the present application also provide a device for verifying a kinship relationship, a computing device, and a computer-readable storage medium.

[0004] According to a first aspect of the embodiments of the present application, a method for verifying a kinship relationship is provided, comprising:

[0005] obtaining at least two images, wherein each image includes different persons;

[0006] inputting each image into a feature extraction model to obtain output features of a plurality of specified residual blocks of each image and a global feature of a person in each image;

[0007] for any image, combining the output features of the plurality of specified residual blocks of the image to obtain first combined features, inputting the first combined features into a local attention model to obtain a local feature of a person, wherein the local attention model is a multi-head attention model;

[0008] combining the local features of the persons in the images and the global features of the persons in the images to obtain second combined features;

[0009] inputting the second combined features into a kinship relationship verification model to obtain a kinship relationship verification result between the persons.

[0010] Optionally, the step of obtaining at least two images comprises:

[0011] obtaining at least two initial images, wherein each initial image includes different persons;

[0012] Based on the character, each initial image is preprocessed to obtain at least two images.

[0013] Optionally, the number of feature extraction models is multiple, and each feature extraction model has the same model structure and the same model parameter.

[0014] The step of inputting each image into the feature extraction model to obtain the output features of the multiple specified residual blocks of each image and the character global feature of each image includes:

[0015] The step of inputting each image into the feature extraction model to obtain the output features of the multiple specified residual blocks of each image and the character global feature of each image includes:

[0016] Optionally, for any image, the step of combining the output features of the multiple specified residual blocks of the image to obtain the first combined feature includes:

[0017] For any image, the step of inputting the output features of the multiple specified residual blocks of the image into the channel layer to cross-combine the output features of the multiple specified residual blocks of the image through the channel layer to obtain the first combined feature.

[0018] Optionally, the local attention model includes a spatial attention sub-model and an attention learning sub-model.

[0019] The step of inputting the first combined feature into the local attention model to obtain the character local feature includes:

[0020] Channel shuffling is performed on the first combined feature.

[0021] Feature fusion is performed on the channel-shuffled first combined feature using a 1*1 convolution kernel to obtain a fused feature.

[0022] The fused feature is split to obtain a preset number of split features.

[0023] Each split feature is input into the spatial attention sub-model to obtain a spatial attention matrix corresponding to the split feature.

[0024] Each spatial attention matrix is input into the attention learning sub-model to obtain multiple character local features.

[0025] Optionally, the step of inputting each spatial attention matrix into the attention learning sub-model to obtain multiple character local features includes:

[0026] A column-wise summation operation is performed on the first spatial attention matrix, and a second spatial attention matrix is constructed based on the summation result, wherein the first spatial attention matrix is any one of the spatial attention matrices.

[0027] Transposing the second spatial attention matrix to obtain a third spatial attention matrix;

[0028] Determining a maximum value in the third spatial attention matrix as a matrix center of the first spatial attention matrix;

[0029] According to the matrix center of each spatial attention matrix, the local features of the person corresponding to each spatial attention matrix are determined under the constraint of a preset loss function, wherein the constraint of the preset loss function includes that each weight value in the same spatial attention matrix converges to the matrix center of the spatial attention matrix, and the matrix centers of different spatial attention matrices are separated from each other.

[0030] Optionally, the step of combining the local features of the person of each image and the global features of the person of each image to obtain the second combined features comprises:

[0031] Combining the k local features of the person of each image to obtain k local combined features of the person;

[0032] Combining the global features of the person of each image to obtain 1 global combined feature of the person;

[0033] Combining the k local combined features of the person and the global combined feature of the person to obtain the second combined features.

[0034] Optionally, the training manner of the kinship verification model comprises:

[0035] Obtaining a training sample set, wherein the training sample set comprises a plurality of training images, and each training image comprises a sample person, a sample family label of the sample person and a sample person label;

[0036] Dividing the training sample set to obtain a plurality of training subsets, wherein each training subset comprises a plurality of training images;

[0037] Inputting each training image in a first training subset into an initial verification model to obtain distance information between each training image in the first training subset and a class center, a predicted family class of each training image and a predicted person class, wherein the class center is updated based on a preset class center update function in a calculation process, and the first training subset is any one of the plurality of training subsets;

[0038] According to the distance information between each training image in the first training subset and the class center, a preset family-level multi-center loss function is used to calculate a family-level multi-center loss value;

[0039] According to the predicted family class of each training image and the sample family label, a family classification loss value is calculated;

[0040] According to the predicted person category of each training image and the sample person label, a person classification loss value is calculated;

[0041] According to the family-level multi-center loss value, the family classification loss value and the person classification loss value, a total loss value is calculated;

[0042] Based on the total loss value, the model parameters of the initial verification model are adjusted, and the step of inputting each training image in the first training subset into the initial verification model to obtain the distance information between each training image and the class center, the predicted family category and the predicted person category of each training image in the first training subset is executed until a preset training stop condition is reached, and a trained kinship verification model is obtained.

[0043] According to a second aspect of the embodiment of the present application, a kinship verification device is provided, comprising:

[0044] An acquisition module is configured to acquire at least two images, wherein each image includes different persons;

[0045] A feature extraction module is configured to input each image into a feature extraction model respectively to obtain output features of a plurality of specified residual blocks of each image and person global features of each image;

[0046] An input module is configured to, for any image, combine the output features of the plurality of specified residual blocks of the image to obtain first combined features, and input the first combined features into a local attention model to obtain person local features, wherein the local attention model is a multi-head attention model;

[0047] A feature combination module is configured to combine the person local features of each image and the person global features of each image to obtain second combined features;

[0048] A verification module is configured to input the second combined features into a kinship verification model to obtain kinship verification results between each person.

[0049] Optionally, the acquisition module is further configured to:

[0050] Acquire at least two initial images, wherein each initial image includes different persons;

[0051] Based on the persons, each initial image is preprocessed to obtain at least two images.

[0052] Optionally, the number of feature extraction models is a plurality, and the model structures and the model parameters of each feature extraction model are the same;

[0053] The feature extraction module is further configured to:

[0054] The images are input into the feature extraction models in parallel to obtain output features of the multiple specified residual blocks of the images and global features of the characters in the images.

[0055] Optionally, the input module is further configured to:

[0056] For any image, the output features of the multiple specified residual blocks of the image are input into the channel layer, the output features of the multiple specified residual blocks of the image are cross combined by the channel layer to obtain first combined features.

[0057] Optionally, the local attention model comprises a spatial attention sub-model and an attention learning sub-model.

[0058] The input module is further configured to:

[0059] The first combined features are subjected to channel shuffling.

[0060] The first combined features subjected to channel shuffling are subjected to feature fusion by using a 1*1 convolution kernel to obtain fused features.

[0061] The fused features are split to obtain a preset number of split features.

[0062] The split features are respectively input into the spatial attention sub-model to obtain spatial attention matrices corresponding to the split features.

[0063] The spatial attention matrices are respectively input into the attention learning sub-model to obtain multiple character local features.

[0064] Optionally, the input module is further configured to:

[0065] The first spatial attention matrix is subjected to column-wise summation operation, and a second spatial attention matrix is constructed based on the summation result, wherein the first spatial attention matrix is any one of the spatial attention matrices.

[0066] The second spatial attention matrix is subjected to transposition operation to obtain a third spatial attention matrix.

[0067] The maximum value in the third spatial attention matrix is determined as the matrix center of the first spatial attention matrix.

[0068] According to the matrix centers of the spatial attention matrices, the character local features corresponding to the spatial attention matrices are respectively determined under the constraint of a preset loss function, wherein the constraint of the preset loss function comprises that each weight value in the same spatial attention matrix converges to the matrix center of the spatial attention matrix, and the matrix centers of different spatial attention matrices are separated from each other.

[0069] Optionally, the feature combination module is further configured to:

[0070] combining the k person local features of each image to obtain k person local combined features;

[0071] combining the person global features of each image to obtain 1 person global combined feature;

[0072] combining the k person local combined features and the person global combined feature to obtain a second combined feature.

[0073] Optionally, the apparatus further comprises a training module;

[0074] The training module is configured to:

[0075] obtain a training sample set, wherein the training sample set comprises a plurality of training images, and each training image comprises a sample person, a sample family label of the sample person, and a sample person label;

[0076] divide the training sample set to obtain a plurality of training subsets, wherein each training subset comprises a plurality of training images;

[0077] input each training image in a first training subset into an initial verification model to obtain distance information between each training image in the first training subset and a class center, a predicted family class of each training image, and a predicted person class, wherein the class center is updated based on a preset class center update function in a calculation process, and the first training subset is any one of the plurality of training subsets;

[0078] calculate a family-level multi-center loss value by using a preset family-level multi-center loss function according to the distance information between each training image in the first training subset and the class center;

[0079] calculate a family classification loss value according to the predicted family class of each training image and the sample family label;

[0080] calculate a person classification loss value according to the predicted person class of each training image and the sample person label;

[0081] calculate a total loss value according to the family-level multi-center loss value, the family classification loss value, and the person classification loss value;

[0082] adjust model parameters of the initial verification model based on the total loss value, and return to execute the step of inputting each training image in the first training subset into the initial verification model to obtain the distance information between each training image in the first training subset and the class center, the predicted family class of each training image, and the predicted person class until a preset training stop condition is reached, to obtain a trained kinship verification model.

[0083] According to a third aspect of the embodiments of the present application, a computing device is provided, comprising a memory and a processor.

[0084] The memory is configured to store computer-executable instructions, and the processor is configured to implement the above-mentioned kinship verification method when executing the computer-executable instructions.

[0085] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer instructions, and the computer instructions are executed by a processor to implement the above-mentioned kinship verification method.

[0086] According to a fifth aspect of the embodiments of the present application, a chip is provided, which stores computer instructions, and the computer instructions are executed by the chip to implement the above-mentioned kinship verification method.

[0087] The kinship verification method provided by the present application, by obtaining at least two images, inputting each image into a feature extraction model respectively, obtaining the output features of a plurality of specified residual blocks of each image and the global features of the person in each image, then for any image, combining the output features of the plurality of specified residual blocks of the image to obtain first combined features, inputting the first combined features into a local attention model to obtain local features of the person, and then combining the local features of the person in each image and the global features of the person in each image to obtain second combined features, inputting the second combined features into a kinship verification model to obtain the kinship verification result between each person. Through the above-mentioned kinship verification method based on deep learning, efficient verification of the kinship between persons is realized. By using the local attention model, the local features with significant differences in each image can be automatically extracted without image key point detection. The local features of the person with significant differences in each image and the global features of the person are combined respectively and input into the kinship verification model, which can make the output result of the kinship verification model more accurate and ensure the accuracy of the kinship verification result. BRIEF DESCRIPTION OF DRAWINGS

[0088] Figure 1 A flowchart of a kinship verification method according to an embodiment of the present application is shown;

[0089] Figure 2 A structural schematic diagram of a local attention model according to an embodiment of the present application is shown;

[0090] Figure 3 An effect schematic diagram of an attention learning sub-model loss function according to an embodiment of the present application is shown;

[0091] Figure 4 A distribution schematic diagram of a single-center feature space and a multi-center feature space according to an embodiment of the present application is shown;

[0092] Figure 5A structural diagram of a kinship verification model is shown according to an embodiment of the present application.

[0093] Figure 6 A structural diagram of a kinship verification device is shown according to an embodiment of the present application.

[0094] Figure 7 A structural block diagram of a computing device is shown according to an embodiment of the present application. DETAILED DESCRIPTION

[0095] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present application. Some embodiments of the present application can be practiced in a variety of ways, and the description below is not intended to limit the present application.

[0096] The terminology used in this disclosure of one or more embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present application. As used in this disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0097] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only as a shorthand notation to distinguish one item from another. For example, in the present disclosure, first can be termed second and, similarly, second can be termed first, without departing from the scope of one or more embodiments of the present application.

[0098] First, the noun terms related to one or more embodiments of the present application are explained.

[0099] Residual Network (ResNet): Residual in mathematical statistics refers to the difference between the actual observation value and the estimated value, and the residual network is composed of a series of residual blocks. In the process of training the neural network model, under the premise that the neural network can converge, as the depth of the neural network increases, the neural network model degradation occurs. Adding a residual network to the neural network model effectively solves the problem of neural network degradation and has been widely used in the fields of image recognition and image segmentation.

[0100] Metric learning: a method of space mapping, which can learn a feature space, in which all data are converted into a feature vector, and the feature vectors between similar samples are small, and the feature vectors between dissimilar samples are large, thereby distinguishing the data.

[0101] Multi-head attention model: a multi-head attention model is composed of multiple attention heads, and each attention head focuses on different features of the input information.

[0102] Face recognition technology is an important biometric technology, which is widely used in various fields of people's daily life. The relative relationship verification based on face images refers to using image processing technology to verify whether the to-be-tested face image pair has a relative relationship. Generally, 1:1 face relative relationship verification is to input two face images, and obtain whether the two input images have a relative relationship through a relative relationship verification algorithm. Face relative relationship verification includes three kinds of relative pairs: the relative relationship verification between grandparents and grandchildren is called intergenerational relative relationship verification; the relative relationship verification between parents and children is called next-generation relative relationship verification; and the relative relationship verification between brothers and sisters is called same-generation relative relationship verification. Face relative relationship verification is a specific application of face recognition technology, and is more difficult than face recognition. It is more difficult to collect face image data with complete relative relationships, resulting in that face relative relationship verification is a great challenge. Face relative relationship verification has important application value, and using this technology can help find missing children, analyze family images to extract family relationships, and aggregate family images in an album.

[0103] However, the existing face relative relationship verification methods are all based on metric learning methods, but using one metric learning feature center for different members in a family will cause all members in the same family to be verified as having a relative relationship, but the husband and wife do not have a relative relationship, and it is unreasonable to identify them as having a relative relationship. In the face relative relationship training, a family-level multi-center loss function is added to each family to separate the family members who do not have a relative feature.

[0104] In addition, due to the differences between genders and ages, the face features of relatives extracted by a general face recognition model have large differences, and the general relative relationship verification method compares and calculates the differences between the face features of the pair using a metric function, which cannot model the relative face features and has low accuracy.

[0105] Based on the above technical problems, in the present application, a relative relationship verification method is provided. The present application also relates to a relative relationship verification device, a computing device, and a computer readable storage medium, which are described in detail one by one in the following embodiments.

[0106] Figure 1 A flowchart of a method for verifying a kinship relationship is shown, according to an embodiment of the present application, and specifically includes the following steps:

[0107] S102: Obtain at least two images.

[0108] The at least two images include different persons, where a person refers to an object to be verified for a kinship relationship. When verifying a kinship relationship for at least two different persons, first, images including different persons are obtained, and then a kinship relationship is verified based on the at least two images including different persons. The format of the images can be.jpg,.png,.raw, etc., and the format of the images is not specifically limited in the present specification.

[0109] There are various ways to obtain the at least two images. The user can directly input the at least two images to the execution subject for kinship relationship verification, or the user can pre-store the at least two images in a database of a server or a terminal. When the user sends a kinship relationship verification instruction to the execution subject, the execution subject obtains the images from the database of the server or the terminal for kinship relationship verification. The way to obtain the at least two images is not limited in the present specification.

[0110] In an implementation manner of the embodiment of the present specification, step S102 can be specifically implemented in the following manner:

[0111] Obtain at least two initial images, where each initial image includes different persons;

[0112] Preprocess each initial image based on the persons included in the initial image, to obtain at least two images.

[0113] When verifying a kinship relationship for at least two different persons, first, images including different persons are obtained as initial images for kinship relationship verification, and then the at least two obtained initial images are preprocessed based on the persons included in each initial image, to obtain at least two images including different persons.

[0114] The reason for preprocessing the at least two initial images is that in each initial image, there are image regions other than persons, and the positions of the persons in the initial images are different, which is not conducive to extracting the features of the persons in the process of kinship relationship verification, and thus leads to inaccurate kinship relationship verification results of the images.

[0115] Preprocessing the at least two initial images based on the persons included in each initial image refers to cutting and correcting the at least two initial images.

[0116] Specifically, by cropping the at least two initial images, image regions other than the person in the at least two initial images can be removed, and only image regions including the person in each initial image are reserved. The at least two initial images can be cropped by using a face detection algorithm, for example, a multi-task cascaded convolutional neural network (MTCNN). After the at least two initial images are cropped, the person in each initial image needs to be corrected. The cropped initial images can be adjusted by using a face key point detection algorithm. For example, when the key points in the cropped at least two initial images are not on the same horizontal line by using the face key point detection algorithm, the at least two initial images are processed by scaling, rotating, stretching, and the like, so that the key points in the at least two initial images are adjusted to be on the same horizontal line. By the above method, when the relationship between the persons is verified, the positions of the persons input into the relationship verification model are consistent, which is beneficial to the extraction of the features of the persons, thereby improving the accuracy of the relationship verification result.

[0117] In the above embodiment, at least two initial images are obtained, and then each initial image is preprocessed based on the person included in the initial image, and at least two images are obtained. By the above method, each initial image is cropped and corrected, which is beneficial to the extraction of the feature information of the images when the relationship between at least two different persons is verified, thereby improving the accuracy of the relationship verification result.

[0118] In S104, each image is input into a feature extraction model to obtain output features of a plurality of specified residual blocks of each image and a person global feature of each image.

[0119] The feature extraction model is a network model for extracting the feature information of the person in the image. Any network model capable of extracting the feature information of the person in the image can be used as the feature extraction model, and the type of the feature extraction model is not limited in the present application.

[0120] In the present embodiment, the feature extraction model can be a residual network (ResNet) model. Specifically, the feature of the person in the image can be extracted by using a ResNet50 residual network structure of a convolutional neural network model (CNN) combined with the ResNet.

[0121] In the residual network structure of the ResNet50, a plurality of residual blocks are included, and the residual blocks are used to extract high-level features and low-level features of the images, so that the high-level features and the low-level features of the images are fused by using convolution kernels in the CNN to obtain local features of the person in each image.

[0122] In the feature extraction model, a global average pooling layer (GAP, Global Average Pooling) is further included, and the global average pooling layer is used to extract global features of the person in the image. By using the global average pooling layer to extract the global features of the person in the image, the parameter information in the feature extraction model can be reduced, and the phenomenon of overfitting of the feature extraction model can be avoided.

[0123] In an implementation manner of the embodiment of the present specification, the number of the feature extraction models is multiple, and the model structures and the model parameters of the feature extraction models are the same. Correspondingly, the step S104 can be implemented by the following manner:

[0124] The images are input into the feature extraction models in parallel to obtain the output features of the plurality of specified residual blocks of the images and the global features of the person in the images.

[0125] After the at least two initial images are preprocessed to obtain the at least two images, the execution subject inputs the images into the feature extraction models respectively to obtain the output features of the plurality of specified residual blocks of the images and the global features of the person in the images.

[0126] The execution subject inputs the images into the feature extraction models respectively, which can use a serial manner to input the images into the feature extraction models in sequence, and based on the feature extraction models, the output features of the plurality of specified residual blocks of the images and the global features of the person in the images are obtained in sequence. Alternatively, a plurality of feature extraction models with the same structure and the same parameters are established at the same time, and then the images are input into the feature extraction models in parallel, and then the output features of the plurality of specified residual blocks of the images and the global features of the person in the images are obtained at the same time.

[0127] In the embodiment, a plurality of feature extraction models with the same structure and the same parameters are established at the same time, and the number of the feature extraction models depends on the number of the images. Then, the at least two preprocessed images are input into the feature extraction models in parallel, the high-level features and the low-level features of the images are extracted by using the plurality of specified residual blocks in the feature extraction models, and the global features of the person in the images are extracted by using the global average pooling layer in the feature extraction models.

[0128] In the above embodiment, the output features of the multiple specified residual blocks of each image and the global features of the person in each image are obtained by inputting each image into each feature extraction model in parallel. By inputting each image into multiple feature extraction models with the same structure and parameters in parallel, the output features of the multiple specified residual blocks of each image and the global features of the person in each image can be obtained simultaneously, thereby improving the efficiency of feature extraction of the person. Meanwhile, inputting each image into the global average pooling layer of the feature extraction model for extraction of the global features of the person can reduce the parameter information in the feature extraction model and avoid overfitting of the feature extraction model.

[0129] In step S106, for any image, the output features of the multiple specified residual blocks of the image are combined to obtain first combined features, and the first combined features are input into a local attention model to obtain local features of the person.

[0130] After obtaining the output features of the multiple specified residual blocks of each image by using the feature extraction model, for any image, the output features of the multiple specified residual blocks of the image are combined, and then the combined output features of the multiple specified residual blocks are input into the local attention model to obtain the local features of the person in each image.

[0131] In this embodiment, the local attention model is a multi-head attention model, which includes multiple attention heads, each of which focuses on different output features of the combined multiple specified residual blocks. By using the multi-head attention model, the significant difference regions of the person in the image can be automatically identified without key point detection, and then the local features of the different significant difference regions of the person in each image are extracted, thereby improving the accuracy of the kinship verification.

[0132] In one implementation manner of the embodiments of the present disclosure, the step of combining the output features of the multiple specified residual blocks of any image to obtain first combined features in step S106 can be implemented in the following manner:

[0133] For any image, the output features of the multiple specified residual blocks of the image are input into a channel layer, and the output features of the multiple specified residual blocks of the image are cross combined by the channel layer to obtain first combined features.

[0134] In this embodiment, after the high-level features and low-level features of any image are extracted by using the multiple specified residual blocks in the feature extraction model, the high-level features and low-level features of any image are input into the channel layer of the feature extraction model for cross combination, and the cross combined high-level features and low-level features are obtained as first combined features.

[0135] In the above embodiment, the high-level features and low-level features of any image extracted by the plurality of specified residual blocks in the feature extraction model are very small and scattered, which is not conducive to further extracting the local features of the image. Therefore, the high-level features and low-level features of any image extracted by the plurality of specified residual blocks need to be cross combined in the channel layer of the feature extraction model, so as to obtain the first combined features after combination, which is more conducive to subsequent extraction of the local features of the image.

[0136] In an implementation form of the embodiment of the present specification, the local attention model includes a spatial attention sub-model and an attention learning sub-model. Correspondingly, the step of inputting the first combined features into the local attention model in step S106 to obtain the local features of the person can be implemented by the following manner:

[0137] performing channel shuffling on the first combined features;

[0138] performing feature fusion on the first combined features after channel shuffling by using a 1*1 convolution kernel to obtain fused features;

[0139] splitting the fused features to obtain a preset number of split features;

[0140] inputting each split feature into the spatial attention sub-model to obtain a spatial attention matrix corresponding to each split feature;

[0141] inputting each spatial attention matrix into the attention learning sub-model to obtain a plurality of local features of the person.

[0142] Figure 2 A structure diagram of a local attention model according to an embodiment of the present application is shown, which includes multi-scale combined features, a channel layer, a 1*1 convolution kernel, a 3*3 convolution kernel, a spatial attention sub-model (SAM, Spatial Attention Mechanisms), a spatial attention matrix, and an attention learning sub-model (ALM, Attention Learning Module).

[0143] In the present embodiment, after obtaining the first combined features, the first combined features are subjected to channel shuffling, and the first combined features after channel shuffling are further fused by using a 1*1 convolution kernel in the feature extraction model to obtain fused features of at least two images. Then, the fused features of the at least two images are averagely split into a preset number of split features in the channel layer of the feature extraction model. The first combined features can be multi-scale combined features.

[0144] Then, each split feature is input into a spatial attention sub-model, a multi-head attention mechanism in the spatial attention sub-model is used to obtain a spatial attention matrix corresponding to each split feature, and the spatial attention matrix corresponding to each split feature is input into an attention learning sub-model to obtain multiple local features of the persons in the at least two images.

[0145] In the above embodiment, the first combined feature is further fused by using a 1*1 convolution kernel to obtain a fused feature, the fused feature is input into a channel layer to split into a preset number of split features, each split feature is input into a spatial attention sub-model, and a multi-head attention mechanism is used to identify image regions that are significantly different in the at least two images, thereby facilitating the verification of the kinship relationship of the images.

[0146] In an implementation form of the embodiment of the present specification, the step of inputting each spatial attention matrix into an attention learning sub-model to obtain multiple local features of the persons can be implemented in the following manner:

[0147] The first spatial attention matrix is subjected to column-wise summation operation, and a second spatial attention matrix is constructed based on the summation result, wherein the first spatial attention matrix is any one of the spatial attention matrices;

[0148] The second spatial attention matrix is subjected to transposition operation to obtain a third spatial attention matrix;

[0149] The maximum value in the third spatial attention matrix is determined as the matrix center of the first spatial attention matrix;

[0150] According to the matrix center of each spatial attention matrix, the local feature of the person corresponding to each spatial attention matrix is determined under the constraint of a preset loss function, wherein the constraint of the preset loss function includes that each weight value in the same spatial attention matrix converges to the matrix center of the spatial attention matrix, and the matrix centers of different spatial attention matrices are separated from each other.

[0151] Figure 3 An attention learning sub-model loss function effect diagram provided by an embodiment of the present application is shown.

[0152] In the present embodiment, after a preset number of split features are input into a spatial attention sub-model (SAM, Spatial Attention Module), a spatial attention matrix corresponding to each split feature is obtained, and then k spatial attention matrices are input into an attention learning sub-model (ALM, Attention Learning Module), wherein the spatial attention matrix is a result processed by a row-wise softmax function, therefore, the row-wise softmax function is first applied to each spatial attention matrix A li The operation S of summing by column, and then constructing the spatial attention matrix based on the sum result , and specifically, constructing the spatial attention matrix As shown in formula (1).

[0153] (1)

[0154] After obtaining the spatial attention matrix , a transposition operation is performed on the spatial attention matrix to obtain the spatial attention matrix :

[0155] (2)

[0156] After obtaining the spatial attention matrix , the maximum value in the spatial attention matrix is determined as the matrix center of the spatial attention matrix A:

[0157] (3)

[0158] Wherein, S represents summing the spatial attention matrix; R represents the Reshape operation.

[0159] After determining the matrix center of the spatial attention matrix, the local features of the characters corresponding to each spatial attention matrix are determined according to the matrix center under the constraint of a preset loss function, wherein the constraint of the preset loss function includes that the weight values in the same spatial attention matrix converge to the matrix center of the spatial attention matrix, and the matrix centers of different spatial attention matrices are separated from each other. Specifically, the preset loss function includes an attention weight center convergence loss function and an attention weight center separation loss function . Specifically, the preset loss function can be calculated by formula (4)-(6):

[0160] (4)

[0161] and

[0162] (5)

[0163] (6)

[0164] Wherein, represents the preset loss function, which is obtained by adding two loss functions and ; and represents the attention weight center convergence loss function; represents the center separation loss function of attention weight; H, W represent the length and width of the spatial attention matrix; i, j represent the i, j th spatial attention matrix in each spatial attention matrix; k represents the total number of spatial attention matrices; x, y represent the coordinates of each element in each spatial attention matrix; 、 represents the center coordinates of each spatial attention matrix.

[0165] In the above embodiment, by introducing the attention weight center aggregation loss function, the attention weights in each spatial attention matrix can be aggregated to the center, and the reciprocal of the distance of each element point in the spatial attention matrix to the center coordinates is used to weight the point-to-point attention matrix difference, so that the pixel points far from the center coordinates of the spatial attention matrix have smaller weights, and the pixel points close to the center coordinates of the spatial attention matrix have larger weights, and the attention weights of the pixel points close to each other will also approach the highest center point. By introducing the attention weight center separation loss function, the centers of attention can be separated from each other, thereby helping to extract attention feature maps of different positions of the person in the image.

[0166] S108, combining the person local features of each image and the person global features of each image to obtain second combined features.

[0167] After the first combined features are input into the local attention model to obtain multiple person local features, each person local feature and the person global feature of each image obtained by using the global average pooling layer are combined to obtain second combined features, so as to input the second combined features into the kinship verification model for kinship verification.

[0168] In an implementation manner of the embodiment of the present specification, step S108 can be implemented in the following manner:

[0169] combining the k person local features of each image to obtain k person local combined features;

[0170] combining the person global features of each image to obtain 1 person global combined feature;

[0171] combining the k person local combined features and the person global combined feature to obtain second combined features.

[0172] In the present embodiment, in the case of verifying the kinship of at least two images, k person local features and 1 person global features in the first image and k person local features and 1 person global features in the second image are first extracted.

[0173] Then, the first image and the second image are combined to obtain k local features of the person, and the first image and the second image are combined to obtain 1 global feature of the person; the k local features of the person and the 1 global feature of the person are combined to obtain the second combined feature.

[0174] By the above method, the local features of each person and the global features of each person are combined to obtain the second combined feature, and the second combined feature is input into the kinship verification model, so that the output result of the kinship verification model is more accurate, and the accuracy of kinship verification between images is improved.

[0175] In S110, the second combined feature is input into the kinship verification model to obtain the kinship verification result between each person.

[0176] After the local features of each person and the global features of each person are combined to obtain the second combined feature, the second combined feature is input into the kinship verification model for kinship verification, and a relationship score value between each person is obtained, and the relationship score value is normalized to obtain the kinship verification result between each person.

[0177] In one possible implementation of the embodiments of the present specification, the kinship verification model can be used to measure the kinship between faces, learn the relationship of the fusion features, and obtain the kinship / non-kinship relationship score value of the person pair through the sigmoid activation function. In addition, the training here uses the BCE loss. This module is similar to a learnable metric function, which can learn the feature relationship of faces in different family relationships through training, so as to overcome the limitations of the hand-designed metric function and better learn the potential relationship between features.

[0178] Specifically, the relationship score value between each person can be calculated by formula (7):

[0179] (7)

[0180] Wherein, represents the relationship score value between each person; represents input image 1; represents input image 2; represents the local feature of each person in each image; represents the mapping of the backbone network, which is used to extract the global feature of each person in each image; represents operation.

[0181] In one possible implementation of the embodiments of the present specification, a training method of a kinship verification model is provided, which specifically includes the following steps:

[0182] obtain a training sample set, wherein the training sample set includes a plurality of training images, and the training images include sample persons and sample family labels and sample person labels of the sample persons;

[0183] divide the training sample set to obtain a plurality of training subsets, wherein each training subset includes a plurality of training images;

[0184] input each training image in a first training subset into an initial verification model to obtain distance information between each training image in the first training subset and a class center, a predicted family category of each training image, and a predicted person category, wherein the class center is updated based on a preset class center update function in a calculation process, and the first training subset is any one of the plurality of training subsets;

[0185] calculate a family-level multi-center loss value by using a preset family-level multi-center loss function according to the distance information between each training image in the first training subset and the class center;

[0186] calculate a family classification loss value according to the predicted family category of each training image and the sample family label;

[0187] calculate a person classification loss value according to the predicted person category of each training image and the sample person label;

[0188] calculate a total loss value according to the family-level multi-center loss value, the family classification loss value, and the person classification loss value;

[0189] adjust model parameters of the initial verification model based on the total loss value, and return to perform the step of inputting each training image in the first training subset into the initial verification model to obtain the distance information between each training image in the first training subset and the class center, the predicted family category of each training image, and the predicted person category until a preset training stop condition is reached, to obtain a trained kinship verification model.

[0190] In a possible implementation manner of the embodiments of the present specification, taking training of a kinship verification model as an example, first, a training sample set is obtained, and the training sample set can be a CASIA dataset. The CASIA dataset is a large-scale face dataset, that is, the CASIA dataset includes a plurality of training images, and is used for identity authentication and face recognition.

[0191] Then the Base network is trained by using the CASIA dataset and the joint method of softmax and center loss. It should be noted that the center loss cannot be given a large weight in the initial stage of training the relative relationship verification model, which will lead to a large center loss and the softmax cannot converge. Therefore, a similar warm-up strategy is adopted to dynamically adjust the weight of the center loss during the training of the relative relationship verification model.

[0192] For example, during the training of the Base network, the first 200,000 iterations are trained by using a preset weight (for example, the preset weight is set to 0.5) to train the Base network. After 200,000 iterations, the preset weight is increased to 1.5 times of the original value every 50,000 iterations, and then the Base network is trained by using the increased preset weight until the iteration is completed, that is, the training of the Base network is completed.

[0193] After the training of the Base network is completed, only the last fully connected layer of the Base network is left, and the network layers other than the fully connected layer are parameter frozen to generate the parameter frozen Base network. Then the CASIA dataset is used to iteratively train the parameter frozen Base network, and 200,000 iterations of the parameter frozen Base network are iteratively trained to obtain an updated Base network. The network layer parameters of the updated Base network are unfrozen, and the CASIA dataset is used to iteratively train the parameter unfrozen Base network to obtain a finally trained relative relationship verification model. Finally, in the inference stage, two Base networks are used to perform the final relative relationship verification in a shared parameter manner.

[0194] By training the relative relationship verification model in the above manner, the phenomenon that the softmax cannot converge due to a large center loss can be avoided, so that the output result of the relative relationship verification model can be more accurate, and the accuracy of the relative relationship verification between images is improved.

[0195] In addition, the present application trains the base network by using the joint method of logistic regression and center loss for the relative relationship verification model, which can generate a feature space with multiple center points. Generally, the metric loss function used by the relative relationship verification model is a single feature center, that is, all face features in a family are aggregated to a center in the feature space, which will lead to the distance of the same face feature in the feature space being greater than the distance of different face features, and the face features of the husband and wife without relative relationship being close to each other, thereby causing misjudgment.

[0196] Figure 4 A distribution diagram of a single center feature space and a multi-center feature space according to an embodiment of the present application is shown. In Figure 4In the figure, (a) represents a single-center feature space, and (b) represents a multi-center feature space. In the single-center feature space and the multi-center feature space, each shape represents a face sample of a different family member, and the asterisked dot represents a feature center in the feature space. In the kinship verification model, a single-center feature space is usually used, that is, through optimization of a metric loss, all face features of the same family can be aggregated, and at the same time, the intra-class distance of a person is greater than the inter-class distance between the person and a relative. For example, we cannot map the features of a father, a mother, and a child to the same point. Because the child has potential similarity with the parents, but the father and the mother do not have such similarity.

[0197] Therefore, it is necessary to generate a feature space with multiple center points by combining a logistic regression with a center loss. Through the above method, the feature boundary between different training sample sets can be increased. The family-level multi-center loss function is calculated by formula (8):

[0198] (8)

[0199] wherein, represents a family-level multi-center loss function; represents the number of training samples in each minibatch; represents the number of accumulated training samples; represents the class of a feature center in a multi-center feature space; represents the number of accumulated feature center classes in the multi-center feature space.

[0200] In the embodiments of the present disclosure, the class center needs to be updated based on a preset class center update function in the calculation process, wherein the class center update function is shown in formulas (9)-(11):

[0201]

[0202] (10)

[0203] (11)

[0204] wherein, represents the number of training samples in each minibatch, represents the class of a feature center in a multi-center feature space; , which is used to prevent the denominator from being zero when updating the class of the multiple feature center points.

[0205] To ensure that the backbone network can extract effective face features, a classification loss function is added behind the backbone network, which includes two parts: a person label classification loss and a family label classification loss Specifically, the classification loss function is calculated by formulas (12)-(14):

[0206] (12)

[0207] (13)

[0208] (14)

[0209] wherein, is the weight of the loss, The classification loss is composed of the cross-entropy loss of fid and mutil-center loss; CE is the cross-entropy loss function, which is used for face recognition feature classification and is used in series with the softmax function.

[0210] The kinship verification method provided in the application obtains at least two images, inputs each image into a feature extraction model to obtain output features of a plurality of specified residual blocks of each image and a person global feature of each image, then for any image, combines the output features of the plurality of specified residual blocks of the image to obtain first combined features, inputs the first combined features into a local attention model to obtain a person local feature, and combines the person local features of each image and the person global features of each image to obtain second combined features, and inputs the second combined features into a kinship verification model to obtain a kinship verification result between each person. Through the above kinship verification method based on deep learning, efficient verification of the kinship between persons is realized. The local attention model can automatically extract local features with significant differences in each image without image key point detection, combines the person local features and the person global features with significant differences in each image respectively, and inputs them into the kinship verification model, so that the output result of the kinship verification model is more accurate, and the accuracy of the kinship verification result is ensured.

[0211] Figure 5 A structure diagram of a kinship verification model according to an embodiment of the application is shown.

[0212] The first face image and the second face image are input into the kinship verification model to obtain whether the first face image and the second face image have a kinship. First, the first face image and the second face image needing to be verified for the kinship are cropped and corrected, and are input into a backbone network composed of an improved network structure of resnet50, to obtain final face global features after a GAP (global average pooling layer). The first backbone network and the second backbone network are of the same structure and share parameters, and are therefore also called a twin network. Meanwhile, output features of C3, C4 and C5 of the residual block are combined in a channel layer to obtain face multi-scale combined features for extracting local features of the face. The combined features are input into a PAM (local attention model), to obtain k face local features, and then 2*k face local features of the input first face image and the input second face image are combined, and the global features output by the backbone network are also combined to obtain k+1 combined features. The k+1 combined features are input into the kinship verification model, and finally a kinship score between the faces is obtained.

[0213] Corresponding to the method embodiments, the present application also provides kinship verification device embodiments, Figure 6 A structure schematic diagram of a kinship verification device according to an embodiment of the present application is shown. As shown in the figure, Figure 6 The device comprises:

[0214] The obtaining module 602 is configured to obtain at least two images, wherein different persons are included in each image;

[0215] The feature extraction module 604 is configured to input each image into a feature extraction model respectively, to obtain output features of a plurality of specified residual blocks of each image and a person global feature of each image;

[0216] The input module 606 is configured to, for any image, combine the output features of the plurality of specified residual blocks of the image to obtain first combined features, and input the first combined features into a local attention model to obtain a person local feature, wherein the local attention model is a multi-head attention model;

[0217] The feature combination module 608 is configured to combine the person local features of each image and the person global features of each image to obtain second combined features;

[0218] The verification module 610 is configured to input the second combined features into a kinship verification model to obtain a kinship verification result between each person.

[0219] Optionally, the obtaining module 602 is further configured to:

[0220] obtaining at least two initial images, wherein different persons are included in each initial image;

[0221] performing preprocessing on each initial image based on the persons, to obtain at least two images.

[0222] Optionally, the number of the feature extraction models is multiple, and each feature extraction model has the same model structure and the same model parameter.

[0223] The feature extraction module 604 is further configured to:

[0224] parallelly input each image into each feature extraction model, to obtain the output features of the multiple specified residual blocks of each image and the person global features of each image.

[0225] Optionally, the input module 606 is further configured to:

[0226] For any image, the output features of the multiple specified residual blocks of the image are input into the channel layer, and the output features of the multiple specified residual blocks of the image are cross-combined by the channel layer to obtain first combined features.

[0227] Optionally, the local attention model includes a spatial attention sub-model and an attention learning sub-model.

[0228] The input module 606 is further configured to:

[0229] perform channel shuffling on the first combined features;

[0230] perform feature fusion on the first combined features after channel shuffling by using a 1*1 convolution kernel, to obtain fused features;

[0231] split the fused features to obtain a preset number of split features;

[0232] input each split feature into the spatial attention sub-model, to obtain a spatial attention matrix corresponding to each split feature;

[0233] input each spatial attention matrix into the attention learning sub-model, to obtain multiple person local features.

[0234] Optionally, the input module 606 is further configured to:

[0235] perform column summation operation on the first spatial attention matrix, and construct a second spatial attention matrix based on the summation result, wherein the first spatial attention matrix is any one of the spatial attention matrices;

[0236] perform transposition operation on the second spatial attention matrix, to obtain a third spatial attention matrix;

[0237] determining a maximum value in the third spatial attention matrix as a matrix center of the first spatial attention matrix;

[0238] determining, according to the matrix center of each spatial attention matrix, a local feature of a person corresponding to each spatial attention matrix under a constraint of a preset loss function, wherein the constraint of the preset loss function includes that each weight value in a same spatial attention matrix converges to the matrix center of the spatial attention matrix, and the matrix centers of different spatial attention matrices are separated from each other.

[0239] Optionally, the feature combination module 608 is further configured to:

[0240] combining the k local features of the person in each image to obtain a k local combined feature of the person;

[0241] combining the global features of the person in each image to obtain a global combined feature of the person;

[0242] combining the k local combined feature of the person and the global combined feature of the person to obtain a second combined feature.

[0243] Optionally, the apparatus further includes a training module;

[0244] the training module is configured to:

[0245] obtain a training sample set, wherein the training sample set includes a plurality of training images, and each training image includes a sample person, a sample family label of the sample person, and a sample person label;

[0246] divide the training sample set to obtain a plurality of training subsets, wherein each training subset includes a plurality of training images;

[0247] input each training image in a first training subset into an initial verification model to obtain distance information between each training image in the first training subset and a class center, a predicted family class of each training image, and a predicted person class, wherein the class center is updated based on a preset class center update function in a calculation process, and the first training subset is any one of the plurality of training subsets;

[0248] calculate a family-level multi-center loss value by using a preset family-level multi-center loss function according to the distance information between each training image in the first training subset and the class center;

[0249] calculate a family classification loss value according to the predicted family class of each training image and the sample family label;

[0250] calculate a person classification loss value according to the predicted person class of each training image and the sample person label;

[0251] According to the family-level multi-center loss value, the family classification loss value and the person classification loss value, a total loss value is calculated;

[0252] Based on the total loss value, the model parameters of the initial verification model are adjusted, and the step of inputting each training image in the first training subset into the initial verification model is performed until a preset training stop condition is reached, and a trained kinship verification model is obtained.

[0253] The kinship verification device provided in the present application comprises: an image acquisition unit, a feature extraction model, a local attention model, and a kinship verification model.

[0254] The above is a schematic scheme of a kinship verification device of the present embodiment. It should be noted that the technical scheme of the kinship verification device belongs to the same concept as the technical scheme of the kinship verification method described above, and the details of the technical scheme of the kinship verification device that are not described in detail can be referred to the description of the technical scheme of the kinship verification method described above. In addition, each component in the device embodiment should be understood as a functional module that must be established to realize each step of the program flow or each step of the method. Each functional module is not limited by actual functional division or separation. The device claim defined by such a group of functional modules should be understood as a functional module architecture for realizing the solution of the computer program mainly by the specification, and should not be understood as an entity device for realizing the solution mainly by hardware.

[0255] Figure 7A structural block diagram of a computing device 700 is shown, according to an embodiment of the present application. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 through a bus 730, and a database 750 is used to store data.

[0256] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of networks such as the Internet. The access device 440 can include one or more of any type of network interface (for example, a network interface card (NIC)), wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0257] In an embodiment of the present application, the above-mentioned components of the computing device 700, as well as other components not shown in the figure, can be connected to each other, for example, through a bus. It should be understood that, Figure 7 the computing device structure block diagram shown is for the purpose of example only, and is not a limitation on the scope of the present application. Other components can be added or replaced as needed by those skilled in the art. Figure 7 the computing device structure block diagram shown is for the purpose of example only, and is not a limitation on the scope of the present application. Other components can be added or replaced as needed by those skilled in the art.

[0258] The computing device 700 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and so on), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and so on), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 700 can also be a mobile or stationary server.

[0259] The processor 720 is configured to execute the computer executable instructions of the above-mentioned kinship verification method.

[0260] The above is a schematic solution of the computing device of the embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned kinship verification method belong to the same concept, and the details of the technical solution of the computing device which are not described in detail can be referred to the description of the technical solution of the above-mentioned kinship verification method.

[0261] The embodiment of the present application further provides a computer readable storage medium which stores computer instructions, and the instructions are executed by a processor to implement the kinship verification method.

[0262] The above is a schematic solution of the computer readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above-mentioned kinship verification method belong to the same concept, and the details of the technical solution of the storage medium which are not described in detail can be referred to the description of the technical solution of the above-mentioned kinship verification method.

[0263] The embodiment of the present application discloses a chip which stores computer instructions, and the computer instructions are executed by a processor to implement the steps of the above-mentioned kinship verification method.

[0264] The specific embodiments of the present application are described above. Other embodiments are within the scope of the appended claims. In some cases, acts or steps recited in the claims can be performed in an order other than that recited in the embodiments, and still achieve desirable results. Also, the process depicted in the figures can not necessarily require the particular order shown, or sequential order to achieve the results desired. In certain implementations, multitasking and parallel processing can be advantageous.

[0265] The computer instructions include computer program codes which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program codes, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0266] It should be noted that, for the aforementioned method embodiments, the sequences of the described actions are not the only ones that can be performed to implement the present application. In some embodiments, the sequences of actions can be performed in different order or simultaneously. In some embodiments, other sequences of actions can be performed, which should be apparent to a person of ordinary skill in the art in light of the teachings of the present application.

[0267] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0268] The preferred embodiments of the present application disclosed above are only used to explain the present application. The alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited by the claims and their full scope and equivalents.

Claims

1. A method of verifying a kinship, characterized by, The method comprises the following steps: acquiring at least two images, wherein different persons are included in each of the images; inputting each of the images into a feature extraction model respectively to obtain output features of a plurality of specified residual blocks of each of the images and person global features of each of the images; for any image, combining the output features of the plurality of specified residual blocks of the image to obtain first combined features, and inputting the first combined features into a local attention model to obtain person local features, wherein the local attention model is a multi-head attention model, and the local attention model comprises a spatial attention sub-model and an attention learning sub-model; the step of inputting the first combined features into the local attention model to obtain the person local features comprises: performing channel shuffling on the first combined features; performing feature fusion on the first combined features after channel shuffling by using a 1*1 convolution kernel to obtain fused features; splitting the fused features to obtain a preset number of split features; inputting each of the split features into the spatial attention sub-model to obtain a spatial attention matrix corresponding to each of the split features; and inputting each of the spatial attention matrices into the attention learning sub-model to obtain a plurality of person local features; the step of inputting each of the spatial attention matrices into the attention learning sub-model to obtain the plurality of person local features comprises: performing column-wise summation on a first spatial attention matrix to obtain a second spatial attention matrix based on the summation result, wherein the first spatial attention matrix is any one of the spatial attention matrices; performing a transpose operation on the second spatial attention matrix to obtain a third spatial attention matrix; determining a maximum value in the third spatial attention matrix as a matrix center of the first spatial attention matrix; and determining a person local feature corresponding to each of the spatial attention matrices under the constraint of a preset loss function according to the matrix center of each of the spatial attention matrices, wherein the constraint of the preset loss function comprises that each weight value in a same spatial attention matrix converges to the matrix center of the spatial attention matrix, and the matrix centers of different spatial attention matrices are separated from each other; combining the person local features of each of the images and the person global features of each of the images to obtain second combined features; inputting the second combined features into a kinship verification model to obtain kinship verification results between the persons.

2. The method of claim 1, wherein, The step of acquiring at least two images comprises: acquiring at least two initial images, wherein different persons are included in each of the initial images; based on the persons, pre-processing each of the initial images to obtain at least two images.

3. The method of claim 1, wherein, The number of the feature extraction models is a plurality, and the model structures and the model parameters of each of the feature extraction models are the same; the step of inputting each of the images into a feature extraction model respectively to obtain output features of a plurality of specified residual blocks of each of the images and person global features of each of the images comprises: The image is input into each feature extraction model in parallel, and the output features of a plurality of specified residual blocks of each image and the global features of the person in each image are obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The step of combining the output features of a plurality of specified residual blocks of the image to obtain the first combined features comprises: The output features of a plurality of specified residual blocks of the image are input into a channel layer, and the output features of a plurality of specified residual blocks of the image are cross combined through the channel layer to obtain the first combined features.

5. The method according to any one of claims 1-3, characterized in that, The step of combining the local features of the person in each image and the global features of the person in each image to obtain the second combined features comprises: The k local features of the person in each image are combined to obtain k local combined features of the person; The global features of the person in each image are combined to obtain one global combined feature of the person; The k local combined features of the person and the global combined feature of the person are combined to obtain the second combined features.

6. The method of claim 1, wherein, The training method of the kinship verification model comprises: obtaining a training sample set, wherein the training sample set comprises a plurality of training images, the training images comprise a sample person and sample family labels and sample person labels of the sample person; dividing the training sample set to obtain a plurality of training subsets, wherein each training subset comprises a plurality of training images; inputting each training image in a first training subset into an initial verification model to obtain distance information between each training image in the first training subset and a class center, a predicted family category of the each training image and a predicted person category, wherein the class center is updated based on a preset class center update function in the calculation process, and the first training subset is any one of the plurality of training subsets; calculating a family-level multi-center loss value by using a preset family-level multi-center loss function according to the distance information between each training image in the first training subset and the class center; calculating a family classification loss value according to the predicted family category of the each training image and the sample family label; calculating a person classification loss value according to the predicted person category of the each training image and the sample person label; calculating a total loss value according to the family-level multi-center loss value, the family classification loss value and the person classification loss value; adjusting model parameters of the initial verification model based on the total loss value, and returning to execute the step of inputting each training image in a first training subset into an initial verification model to obtain distance information between each training image in the first training subset and a class center, a predicted family category of the each training image and a predicted person category until a preset training stop condition is reached, and obtaining a trained kinship verification model.

7. A kinship verification apparatus characterized by comprising: comprises: an acquisition module configured to acquire at least two images, wherein each image comprises different persons; a feature extraction module configured to input each image into a feature extraction model to obtain output features of a plurality of specified residual blocks of each image and global features of the person in each image; The input module is configured to, for any image, combine the output features of a plurality of specified residual blocks of the image to obtain first combined features, input the first combined features into a local attention model to obtain a local feature of a person, wherein the local attention model is a multi-head attention model, and the local attention model includes a spatial attention sub-model and an attention learning sub-model; the step of inputting the first combined features into the local attention model to obtain the local feature of the person includes: performing channel shuffling on the first combined features; performing feature fusion on the first combined features after channel shuffling by using a 1*1 convolution kernel to obtain fused features; splitting the fused features to obtain a preset number of split features; inputting each split feature into the spatial attention sub-model to obtain a spatial attention matrix corresponding to each split feature; inputting each spatial attention matrix into the attention learning sub-model to obtain a plurality of local features of the person; the step of inputting each spatial attention matrix into the attention learning sub-model to obtain the plurality of local features of the person includes: performing column-wise summation on a first spatial attention matrix, and constructing a second spatial attention matrix based on a summation result, wherein the first spatial attention matrix is any one of the spatial attention matrices; performing a transpose operation on the second spatial attention matrix to obtain a third spatial attention matrix; determining a maximum value in the third spatial attention matrix as a matrix center of the first spatial attention matrix; determining a local feature of a person corresponding to each spatial attention matrix under the constraint of a preset loss function according to the matrix center of each spatial attention matrix, wherein the constraint of the preset loss function includes that each weight value in a same spatial attention matrix converges to the matrix center of the spatial attention matrix, and the matrix centers of different spatial attention matrices are separated from each other; The feature combination module is configured to combine the local features of the persons in each image and the global features of the persons in each image to obtain second combined features. The verification module is configured to input the second combined features into a kinship verification model to obtain a kinship verification result between the persons.

8. A computing device, comprising: comprising: a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to implement the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, a computer instruction is stored, and the computer instruction is executed by a processor to implement the method in any one of claims 1-6.

Images