Portrait aggregation method and device based on progressive multi-source feature fusion and alignment

Abstract

The present invention discloses a portrait clustering method and device based on progressive multi-source feature fusion and alignment, which relates to the field of public video intelligent analysis, including: constructing a portrait clustering model, in which the features of a face image, a face simulation sketch portrait, a body image, and a body simulation sketch portrait are progressively fused, and the body features and face features are aligned through pseudo-label clustering and similarity constraints, so as to effectively fuse multi-source portrait features and generate progressive multi-source fused pedestrian features; the face image, body image and the corresponding face simulation sketch portrait and body simulation sketch portrait of a person to be queried are input into the trained portrait clustering model to obtain the progressive multi-source fused pedestrian features of the person to be queried, and the similarity between the progressive multi-source fused pedestrian features and the progressive multi-source fused pedestrian features of each reference sample in a database is calculated to perform portrait clustering.

Description

Technical Field

[0001] The present invention relates to the field of public video intelligent analysis, and in particular to a portrait aggregation method and device based on progressive multi-source feature fusion and alignment. Background Art

[0002] With the development of economy and society, the demand for identity confirmation is growing. Portrait clustering technology groups pictures with the same identity into one file, and it has a wide range of applications in the field of public security. Human images and facial images captured by cameras are the main data sources of portrait clustering technology. However, in practice, not all events can be captured clearly by cameras. For example, some scenes may not have cameras installed, or the target information may be missing or blurred due to shooting angles, occlusions, insufficient light, etc. In this case, human / face simulation sketches drawn based on eyewitness descriptions become an important source of supplementary data. Although these data sources provide diversity for portrait clustering, they are quite different from each other. The clothing styles and colors in human images are varied, and facial images vary significantly due to angles, light and other conditions, while simulated sketches are highly subjective and have inaccurate details. These factors lead to large inconsistencies in feature extraction and representation between different data sources, and the effect is poor when clustering is performed directly. Therefore, how to effectively integrate multiple data sources and solve the problem of data differences is a key challenge facing portrait clustering technology. Summary of the invention

[0003] The purpose of this application is to propose a portrait aggregation method and device based on progressive multi-source feature fusion and alignment to address the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a portrait clustering method based on progressive multi-source feature fusion and alignment, comprising the following steps:

[0005] A portrait clustering model is constructed, which includes a face feature extraction unit and a body feature extraction unit. The face feature extraction unit includes a first shallow network, a second shallow network and a first deep network, and the body feature extraction unit includes a third shallow network, a fourth shallow network and a second deep network. The first shallow network and the second shallow network are used to extract shallow features of face images and shallow features of face simulation sketches, respectively. The shallow features of face images and shallow features of face simulation sketches are connected in series through channels and then input into the first deep network to generate face features. The third shallow network and the fourth shallow network are used to extract shallow features of body images and shallow features of body simulation sketches, respectively. The shallow features of the body image and the shallow features of the human body simulation sketch are channel-connected and then input into the second deep network to generate human body features, and the facial features and human body features are superimposed to obtain the pedestrian features of progressive multi-source fusion; in the training process of the portrait clustering model, the pedestrian features of progressive multi-source fusion of all samples in the training data are clustered to generate pseudo labels, a cross entropy loss function is constructed according to the pseudo labels and the pedestrian features of progressive multi-source fusion, a multi-source feature alignment loss function is constructed according to the facial features and the human body features, a total loss function is constructed according to the cross entropy loss function and the multi-source feature alignment loss function, and the portrait clustering model is trained based on the total loss function to obtain a trained portrait clustering model;

[0006] Collecting several reference samples of face images, body images and their corresponding face simulation sketches and body simulation sketches and inputting them into a trained portrait clustering model, obtaining progressive multi-source fused pedestrian features of each reference sample, clustering based on the progressive multi-source fused pedestrian features of all reference samples, generating pseudo labels corresponding to each reference sample, and storing the progressive multi-source fused pedestrian features of each reference sample and their corresponding pseudo labels in a database;

[0007] The face image, body image and their corresponding face simulation sketch portrait and body simulation sketch portrait of the person to be queried are obtained and input into the trained portrait clustering model to obtain the progressive multi-source fused pedestrian features of the person to be queried, and the similarity between the progressive multi-source fused pedestrian features of the person to be queried and the progressive multi-source fused pedestrian features of each reference sample in the database is calculated, and the pseudo-label corresponding to the progressive multi-source fused pedestrian features of the reference sample in the database with the highest similarity is used as the portrait clustering result of the person to be queried.

[0008] Preferably, the first shallow network, the second shallow network, the third shallow network and the fourth shallow network all use shallow neural networks, and the structure of the shallow neural network is the block linear mapping layer and the first 6 layers of Transformer blocks in the ViT-Base model; the first deep network and the second deep network use deep neural networks, and the structure of the deep neural network is the last 6 layers of Transformer blocks in the ViT-Base model.

[0009] Preferably, the first shallow network and the second shallow network are used to extract shallow features of the face image and shallow features of the face simulation sketch respectively, and the shallow features of the face image and the shallow features of the face simulation sketch are connected in series and then input into the first deep network to generate face features, specifically including:

[0010] The face image and the face simulation sketch are input into the first shallow network and the second shallow network respectively, and the shallow features of the face image and the shallow features of the face simulation sketch are extracted, as shown in the following formula:

[0011] ;

[0012] ;

[0013] in, and They represent face images and face simulation sketches respectively, B is the number of input samples, the number of channels of each sample is 3, the height is H, and the width is W; represents the first shallow network, represents the second shallow network, and denote the learnable parameters of the first shallow network and the second shallow network respectively, is the shallow feature of face image, is the shallow feature of the face simulation sketch, N is the number of feature blocks of the face, and D is the number of channels of the shallow feature;

[0014] The shallow features of the face image and the shallow features of the face simulation sketch are connected in series to obtain the first series feature, as shown in the following formula:

[0015] ;

[0016] in, represents the first series feature, Indicates that channels are connected in series;

[0017] The first concatenated feature is input into the first deep network to obtain the face feature, as shown in the following formula:

[0018] ;

[0019] in, Represents facial features, represents the first deep network, are the learnable parameters of the first deep network.

[0020] Preferably, the third shallow network and the fourth shallow network are used to extract shallow features of the human body image and shallow features of the human body simulation sketch respectively, and the shallow features of the human body image and the shallow features of the human body simulation sketch are connected in series through channels, and then input into the second deep network to generate human body features, specifically including:

[0021] The human body image and the human body simulation sketch image are input into the third shallow network and the fourth shallow network respectively, and the shallow features of the human body image and the shallow features of the human body simulation sketch image are extracted, as shown in the following formula:

[0022] ;

[0023] ;

[0024] in, and are human body images and human body simulation sketch portraits, respectively. B is the number of input samples. The number of channels of each sample is 3, the height is h, and the width is w. represents the third shallow network, represents the fourth shallow network, and Represent the learnable parameters of the third shallow network and the fourth shallow network respectively, is the shallow feature of human image, is the shallow feature of the human body simulation sketch, M is the number of feature blocks of the human body, and D is the number of channels of the shallow feature;

[0025] The shallow features of the human body image and the shallow features of the human body simulation sketch are connected in series to obtain the second series feature, as shown in the following formula:

[0026] ;

[0027] in, represents the second series feature, Indicates that channels are connected in series;

[0028] The second concatenated features are input into the second deep network to obtain the human features, as shown in the following formula:

[0029] ;

[0030] in, Represents human characteristics, represents the second deep network, are the learnable parameters of the second deep network.

[0031] As a preferred method, facial features and body features are superimposed to obtain pedestrian features of progressive multi-source fusion, specifically including:

[0032] The facial features and body features are superimposed to obtain the pedestrian features of progressive multi-source fusion, as shown in the following formula:

[0033] ;

[0034] in, Represents facial features, Represents human characteristics, It represents the pedestrian features of progressive multi-source fusion, B is the number of input samples, and D is the number of channels of shallow features.

[0035] As a preferred method, clustering is performed based on the pedestrian features of progressive multi-source fusion of all samples in the training data to generate pseudo labels, a cross entropy loss function is constructed based on the pseudo labels and the pedestrian features of progressive multi-source fusion, a multi-source feature alignment loss function is constructed based on the face features and the body features, and a total loss function is constructed based on the cross entropy loss function and the multi-source feature alignment loss function, which specifically includes:

[0036] The progressive multi-source fused pedestrian features of all samples in the training data are clustered and divided into C clusters through clustering. The samples in each cluster have the same pseudo label, and the pseudo label of each sample in the training data is obtained as shown in the following formula:

[0037] ;

[0038] in, represents pedestrian features of progressive multi-source fusion, Cluster represents clustering operation, represents the pseudo-label of the sample in the training data, B is the number of input samples, and C represents the number of pseudo-label clusters;

[0039] The cross entropy loss function is expressed as:

[0040] ;

[0041] in, represents the cross entropy ( ) loss function, It represents the linear projection of the pedestrian features of progressive multi-source fusion into learnable parameters of the posterior probability. Respectively represent the learnable parameters of the first shallow network, the second shallow network, the first deep network, the third shallow network, the fourth shallow network, and the second deep network;

[0042] The multi-source feature alignment loss function is expressed as:

[0043] ;

[0044] in, Represents facial features, Represents human characteristics, represents the multi-source feature alignment loss function, represents the L2 norm;

[0045] The total loss function is expressed as:

[0046] ;

[0047] in, represents the total loss function, is a hyperparameter.

[0048] In the second aspect, the present invention provides a portrait clustering device based on progressive multi-source feature fusion and alignment, including: a model building module, configured to build a portrait clustering model, the portrait clustering model includes a face feature extraction unit and a body feature extraction unit, the face feature extraction unit includes a first shallow network, a second shallow network and a first deep network, the body feature extraction unit includes a third shallow network, a fourth shallow network and a second deep network; the first shallow network and the second shallow network are respectively used to extract shallow features of face images and shallow features of face simulation sketch portraits, the shallow features of face images and shallow features of face simulation sketch portraits are channel-connected and input into the first deep network to generate face features; the third shallow network and the fourth shallow network are respectively used to extract shallow features of face images and shallow features of face simulation sketch portraits shallow features of the human body image and shallow features of the human body simulation sketch, channel-connect the shallow features of the human body image and the shallow features of the human body simulation sketch, and then input them into the second deep network to generate human body features, and superimpose the facial features and human body features to obtain progressive multi-source fused pedestrian features; in the training process of the portrait clustering model, cluster the progressive multi-source fused pedestrian features of all samples in the training data to generate pseudo labels, and construct a cross entropy loss function according to the pseudo labels and the progressive multi-source fused pedestrian features, and construct a multi-source feature alignment loss function according to the facial features and the human body features. A total loss function is constructed based on the cross entropy loss function and the multi-source feature alignment loss function, and the portrait clustering model is trained based on the total loss function to obtain a trained portrait clustering model;

[0049] A database construction module is configured to collect face images, body images and their corresponding face simulation sketch portraits and body simulation sketch portraits of a number of reference samples and input them into a trained portrait clustering model to obtain progressive multi-source fused pedestrian features of each reference sample, perform clustering based on the progressive multi-source fused pedestrian features of all reference samples, generate a pseudo label corresponding to each reference sample, and store the progressive multi-source fused pedestrian features of each reference sample and its corresponding pseudo label in a database;

[0050] The profiler module is configured to obtain the face image, body image and their corresponding face simulation sketch portrait and body simulation sketch portrait of the person to be queried and input them into the trained portrait profiler model to obtain the progressive multi-source fused pedestrian features of the person to be queried, calculate the similarity between the progressive multi-source fused pedestrian features of the person to be queried and the progressive multi-source fused pedestrian features of each reference sample in the database, and take the pseudo-label corresponding to the progressive multi-source fused pedestrian features of the reference sample in the database with the highest similarity as the portrait profiler result of the person to be queried.

[0051] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0052] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0053] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] By progressively fusing face images, face simulation sketches, body images, and body simulation sketches, multi-source portrait features are effectively fused. By constructing a cross entropy loss function and a multi-source feature alignment loss function to train the portrait clustering model, the pedestrian features of progressive multi-source fusion are extracted, which can improve the accuracy of unsupervised portrait clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 A schematic diagram of a process of a portrait aggregation method based on progressive multi-source feature fusion and alignment according to an embodiment of the present application;

[0058] Figure 2 A schematic diagram of a training process of a portrait clustering model in a portrait clustering method based on progressive multi-source feature fusion and alignment according to an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a portrait gathering device based on progressive multi-source feature fusion and alignment according to an embodiment of the present application;

[0060] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Figure 1 A portrait file aggregation method based on progressive multi-source feature fusion and alignment provided by an embodiment of the present application is shown, comprising the following steps:

[0063] S1, construct a portrait polymorphism model, the portrait polymorphism model includes a face feature extraction unit and a body feature extraction unit, the face feature extraction unit includes a first shallow network, a second shallow network and a first deep network, and the body feature extraction unit includes a third shallow network, a fourth shallow network and a second deep network; the first shallow network and the second shallow network are used to extract shallow features of face images and shallow features of face simulation sketches, respectively, the shallow features of face images and shallow features of face simulation sketches are connected in series through channels and then input into the first deep network to generate face features; the third shallow network and the fourth shallow network are used to extract shallow features of body images and shallow features of body simulation sketches, respectively, The shallow features of the human image and the shallow features of the human simulation sketch are channel-connected and then input into the second deep network to generate human features, and the facial features and human features are superimposed to obtain the progressive multi-source fused pedestrian features; in the training process of the portrait clustering model, the progressive multi-source fused pedestrian features of all samples in the training data are clustered to generate pseudo labels, a cross entropy loss function is constructed according to the pseudo labels and the progressive multi-source fused pedestrian features, a multi-source feature alignment loss function is constructed according to the facial features and the human features, a total loss function is constructed according to the cross entropy loss function and the multi-source feature alignment loss function, and the portrait clustering model is trained based on the total loss function to obtain a trained portrait clustering model.

[0064] In a specific embodiment, the first shallow network, the second shallow network, the third shallow network and the fourth shallow network all use shallow neural networks, and the structure of the shallow neural network is the block linear mapping layer and the first 6 layers of Transformer blocks in the ViT-Base model; the first deep network and the second deep network use deep neural networks, and the structure of the deep neural network is the last 6 layers of Transformer blocks in the ViT-Base model.

[0065] Specifically, refer to Figure 2 , the structures of the first shallow network, the second shallow network, the third shallow network and the fourth shallow network are all shallow neural networks, and the structures of the first deep network and the second deep network are all deep neural networks. The embodiment of the present application selects the ViT-Base model for feature extraction, wherein the block linear mapping layer and the first 6 layers of Transformer blocks of the ViT-Base model are used as shallow neural networks, the last 6 layers of Transformer blocks are used as deep neural networks, and the classification handle output by the last layer of Transformer blocks is regarded as facial features or portrait features for feature fusion and feature alignment.

[0066] In a specific embodiment, the first shallow network and the second shallow network are used to extract shallow features of the face image and shallow features of the face simulation sketch respectively. The shallow features of the face image and the shallow features of the face simulation sketch are connected in series and then input into the first deep network to generate face features, which specifically includes:

[0067] The face image and the face simulation sketch are input into the first shallow network and the second shallow network respectively, and the shallow features of the face image and the shallow features of the face simulation sketch are extracted, as shown in the following formula:

[0068] ;

[0069] ;

[0070] in, and They represent face images and face simulation sketches respectively, B is the number of input samples, the number of channels of each sample is 3, the height is H, and the width is W; represents the first shallow network, represents the second shallow network, and denote the learnable parameters of the first shallow network and the second shallow network respectively, is the shallow feature of face image, is the shallow feature of the face simulation sketch, N is the number of feature blocks of the face, and D is the number of channels of the shallow feature;

[0071] The shallow features of the face image and the shallow features of the face simulation sketch are connected in series to obtain the first series feature, as shown in the following formula:

[0072] ;

[0073] in, represents the first series feature, Indicates that channels are connected in series;

[0074] The first concatenated feature is input into the first deep network to obtain the face feature, as shown in the following formula:

[0075] ;

[0076] in, Represents facial features, represents the first deep network, are the learnable parameters of the first deep network.

[0077] In a specific embodiment, the third shallow network and the fourth shallow network are used to extract shallow features of a human body image and shallow features of a human body simulation sketch respectively, and the shallow features of the human body image and the shallow features of the human body simulation sketch are connected in series through channels, and then input into the second deep network to generate human body features, specifically including:

[0078] The human body image and the human body simulation sketch image are input into the third shallow network and the fourth shallow network respectively, and the shallow features of the human body image and the shallow features of the human body simulation sketch image are extracted, as shown in the following formula:

[0079] ;

[0080] ;

[0081] in, and are human body images and human body simulation sketch portraits, respectively. B is the number of input samples. The number of channels of each sample is 3, the height is h, and the width is w. represents the third shallow network, represents the fourth shallow network, and Represent the learnable parameters of the third shallow network and the fourth shallow network respectively, is the shallow feature of human image, is the shallow feature of the human body simulation sketch, M is the number of feature blocks of the human body, and D is the number of channels of the shallow feature;

[0082] The shallow features of the human body image and the shallow features of the human body simulation sketch are connected in series to obtain the second series feature, as shown in the following formula:

[0083] ;

[0084] in, represents the second series feature, Indicates that channels are connected in series;

[0085] The second concatenated features are input into the second deep network to obtain the human features, as shown in the following formula:

[0086] ;

[0087] in, Represents human characteristics, represents the second deep network, are the learnable parameters of the second deep network.

[0088] In a specific embodiment, the facial features and the body features are superimposed to obtain the pedestrian features of progressive multi-source fusion, which specifically includes: superimposing the facial features and the body features to obtain the pedestrian features of progressive multi-source fusion, as shown in the following formula:

[0089] ;

[0090] in, Represents facial features, Represents human characteristics, It represents the pedestrian features of progressive multi-source fusion, B is the number of input samples, and D is the number of channels of shallow features.

[0091] Specifically, in the face feature extraction unit, the shallow features of the face image and the shallow features of the face simulation sketch are extracted through the first shallow network and the second shallow network respectively, and the channels are connected in series and input into the first deep network to generate face features; in the body feature extraction unit, the shallow features of the body image and the shallow features of the body simulation sketch are extracted through the third shallow network and the fourth shallow network respectively, and the channels are connected in series and input into the second deep network to generate body features. After further superimposing the face features and the body features, the pedestrian features of progressive multi-source fusion can be obtained.

[0092] In a specific embodiment, clustering is performed based on the pedestrian features of progressive multi-source fusion of all samples in the training data to generate pseudo labels, a cross entropy loss function is constructed based on the pseudo labels and the pedestrian features of progressive multi-source fusion, a multi-source feature alignment loss function is constructed based on the face features and the body features, and a total loss function is constructed based on the cross entropy loss function and the multi-source feature alignment loss function, which specifically includes:

[0093] The progressive multi-source fused pedestrian features of all samples in the training data are clustered and divided into C clusters through clustering. The samples in each cluster have the same pseudo label, and the pseudo label of each sample in the training data is obtained as shown in the following formula:

[0094] ;

[0095] in, represents pedestrian features of progressive multi-source fusion, Cluster represents clustering operation, represents the pseudo-label of the sample in the training data, B is the number of input samples, and C represents the number of pseudo-label clusters;

[0096] The cross entropy loss function is expressed as:

[0097] ;

[0098] in, represents the cross entropy ( ) loss function, It represents the linear projection of the pedestrian features of progressive multi-source fusion into learnable parameters of the posterior probability. Respectively represent the learnable parameters of the first shallow network, the second shallow network, the first deep network, the third shallow network, the fourth shallow network, and the second deep network;

[0099] The multi-source feature alignment loss function is expressed as:

[0100] ;

[0101] in, Represents facial features, Represents human characteristics, represents the multi-source feature alignment loss function, represents the L2 norm;

[0102] The total loss function is expressed as:

[0103] ;

[0104] in, represents the total loss function, is a hyperparameter.

[0105] Specifically, face images, body images and their corresponding face simulation sketches and body simulation sketches of several persons are collected, and the height of face images and face simulation sketches are unified to H, and the width is unified to W, and the height of body images and body simulation sketches are unified to h, and the width is unified to w, and random flipping is used for data enhancement to construct training data. Each sample in the training data is a face image, a body image and its corresponding face simulation sketch and body simulation sketch of the same person, and the number of samples in the training data is B, and the number of channels of the face image and face simulation sketch of each sample is 3, the height is H, and the width is W, and the number of channels of the body image and body simulation sketch of each sample is 3, the height is h, and the width is w. The portrait clustering model is trained using training data. Similarly, the facial image and facial simulation sketch of each sample in the training data are input into the facial feature extraction unit. The shallow features of the facial image and the shallow features of the facial simulation sketch of each sample in the training data are extracted through the first shallow network and the second shallow network. After the channels are connected in series, they are input into the first deep network to extract the facial features of each sample in the training data. The human body image and human body simulation sketch of each sample in the training data are input into the human body feature extraction unit. The shallow features of the human body image and the shallow features of the human body simulation sketch of each sample in the training data are extracted through the third shallow network and the fourth shallow network. After the channels are connected in series, they are input into the second deep network to extract the human body features of each sample in the training data. The facial features and human body features of each sample in the training data are superimposed to obtain the progressive multi-source fused pedestrian features of each sample in the training data. The progressive multi-source fused pedestrian features of all samples in the training data are clustered to obtain the corresponding pseudo labels of each sample in the training data. , through clustering operation, the progressive multi-source fused pedestrian features of all samples in the training data are divided into C clusters, and the samples in each cluster are assigned the same pseudo label, so each sample in the training data has a corresponding pseudo label. The cross entropy loss function is constructed by pseudo labels and progressive multi-source fused pedestrian features , which is used to measure the difference between the pseudo label of each sample in the training data and the pedestrian features of progressive multi-source fusion, and construct a multi-source feature alignment loss function based on the L2 norm , which is used to measure the Euclidean distance between the facial feature d and the body feature q of each sample in the training data. The total loss function is constructed based on the cross entropy loss function and the multi-source feature alignment loss function, and the total loss function is minimized by the gradient descent method, where the hyperparameter Used to adjust the multi-source feature alignment loss function In the total loss function Based on the derivative chain and back propagation method, the gradient descent method will update And all the neural network parameters, namely , realize neural network optimization, and then obtain the trained portrait clustering model.

[0106] In one of the examples, the clustering operation uses the DBSCAN clustering algorithm. In other examples, other clustering algorithms such as K-means can also be used. The clustering method used in the embodiments of the present application is not specifically limited. In one of the examples, the pedestrian features of progressive multi-source fusion can be linearly projected using a fully connected layer and a Softmax layer to obtain the corresponding posterior probability. In other examples, other linear projection methods can also be used, which are not limited here.

[0107] S2, collects face images, body images and their corresponding face simulation sketch portraits and body simulation sketch portraits of several reference samples and inputs them into the trained portrait clustering model to obtain the progressive multi-source fused pedestrian features of each reference sample, clusters the progressive multi-source fused pedestrian features of all reference samples, generates a pseudo-label corresponding to each reference sample, and stores the progressive multi-source fused pedestrian features of each reference sample and its corresponding pseudo-label in a database.

[0108] Specifically, after obtaining the trained portrait clustering model, it can be used to construct a database. First, collect the face images, body images and their corresponding face simulation sketch portraits and body simulation sketch portraits of the reference samples and input them into the trained portrait clustering model to obtain the progressive multi-source fused pedestrian features of the reference samples; secondly, perform clustering operations on the progressive multi-source fused pedestrian features of all reference samples to generate pseudo labels corresponding to each reference sample, and through clustering operations, divide the progressive multi-source fused pedestrian features of all reference samples into several clusters, and assign the same pseudo labels to the reference samples in each cluster; finally, store the progressive multi-source fused pedestrian features of each reference sample and their corresponding pseudo labels in the database, which can be used as a reference for portrait archiving. In one embodiment, the reference sample can be a sample in the training data.

[0109] S3, obtain the face image, body image and their corresponding face simulation sketch portrait and body simulation sketch portrait of the person to be queried and input them into the trained portrait clustering model to obtain the progressive multi-source fused pedestrian features of the person to be queried, calculate the similarity between the progressive multi-source fused pedestrian features of the person to be queried and the progressive multi-source fused pedestrian features of each reference sample in the database, and take the pseudo-label corresponding to the progressive multi-source fused pedestrian features of the reference sample in the database with the highest similarity as the portrait clustering result of the person to be queried.

[0110] Specifically, the face images and body images of the person to be queried that have not been archived and their corresponding face simulation sketch portraits and body simulation sketch portraits are input into the trained portrait clustering model to obtain the progressive multi-source fused pedestrian features of the person to be queried, and the progressive multi-source fused pedestrian features of the person to be queried are used as query features, and the query features are queried in the database, that is, the similarity between the query features and the progressive multi-source fused pedestrian features of each reference sample in the database is calculated, and the person to be queried is archived as a pseudo-label corresponding to the progressive multi-source fused pedestrian features of the reference sample in the database with the highest similarity to the query features. Therefore, the face images and body images of the person to be queried and their corresponding face simulation sketch portraits and body simulation sketch portraits can be archived together with the reference sample in the database with the highest similarity to obtain the final portrait clustering result.

[0111] In one of the examples, Euclidean distance can be used as the similarity measurement method. Therefore, the pseudo-label corresponding to the pedestrian feature of the progressive multi-source fusion of the reference sample in the database with the smallest Euclidean distance to the query feature can be selected as the portrait clustering result of the person to be queried. In other examples, other similarity measurement methods such as cosine similarity can also be selected.

[0112] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a portrait gathering device based on progressive multi-source feature fusion and alignment. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0113] The embodiment of the present application provides a portrait gathering device based on progressive multi-source feature fusion and alignment, comprising:

[0114] The model building module 1 is configured to build a portrait cluster model, which includes a face feature extraction unit and a body feature extraction unit. The face feature extraction unit includes a first shallow network, a second shallow network and a first deep network, and the body feature extraction unit includes a third shallow network, a fourth shallow network and a second deep network. In the training process of the portrait cluster model, the first shallow network and the second shallow network are used to extract shallow features of face images and shallow features of face simulation sketches, respectively. The shallow features of face images and shallow features of face simulation sketches are connected in series and then input into the first deep network to generate face features. The third shallow network and the fourth shallow network are used to extract human features, respectively. shallow features of the human body image and shallow features of the human body simulation sketch, connect the shallow features of the human body image and the shallow features of the human body simulation sketch in series, and then input them into the second deep network to generate human body features; superimpose the facial features and the human body features to obtain the pedestrian features of progressive multi-source fusion, cluster based on the pedestrian features of progressive multi-source fusion, generate pseudo labels, construct a cross entropy loss function according to the pseudo labels and the pedestrian features of progressive multi-source fusion, construct a multi-source feature alignment loss function according to the facial features and the human body features, construct a total loss function according to the cross entropy loss function and the multi-source feature alignment loss function, train the portrait clustering model based on the total loss function to obtain the trained portrait clustering model;

[0115] Database construction module 2 is configured to collect face images, body images and their corresponding face simulation sketch portraits and body simulation sketch portraits of several reference samples and input them into the trained portrait clustering model to obtain the pedestrian features of each reference sample through progressive multi-source fusion, perform clustering based on the pedestrian features of all reference samples through progressive multi-source fusion, generate a pseudo label corresponding to each reference sample, and store the pedestrian features of each reference sample through progressive multi-source fusion and its corresponding pseudo label in the database;

[0116] The clustering module 3 is configured to obtain the face image, body image and their corresponding face simulation sketch portrait and body simulation sketch portrait of the person to be queried and input them into the trained portrait clustering model to obtain the progressive multi-source fused pedestrian features of the person to be queried, calculate the similarity between the progressive multi-source fused pedestrian features of the person to be queried and the progressive multi-source fused pedestrian features of each reference sample in the database, and take the pseudo-label corresponding to the progressive multi-source fused pedestrian features of the reference sample in the database with the highest similarity as the portrait clustering result of the person to be queried.

[0117] Figure 4 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 4As shown, the electronic device of this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer-executable instructions; the processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0118] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0119] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0120] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 401 executes the computer execution instructions, the above method is implemented.

[0121] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 401, the above method is implemented.

[0122] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0123] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0124] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0125] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 401 to perform some steps of the methods of various embodiments of the present application.

[0126] It should be understood that the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 401 may be any conventional processor 401, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 401 for execution, or may be executed by a combination of hardware and software modules in the processor 401.

[0127] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0128] The bus 403 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 403 in the drawings of the present application is not limited to only one bus 403 or one type of bus 403.

[0129] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0130] An exemplary storage medium is coupled to the processor 401, so that the processor 401 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 401. The processor 401 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 401 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0131] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A portrait clustering method based on progressive multi-source feature fusion and alignment, characterized in that: The following steps are involved: Constructing a portrait polyarch model, the portrait polyarch model includes a face feature extraction unit and a body feature extraction unit, the face feature extraction unit includes a first shallow network, a second shallow network and a first deep network, and the body feature extraction unit includes a third shallow network, a fourth shallow network and a second deep network; the first shallow network and the second shallow network are used to extract shallow features of face images and shallow features of face simulation sketch images, respectively, the shallow features of face images and shallow features of face simulation sketch images are connected in series through channels and then input into the first deep network to generate face features; the third shallow network and the fourth shallow network are used to extract shallow features of body images and shallow features of body simulation sketch images, respectively, the shallow features of body images and shallow features of body simulation sketch images are connected in series through channels and then input into the second deep network to generate body features, and the face features and body features are superimposed to obtain pedestrian features of progressive multi-source fusion; In the training process of the portrait clustering model, the pedestrian features of the progressive multi-source fusion of all samples in the training data are clustered to generate pseudo labels, a cross entropy loss function is constructed according to the pseudo labels and the pedestrian features of the progressive multi-source fusion, a multi-source feature alignment loss function is constructed according to the facial features and the body features, a total loss function is constructed according to the cross entropy loss function and the multi-source feature alignment loss function, and the portrait clustering model is trained based on the total loss function to obtain a trained portrait clustering model; Collecting face images, body images and their corresponding face simulation sketch portraits and body simulation sketch portraits of several reference samples and inputting them into the trained portrait clustering model, obtaining the progressive multi-source fused pedestrian features of each reference sample, clustering based on the progressive multi-source fused pedestrian features of all reference samples, generating pseudo labels corresponding to each reference sample, and storing the progressive multi-source fused pedestrian features of each reference sample and their corresponding pseudo labels in a database; Acquire the face image, body image and their corresponding face simulation sketch portrait and body simulation sketch portrait of the person to be queried and input them into the trained portrait clustering model to obtain the progressive multi-source fused pedestrian features of the person to be queried, calculate the similarity between the progressive multi-source fused pedestrian features of the person to be queried and the progressive multi-source fused pedestrian features of each reference sample in the database, and take the pseudo-label corresponding to the progressive multi-source fused pedestrian features of the reference sample in the database with the highest similarity as the portrait clustering result of the person to be queried.

2. The portrait clustering method based on progressive multi-source feature fusion and alignment according to claim 1, characterized in that: The first shallow network, the second shallow network, the third shallow network and the fourth shallow network all use shallow neural networks, and the structure of the shallow neural network is the block linear mapping layer and the first 6 layers of Transformer blocks in the ViT-Base model; the first deep network and the second deep network use deep neural networks, and the structure of the deep neural network is the last 6 layers of Transformer blocks in the ViT-Base model.

3. The portrait clustering method based on progressive multi-source feature fusion and alignment according to claim 1, characterized in that: The first shallow network and the second shallow network are used to extract shallow features of the face image and shallow features of the face simulation sketch respectively, and the shallow features of the face image and the shallow features of the face simulation sketch are connected in series and then input into the first deep network to generate face features, specifically including: The face image and the face simulation sketch are input into the first shallow network and the second shallow network respectively, and the shallow features of the face image and the shallow features of the face simulation sketch are extracted, as shown in the following formula: ； ； in, and They represent face images and face simulation sketches respectively, B is the number of input samples, the number of channels of each sample is 3, the height is H, and the width is W; represents the first shallow network, represents the second shallow network, and denote the learnable parameters of the first shallow network and the second shallow network respectively, is the shallow feature of face image, is the shallow feature of the face simulation sketch, N is the number of feature blocks of the face, and D is the number of channels of the shallow feature; The shallow features of the face image and the shallow features of the face simulation sketch are connected in series to obtain a first series feature, as shown in the following formula: ； in, represents the first series feature, Indicates that channels are connected in series; The first series feature is input into the first deep network to obtain the face feature, as shown in the following formula: ； in, Represents facial features, represents the first deep network, are the learnable parameters of the first deep network.

4. The portrait clustering method based on progressive multi-source feature fusion and alignment according to claim 1, characterized in that: The third shallow network and the fourth shallow network are used to extract shallow features of the human body image and shallow features of the human body simulation sketch respectively, connect the shallow features of the human body image and the shallow features of the human body simulation sketch in series, and then input them into the second deep network to generate human body features, specifically including: The human body image and the human body simulation sketch image are respectively input into the third shallow network and the fourth shallow network, and the shallow features of the human body image and the shallow features of the human body simulation sketch image are extracted, as shown in the following formula: ； ； in, and are human body images and human body simulation sketch portraits, respectively. B is the number of input samples. The number of channels of each sample is 3, the height is h, and the width is w. represents the third shallow network, represents the fourth shallow network, and Represent the learnable parameters of the third shallow network and the fourth shallow network respectively, is the shallow feature of human image, is the shallow feature of the human body simulation sketch, M is the number of feature blocks of the human body, and D is the number of channels of the shallow feature; The shallow features of the human body image and the shallow features of the human body simulation sketch are connected in series to obtain the second series feature, as shown in the following formula: ； in, represents the second series feature, Indicates that channels are connected in series; The second series feature is input into the second deep network to obtain the human body feature, as shown in the following formula: ； in, Represents human characteristics, represents the second deep network, are the learnable parameters of the second deep network.

5. The portrait clustering method based on progressive multi-source feature fusion and alignment according to claim 1, characterized in that: The facial features and body features are superimposed to obtain pedestrian features of progressive multi-source fusion, specifically including: The facial features and body features are superimposed to obtain pedestrian features of progressive multi-source fusion, as shown in the following formula: ； in, Represents facial features, Represents human characteristics, It represents the pedestrian features of progressive multi-source fusion, B is the number of input samples, and D is the number of channels of shallow features.

6. The portrait clustering method based on progressive multi-source feature fusion and alignment according to claim 1, characterized in that: Clustering is performed based on the progressive multi-source fused pedestrian features of all samples in the training data to generate pseudo labels, a cross entropy loss function is constructed according to the pseudo labels and the progressive multi-source fused pedestrian features, a multi-source feature alignment loss function is constructed according to the face features and the body features, and a total loss function is constructed according to the cross entropy loss function and the multi-source feature alignment loss function, specifically including: The progressive multi-source fused pedestrian features of all samples in the training data are clustered and divided into C clusters through clustering. The samples in each cluster have the same pseudo label, and the pseudo label of each sample in the training data is obtained as shown in the following formula: ； in, represents pedestrian features of progressive multi-source fusion, Cluster represents clustering operation, represents the pseudo label, B is the number of input samples, and C represents the number of pseudo label clusters; The cross entropy loss function is expressed as: ； in, represents the cross entropy ( ) loss function, It represents the linear projection of the pedestrian features of progressive multi-source fusion into learnable parameters of the posterior probability. Respectively represent the learnable parameters of the first shallow network, the second shallow network, the first deep network, the third shallow network, the fourth shallow network, and the second deep network; The multi-source feature alignment loss function is expressed as: ； in, Represents facial features, Represents human characteristics, represents the multi-source feature alignment loss function, represents the L2 norm; The total loss function is expressed as: ； in, represents the total loss function, is a hyperparameter.

7. A portrait gathering device based on progressive multi-source feature fusion and alignment, characterized in that: include: A model building module is configured to build a portrait polymorphic model, the portrait polymorphic model includes a face feature extraction unit and a body feature extraction unit, the face feature extraction unit includes a first shallow network, a second shallow network and a first deep network, and the body feature extraction unit includes a third shallow network, a fourth shallow network and a second deep network; the first shallow network and the second shallow network are used to extract shallow features of face images and shallow features of face simulation sketches, respectively, the shallow features of face images and shallow features of face simulation sketches are connected in series through channels and then input into the first deep network to generate face features; the third shallow network and the fourth shallow network are used to extract shallow features of body images and shallow features of body simulation sketches, respectively, the shallow features of body images and shallow features of body simulation sketches are connected in series through channels and then input into the second deep network to generate body features, and the face features and body features are superimposed to obtain pedestrian features of progressive multi-source fusion; In the training process of the portrait clustering model, the pedestrian features of the progressive multi-source fusion of all samples in the training data are clustered to generate pseudo labels, a cross entropy loss function is constructed according to the pseudo labels and the pedestrian features of the progressive multi-source fusion, a multi-source feature alignment loss function is constructed according to the facial features and the body features, a total loss function is constructed according to the cross entropy loss function and the multi-source feature alignment loss function, and the portrait clustering model is trained based on the total loss function to obtain a trained portrait clustering model; A database construction module is configured to collect face images, body images and their corresponding face simulation sketch portraits and body simulation sketch portraits of a plurality of reference samples and input them into the trained portrait clustering model, obtain the pedestrian features of each reference sample through progressive multi-source fusion, perform clustering based on the pedestrian features of all reference samples through progressive multi-source fusion, generate a pseudo label corresponding to each reference sample, and store the pedestrian features of each reference sample through progressive multi-source fusion and its corresponding pseudo label in the database; The profiler module is configured to obtain the face image, body image and their corresponding face simulation sketch portrait and body simulation sketch portrait of the person to be queried and input them into the trained portrait profiler model to obtain the progressive multi-source fused pedestrian features of the person to be queried, calculate the similarity between the progressive multi-source fused pedestrian features of the person to be queried and the progressive multi-source fused pedestrian features of each reference sample in the database, and use the pseudo-label corresponding to the progressive multi-source fused pedestrian features of the reference sample in the database with the highest similarity as the portrait profiler result of the person to be queried.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images