Age estimation method and device based on multi-modal feature fusion
By extracting multimodal features using the CLIP model and performing adaptive weighted fusion, the problem of low age estimation accuracy under single modality is solved, achieving higher age estimation accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 709TH RESEARCH INSTITUTE CHINA STATE SHIPBUILDING CORP LTD
- Filing Date
- 2026-01-07
- Publication Date
- 2026-05-15
AI Technical Summary
Existing age estimation methods rely on a single image modality, resulting in low accuracy and poor adaptability in complex scenes.
The CLIP model is used to extract facial image and gender text feature vectors. Multimodal feature fusion is performed through the attention fusion module, and the joint loss function is used to optimize the model parameters to achieve adaptive weighted fusion and age regression.
It improves the accuracy and robustness of age estimation and is suitable for various face analysis and intelligent recognition scenarios.
Smart Images
Figure CN122046291A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning and multimodal feature fusion technology, and in particular to an age estimation method and apparatus based on multimodal feature fusion. Background Technology
[0002] Age estimation is a fundamental task in computer vision, widely applied in real-world scenarios such as security monitoring, financial risk control, and intelligent recommendation. Currently, most common age estimation methods rely on visual information from facial images, extracting facial features using deep convolutional neural networks and predicting age through regression or classification models. While these methods have achieved some success on standardized datasets, they still face numerous challenges in practical applications.
[0003] On the one hand, factors such as lighting, occlusion, and changes in facial expressions can lead to insufficient information and ambiguous representations in image features under a single modality, resulting in unstable prediction results. On the other hand, existing methods generally ignore other key factors affecting age perception, such as gender and context, which makes the model's generalization ability and individual adaptability poor.
[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention
[0005] The technical problem to be solved by this invention is that existing age estimation methods rely on a single image modality, resulting in low accuracy and poor adaptability in complex scenes.
[0006] To achieve the above objectives, in a first aspect, the present invention provides an age estimation method based on multimodal feature fusion, comprising: The image encoder in the CLIP model is used to extract the image feature vector of the face image, and the text encoder in the CLIP model is used to extract the text feature vector of the gender text description generated by the gender label; The image feature vector and the text feature vector are concatenated and input into the attention fusion module to generate a fused feature vector; The fused feature vector is input into the regression module for regression processing; The trainable parameters of the attention fusion module, the regression module, and the image encoder and text encoder in the CLIP model are updated using a joint loss function to complete the training process of the CLIP model.
[0007] Furthermore, the method further includes: Obtain a sample dataset containing face images, gender labels, and age labels, and perform face image detection, cropping, and standardization processing; The gender label is converted into a gender text description to serve as input for the text encoder.
[0008] Furthermore, the age estimation method also includes: The concatenated feature vector is input into the first attention fully connected layer for nonlinear transformation to generate an intermediate feature vector. The intermediate feature vector is input into the second attention fully connected layer for nonlinear transformation to generate attention weights; The image feature vector and the text feature vector are weighted and summed using the attention weights to obtain the fused feature vector.
[0009] Furthermore, the expression for the attention weight is: ; in, For attention weights, The weights are: W1 = Sigmoid function, W2 = second-layer weight matrix, and W1 = first-layer weight matrix. I For image feature vectors, T Let b1 be the bias term of the first attention fully connected layer and b2 be the bias term of the second attention fully connected layer. The expression for the fused feature vector is: ; in, F To fuse feature vectors, For attention weights, I For image feature vectors, T This is the gender text feature vector.
[0010] Furthermore, the regression module includes a first fully connected regression layer and a second fully connected regression layer connected in sequence: The first fully connected regression layer is used to perform non-linear transformations on the fused feature vector to extract intermediate features; The second fully connected regression layer is used to perform age regression calculations based on the intermediate features.
[0011] Furthermore, the joint loss function includes: L1 loss based on the difference between predicted age and actual age label; And a contrastive loss based on the cosine similarity between image feature vectors and text feature vectors.
[0012] Furthermore, the contrast loss is calculated based on the cosine similarity between the image feature vector and the text feature vector corresponding to the gender text description generated from the real gender label, as well as the cosine similarity between the image feature vector and the text feature vector corresponding to the gender text description generated from a gender label unrelated to the image.
[0013] Furthermore, the expression for the joint loss function is as follows: ; Where L is the joint loss function, These are the weighting coefficients for L1 loss. y represents the predicted age value, and y represents the actual age label. The weighting coefficients for the contrastive loss are given, and m represents the boundary of the contrastive loss. The cosine similarity between the image feature vector and the gender text feature vector is given. This represents the text feature vector corresponding to the gender text description generated from the real gender labels. This represents the text feature vector corresponding to the incorrect gender text description generated by gender labels unrelated to the image; To predict the L1 loss between age and the actual age label.
[0014] Furthermore, the formula for calculating the cosine similarity is as follows: ; in, I For image feature vectors, T This is the gender text feature vector.
[0015] Secondly, the present invention also provides an age estimation device based on multimodal feature fusion, used to implement the age estimation method based on multimodal feature fusion described in the first aspect, the device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the age estimation method based on multimodal feature fusion as described in the first aspect.
[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method described in the first aspect and / or the method described in the second aspect.
[0017] Unlike existing technologies, the present invention has at least the following beneficial effects: This invention acquires multimodal sample data, extracts facial image feature vectors and gender text feature vectors using the CLIP model, and then uses an attention fusion module to adaptively weight and fuse the multimodal features to obtain a more discriminative age representation. Finally, a regression module completes age prediction. This method effectively improves the accuracy and robustness of age estimation and is applicable to various face analysis and intelligent recognition scenarios. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0019] Figure 1 This is a flowchart illustrating an age estimation method based on multimodal feature fusion provided in Embodiment 1. Figure 2 This is a schematic diagram of the data preprocessing process for an age estimation method based on multimodal feature fusion provided in this embodiment 1; Figure 3 This is a flowchart illustrating the process of concatenating the image feature vector and the text feature vector, and inputting them into the attention fusion module to generate a fused feature vector. Figure 4 This is a schematic diagram of the process of inputting the fused feature vector into the regression module for regression processing; Figure 5 This is a flowchart illustrating the process of updating the CLIP model using the joint loss function for the attention fusion module, the regression module, and the trainable parameters. Figure 6 This is a schematic diagram of the inference process of the CLIP model after training, provided in this second embodiment. Figure 7 This is an architecture diagram of an age estimation device based on multimodal feature fusion provided in Embodiment 3 of the present invention. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0021] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0022] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0023] Example 1: Embodiment 1 of this invention provides an age estimation method based on multimodal feature fusion, such as... Figure 1 As shown, it includes: Step 10: Use the image encoder in the CLIP model to extract the image feature vector of the face image, and use the text encoder in the CLIP model to extract the text feature vector of the gender text description generated by the gender label.
[0024] Contrast Languages Contrastive Language-Image Pre-training (CLIP) is a multimodal pre-trained model trained on large-scale image-text pairing data. It embeds images and text into a unified semantic space through contrastive learning, exhibiting excellent cross-modal representation capabilities and has been widely applied in tasks such as image-text retrieval and image classification. The image encoder, a visual feature extraction module in the CLIP model, converts the input face image into a fixed-dimensional image feature vector to represent image information. The text encoder, a text feature extraction module in the CLIP model, converts the input gender text description into a text feature vector with the same dimensionality as the image feature vector.
[0025] Step 20: Concatenate the image feature vector and the text feature vector, and input them into the attention fusion module to generate a fused feature vector.
[0026] Step 30: Input the fused feature vector into the regression module for regression processing.
[0027] Step 40: Update the trainable parameters of the attention fusion module, the regression module, and the image encoder and text encoder in the CLIP model using the joint loss function to complete the training process of the CLIP model.
[0028] The joint loss function is a loss function composed of multiple loss terms, used to simultaneously constrain different optimization objectives during model training, guiding it to update in a direction favorable to age prediction. The trainable parameters include weight and bias parameters from the image encoder, text encoder, attention fusion module, and regression module.
[0029] This invention acquires multimodal sample data, extracts facial image feature vectors and gender text feature vectors using the CLIP model, and then achieves adaptive weighted fusion of multimodal features through an attention fusion module to enhance the effective representation of age-related information. This method effectively improves the accuracy and robustness of age estimation and is applicable to various face analysis and intelligent recognition scenarios.
[0030] In another embodiment, refer to Figure 2 As shown, the age estimation method based on multimodal feature fusion further includes a data collection step, comprising: Step 11: Obtain a sample dataset containing face images, gender labels, and age labels; perform detection, cropping, and standardization processing on the face images; and use the processed face images as input to the image encoder. Step 12: Convert the gender tags into gender text descriptions to serve as input for the text encoder; This embodiment uses the UTKFace public dataset, which contains approximately 23,708 face images ranging in age from 0 to 116 years old, with built-in age and gender labels. MTCNN is used to detect face regions and locate key points in each image. The detected face regions are then cropped and uniformly scaled to a standard size of 224×224 pixels to meet the input requirements of the CLIP model image encoder. The dataset is then randomly divided into training, validation, and test sets in an 8:1:1 ratio. For each image, a corresponding gender description is automatically generated based on its gender label, such as "a photo of a male face" and "a photo of a female face," for subsequent feature matching. A multi-task cascaded convolutional network (MTCNN) is specifically designed for face detection and facial landmark alignment. The MTCNN has excellent face detection and five-point key point localization capabilities, which can filter out background interference and irrelevant areas, retaining only clear face regions as input models, thus helping to improve the semantic focus in the image feature extraction stage.
[0031] By using the MTCNN to accurately detect and crop faces, and uniformly scaling them to a 224×224 pixel size suitable for the CLIP model, the proportions and format of each input image are consistent, avoiding feature representation bias caused by inconsistent image scales and improving the stability and generalization ability of the image encoder. In the above steps, the dataset is divided into training, validation, and test sets in an 8:1:1 ratio, which conforms to the conventional partitioning principles of machine learning tasks and helps in evaluating the model's generalization ability and optimizing parameter tuning.
[0032] The preprocessed 224×224 pixel face image is input into the CLIP model's image encoder.
[0033] In this embodiment, the image encoder of the CLIP model adopts a visual transformation architecture, specifically the ViT-B / 32 architecture (Vision Transformer Base). Here, "32" represents that the image segmentation block size processed by the model is 32x32 pixels.
[0034] The encoder outputs a 512-dimensional visual feature vector. The text encoder receives template text (e.g., "a photo of a male face" and "a photo of a female face") and outputs a 512-dimensional text feature vector. This pre-trained model is used because, as one of the CLIP standard models, it can generate high-quality, universal image feature vector representations, and its 512-dimensional output feature vector has the same dimension as the text encoder's output, facilitating subsequent feature fusion.
[0035] In another embodiment, the concatenated vector is input to an attention module, which consists of two fully connected layers: The first attention fully connected layer is used to perform non-linear transformations on the concatenated feature vectors, with an output dimension of 512 and an activation function of ReLU. The second attention fully connected layer is used to generate attention weights based on the output of the first attention fully connected layer. The output dimension is 1, and the activation function is Sigmoid to obtain the attention weights.
[0036] The step of generating the fused feature vector includes: The concatenated feature vector is input into the first attention fully connected layer for nonlinear transformation to generate an intermediate feature vector; the intermediate feature vector is input into the second attention fully connected layer for nonlinear transformation to generate attention weights; the attention weights are used to perform a weighted summation of the image feature vector and the text feature vector to obtain the fused feature vector.
[0037] The first fully connected attention layer, through nonlinear transformation, fully mines the deep correlation information in the spliced features, providing a more discriminative intermediate representation for the generation of attention weights. The second fully connected attention layer uses the sigmoid activation function to generate normalized attention weights, enabling adaptive modeling of the importance of image feature vectors and gender text feature vectors. By weighted summing of the two types of features according to the attention weights, the modal information that is more critical to age prediction can be highlighted, thereby improving the expressive power of the fused feature vector and the accuracy of age estimation.
[0038] In one embodiment, refer to Figure 3 As shown, step 20 includes: Step 201: Concatenate the image feature vector with the gender text feature vector to obtain a concatenated vector.
[0039] Step 202: Input the concatenated vector into an attention fusion module containing at least two fully connected layers, calculate the attention weights through the attention fusion module, and perform weighted processing on the concatenated vector according to the attention weights to obtain the fused feature vector.
[0040] By concatenating the image feature vector with the gender text feature vector before feature fusion, the complete representation of both modalities can be preserved in a single vector. Subsequently, an attention fusion module is used to weight the concatenated features, enabling the model to adaptively adjust the contribution ratio of the image feature vector and the gender text feature vector based on the characteristics of different samples during the fusion stage. This two-step feature fusion method avoids modal conflicts that may result from simple concatenation and improves the consistency and effectiveness of the fused features in age prediction tasks, thereby enhancing multimodal collaborative capabilities and prediction accuracy.
[0041] The expression for the attention weight is: ; in, For attention weights, W1 is the weight matrix of the first attention fully connected layer, where W1 is the weight matrix of the second attention fully connected layer and W2 is the weight matrix of the second attention fully connected layer. I For image feature vectors, T Let b1 be the bias term of the first attention fully connected layer and b2 be the bias term of the second attention fully connected layer. The expression for the fusion feature is: ; in, F As a feature of fusion, For attention weights, I For image feature vectors,T This is the gender text feature vector.
[0042] In this embodiment, the attention weight value is between 0 and 1. W1 is the Sigmoid function, W2 is the weight matrix of the second attention fully connected layer with dimensions (1,256), and W1 is the weight matrix of the first attention fully connected layer with dimensions (256,1024). I For image feature vectors, T This is the gender text feature vector. b1 is the bias term of the first attention fully connected layer, with dimensions (256, 1), and b2 is the bias term of the second attention fully connected layer, with dimensions (1, 1). The image feature vector... I With gender text feature vector T Concatenate along the feature dimension to obtain the concatenated feature vector. I ; T ], its dimensions are (1, 1024).
[0043] At this point, the image feature vector I (1, 512) and gender text feature vector T (1, 512) is weighted and fused to form a multimodal feature representation, resulting in the generated fused feature vector. F (1, 512) Compared to simple concatenation or fixed-weight fusion methods, this attention mechanism adaptively learns the respective weights of image features and gender text features in the age prediction task for the current input sample. For example, for samples with obvious gender characteristics, the model may assign higher weights to gender text features; while for samples with unique age features or better image quality, the model may rely more on image feature vectors. This setting can adaptively allocate the weights of image and text information, effectively alleviating the problem of insufficient single-modal representation information.
[0044] In one embodiment, the regression module includes a first fully connected regression layer and a second fully connected regression layer connected in sequence: the first fully connected regression layer is used to perform a nonlinear transformation on the fused feature vector to extract intermediate features; the second fully connected regression layer is used to perform age regression operations based on the intermediate features.
[0045] Furthermore, referring to Figure 4 As shown, step 30 includes: Step 301: Input the fused feature vector into the first fully connected regression layer. The first fully connected layer uses an activation function to perform a nonlinear transformation to extract intermediate representation features.
[0046] Step 302: Input the intermediate representation features into the second fully connected regression layer, which is a linear output layer, to obtain the age prediction value.
[0047] This two-layer regression structure achieves nonlinear feature extraction in the first fully connected regression layer and linear regression output in the second fully connected regression layer, enabling the regression module to have both effective feature representation capabilities and stable prediction capabilities, thereby improving the overall accuracy of age estimation.
[0048] In one embodiment, in step 40, the joint loss function includes: The loss is based on the L1 loss between the predicted age and the actual age label; and the contrast loss is based on the cosine similarity between the image feature vector and the text feature vector. The contrast loss is calculated based on the cosine similarity between the image feature vector and the text feature vector corresponding to the gender text description generated from the actual gender label, and the cosine similarity between the image feature vector and the text feature vector corresponding to the gender text description generated from the gender label unrelated to the image.
[0049] The L1 loss, also known as the mean absolute error loss, is a loss function used in regression tasks. Its goal is to make the model's predicted values as close as possible to the true values. The core idea of the contrastive loss is to bring the feature representations of similar samples closer together and push away the feature representations of dissimilar samples. In multimodal scenarios, it is used to align image feature vectors with text feature vectors in a unified semantic space.
[0050] In another embodiment, refer to Figure 5 As shown, step 40 includes: Step 401: Construct a joint loss function, which includes an L1 loss to measure the difference between the predicted age and the true age label, and a contrastive loss to measure the semantic consistency between image features and gender text features. By constructing a joint loss function that includes both L1 loss and contrastive loss, the model can simultaneously optimize the age regression accuracy and the semantic consistency between image features and gender text features during training. Step 402: Input the cropped and standardized face image, the gender text description generated by the gender label, and the corresponding age label into the model, and perform forward propagation to obtain the predicted age and loss value.
[0051] Step 403: Based on the loss value, the trainable parameters of the attention fusion module, the regression module, and the image encoder and text encoder in the CLIP model are updated through the backpropagation algorithm, so that each module can be optimized collaboratively under the constraint of a unified loss function, thereby improving the discriminativeness and stability of the fused features.
[0052] Step 404: Repeat steps 402 to 403 until the loss function converges to complete the training process of the CLIP model.
[0053] By setting steps 401 to 404 above, the present invention introduces age regression error constraints and multimodal semantic consistency constraints simultaneously during model training. This enables the model to optimize age prediction accuracy while further enhancing the alignment and fusion quality of image features and gender text features in a unified feature space, thereby improving the overall accuracy and generalization ability of the CLIP model.
[0054] In another embodiment, the joint loss function is defined as follows: ; Where L is the joint loss function, These are the weighting coefficients for L1 loss. y represents the predicted age value, and y represents the actual age label. The weighting coefficients for the contrastive loss are given, and m represents the boundary of the contrastive loss. The cosine similarity between the image feature vector and the gender text feature vector is given. This represents the text feature vector corresponding to the gender text description generated from the real gender labels. This represents the text feature vector corresponding to the incorrect gender text description generated by gender labels unrelated to the image; To predict the L1 loss between age and the actual age label.
[0055] The contrast loss is calculated using cosine similarity, and the calculation formula is as follows: ; in, I For image feature vectors, T Let be the gender text feature vector. In the above formula, the numerator is the sum of the product of the corresponding components of the two vectors, and the denominator is the product of the square roots of the squares of each vector's components.
[0056] Using the training set data, the model parameters are updated via standard backpropagation; the update scope includes all parameters of the attention fusion module, regression module, and CLIP encoder. The loss is monitored on the validation set, and once the loss converges, the model with the best performance on the test set is selected. This setup ensures that the L1 loss guarantees accuracy for the primary task of age prediction. The CLIP contrastive loss continuously narrows the distance between the image and its true semantic description during training, while widening the distance between it and incorrect descriptions, thereby strengthening the alignment of multimodal features in the semantic space. This improved alignment results in higher quality fused features, ultimately improving the accuracy of age estimation and effectively suppressing bias caused by gender distribution bias in the training data.
[0057] To select the best-performing model parameter configuration, mean absolute error and root mean square error are introduced as evaluation metrics during training to measure the model's prediction accuracy on the validation set. The model with the smallest error is selected as the final deployment version.
[0058] In this embodiment, 10,000 face images were randomly selected from the UTKFace dataset for a comparative experiment. The experimental results are shown in Table 1. Table 1. Performance Comparison Results of Different Age Estimation Methods
[0059] In the table above, CLIP (image encoder) + fully connected layer refers to the result without feature fusion, directly using the image encoder to extract features. ResNet-50 (image) is a benchmark comparison model that uses only a ResNet-50 convolutional neural network to extract features from face images and perform age regression, representing a traditional unimodal age estimation scheme that does not introduce any multimodal information. As shown in Table 1, the method of this invention outperforms the traditional ResNet-50 model and the CLIP image encoder method without feature fusion in both Mean Absolute Error (MAE) and Mean Squared Error (MSE). Specifically, MAE is reduced from 5.2 in ResNet-50 to 4.1, and MSE is reduced from 34.1 to 22.8, indicating that the present invention has a significant improvement in regression accuracy and stability. Compared with the CLIP+ fully connected structure, this invention introduces a multimodal feature fusion mechanism and a joint loss function, which further enhances the image-text alignment capability, enabling the model to make fuller use of gender semantic information and achieve better performance under the same training data conditions, thus verifying the effectiveness of the method of this invention in the age estimation task.
[0060] Example 2: In the above-described Embodiment 1, the model training process of the age estimation method based on multimodal feature fusion has been described in detail. This Embodiment 2, based on Embodiment 1, further explains how to use the trained CLIP model to infer new input face images in order to obtain the corresponding age prediction results.
[0061] In this embodiment, refer to Figure 6 As shown, the reasoning steps include: Step 31: Perform face detection and cropping on the image to be predicted to obtain a standardized face image that meets the model input requirements.
[0062] Step 32: Input the standardized face image into the image encoder of the CLIP model to extract the image feature vector.
[0063] Step 33: Construct text prompt templates to distinguish between different genders, as candidate gender text descriptions.
[0064] Step 34: Input the candidate gender text descriptions into the CLIP text encoder to obtain the corresponding text feature vectors.
[0065] Step 35: Calculate the cosine similarity between the image feature vector and each candidate text feature vector, and select the gender text description with the highest cosine similarity as the gender text description of the image.
[0066] Step 36: Input the image feature vector and the text feature vector corresponding to the gender text description into the attention fusion module to obtain the fused feature vector.
[0067] Step 37: Input the fused feature vector into the regression module to obtain the predicted age value of the image.
[0068] In this embodiment, firstly, face detection and cropping are performed on the image to be predicted to obtain a standardized face image that meets the model input requirements. For better face detection alignment, a multi-task cascaded convolutional neural network (MTCNN) based on deep learning is used. The standardized face image, after detection and cropping by MTCNN, is uniformly adjusted to 224×224 pixels to ensure consistency with the input format during the training phase. Subsequently, the standardized face image is input into the image encoder of the CLIP model to extract the corresponding image feature vector I. To obtain gender-related text features, gender text descriptions for distinguishing different genders are pre-constructed, such as "a photo of a male face" and "a photo of a female face," and these candidate gender text descriptions are respectively input into the text encoder of the CLIP model to obtain their respective corresponding text feature vectors T. male With T pemale Next, the cosine similarity between the image feature vector I and each candidate text feature vector is calculated, and the gender text description with the highest cosine similarity is selected as the gender text description of the image. Simultaneously, its corresponding text feature vector is determined. T The text feature vector is then used as the subsequent fusion vector. Next, the image feature vector I and the text feature vector T are input together into the trained attention fusion module, which generates a fused feature vector F through an attention weighting mechanism. Finally, the fused feature vector F is input into the regression module, which calculates and outputs the predicted age value of the image through a two-layer fully connected network (512→256→1). .
[0069] This invention utilizes zero-sample gender features by leveraging the open-world visual and language understanding capabilities of the CLIP model. It directly reuses the CLIP pre-trained image encoder and text encoder, obtaining gender information without the need for additional training of a gender classifier. It also suppresses cross-modal bias by introducing gender-specific text as a conditional input and fusing text feature vectors and image feature vectors to reduce prediction bias caused by gender imbalance.
[0070] Example 3: like Figure 7 The diagram shown is an architectural schematic of an age estimation device based on multimodal feature fusion according to an embodiment of the present invention. This age estimation device based on multimodal feature fusion includes one or more processors 21 and a memory 22. Figure 7 Take a processor 21 as an example.
[0071] Processor 21 and memory 22 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.
[0072] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the method in Embodiment 1. The processor 21 executes the method by running the non-volatile software program and instructions stored in the memory 22.
[0073] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0074] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they perform the method described in Embodiment 1 above, for example, the method described above. Figures 1 to 6 The steps shown.
[0075] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 7 One of the processors 21 can enable the one or more processors to execute the social methods in the first and second embodiments, for example, to execute the methods described above. Figures 1 to 6 The steps shown.
[0076] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.
[0077] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.
[0078] Those skilled in the art will readily understand that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An age estimation method based on multimodal feature fusion, characterized in that, The age estimation method includes: The image encoder in the CLIP model is used to extract the image feature vector of the face image, and the text encoder in the CLIP model is used to extract the text feature vector of the gender text description generated by the gender label; The image feature vector and the text feature vector are concatenated and input into the attention fusion module to generate a fused feature vector; The fused feature vector is input into the regression module for regression processing; The trainable parameters of the attention fusion module, the regression module, and the image encoder and text encoder in the CLIP model are updated using a joint loss function to complete the training process of the CLIP model, so as to use the trained CLIP model for age estimation.
2. The age estimation method based on multimodal feature fusion according to claim 1, characterized in that, The age estimation method also includes: Obtain a sample dataset containing face images, gender labels, and age labels; detect, crop, and standardize the face images; and use the processed face images as input to the image encoder. The gender label is converted into a gender text description to serve as input for the text encoder.
3. The age estimation method based on multimodal feature fusion according to claim 1, characterized in that, The age estimation method also includes: The concatenated feature vector is input into the first attention fully connected layer for nonlinear transformation to generate an intermediate feature vector. The intermediate feature vector is input into the second attention fully connected layer for nonlinear transformation to generate attention weights; The image feature vector and the text feature vector are weighted and summed using the attention weights to obtain the fused feature vector.
4. The age estimation method based on multimodal feature fusion according to claim 3, characterized in that, The expression for the attention weight is: ； in, For attention weights, W1 is the weight matrix of the first attention fully connected layer, where W1 is the weight matrix of the second attention fully connected layer and W2 is the weight matrix of the second attention fully connected layer. I For image feature vectors, T Let b1 be the bias term of the first attention fully connected layer and b2 be the bias term of the second attention fully connected layer. The expression for the fused feature vector is: ； in, F To fuse feature vectors, For attention weights, I For image feature vectors, T This is the text feature vector.
5. The age estimation method based on multimodal feature fusion according to claim 1, characterized in that, The regression module includes a first fully connected regression layer and a second fully connected regression layer connected in sequence: The first fully connected regression layer is used to perform non-linear transformations on the fused feature vector to extract intermediate features; The second fully connected regression layer is used to perform age regression calculations based on the intermediate features.
6. The age estimation method based on multimodal feature fusion according to claim 1, characterized in that, The joint loss function includes: L1 loss based on the difference between the predicted age and the actual age label, and contrast loss based on the cosine similarity between the image feature vector and the text feature vector.
7. The age estimation method based on multimodal feature fusion according to claim 6, characterized in that, The expression for the joint loss function is as follows: ； Where L is the joint loss function, These are the weighting coefficients for L1 loss. y represents the predicted age value, and y represents the actual age label. The weighting coefficients for the contrastive loss are given, and m represents the boundary of the contrastive loss. The cosine similarity between the image feature vector and the gender text feature vector is given. This represents the text feature vector corresponding to the gender text description generated from the real gender labels. This represents the text feature vector corresponding to the incorrect gender text description generated by gender labels unrelated to the image; To predict the L1 loss between age and the actual age label.
8. The age estimation method based on multimodal feature fusion according to claim 6, characterized in that, The formula for calculating the cosine similarity is as follows: ； in, I For image feature vectors, T This is a gender-specific text feature vector.
9. An age estimation device based on multimodal feature fusion, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the age estimation method based on multimodal feature fusion as described in any one of claims 1-8.
10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the age estimation method based on multimodal feature fusion as described in any one of claims 1-8.