Multi-modal age estimation system and method based on embedded regularization double-branch fusion
By embedding regularized dual-branch fusion network to process speech and face data, generate external product sum summation age embedding features, solving the problem of modal missing in single-modal age estimation and multimodal age estimation, achieving a lower Estimation error and system robustness, especially in the absence of data in criminal investigation scenarios.
Patent Information
- Application Number
- CN202310507061.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-08
- Publication Date
- 2024-11-08
AI Technical Summary
When facing complex and multimodal data, existing single-modal age estimation technology is difficult to deal with data loss, resulting in performance degradation and unable to meet the requirements of system robustness. Especially in criminal investigation scenarios, inference cannot be completed when data is missing. Moreover, the development of two age estimation systems is complex and has low cost performance.
Using a multimodal age estimation system based on embedded regularization dual-branch fusion, the voice and face data are processed separately through the embedded extraction network and the dual-branch fusion network, and the embedded extraction, regularization processing and information interaction are performed to generate external product sum finding. and age embedded features to achieve age estimation.
Low average error can be obtained when the modal is missing, such as 6.47 years old when the face mode is missing, 5.56 years old when the voice mode is missing, and 5.09 years old when the modal is not missing, solving the mode in multimodal system state missing problem and achieves an average error of 5.05 years on the AgeVoxCeleb dataset.
Smart Images

Figure CN118918619A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technology in the field of image processing, specifically a multimodal age estimation system and method based on embedded regularized dual-branch fusion. Background Art
[0002] At present, age estimation technology has been widely used in customer analysis and security supervision. With the development of deep learning and pattern recognition technology, more and more deep learning technologies are applied to age estimation technology. However, existing methods are often single-modal, such as face age estimation and voice age estimation, which are difficult to handle more complex and multi-modal data. When faced with complex data, such as data missing and data loss, the performance of single-modal systems will be catastrophically reduced and cannot meet the requirements of system robustness. When faced with multi-modal data, single-modal systems cannot effectively use the complementary information between different modalities for more accurate estimation, which is very inconvenient in actual use, especially in criminal investigation scenarios, where data missing and multi-modal data are often encountered. When deploying a face age estimation system, but the video image is missing and there is only voice, this cannot complete the reasoning, and vice versa. Developing two age estimation systems to deal with data missing is not only complex but also cost-effective. Summary of the invention
[0003] Aiming at the modality missing problem of existing multimodal estimation technology, the present invention proposes a multimodal age estimation method based on embedded regularized dual-branch fusion, which can process single-modal and multi-modal age estimation at the same time.
[0004] The present invention is achieved through the following technical solutions:
[0005] The present invention relates to a multimodal age estimation system based on embedding regularization dual-branch fusion, comprising: an embedding extraction network and a dual-branch fusion network, wherein: the embedding extraction network performs embedding extraction according to speech data and face images respectively to obtain speech embedding features and face embedding features; the dual-branch fusion network performs embedding regularization processing according to the speech embedding features and face embedding features respectively to obtain speech structured sparse embedding features and face structured sparse embedding features, and after information interaction, outer product age embedding features and sum age embedding features are obtained respectively, and then the estimated age is obtained according to the outer product age embedding features and the sum age embedding prediction.
[0006] The present invention relates to a multimodal age estimation method based on the above system. In the offline stage, an embedding extraction network is firstly trained to generate embedded data; the generated embedded data is then used to supervise the training of a dual-branch fusion network; in the online stage, face images and voice data are respectively collected and input into the embedding extraction network to obtain embedded data, and then the embedded data is sent to the dual-branch fusion network to obtain an age estimation result. Technical Effects
[0007] The present invention uses embedding regularization to enhance the single-modal embedding representation capability and solve the modality missing problem in the multimodal system; adopts matrix outer product as the information interaction method to enhance the multimodal embedding interaction capability and reduce the multimodal age estimation error; adopts deep neural network to fuse single-modal enhancement branch and multi-modal enhancement branch to solve single-modal age estimation and multi-modal age estimation at the same time.
[0008] When modality loss occurs, the present invention can rely only on the remaining modality data for age estimation through embedded regularization. When the face modality is missing, an average error of 6.47 years can be obtained, and when the voice modality is missing, an average error of 5.56 years can be obtained. When modality loss does not occur, the present invention can obtain an average error of 5.09 years through matrix outer product. The present invention can simultaneously process single-modality age estimation and multi-modality age estimation through a dual-branch fusion network, solving the common modality loss problem in multi-modal systems while having a lower estimation error, reaching an average error of 5.05 years in the AgeVoxCeleb public data set. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 It is a schematic diagram of the system of the present invention;
[0010] Figure 2 This is a schematic diagram of the speech estimation network;
[0011] Figure 3 This is a schematic diagram of the face estimation network;
[0012] Figure 4 Detailed diagram of the embedding regularization module;
[0013] Figure 5 It is a schematic diagram of the information interaction module;
[0014] Figure 6 This is a detailed diagram of the information interaction module;
[0015] Figure 7 Schematic diagram of the embedding regularization module;
[0016] Figure 8 It is a schematic diagram of information interaction;
[0017] In the figure: (a) sum information interaction, (b) concatenation information interaction, (c) outer product information interaction. DETAILED DESCRIPTION
[0018] like Figure 1As shown, this embodiment relates to a multimodal age estimation system based on embedding regularization dual-branch fusion, including: an embedding extraction network and a dual-branch fusion network, wherein: the embedding extraction network performs embedding extraction according to speech data and face images respectively to obtain speech embedding features and face embedding features; the dual-branch fusion network performs embedding regularization processing according to the speech embedding features and the face embedding features respectively to obtain speech structured sparse embedding features and face structured sparse embedding features, and after information interaction, the outer product age embedding features and the sum age embedding features are obtained respectively, and then the estimated age is obtained according to the outer product age embedding features and the sum age embedding predictions.
[0019] The embedding extraction network comprises: a speech embedding extraction network composed of a speech backbone network and a speech estimation network, and a face embedding extraction network composed of a face detection and alignment network, an image backbone network and an image estimation network, wherein: the speech backbone network extracts Fbank features from the input speech and performs spectrum enhancement and noise enhancement, and generates the speaker's identity embedding features by training the speaker's tasks and locking the parameters; the speech estimation network learns the mapping from the identity embedding features to the speech embedding features to obtain the speech embedding features; the face detection and alignment network crops the input image, performs data enhancement by random horizontal flipping and random grayscale, and the image backbone network performs feature extraction according to the enhanced image to obtain the general image embedding features; the image estimation network learns the mapping from the general image embedding features to the face embedding features to obtain the face embedding features.
[0020] The voice backbone network in this embodiment adopts a time-delay neural network, and its network parameters are shown in Table 1:
[0021] Table 1
[0022] like Figure 2As shown, the speech estimation network includes: two dropout layers, two ReLU layers and several linear layers, wherein: the first linear layer performs feature extraction processing based on the input identity embedding feature while keeping the dimension unchanged, and extracts the speech age feature from the identity embedding feature; the first ReLU layer introduces a nonlinear transformation based on the speech age feature to obtain the ReLU speech age feature; the first dropout layer randomly discards some neurons from the ReLU speech age feature to reduce the coupling degree between neurons, reduce the risk of network overfitting, and obtain the ReLU-dropout speech age feature; the second linear layer performs dimensionality reduction based on the ReLU-dropout speech age feature to obtain the speech embedding feature; the second ReLU layer introduces a nonlinear transformation based on the speech embedding feature to obtain the ReLU speech embedding feature; the second dropout layer randomly discards some neurons from the ReLU speech age embedding to reduce the coupling degree between neurons, reduce the risk of network overfitting, and obtain the ReLU-dropout speech embedding feature; the third linear layer performs dimensionality reduction based on the ReLU-dropout speech embedding feature to obtain the speech classification estimation result; the fourth linear layer performs dimensionality reduction based on the classification estimation result to obtain the reduced-dimensional speech classification estimation result. The fifth linear layer performs dimensionality reduction according to the dimensionality reduction speech classification estimation result to obtain the speech regression estimation result.
[0023] The image backbone network in this embodiment is composed of a 50-layer residual network, and its network parameters are shown in Table 2:
[0024] Table 2
[0025] like Figure 3 As shown, the image estimation network includes: a dropout layer, a ReLU layer and several linear layers, wherein: the third dropout layer randomly discards some neurons for the universal image embedding feature to reduce the coupling degree between neurons, reduce the risk of network overfitting, and obtains the dropout universal image embedding feature; the sixth linear layer performs dimensionality reduction based on the dropout universal image embedding feature to obtain the face embedding feature; the third ReLU layer introduces nonlinear transformation based on the face embedding feature to obtain the ReLU face embedding feature; the seventh linear layer performs dimensionality reduction based on the ReLU face embedding feature to obtain the face classification estimation result; the eighth linear layer performs dimensionality reduction based on the face classification estimation result to obtain the reduced dimensionality face classification estimation result; the ninth linear layer performs dimensionality reduction based on the reduced dimensionality face classification estimation result to obtain the face regression estimation result.
[0026] The dual-branch fusion network includes: an embedding regularization module, an information interaction module and a prediction layer, wherein: the embedding regularization module calculates speech structured sparse embedding features and face structured sparse embedding features according to speech embedding features and face embedding features respectively; the information interaction module performs outer product calculation on the regularized speech structured sparse embedding features and face structured sparse embedding features to obtain outer product age embedding features, and performs sum calculation to obtain sum age embedding features; the prediction layer performs classification age estimation and regression age estimation according to the outer product age embedding features and the sum age embedding features.
[0027] like Figure 4 As shown, the embedding regularization module includes: a dimension expansion linear layer and an auxiliary embedding layer, wherein: the dimension expansion linear layer expands the dimension of the speech embedding features and the face embedding features to twice the target embedding dimension, and the auxiliary embedding layer performs point multiplication with the speech embedding features and the face embedding features after the expanded dimensions, respectively, and maps the speech embedding features and the face embedding features into speech structured sparse embedding features and face structured sparse embeddings.
[0028] like Figure 5 As shown, the information interaction module includes: three linear layers, a summation layer, an outer product layer and a flattening layer, wherein: the outer product layer performs matrix outer product according to the input face regularization embedding feature and the speech regularization embedding feature to obtain the outer product interaction embedding feature; the flattening layer performs flattening according to the input outer product interaction embedding feature to obtain the flattened outer product interaction embedding feature; the tenth linear layer performs dimensionality reduction according to the input flattened outer product interaction embedding feature to obtain the reduced dimensionality outer product age embedding feature; the eleventh linear layer performs dimensionality reduction according to the reduced dimensionality outer product age embedding feature to obtain the outer product age embedding feature; the summation layer performs summation according to the input face regularization embedding feature and the speech regularization embedding feature to obtain the summed interaction embedding feature; the twelfth linear layer performs dimensionality reduction according to the summed interaction embedding feature to obtain the summed age embedding.
[0029] like Figure 6As shown, the prediction layer includes: a summation operation layer, a dropout layer, a ReLU layer and three linear layers, wherein: the summation operation layer performs summation according to the input outer product age embedding feature and the summed age embedding feature to obtain a fused age embedding feature; the fourth dropout layer randomly discards some neurons from the fused age embedding to reduce the coupling degree between neurons and reduce the risk of network overfitting to obtain the dropout fused age embedding feature; the fourth ReLU layer introduces nonlinear transformation according to the dropout fused age embedding feature to obtain the ReLU-dropout fused age embedding feature; the thirteenth linear layer performs dimensionality reduction according to the ReLU-dropout fused age embedding feature to obtain a fused classification estimation result; the fourteenth linear layer performs dimensionality reduction according to the fused classification estimation result to obtain a reduced dimensionality fused classification estimation result; the fifteenth linear layer performs dimensionality reduction according to the reduced dimensionality fused classification estimation result to obtain a fused regression estimation result.
[0030] The multimodal age estimation system first trains an embedding extraction network in an offline stage and then uses it to generate embedded data; then supervises the training of a dual-branch fusion network with the generated embedded data; in an online stage, face images and voice data are collected respectively and input into the embedding extraction network respectively to obtain embedded data, and then the embedded data is sent to the dual-branch fusion network to obtain an age estimation result.
[0031] The supervised training refers to: using the classification estimation error and the regression estimation error as the supervised loss for training, specifically: using the cross entropy loss and the mean square error loss as the classification estimation loss and the regression estimation loss in the training phase, and weighted addition of the two loss functions for collaborative training, where: the cross entropy loss Mean square error loss Where: y ij is the true label of the i-th sample belonging to the j-th class, p ij is the predicted probability that the i-th sample belongs to the j-th class, y i represents the true label of the i-th sample, represents the predicted value of the i-th sample, and N is the total number of samples.
[0032] After specific actual experiments: it was carried out on the experimental test platform (CPU: Intel(R) Xeon(R) Gold6226R, GPU: RTX3080Ti), and the deep learning framework used was Pytorch.
[0033] The actual experiment is as follows: The public multimodal age dataset AgeVoxCeleb is used and divided into three parts: training set (4000 people, 137946 samples), validation set (479 people, 14237 samples), and test set (497 people, 16050 samples).
[0034] In the experiment, a comparison experiment and ablation experiment were conducted. As shown in Tables 3, 4, and 5, the mean absolute error (MAE) and Pearson correlation coefficient (PCC) were used as evaluation indicators. The lower the MAE, the better, and the higher the PCC, the better. The following experimental analysis is mainly based on MAE, and similar conclusions can be obtained by analyzing PCC.
[0035] In the experiment, to ensure fair comparison, the results published on AgeVoxCeleb are used. In addition, the same experimental settings are used to reproduce related methods without published results. Table 3 is a comparative experiment of different age estimation frameworks, mainly analyzing the effectiveness of the proposed multimodal age estimation method based on embedding regularization dual-branch fusion. Table 4 is an ablation experiment of the embedding regularization module. Table 5 is an ablation experiment of the information interaction module.
[0036] In the experiment, the experimental analysis of Table 3 is as follows: speech modality and face modality represent a single modal framework, and face + speech modality represents a multimodal framework. Among them, under the multimodal framework, the results are displayed as follows: modality not missing / speech modality missing / face modality missing / . First, the performance of the present invention is significantly better than that of the existing single modal framework. Secondly, when the modality is missing, the performance of the present invention is still better than that of the existing single modal method. That is, the method can handle multimodal age estimation and single modal age estimation at the same time.
[0037] Table 3
[0038] In the experiment, the experimental analysis of Table 4 is as follows: face + voice modality represents a multimodal framework. Under the multimodal framework, the results are presented as follows: modality not missing / voice modality missing / face modality missing / . Sum represents a multimodal age estimation method that does not use an embedding regularization module but uses an information interaction-summation branch. ER represents an embedding regularization module. Sum+ER represents a multimodal age estimation method that uses an embedding regularization module and an information interaction-summation branch. Aug-MM is a general data augmentation method. Sum+Aug-MM represents a multimodal age estimation method that uses data augmentation and an information interaction-summation branch. Sum+ER+Aug-MM represents a multimodal age estimation method that uses an information interaction-summation branch and an embedding regularization module as well as data augmentation. By comparing Sum and Sum+ER, it can be found that ER has a significant improvement on the face modality, which shows that ER has the function of enhancing the single-modal representation capability. The same conclusion can be obtained by comparing Sum+Aug-MM and Sum+ER+Aug-MM.
[0039] Table 4
[0040] In the experiment, the experimental analysis of Table 5 is as follows: face + voice modality represents a multimodal framework. Under the multimodal framework, the results are displayed as follows: modality not missing / voice modality missing / face modality missing. SC represents the information interaction-concatenation branch, Sum represents the information interaction-summation branch, and MP represents the information interaction-outer product branch. By comparing the three different information interactions, it can be seen that in the information interaction module, the information interaction-outer product branch can achieve the best result when the modality is not missing, but cannot cope with the problem of modality missing. The information interaction-summation branch performs slightly better than the information interaction-concatenation branch. Therefore, by fusing the information interaction-outer product branch and the information interaction-summation branch to simultaneously process multimodal age estimation and single modal age estimation, the information interaction-summation branch plays a role when the modality is missing, and the information interaction-outer product branch plays a role when the modality is not missing.
[0041] Table 5
[0042] In this embodiment, the speech embedding extraction network extracts 40-dimensional Fbank features from the input speech of length 3s and then feeds them into the backbone network, using spectral enhancement and noise enhancement. The task of training speakers on VoxCeleb1 and VoxCeleb2 is Then lock the parameters to generate the speaker's identity embedding feature with a dimension of 192, send the identity embedding to the estimation network, learn the mapping from identity embedding to voice age embedding, and the voice age embedding is 128 dimensions; the face embedding extraction network uses random horizontal flipping and random grayscale for data enhancement based on the input image of size 3*224*224 (when the input size is not specified, the face detection and alignment network is used for cropping). The backbone network is pre-trained on the image classification task, outputs 2048-dimensional embedding features, and then sent to the estimation network for training.
[0043] In this embodiment, the embedding regularization module provides structured sparse representation to enhance the single-modal embedding representation capability and solve the problem of modality missing. Figure 7 As shown in the figure, the dimension expansion linear layer in the embedding regularization module embeds the face age embedding features and the voice age embedding e f and e s Perform dimension expansion, that is, double the dimension of the original embedding, and obtain the expanded face age embedding e′ f and the expanded voice age embedding e′ s After that, we embed a f and voice-assisted embedding s Embed the expanded age into e′ f and e′ s Perform point multiplication to obtain the regularized face age embedding features and voice age embedding respectively.
[0044] This embodiment constrains the outputs of the speech estimation network and the image estimation network through an embedding regularization module and saves the regularized age embedding.
[0045] like Figure 8 As shown, the information interaction module uses matrix outer product as a way to enhance the multi-modal embedding interaction capability and uses summation as a way to enhance the unimodal embedding representation capability, thereby providing the network with additional structural information and solving the problem of missing modalities.
[0046] When the modality is not missing, the matrix outer product fully interacts each dimension of the two embeddings, while the interaction between summation and splicing is limited. When modality missing occurs, summation can retain the embedding of the remaining modalities. Therefore, this embodiment adopts the information interaction method of summation to better cooperate with embedding regularization. When the modality is missing, the embedding of the remaining modalities can be retained intact. In addition, this embodiment adopts the information interaction method of matrix outer product to enhance the interactive ability of multimodal embedding.
[0047] Table 6 AgeVoxCeleb error test
[0048] Compared with the prior art, the present invention adopts the network design of speech estimation network and face estimation network as well as the enhancement of unimodal features by embedding regularization and the enhancement of multimodal features by information interaction-outer product. Compared with the prior art, the present invention has lower estimation error and achieves an average error of 5.05 years in the AgeVoxCeleb public dataset.
[0049] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.
Claims
1. A multimodal age estimation system based on embedding regularization dual-branch fusion, characterized in that: include: Embedding extraction network and dual-branch fusion network, wherein: the embedding extraction network performs embedding extraction according to the speech data and face image respectively to obtain speech embedding features and face embedding features; the dual-branch fusion network performs embedding regularization processing according to the speech embedding features and face embedding features respectively to obtain speech structured sparse embedding features and face structured sparse embedding features, and after information interaction, the outer product age embedding features and the summed age embedding are obtained respectively and the estimated age is obtained by summing them.
2. The multimodal age estimation system based on embedded regularized dual-branch fusion according to claim 1, characterized in that: The embedding extraction network comprises: a speech embedding extraction network composed of a speech backbone network and a speech estimation network, and a face embedding extraction network composed of a face detection and alignment network, an image backbone network and an image estimation network, wherein: the speech backbone network extracts Fbank features from the input speech and performs spectrum enhancement and noise enhancement, and generates the speaker's identity embedding features by training the speaker's tasks and locking the parameters; the speech estimation network learns the mapping from the identity embedding features to the speech embedding features to obtain the speech embedding features; the face detection and alignment network crops the input image, performs data enhancement by random horizontal flipping and random grayscale, and the image backbone network performs feature extraction according to the enhanced image to obtain the general image embedding features; the image estimation network learns the mapping from the general image embedding features to the face embedding features to obtain the face embedding features.
3. The multimodal age estimation system based on embedded regularized dual-branch fusion according to claim 2, characterized in that: The speech estimation network comprises: two dropout layers, two ReLU layers and several linear layers, wherein: the first linear layer performs feature extraction processing according to the input identity embedding feature while keeping the dimension unchanged, and extracts the speech age feature from the identity embedding feature; the first ReLU layer introduces nonlinear transformation according to the speech age feature to obtain the ReLU speech age feature; the first dropout layer randomly discards some neurons from the ReLU speech age feature to reduce the coupling degree between neurons, reduce the risk of network overfitting, and obtain the ReLU-dropout speech age feature; the second linear layer performs dimensionality reduction according to the ReLU-dropout speech age feature to obtain the speech embedding feature; the second ReLU layer introduces nonlinear transformation according to the speech embedding feature to obtain the ReLU speech embedding feature; the second dropout layer randomly discards some neurons from the ReLU speech age embedding to reduce the coupling degree between neurons, reduce the risk of network overfitting, and obtain the ReLU-dropout speech embedding feature; the third linear layer performs dimensionality reduction according to the ReLU-dropout speech embedding feature to obtain the speech classification estimation result; the fourth linear layer performs dimensionality reduction according to the classification estimation result to obtain the reduced-dimensional speech classification estimation result; the fifth linear layer performs dimensionality reduction according to the reduced-dimensional speech classification estimation result to obtain the speech regression estimation result.
4. The multimodal age estimation system based on embedded regularized dual-branch fusion according to claim 2, characterized in that: The image estimation network comprises: a dropout layer, a ReLU layer and several linear layers, wherein: the third dropout layer randomly discards some neurons for the universal image embedding feature to reduce the coupling degree between neurons, reduce the risk of network overfitting, and obtains the dropout universal image embedding feature; the sixth linear layer performs dimensionality reduction according to the dropout universal image embedding feature to obtain the face embedding feature; the third ReLU layer introduces nonlinear transformation according to the face embedding feature to obtain the ReLU face embedding feature; the seventh linear layer performs dimensionality reduction according to the ReLU face embedding feature to obtain the face classification estimation result; the eighth linear layer performs dimensionality reduction according to the face classification estimation result to obtain the reduced dimensionality face classification estimation result; the ninth linear layer performs dimensionality reduction according to the reduced dimensionality face classification estimation result to obtain the face regression estimation result.
5. The multimodal age estimation system based on embedded regularized dual-branch fusion according to claim 1, characterized in that: The dual-branch fusion network includes: an embedding regularization module, an information interaction module and a prediction layer, wherein: the embedding regularization module calculates speech structured sparse embedding features and face structured sparse embedding features according to speech embedding features and face embedding features respectively; the information interaction module performs outer product calculation on the regularized speech structured sparse embedding features and face structured sparse embedding features to obtain outer product age embedding features, and performs sum calculation to obtain sum age embedding features; the prediction layer performs classification age estimation and regression age estimation according to the outer product age embedding features and the sum age embedding features.
6. The multimodal age estimation system based on embedded regularized dual-branch fusion according to claim 5, characterized in that: The embedding regularization module includes: a dimension expansion linear layer and an auxiliary embedding layer, wherein: the dimension expansion linear layer expands the dimension of the speech embedding features and the face embedding features to twice the target embedding dimension, and the auxiliary embedding layer performs point multiplication with the speech embedding features and the face embedding features after the expanded dimension, respectively, and maps the speech embedding features and the face embedding features into speech structured sparse embedding features and face structured sparse embedding.
7. The multimodal age estimation system based on embedded regularized dual-branch fusion according to claim 5, characterized in that: The information interaction module includes: three linear layers, a summation layer, an outer product layer and a flattening layer, wherein: the outer product layer performs matrix outer product according to the input face regularization embedding feature and the speech regularization embedding feature to obtain the outer product interaction embedding feature; the flattening layer performs flattening according to the input outer product interaction embedding feature to obtain the flattened outer product interaction embedding feature; the tenth linear layer performs dimensionality reduction according to the input flattened outer product interaction embedding feature to obtain the reduced dimensionality outer product age embedding feature; the eleventh linear layer performs dimensionality reduction according to the reduced dimensionality outer product age embedding feature to obtain the outer product age embedding feature; the summation layer performs summation according to the input face regularization embedding feature and the speech regularization embedding feature to obtain the summed interaction embedding feature; the twelfth linear layer performs dimensionality reduction according to the summed interaction embedding feature to obtain the summed age embedding.
8. The multimodal age estimation system based on embedded regularized dual-branch fusion according to claim 5, characterized in that: The prediction layer includes: a summation operation layer, a dropout layer, a ReLU layer and three linear layers, wherein: the summation operation layer performs summation according to the input outer product age embedding feature and the summed age embedding feature to obtain a fused age embedding feature; the fourth dropout layer randomly discards some neurons from the fused age embedding to reduce the coupling degree between neurons and reduce the risk of network overfitting to obtain the dropout fused age embedding feature; the fourth ReLU layer introduces nonlinear transformation according to the dropout fused age embedding feature to obtain the ReLU-dropout fused age embedding feature; the thirteenth linear layer performs dimensionality reduction according to the ReLU-dropout fused age embedding feature to obtain a fused classification estimation result; the fourteenth linear layer performs dimensionality reduction according to the fused classification estimation result to obtain a reduced dimensionality fused classification estimation result; the fifteenth linear layer performs dimensionality reduction according to the reduced dimensionality fused classification estimation result to obtain a fused regression estimation result.
9. A multimodal age estimation method based on embedded regularized dual-branch fusion based on any one of the systems of claims 1-8, characterized in that: In the offline stage, the embedding extraction network is first trained to generate embedded data; the generated embedded data is then used to supervise the training of the dual-branch fusion network; In the online stage, face images and voice data are collected and input into the embedding extraction network to obtain embedded data, which are then fed into the dual-branch fusion network to obtain the age estimation result. The supervised training refers to: using the classification estimation error and the regression estimation error as the supervised loss for training, specifically: using the cross entropy loss and the mean square error loss as the classification estimation loss and the regression estimation loss in the training phase, and weighted addition of the two loss functions for collaborative training, where: the cross entropy loss Mean square error loss Where: y ij is the true label of the i-th sample belonging to the j-th class, p ij is the predicted probability that the i-th sample belongs to the j-th class, y i represents the true label of the i-th sample, represents the predicted value of the i-th sample, and N is the total number of samples.
Citation Information
Patent Citations
Face age estimation method and system fusing gender and racial information
CN113920413A
