A multimodal face retrieval method based on generative language models
By constructing a generative language model and combining it with Roberta and ResNet101 models, the problem of insufficient accuracy in multimodal face detection and retrieval was solved, achieving efficient face recognition and retrieval and enhancing the model's understanding and application capabilities.
Patent Information
- Application Number
- CN202410798511.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-20
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-06-20
AI Technical Summary
Existing technologies lack accuracy and robustness in processing multimodal data, especially when image quality is poor, facial expressions are varied, occlusion is present, and lighting conditions are changing. Furthermore, the application of generative language models in the field of multimodal face detection and retrieval has not been fully developed.
A generative language model is constructed, using the Roberta model as the text feature extractor and the ResNet101 model as the image feature extractor. The image-text contrast loss, image-text matching loss, and language modeling loss are calculated through the encoding layer. Multi-task joint training is performed to optimize the parameters of the encoding and decoding layers, thereby achieving effective matching and generation of images and text.
It improves the accuracy and efficiency of face recognition, enhances the model's understanding ability, simplifies model training and use, lowers the application threshold, and realizes multi-task joint learning and generalization capabilities.
Smart Images

Figure CN118656511B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of face recognition, specifically relating to a multimodal face retrieval method based on a generative language model. Background Technology
[0002] With the rapid development of artificial intelligence and computer vision technologies, face detection and retrieval have become key technologies in fields such as security monitoring, personal authentication, social media, and digital entertainment. Traditional face detection and retrieval methods rely on specific image processing techniques and pattern recognition algorithms, such as deep learning models, to identify and match facial features. However, these methods face certain challenges when processing multimodal data (such as datasets containing images, videos, and text information simultaneously). Especially under conditions of poor image quality, varied facial expressions, occlusion, and changing lighting conditions, the accuracy and robustness of traditional methods still need improvement.
[0003] In recent years, generative language models, such as OpenAI's GPT series, have demonstrated powerful capabilities in text generation, understanding, and multimodal tasks. These models can understand and generate human language, providing a novel perspective for processing and understanding multimodal data. In particular, by combining generative language models with face detection technology, more accurate and flexible face retrieval systems can be developed. Furthermore, although many face detection and retrieval methods have been proposed, there are few methods based on generative language models. This multimodal face detection and retrieval method based on generative language models can not only improve the accuracy of face recognition but also effectively identify and retrieve faces in complex multimodal scenarios.
[0004] Despite the significant success of generative language models in text processing, their application in multimodal face detection and retrieval still faces numerous challenges. These include designing effective model architectures to integrate information from different modalities, leveraging contextual information understood by the model to improve the accuracy and efficiency of face retrieval, and handling large-scale multimodal datasets. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides a multimodal face retrieval method based on a generative language model, comprising: establishing a generative language model, training the generative language model, inputting pedestrian images and corresponding descriptive text into the trained generative language model to perform face retrieval, and obtaining retrieval results;
[0006] The generative language model includes: a text feature extractor, an image feature extractor encoding layer, and a decoding layer;
[0007] The training process of the generative language model includes:
[0008] S1: The training set is composed of face image text pairs obtained from the Internet. After processing the text and corresponding image of each data in the training set, they are input into the text feature extractor and the image feature extractor respectively to obtain their respective feature representations.
[0009] The text feature extractor uses the Roberta model;
[0010] The image feature extractor uses the ResNet101 model;
[0011] S2: Encode the feature representations of the image and text separately through the coding layer, and calculate the contrast loss (ITC) between the encoded image and text;
[0012] S3: Construct positive and negative sample pairs for the image, combine the positive and negative sample pairs together and encode them through the coding layer, and calculate the image-text matching loss ITM based on the encoding result;
[0013] S4: The feature representations of the image and text are used together as input to the encoding layer for encoding processing. The decoding layer decodes and generates the corresponding text R based on the encoding result. The language modeling loss LM is calculated to measure the difference between the generated text R and the target text T.
[0014] S5: Perform reverse gradient updates on ITC loss, ITM loss, and LM loss to optimize the parameters of the encoding and decoding layers and complete the training of the model.
[0015] The beneficial effects of this invention are:
[0016] This invention constructs a generative language model with powerful modeling capabilities, capable of acquiring semantically richer text representations. By introducing special markers, text descriptions and image content can better complement each other, enhancing the model's understanding ability. It also achieves joint training of multiple tasks, including text generation, visual question answering, face detection, and face retrieval. The learning of different tasks can mutually promote each other, improving the model's generalization ability and filling the gap in face detection and face retrieval tasks based on generative language models. At the same time, its unified multi-task learning framework simplifies model training and use, lowering the application threshold. Attached Figure Description
[0017] Figure 1 This is a schematic diagram of the generative language model structure of the present invention;
[0018] Figure 2 This is a flowchart illustrating the training process of the generative language model of this invention.
[0019] Figure 3 This is an example diagram of the training set for this invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] A multimodal face retrieval method based on a generative language model, such as Figure 1 As shown, it includes: establishing a generative language model, training the generative language model, inputting pedestrian images and corresponding descriptive text into the trained generative language model to perform face retrieval, and obtaining retrieval results;
[0022] The generative language model includes: a text feature extractor, an image feature extractor encoding layer, and a decoding layer;
[0023] The training process of the generative language model, such as Figure 2 As shown, it includes:
[0024] S1: The training set is composed of face image text pairs obtained from the Internet. After processing the text and corresponding image of each data in the training set, they are input into the text feature extractor and the image feature extractor respectively to obtain their respective feature representations.
[0025] The text feature extractor uses the Roberta model;
[0026] The image feature extractor uses the ResNet101 model;
[0027] S2: Encode the feature representations of the image and text separately through the coding layer, and calculate the contrast loss (ITC) between the encoded image and text;
[0028] S3: Construct positive and negative sample pairs for the image, combine the positive and negative sample pairs together and encode them through the coding layer, and calculate the image-text matching loss ITM based on the encoding result;
[0029] S4: The feature representations of the image and text are used together as input to the encoding layer for encoding processing. The decoding layer decodes and generates the corresponding text R based on the encoding result. The language modeling loss LM is calculated to measure the difference between the generated text R and the target text T.
[0030] S5: Perform reverse gradient updates on ITC loss, ITM loss, and LM loss to optimize the parameters of the encoding and decoding layers and complete the training of the model.
[0031] S1: After processing the text and corresponding image of each data point in the training set, input them into the text feature extractor and image feature extractor respectively to obtain their respective feature representations.
[0032] In this embodiment, the text and corresponding image of each data point in the training set are processed and then input into the text feature extractor and image feature extractor respectively to obtain their respective feature representations, including:
[0033] The text is converted into a numerical representation using a word segmenter, and then a vector representation of the text is obtained using a text feature extractor. The image is filled with a square along its longest side and resized to the size required by the image feature extractor. This square is then input into the image feature extractor to obtain a vector representation of the image.
[0034] The image and text datasets used in this embodiment are all derived from different publicly available datasets, which cover a rich variety of image and text information. The data in these publicly available datasets is explicitly authorized content, and the online data sources must comply with the robots.txt protocol and website terms of use, and meet the privacy regulations such as the Personal Information Protection Law and GDPR. In order to better meet the needs of the four tasks of face detection, face retrieval, visual question answering, and text generation, this invention has re-divided and integrated these datasets to ensure the validity and relevance of the data.
[0035] In this embodiment, the accuracy of the face retrieval task primarily relies on the face detection and text generation tasks. Therefore, no corresponding target text was set during training. Different coded texts and target texts were constructed for the three tasks: face detection, visual question answering, and text generation. Each task has different prompt texts or special markers to enable the model to distinguish between different tasks. These coded and target texts were carefully designed based on the characteristics and requirements of the tasks, aiming to improve the model's performance and accuracy. Figure 3 As shown, this invention demonstrates an example of how face image text pairs obtained online form the encoded text and target text of the corresponding task in the training set.
[0036] In this embodiment of the invention, the industry-leading text feature encoder Roberta and the image feature encoder ResNet101 were selected. The Roberta encoder is renowned for its powerful text representation capabilities, while the ResNet101 encoder is widely recognized for its excellent image feature extraction capabilities. The preprocessed text and images are input into these two encoders respectively, and through their deep learning and computational processing, their respective feature representations are obtained. These feature representations will serve as the foundation for subsequent model training and inference, providing strong support for achieving efficient and accurate face detection, face retrieval, visual question answering, and text generation tasks.
[0037] S2: Input the feature representations of the image and text into the encoding layer respectively, and then calculate the image-text contrast loss (ITC) for both. Figure 2 As shown in (a);
[0038] In this embodiment of the invention, the feature representations of the image and text obtained by the feature extractor are first input into the corresponding encoding layers. The encoding layers further encode these features to extract higher-level information and generate their respective outputs. Then, the image-text contrast loss (ITC) is calculated.
[0039] Specifically, the image and text outputs are first standardized to ensure they have a unit norm. Then, the standardized image feature vector and the standardized text feature vector are multiplied by their transposes and divided by a predetermined temperature parameter to obtain the image-to-text similarity matrix. Next, the image-to-text similarity matrix is transposed to obtain the text-to-image similarity matrix. Both similarity matrices are then subjected to a softmax function and a logarithmic transformation, respectively, and the cross-entropy loss function is used to calculate the loss value. Finally, the image-to-text and text-to-image losses are summed, divided by 2, and averaged to obtain the final ITC loss value.
[0040] In this embodiment of the invention, the ITC loss is calculated to narrow the distance between positive samples (i.e., matched image-text pairs) while widening the distance between negative samples (i.e., mismatched image-text pairs). By minimizing this loss, the model can learn how to extract useful information from images and text and match them effectively.
[0041] S3: Construct positive and negative sample pairs for the image, then combine the positive and negative sample pairs and input them into the encoding layer. Calculate the image-text matching loss (ITM) from the output results, such as... Figure 2 As shown in (b);
[0042] In this embodiment, the images and text input to the model correspond directly, forming positive sample pairs. Next, negative sample pairs are constructed. To ensure the model learns to distinguish between different categories of samples, a negative sampling strategy is adopted, constructing two types of negative sample pairs. The first type uses the image as the baseline, referring to the image-to-text similarity matrix obtained in step S3, and selects text from the dataset that does not match the current image, forming an image-non-corresponding text negative sample pair. The second type uses the text as the baseline, based on the text-to-image similarity matrix obtained in step S3, selecting images that do not match the current text, and constructing a text-non-corresponding image negative sample pair.
[0043] Using the method described above, a set of positive sample pairs and two sets of negative sample pairs were constructed. These sample pairs were normalized and then fed into the encoding layer for encoding, resulting in a unified output. Next, labels were assigned to these sample pairs, with the positive sample pairs labeled 0 and the two negative sample pairs labeled 1. Finally, using the output of the encoding layer and the corresponding labels, the cross-entropy loss was calculated as the ITM loss.
[0044] In this embodiment, the ITM loss includes:
[0045]
[0046] in, Indicates ITM loss, N represents batch size, and I i Let T represent the i-th image. i Let y represent the i-th text. i Indicates the label, y i =1 indicates a match, y i =0 indicates no match, P(y) i =1|I i , T i ) represents the probability of a prediction being a match.
[0047] In this way, the model can not only learn the correct correspondence between images and text, but also effectively distinguish between mismatched images and text, thereby improving the model's accuracy and robustness.
[0048] S4: In this step, the feature representations of both the image and text are used as input and fed into the encoding layer for further processing. Subsequently, the decoding layer is invoked to generate the corresponding text R based on the output of the encoding layer. To measure the difference between the generated text R and the target text T, the language modeling loss (LM) is calculated. The specific operation process is as follows: Figure 2 As shown in (c);
[0049] In this embodiment, this joint input and output generation method aims to enable the model to more comprehensively understand the relationship between the image and the text, and accordingly generate text R that is closer to the target text T. The encoding layer provides rich contextual information to the decoding layer by fusing features from the image and the text. The decoding layer then uses this information to generate a text description that matches the content of the input image.
[0050] To quantify the degree of similarity between the generated text R and the target text T, cross-entropy loss was used as the language modeling loss (LM). This loss function can well reflect the difference in probability distribution between the predicted text and the real text, thereby guiding the model to continuously optimize its generation ability during training.
[0051] In this embodiment, the language modeling loss LM includes:
[0052]
[0053] in, The language modeling loss is represented by LM, where M represents the number of words in the sentence, and P(w i | I, T <i ) represents the language model predicting the word w i The probability, I represents the probability of a given image, T <i T represents the sequence of all words preceding position i. <i =(w1, w2,…,w i-1 ).
[0054] In this way, the model can not only learn how to generate corresponding text descriptions based on image content, but also adjust its output to better match the target text for different tasks. This not only enhances the model's versatility but also improves its performance in different application scenarios.
[0055] S5: After completing steps S1 to S5, three key loss values are obtained: ITC loss, ITM loss, and LM loss. These loss values are then used for backward gradient updates to optimize the parameters of the encoder and decoder layers. This step is the core of model training; through continuous iteration and updates, the model gradually learns better image-text matching capabilities.
[0056] In this embodiment, it is worth noting that the encoding layer always shares its parameters when calculating the ITC loss, ITM loss, and LM loss. This design not only reduces the complexity of the model but also improves the efficiency of parameter utilization. By sharing the parameters of the encoding layer, the model can better integrate information from images and text, thereby improving the overall matching and generation performance.
[0057] Backward gradient update is achieved by calculating the gradient of the loss value with respect to the model parameters and updating the parameters in the opposite direction of the gradient. In this way, the model can gradually reduce the difference between the predicted and the true values, thereby continuously improving its performance.
[0058] Through iterative optimization in this step, the model will be able to better understand and match the relationship between images and text, providing more accurate and reliable support for subsequent tasks such as text generation, visual question answering, face retrieval, and face detection.
[0059] For face retrieval tasks, certain steps need to be added to the existing model training process to achieve accurate face retrieval. The following is a detailed description of these steps:
[0060] S1: Input and parse pedestrian images and prompt text;
[0061] Pedestrian images and relevant prompt text are input into a pre-trained model. After training, the model possesses the ability to detect faces and generate text. Specific markers are then searched for in the model's output. <obj>It typically contains the bounding box information of the face region, followed by descriptive text for that face. By parsing the model's output, the detected face region and its corresponding descriptive text are extracted.
[0062] S2: Perform feature encoding on the face image to be retrieved.
[0063] The image of the face to be retrieved is fed into a pre-trained image feature encoder. This is not intended to correspond to the face region to be detected.
[0064] S3: Similarity Calculation and Result Filtering
[0065] The similarity between the feature representation of the face image to be retrieved and the feature representation of the face region extracted in S1 is calculated. Cosine similarity is typically used as the metric because it effectively measures the directional difference between two vectors. After calculation, the similarity value is compared with a set threshold. If the similarity value of a face region is greater than the threshold, the face is considered similar to the face to be retrieved, and its corresponding bounding box and descriptive text are output as the search result.
[0066] By following the steps above, a face retrieval function based on facial features can be achieved. The model can accurately detect faces from pedestrian images and filter out similar facial regions by comparing their similarity with the face to be retrieved.
[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.< / obj>
Claims
1. A multimodal face retrieval method based on a generative language model, characterized in that, include: A generative language model is established, trained, and pedestrian images and corresponding descriptive text are input into the trained generative language model to perform face retrieval and obtain retrieval results. Input pedestrian images and corresponding descriptive text into the trained generative language model to perform face retrieval and obtain retrieval results, including: Input a face image and a pedestrian image, and input special markers for face retrieval. <r> The generative language model first performs a face detection task, generates a text description containing the bounding boxes of each face in the pedestrian image, and extracts the vector representation of each detected face region based on the coordinates of the detection boxes.< / r> Input the image of the face to be searched, perform image encoding, and obtain the feature vector of the image of the face to be searched; The cosine similarity is calculated between the vectors of the face to be retrieved and the vectors of each face detected by the generative language model to obtain the corresponding similarity value. In a generative language model, a similarity threshold is set. If the similarity value is greater than this threshold, the face is considered similar to the face to be retrieved, and its corresponding bounding box and descriptive text are output as the retrieval result; if the similarity value is not greater than this threshold, the person is not found in the output image. The generative language model includes: a text feature extractor, an image feature extractor encoding layer, and a decoding layer; The training process of the generative language model includes: S1: The face image text pairs are combined into a training set. After processing the text and the corresponding image of each data in the training set, they are input into the text feature extractor and the image feature extractor respectively to obtain their respective feature representations. The text feature extractor uses the Roberta model. The image feature extractor uses the ResNet101 model; S2: Encode the feature representations of the image and text separately through the coding layer, and calculate the contrast loss ITC between the encoded image and text; S3: Construct positive and negative sample pairs for the image, combine the positive and negative sample pairs together and encode them through the coding layer, and calculate the image-text matching loss ITM based on the encoding result; S4: The feature representations of the image and text are used together as input to the encoding layer for encoding processing. The decoding layer decodes and generates the corresponding text R based on the encoding result. The language modeling loss LM is calculated to measure the difference between the generated text R and the target text T. S5: Perform reverse gradient updates on ITC loss, ITM loss, and LM loss to optimize the parameters of the encoding and decoding layers and complete the training of the model.
2. The multimodal face retrieval method based on a generative language model according to claim 1, characterized in that, After processing the text and corresponding image of each data point in the training set, the results are input into the text feature extractor and image feature extractor respectively to obtain their respective feature representations, including: The text is converted into a numerical representation using a word segmenter, and then a vector representation of the text is obtained using a text feature extractor. The image is filled with a square along its longest side and resized to the size required by the image feature extractor. This square is then input into the image feature extractor to obtain a vector representation of the image.
3. The multimodal face retrieval method based on a generative language model according to claim 1, characterized in that, The calculation of the image-to-text contrast loss (ITC) includes: The encoded image feature vector and text feature vector are standardized. The standardized image feature vector and the standardized text feature vector are then multiplied by their transposes and divided by a set temperature parameter to obtain the image-to-text similarity matrix. The image-to-text similarity matrix is then transposed to obtain the text-to-image similarity matrix. The two similarity matrices are then subjected to the softmax function and logarithmic transformation, respectively. The cross-entropy loss function is used to calculate the image-to-text and text-to-image loss values. The image-to-text and text-to-image loss values are added together, divided by 2, and averaged to obtain the final ITC loss value.
4. The multimodal face retrieval method based on a generative language model according to claim 1, characterized in that, Constructing positive and negative sample pairs for an image includes: The images and text input to the model correspond to each other, directly forming positive sample pairs; Two types of negative sample pairs are constructed. The first type uses the image as the reference and selects text from the training set that does not match the current image to form an image-non-corresponding text negative sample pair. The second type uses the text as the reference and selects an image that does not match the current text to construct a text-non-corresponding image negative sample pair.
5. The multimodal face retrieval method based on a generative language model according to claim 1, characterized in that, The image-text matching loss (ITM) is calculated, including: Label the sample pairs, with positive sample pairs labeled as 0 and both negative sample pairs labeled as 1. Calculate the cross-entropy loss as the ITM loss using the output of the encoding layer and the corresponding labels.
6. The multimodal face retrieval method based on a generative language model according to claim 5, characterized in that, The ITM loss includes: ; in, Indicates ITM loss, N represents batch size, and I i Let T represent the i-th image. i Let y represent the i-th text. i Indicates the label, y i =1 indicates a match, y i =0 indicates no match, P(y) i =1|I i , T i ) represents the probability of a prediction being a match.
7. The multimodal face retrieval method based on a generative language model according to claim 1, characterized in that, The feature representations of both images and text are used as input for encoding, and the corresponding text R is generated by decoding based on the encoding results, including: The feature representations of the image and text are input into the encoding layer. After processing through a multi-head self-attention mechanism, skip connections, and layer normalization, the encoded representation is generated. The decoding layer uses masking multi-head self-attention to ensure the sequential nature of the generation process. Then, the output of the decoding layer is subjected to encoding-decoding attention interaction. After processing through a feedforward neural network, skip connections, and layer normalization, the probability of the vocabulary is generated through a softmax layer. The word with the highest probability is selected as the next word to be generated, and the text R is gradually constructed.
8. The multimodal face retrieval method based on a generative language model according to claim 1, characterized in that, To measure the difference between the generated text R and the target text T, the language modeling loss LM is calculated, including: ; in, The language modeling loss is represented by LM, where M represents the number of words in the sentence, and P(w i | I, T <i ) represents the language model predicting the word w i The probability, I represents the probability of a given image, T <i T represents the sequence of all words preceding position i. <i = (w1,w2,…,w i-1 ).
Citation Information
Patent Citations
Pedestrian image retrieval method based on text description
CN112364197A
Mass face library retrieval method based on man-machine collaboration
CN112949369A
