Face searching method and system based on multiple modes

Abstract

The invention relates to the technical field of face search, and discloses a multi-mode-based face search method and system, and the method comprises the steps: obtaining a face image sample set, and carrying out the image analysis and semantic coding of the face image sample set, so as to obtain an image text pair data set; constructing a hierarchical face search index database based on the image text pair data set and the trained multi-modal face representation model; obtaining a search request initiated by a user, and carrying out multi-modal search request coding on the search request to obtain a query vector; and based on the hierarchical face search index database, performing retrieval and result matching on the query vector to obtain a matched face sample set. Therefore, unified feature mapping of image and text description is realized by constructing the multi-modal index, image search by using a text and image search by using an image can be supported, and the applicability and retrieval efficiency of face search in a non-image scene are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face search, and more particularly, to a multi-modal based face search method and system. BACKGROUND

[0002] Face search technology plays an important role in public security, identity verification, social media, and intelligent security, etc. At present, the mainstream face search technology is mainly based on face recognition algorithm. The standard workflow is usually as follows: the system receives an input face image to be searched, extracts the face feature vector in the image by using a deep learning model, and then calculates the feature similarity in the pre-constructed face database to search for the most similar face sample.

[0003] However, this traditional "image search" method is significantly limited in actual application scenarios, and its core defect is that it cannot search for faces through natural language semantic description. In many actual cases, such as in the process of criminal investigation and pursuit or lost person searching in the field of public security, there is often a "0 sample" dilemma, that is, the initial face image of the target person cannot be obtained, and only the semantic description of the target person (such as "a young man with glasses and short hair") is known.

[0004] In the above no-image scenario, the existing face search technology cannot directly map the text description to the face feature space, resulting in the system being unable to automatically perform the search task. This makes it still necessary to rely on a large number of manual investigation operations when searching for a specific target in a monitoring angle or a large database, which is not only inefficient, but also easily leads to missed detection due to fatigue. Therefore, how to break through the limitation of a single mode and realize cross-modal face search supporting both image input and text description input is a problem to be solved in the current face search technology field. SUMMARY

[0005] In order to overcome the defect that the existing face search technology highly depends on the face database and the initial image sample, resulting in being unable to effectively search in the zero sample scenario with only semantic description, the present application aims to provide a multi-modal based face search method and system, and the core purpose is to break the barrier between visual features and semantic features, and to construct a general search framework that can not only support traditional image search, but also support search with natural language description.

[0006] According to an aspect of the present application, a multi-modal based face search method is provided, comprising: obtaining a face image sample set; performing image analysis and semantic coding on the face image sample set to obtain an image-text pair data set; based on the image-text pair data set and a trained multi-modal face representation model, constructing a hierarchical face search index library; obtaining a search request initiated by a user; performing multi-modal search request coding on the search request initiated by the user to obtain a query vector; based on the hierarchical face search index library, performing retrieval and result matching on the query vector to obtain a matched face sample set.

[0007] In a possible implementation, the image analysis and semantic coding on the face image sample set to obtain the image-text pair data set comprises: performing face attribute feature extraction based on a deep neural network on the face image sample to obtain face attribute labels; performing semantic text generation based on a large language model on the face attribute labels to obtain semantic text descriptions; and performing cross-modal data mapping and binding on the face image sample and the semantic text descriptions to obtain the image-text pair data.

[0008] In a possible implementation, the face attribute labels comprise gender, age range, race / skin color, face shape, cheekbone feature, hair color, hair style, beard, eye size, nose bridge, mouth state, glasses, hat, mask and expression.

[0009] In a possible implementation, the construction of the hierarchical face search index library based on the image-text pair data set and the trained multi-modal face representation model comprises: using the trained multi-modal face representation model, performing multi-modal feature extraction and fusion vector calculation on the image-text pair data set to obtain a fusion feature vector set containing fusion features of all samples; based on the face attribute labels and a preset index level configuration, constructing a hierarchical index skeleton structure without filling data; and filling the fusion feature vector set containing the fusion features of all samples into the hierarchical index skeleton structure to obtain the hierarchical face search index library.

[0010] In a possible implementation, the multi-modal search request coding on the search request initiated by the user to obtain the query vector comprises: judging a data modal type of the search request; in response to the data modal type being an image, inputting the search request into an image encoder of the trained multi-modal face representation model to perform feature extraction to obtain the query vector; and in response to the data modal type being text, inputting the search request into a text encoder of the trained multi-modal face representation model to perform feature extraction to obtain the query vector.

[0011] In one possible implementation, based on a hierarchical face search index, query vectors are retrieved and results are matched to obtain a set of matching face samples. This includes: based on the hierarchical face search index, the query vectors are coarsely screened in the search space and candidate set is located using a tree-like index to obtain a subset of candidate vectors; similarity is calculated and scored between the query vectors and the subset of candidate vectors to obtain a candidate score list; and the candidate score list is sorted and the final sample is output to obtain a set of matching face samples.

[0012] In one possible implementation, based on a hierarchical face search index, the query vector is subjected to a tree-indexed search space coarse screening and candidate set localization to obtain a subset of candidate vectors. This includes: parsing the semantic attributes of the query intent of the query vector to obtain a set of attribute constraints; in the hierarchical face search index, the attribute constraint set is located based on the constraints to obtain a list of target leaf node IDs; and based on the list of target leaf node IDs, the subset of candidate vectors is loaded from the hierarchical face search index.

[0013] In one possible implementation, similarity calculation and scoring are performed on the query vector and a subset of candidate vectors to obtain a candidate score list, including: constructing a feature saliency mask for the query vector; based on the feature saliency mask, performing adaptive reweighting on the query vector and candidate vectors to obtain a reweighted query vector and a reweighted input vector; and calculating the weighted cosine similarity score between the reweighted query vector and the reweighted input vector to obtain the candidate scores.

[0014] According to another aspect of this application, a multimodal face search system is provided, comprising: a face image acquisition module for acquiring a face image sample set; a multimodal semantic encoding module for performing image analysis and semantic encoding on the face image sample set to obtain an image-text pair dataset; a hierarchical index construction module for constructing a hierarchical face search index library based on the image-text pair dataset and a trained multimodal face representation model; a user request receiving module for acquiring search requests initiated by users; a multimodal query encoding module for encoding the user-initiated search requests into a query vector; and a hierarchical face retrieval and matching module for retrieving and matching the query vector with the results based on the hierarchical face search index library to obtain a matching face sample set.

[0015] Compared with existing technologies, the multimodal face search method and system provided in this application can significantly expand the applicability of face search technology. Especially in scenarios where photos are lacking, it can directly use the verbal descriptions of witnesses for semantic search, solving the problem that traditional technologies cannot search from scratch. At the same time, by leveraging the alignment capabilities of multimodal models for semantic and visual features and the hierarchical indexing mechanism, it not only achieves flexible cross-modal retrieval, but also effectively improves the retrieval efficiency and matching accuracy in large-scale face databases through the hierarchical filtering mechanism, providing users with a more intelligent and convenient face search experience. Attached Figure Description

[0016] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 The illustration shows a schematic flowchart of a multimodal face search method according to an embodiment of this application.

[0018] Figure 2 The illustration shows a schematic architecture diagram of a multimodal face search method according to an embodiment of this application.

[0019] Figure 3 The figure shows a schematic flowchart of step S2 in the multimodal face search method according to an embodiment of the present application.

[0020] Figure 4 The figure shows a schematic flowchart of step S3 in the multimodal face search method according to an embodiment of the present application.

[0021] Figure 5 The figure shows a schematic flowchart of step S5 in the multimodal face search method according to an embodiment of the present application.

[0022] Figure 6 The figure shows a schematic flowchart of step S6 in the multimodal face search method according to an embodiment of the present application.

[0023] Figure 7 The illustration shows a schematic block diagram of a multimodal face search system according to an embodiment of this application. Detailed Implementation

[0024] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0025] Figure 1 The illustration shows a schematic flowchart of a multimodal face search method according to an embodiment of this application. Figure 2 The illustration shows a schematic architecture diagram of a multimodal face search method according to an embodiment of this application. Figure 1 and Figure 2 As shown, this application provides a multimodal face search method, including: S1: obtaining a face image sample set; S2: performing image analysis and semantic encoding on the face image sample set to obtain an image-text pair dataset; S3: constructing a hierarchical face search index library based on the image-text pair dataset and the trained multimodal face representation model; S4: obtaining a search request initiated by a user; S5: performing multimodal search request encoding on the search request initiated by the user to obtain a query vector; S6: retrieving and matching the query vector with the results based on the hierarchical face search index library to obtain a matching face sample set.

[0026] For example, in step S1, a set of face image samples is acquired. It should be understood that the core of this application lies in utilizing the capabilities of a large language model to provide a detailed description of face images; the more external descriptions of the face acquired, the more accurate the generated text description. Therefore, face image samples containing rich visual information are first acquired so that subsequent steps can extract detailed attribute labels, including age, gender, hairstyle, and face shape. Only with these basic image samples can the system perform attribute analysis through deep neural networks and further utilize the rich face label information to provide sufficient support for the large language model to generate text descriptions, thereby achieving cross-modal conversion from visual information to semantic information.

[0027] In one embodiment, the system reads image data from a pre-set database, surveillance video stream, or publicly available online dataset via a data acquisition interface or file reading module. During the reading process, the system performs preliminary cleaning and formatting of the data to construct a structured sample set. Specifically, the system instantiates each image data in the input stream as a tensor or data object. If the dataset contains... If there are 10 samples, the acquisition process can be represented as constructing a set. ,in Representing the Each acquired facial image sample. This step ensures that each... All of them have a format (e.g., resolution, color space, etc.) that meets the input requirements of subsequent deep neural networks (such as the ResNet50 backbone network used for attribute analysis).

[0028] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between the user terminal and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, information or data such as "facial images" and "search requests" involved in this application were obtained with full authorization.

[0029] For example, in step S2, image analysis and semantic encoding are performed on the face image sample set to obtain an image-text pair dataset. It should be understood that, since a large language model is needed to provide a detailed description of face images, and the more external descriptions of the face obtained, the more accurate the generated text description, it is necessary to label the face images with basic attributes to obtain rich face label information. This rich face label information can provide sufficient support for the large language model to generate text descriptions, enabling more granular analysis capabilities when describing faces, thereby refining the face search range and ultimately achieving an effective association between visual representation and text semantics.

[0030] like Figure 3 As shown, in one embodiment, image analysis and semantic encoding are performed on a set of face image samples to obtain an image-text pair dataset, including: S21, extracting face attribute features from the face image samples based on a deep neural network to obtain face attribute labels; S22, generating semantic text from the face attribute labels based on a large language model to obtain semantic text descriptions; S23, performing cross-modal data mapping and binding on the face image samples and semantic text descriptions to obtain image-text pair data.

[0031] Specifically, firstly, facial attribute features are extracted from face image samples using a deep neural network to obtain facial attribute labels. The system initializes a pre-trained facial attribute analysis model. In this embodiment, a deep convolutional neural network based on the ResNet50 (Residual Network 50-layer) backbone network is used as the attribute analysis model. This model architecture includes an input layer, five-stage convolutional layers (containing multiple residual blocks), a global average pooling layer, and a multi-head fully connected layer. The residual blocks, by introducing skip connections, solve the gradient vanishing problem in deep networks, ensuring the effectiveness of feature extraction. In terms of training methods, the model adopts a multi-task learning strategy. The loss function is composed of a weighted sum of the loss functions of each attribute task. For binary classification attributes (such as whether glasses are worn), a binary cross-entropy loss is used, and for multi-class attributes (such as age group), a cross-entropy loss is used. The system inputs each image sample from the face image sample set into the model, extracts the deep feature map of the image through convolution operations, and maps it to the probability distribution of each attribute in the multi-head fully connected layer. Subsequently, the system performs maximum likelihood estimation or thresholding on the output probability distribution to extract the specific attribute value corresponding to each image sample. This calculation process can be represented as a mapping relationship: ,in Let represent the facial attribute label vector output by the i-th sample. The facial attribute labels include gender, age group, race / skin color, face shape, cheekbone features, hair color, hairstyle, beard, eye size, nose bridge, mouth condition, glasses, hat, mask, and expression. This represents a deep convolutional neural network function used for attribute extraction. This represents the set of weight parameters for the attribute analysis model.

[0032] Secondly, semantic text descriptions are generated from facial attribute tags using a large language model. In this embodiment, a decoder-only model based on the Transformer architecture (such as the GPT series or LLaMA series) is used as the generative large language model (LLM). This model architecture consists of multiple stacked Transformer decoder layers, each containing a multi-head self-attention mechanism and a feedforward neural network, possessing powerful contextual understanding and sequence generation capabilities. In terms of training, the model is pre-trained on a large-scale text corpus, with the optimization objective being to maximize the prediction probability of the next lexical unit, and further fine-tuned through instructions to enhance its ability to follow prompt word instructions. During execution, the system constructs prompt words containing facial attribute tags. Specifically, a prompt word template is preset, which includes role settings (such as "You are a professional police sketch artist"), task instructions, and placeholders. Structured facial attribute tags are filled into the corresponding positions in the template to construct attribute prompt instructions. Subsequently, these instructions are input into the LLM. The LLM, based on an autoregressive mechanism, transforms discrete attribute tags into coherent, natural, and detailed natural language descriptions. The relationships between generative models are described as follows: ,in This represents the generated semantic text description. This represents the generation function of a large language model.

[0033] It is worth mentioning that, in the embodiments of this application, a deep convolutional neural network based on the ResNet50 (Residual Network 50-layer) backbone network is used as the attribute analysis model, and a decoder-only model based on the Transformer architecture (such as the GPT series or LLaMA series) is used as the generative large language model (LLM). However, it is not limited to the above models, and other deep neural network models can also be used, which is not limited by this application.

[0034] Finally, cross-modal data mapping and binding are performed on the face image samples and semantic text descriptions to obtain image-text pair data. The system establishes a tuple or key-value pair structure, using a unique sample index i as the anchor point, to link the original face image sample ( ) and its corresponding semantic text description (

[0035] The system performs one-to-one binding. During this process, the system can also perform data cleaning, removing samples with text descriptions shorter than a preset threshold or containing garbled characters, and combining the remaining high-quality paired data to construct the final image-text pair dataset. The dataset is represented as follows: .

[0036] In one specific embodiment, the system processes an image sample. First, the deep neural network identifies its attribute labels. The data is in JSON format: {"gender": "Male", "age_group": "Young Adult (25-30)", "face_shape": "Square", "hair_style": "Short", "glasses": "Yes (Reading Glasses)"...}. Next, the system constructs prompt words and inputs an LLM (Local Language Model), which generates a semantic text description. "The image shows the facial features of a young man, approximately 25 to 30 years old. He has a well-defined square face, short hair, and most notably, he wears glasses." Finally, the system will...

[0037] With this text The data is bound and stored in an image-text pair dataset for subsequent training of the multimodal model.

[0038] For example, in step S3, a hierarchical face search index is constructed based on the image-text pair dataset and the trained multimodal face representation model. It should be understood that during training, since the model is trained to map matched images and text to similar feature spaces, a fused feature vector containing both visual and semantic information can be generated by calculating the average of the two, preserving both textual and image information. This allows it to adapt to different modalities during face search. Furthermore, a simple hierarchical index is constructed based on face attribute results, such as distinguishing by gender, then by age range, and then by face shape. This simple hierarchical index setting significantly accelerates the face search process, quickly obtaining results and saving time.

[0039] like Figure 4 As shown, in one embodiment, a hierarchical face search index is constructed based on an image-text pair dataset and a trained multimodal face representation model, including: S31, using the trained multimodal face representation model to perform multimodal feature extraction and fusion vector calculation on the image-text pair dataset to obtain a set of fused feature vectors containing the fused features of all samples; S32, based on face attribute labels and a preset index hierarchy configuration, a hierarchical index skeleton structure without data filling is constructed; S33, the set of fused feature vectors containing the fused features of all samples is filled into the hierarchical index skeleton structure to obtain the hierarchical face search index.

[0040] Specifically, the execution of step S3 first relies on a trained multimodal face representation model. In one embodiment of this application, the model adopts a CLIP (Contrastive Language-Image Pre-training) dual-tower architecture. The image encoder uses a Vision Transformer (ViT-B / 32) structure, dividing the input image into multiple patches and extracting global visual features through linear projection and a multi-layer Transformer encoder. The text encoder uses a Transformer-based language model (such as BERT or RoBERTa), extracting semantic features of the text through a self-attention mechanism. In terms of training methods, the model employs a contrastive learning strategy. The system constructs a batch containing N image-text pairs, inputting the image and text into their respective encoders to obtain feature vectors, which are then mapped to a shared embedding space. The objective function for training uses a symmetric InfoNCE loss (or NT-Xent Loss), aiming to maximize the cosine similarity of the N positive sample pairs (matched image-text pairs) within the batch, while minimizing... The similarity of negative sample pairs (mismatched image-text pairs). Through this training, the model learns to bring semantically consistent images and text closer together in the feature space, achieving cross-modal alignment.

[0041] Furthermore, to further enhance the model's understanding of fine-grained features of facial images and its cross-modal generation capabilities, the multimodal face representation model in this embodiment also includes a text generation module during the training phase. This module is jointly trained based on image-text modeling loss. Specifically, during training, the model not only needs to determine whether the image matches the text, but also needs to perform a text prediction task based on the image content. The system inputs image features into a decoder network, requiring the model to reconstruct or predict the corresponding original semantic text description based on the input image features. By introducing this generative training objective, the model is forced to capture more visual details directly related to the semantic description (such as specific hairstyle textures, glasses styles, etc.) during the encoding phase, thereby improving the accuracy of the model in subsequent facial image text descriptions and also promoting the robustness of feature representation.

[0042] Furthermore, during the construction and training of the multimodal face representation model, the system also needs to evaluate the model's ability to generate or predict text sequences. Specifically, by calculating evaluation metrics (such as BLEU or ROUGE scores) between the generated text and the true semantic description, it verifies whether the model can accurately convert the visual features of face images into natural language descriptions. This evaluation step is crucial because it directly determines whether the model can provide semantic feedback consistent with visual facts for the result refinement stage in the subsequent face search process, thereby ensuring the closed-loop accuracy of the entire search system.

[0043] It is worth mentioning that, in the embodiments of this application, the trained multimodal face representation model adopts a CLIP (Contrastive Language-Image Pre-training) dual-tower architecture. However, it is not limited to the above model architecture, and other multimodal model architectures can also be used, which is not limited by this application.

[0044] Based on the model trained above, the system performs multimodal feature extraction and fusion vector calculation on the image-text pair dataset to obtain a fusion feature vector set containing the fusion features of all samples. That is, the system iterates through the image-text pair dataset (…). Each sample pair in the dataset is processed using the model. Specifically, the system calls the image encoder. Face image samples Encode the image to obtain a normalized high-dimensional image feature vector; simultaneously call the text encoder. Semantic text description Encoding is performed to obtain text feature vectors in the same feature space. During the training of the multimodal model, the contrastive learning mechanism continuously brings the matched image encoding results and text encoding results closer in the feature space, causing the cosine similarity of these two feature vectors to approach 1. Therefore, they represent different modal views of the same entity in the feature space. Based on this premise, in order to generate input vectors that combine visual details and semantic information, the system calculates the average of these two vectors. This calculation process can be expressed as:

[0045] in, This represents the fused feature vector of the i-th sample. In this way, the system generates a set of fused feature vectors containing the fused features of all samples.

[0046] Subsequently, based on facial attribute labels and a pre-defined index hierarchy configuration, a hierarchical index skeleton structure for the unpopulated data is constructed. The system initializes a root node and creates branches from top to bottom according to attribute priority. For example, the first level divides the data into two child nodes, "Male / Female," based on the "Gender" attribute; the second level further divides each gender node into child nodes such as "Child / Youth / Elderly" based on the "Age Group" attribute. The third level can continue to divide each age group node into child nodes such as "Round Face / Square Face / Oval Face" based on the "Face Shape" attribute. Through this multi-level attribute division, the system assigns a unique identifier to each leaf node representing a specific attribute combination (such as "Male + Youth"), thereby completing the physical or logical bucket partitioning.

[0047] Finally, the system performs vector mounting and index persistence. The system populates the hierarchical index skeleton structure with a set of fused feature vectors containing the fused features of all samples. Specifically, this involves iterating through each fused feature vector (…). Based on the original facial attribute labels corresponding to the sample, path matching is performed in the hierarchical index skeleton structure to find the corresponding leaf node. Subsequently, the fused feature vector and its corresponding original data index are inserted or mounted into the vector storage list of the leaf node. After mounting all samples, the system serializes and stores the structure, ultimately obtaining a hierarchical face search index library with multimodal compatibility.

[0048] In one specific embodiment, the system processes a sample of a "young male". The system first extracts features using the ViT image encoder and the Transformer text encoder, respectively, and calculates their average value. Then, in the index skeleton, the system starts from the root node, matches the "Gender: Male" branch, then matches the "Age: Young" branch, and finally locates the leaf node "Male-Young". The system will... It is stored in the inverted list of this leaf node. When a search is performed later, if the query conditions contain "male" and "young", the system can directly locate this leaf node for retrieval without traversing the entire database.

[0049] For example, in step S4, a search request initiated by the user is obtained. It should be understood that this application aims to address the current difficulty of searching from scratch by proposing a multimodal face search method. This method can accept both images as input for face search and textual descriptions of faces, performing face search based on the content of the description. Therefore, the system must be capable of receiving multiple modal inputs so that users can flexibly initiate searches based on available clues (such as a photograph or a verbal description).

[0050] Specifically, step S4 mainly includes three stages: listening to user input, receiving data streams, and preliminary parsing. The system is in a listening state through a preset application programming interface (API) or graphical user interface (GUI). When a user uploads an image file or enters text in the search box and clicks the "search" button, the system captures this operation. The system backend receives the data stream submitted by the user and performs preliminary format validation. At this time, the system does not immediately perform feature extraction, but instead encapsulates the received raw data into a request object to be processed. This object not only contains the core content of the user input (image pixel data or text character sequence), but may also contain auxiliary parameters set by the user (such as search range, time limit, etc.).

[0051] In one specific embodiment, a user may initiate a request in two ways. First, the user uploads a low-resolution image of a face captured from surveillance video; the system retrieves this image as a search request, in which case the data modality is image. Second, the user only has a verbal description, so they enter the text "Looking for a young man with glasses and short hair" into the system's search box; the system retrieves this text string as a search request, in which case the data modality is text. Regardless of the form, the system can accurately retrieve and pass it to the subsequent encoding module.

[0052] For example, in step S5, the user-initiated search request is encoded using multimodal search request to obtain a query vector. It should be understood that in the face search stage, when the input is an image, the multimodal representation model's image encoding stage outputs an image feature vector, which can be used to search in the face database. When the input is text, the proposed method can still perform face search by encoding the descriptive text in the text encoding module to obtain a text feature vector, which is then used to search in the face database. In this way, regardless of whether the user inputs an image or text, the system can map it to the same feature space as the vectors stored in the index, thus enabling subsequent similarity calculations.

[0053] like Figure 5 As shown, in one embodiment, a multimodal search request encoding is performed on a user-initiated search request to obtain a query vector, including: S51, determining the data modality type of the search request; S52, in response to the data modality type being an image, inputting the search request into the image encoder of the trained multimodal face representation model for feature extraction to obtain a query vector; S53, in response to the data modality type being text, inputting the search request into the text encoder of the trained multimodal face representation model for feature extraction to obtain a query vector.

[0054] Specifically, the first step is to determine the data modality type of the search request. The system analyzes the received search request object and identifies its data attributes. If the data is in pixel matrix format, it is determined to be in image modality; if the data is in character sequence format, it is determined to be in text modality.

[0055] Secondly, the system selects the appropriate encoding path based on the determined modality type. In response to the data modality type being image, the system inputs the search request into the image encoder of the trained multimodal face representation model for feature extraction. Specifically, the image encoder... The input image data is subjected to convolution and pooling operations to extract high-dimensional visual features, which are then mapped to a query vector. This process can be represented as: In response to the data modality being text, the system inputs the search request into the text encoder of the trained multimodal face representation model for feature extraction. Specifically, the text encoder... The input text sequence undergoes word embedding and context encoding to extract semantic features, which are then mapped to query vectors of the same dimension. This process can be represented as: For example, in step S6, based on the hierarchical face search index, the query vector is retrieved and matched with the results to obtain a set of matching face samples. It should be understood that building a simple hierarchical index based on face attribute results can significantly accelerate the face search process, quickly obtain results, and save time. Through this hierarchical retrieval mechanism, the system avoids brute-force scanning of the entire database, significantly improving retrieval efficiency. Simultaneously, by calculating the similarity between the query vector and the input vector, the system can quantify the degree of matching, thereby outputting the face sample set that best meets the user's needs.

[0056] like Figure 6 As shown, in one embodiment, based on a hierarchical face search index library, the query vector is retrieved and the results are matched to obtain a set of matching face samples, including: S61, based on the hierarchical face search index library, the query vector is coarsely screened in the search space based on a tree index and the candidate set is located to obtain a subset of candidate vectors; S62, the query vector and the subset of candidate vectors are similar to each other and scored to obtain a candidate score list; S63, the candidate score list is sorted and the final sample is output to obtain a set of matching face samples.

[0057] Specifically, firstly, based on a hierarchical face search index, the query vectors are subjected to a tree-indexed search space coarse screening and candidate set localization to obtain a subset of candidate vectors. The system then processes the query vectors ( To perform semantic attribute parsing of the query intent, in order to obtain the attribute constraint set ( Specifically, the system utilizes a lightweight attribute prediction module to predict the discrete attribute distribution (such as gender, whether or not glasses are worn) corresponding to the query vector. In this embodiment, the lightweight attribute prediction module is a multilayer perceptron (MLP) classification head. Specifically, the architecture includes an input layer, a hidden layer, and an output layer. The input layer receives data in dimensions of... (e.g., 512-dimensional) query vector; hidden layer contains (For example, 256) neurons, using ReLU as the activation function; the output layer contains 1 neuron, corresponding to The module uses predefined attribute categories (such as gender, glasses, hairstyle, etc.) and employs the sigmoid activation function to output the independent probability of each attribute. The training method for this module adopts a supervised learning strategy. After the multimodal face representation model is trained, its encoder parameters are frozen. To enable the prediction module to adapt to both image and text inputs, the system constructs hybrid training data. Specifically, image vectors and their corresponding text description vectors from the image-text pair dataset are used as input data, and the corresponding face attribute labels are used as ground truth labels. The binary cross-entropy loss function is used to fine-tune the MLP classifier head. Through this cross-modal joint training, the MLP classifier head learns the attribute discrimination boundary in a unified feature space, thus enabling accurate attribute prediction for query vectors of any modality.

[0058] During the execution phase, the system inputs the query vector into the trained lightweight attribute prediction module to predict its corresponding discrete attribute distribution. The system sets a specific confidence threshold. (In this embodiment, (Set to 0.85). Only attributes with a predicted probability greater than 0.85 are retained as valid constraints. This processing logic can be expressed as: Next, the system searches the hierarchical face search index (... In the context of the system, the system performs constraint-based index node location on the set of attribute constraints. If a clear attribute constraint exists (such as "female"), the system directly navigates to the corresponding branch node and locks the target leaf node ID list. If there are no explicit constraints, calculate the distance between the query vector and each cluster center, and select the closest one. There are 10 leaf nodes. Finally, based on the list of target leaf node IDs, the system loads all the vectors from the hierarchical face search index to construct a narrowed search range, i.e., a subset of candidate vectors. ).

[0059] Secondly, similarity calculations and scoring are performed on the query vector and the candidate vector subset to obtain a candidate score list. It should be understood that although the tree-indexed search space coarse screening has effectively eliminated a large number of irrelevant samples with mismatched attributes (e.g., eliminating all "female" samples), resulting in a candidate vector subset, the vectors within this subset are still unordered and may still be numerous (e.g., containing hundreds or even thousands of samples belonging to the "male-short hair" category). At this point, the system cannot distinguish which of these candidate samples is closer to the user's specific query intent (e.g., a specific combination of facial features or a specific semantic description). Therefore, it is necessary to mathematically calculate the geometric distance (e.g., cosine similarity) between the query vector and each candidate vector in the feature space, quantifying this abstract feature difference into a concrete numerical score.

[0060] In one embodiment, the system uses the standard cosine similarity algorithm to calculate the query vector and the subset of candidate vectors. Since the feature space has already undergone unit hypersphere normalization using contrastive learning during model training, the system directly calculates the query vector ( ) and each candidate input vector ( The dot product or cosine distance of (). The formula for calculation is:

[0061] This method is fast and suitable for scenarios with relatively uniform feature distribution and generalized query intent.

[0062] However, the vector similarity calculation mechanism in the above embodiments has an inherent technical flaw: the semantic intent dilution problem caused by equal weighting of feature dimensions. The technical reason is that this mechanism fails to fully consider the asymmetric feature relationship between the query modality and the base database modality. Specifically, when the search request originates from natural language text descriptions, such as "finding a short-haired man wearing glasses," the generated query vector exhibits significant sparsity in the feature space, showing strong activation only in a few dimensions corresponding to key semantics such as "glasses," "short hair," and "male," while the remaining dimensions carry less effective information.

[0063] In contrast, the input vectors stored in the base database are generated by fusing image and text information. Their features are dense, encoding rich facial identity details in most dimensions, such as facial texture, skeletal structure, and lighting information.

[0064] Standard cosine similarity algorithms assign equal weight to all feature dimensions. In the asymmetric scenarios described above, this equal-weighting approach causes a few high-energy intent dimensions in the query vector to be overwhelmed by a large number of numerically active identity dimensions in the underlying database vector that are irrelevant to the query intent. The direct consequence is that an unrelated sample with highly matched features but a different identity may have a higher similarity score than a target sample with a different identity but a perfect match for key features (such as wearing glasses). This severely limits retrieval accuracy and user experience, especially in text-to-image search scenarios.

[0065] To address the aforementioned technical shortcomings, this solution proposes a feature reweighted similarity calculation method based on query saliency. This method achieves focus on key semantics by dynamically identifying query intent and adjusting the measurement method of the feature space.

[0066] That is, in a preferred embodiment, similarity calculation and scoring are performed on the query vector and a subset of candidate vectors to obtain a candidate score list, including: constructing a feature saliency mask for the query vector; based on the feature saliency mask, performing feature adaptive reweighting on the query vector and candidate vectors to obtain a reweighted query vector and a reweighted input vector; and calculating the weighted cosine similarity score between the reweighted query vector and the reweighted input vector to obtain the candidate score.

[0067] Specifically, the first step is to construct a feature saliency mask for the query vector. It should be understood that in this embodiment, it is essential to first quantify which dimensions the user is truly interested in from the original query vector. Specifically, the feature saliency mask is calculated using an energy-normalized saliency formula. This formula amplifies the influence of high-activation dimensions while suppressing noise in low-activation dimensions. It simulates the attention mechanism of human visual search, automatically extracting a set of dimensions corresponding to core semantics such as "glasses" and "hairstyle" from complex feature vectors to generate a weight blueprint to guide subsequent calculations, laying the foundation for intent-driven adaptive comparison.

[0068] The calculation formula is as follows:

[0069] in, Representing the The significance weight of the feature; Is the query vector at the th The numerical value of the dimension; Is the query vector at the th The numerical value of the dimension; It is a temperature coefficient used to adjust the smoothness of the weight distribution (in this embodiment, (Set to 0.1 to enhance the discriminative power of the weights). It is the total dimension of the feature vector.

[0070] Next, the generated mask needs to be used to adaptively reweight the query vector and candidate input vectors. It should be understood that weight information alone is insufficient to change the comparison results; an actual vector transformation is necessary to construct a feature subspace focused on the query intent.

[0071] Specifically, the feature saliency mask is subjected to a Hadamard product (element-wise product) operation with both the original query vector and the input vector. In other words, a filter tailored to the query intent is applied to both the query request and each sample in the database to be matched, darkening irrelevant feature dimensions while highlighting key features. This physically alters the energy distribution of the original vectors, generating a pair of reweighted query and input vectors, thus eliminating potential interference from irrelevant dimensions before the final computation.

[0072] The calculation formula is as follows:

[0073]

[0074] in, The symbol represents the Hadamard product operation, while for... Taking the square root is to restore the linear weight relationship in subsequent dot product operations. Furthermore, it should be understood that in vector algebra and deep learning programming (such as Python's NumPy or PyTorch), applying the square root function to a vector means performing a square root operation on each value in the vector individually, generating a new vector. This represents a reweighted query vector. This represents the reweighted data entry vector.

[0075] Finally, the final similarity score is calculated in the reweighted feature space. That is, the final similarity measurement needs to be completed in a new space aligned with the query intent. Specifically, the cosine similarity between the reweighted query vector and the reweighted input vector generated in the previous step is calculated. This evaluates the directional consistency of the two vectors under the illumination of the query intent, rather than their geometric contour similarity in the original full space. The final output is a more accurate candidate score list, whose scores better reflect the degree of matching of samples on key features that users care about, thereby achieving the ultimate goal of improving retrieval accuracy.

[0076] The calculation process can be expressed as follows:

[0077] in, This represents the final weighted similarity score;

[0078] Represents a vector and Perform dot product operation; The L2 norm of a vector; It is a local minimum constant to prevent the denominator from being zero. In this embodiment, Set to 1×10 -8 .

[0079] This embodiment aims to address the semantic intent dilution problem caused by equal weighting in traditional vector similarity calculation methods for multimodal face search. By introducing an adaptive weighting mechanism based on query vector energy distribution, this embodiment can dynamically identify and focus on the core semantic features implicit in the user query, effectively suppressing interference from irrelevant feature dimensions when calculating similarity. This significantly improves retrieval accuracy and recall, especially in face search scenarios using text descriptions, enabling the system to return results that better match the user description in key features, greatly optimizing the system's usability and intelligence.

[0080] Finally, the candidate score list is sorted and the final sample output is obtained to get the matching face sample set. The system sorts the candidate score list in descending order according to the similarity score and truncates the top results. The system generates a Top-N set of results. Then, based on the index ID, it backtracks to the index database to retrieve the corresponding original image path and attribute labels, encapsulates them into the final matched face sample set, and outputs it to the user.

[0081] Furthermore, to address the issue that traditional face search results often contain a large number of similar but non-target "noise" samples, the method in this application includes a result refinement and semantic feedback step after outputting the matched face sample set. Specifically, the system utilizes a trained multimodal face representation model (or the aforementioned text generation module) to generate reverse semantic text descriptions for each face image in the final output Top-N face sample set. Particularly, when the user's search request is image-based (i.e., image-based search scenario), this step functions similarly to the "post-processing text information supplementation" in traditional face recognition. That is, after obtaining visually similar search results, it automatically supplements these results with detailed semantic descriptions. The system compares the generated description text (e.g., "This result shows a man wearing sunglasses, but older") with the user's original search request (e.g., "young man"), or directly displays the generated description to the user. Users or systems can use this supplementary text information to perform a secondary filtering (i.e., "refinement") of search results, thereby eliminating false positives that, although geographically close in vector space, differ significantly in key semantic descriptions. This closed-loop process of retrieval plus description greatly improves the accuracy of the final delivered results.

[0082] In summary, the multimodal face search method provided in this application has been elucidated. It can significantly expand the applicability of face search technology, especially in scenarios where photos are lacking. It can directly utilize the verbal descriptions of witnesses for semantic search, solving the problem that traditional technologies cannot search from scratch. At the same time, by leveraging the alignment capabilities of multimodal models for semantic and visual features and the hierarchical indexing mechanism, it not only achieves flexible cross-modal retrieval but also effectively improves retrieval efficiency and matching accuracy in large-scale face databases through hierarchical filtering mechanisms, providing users with a more intelligent and convenient face search experience.

[0083] This application also provides a multimodal face search system for performing the aforementioned multimodal face search method, such as... Figure 7As shown, the multimodal face search system 700 includes: a face image acquisition module 710 for acquiring a face image sample set; a multimodal semantic encoding module 720 for performing image analysis and semantic encoding on the face image sample set to obtain an image-text pair dataset; a hierarchical index construction module 730 for constructing a hierarchical face search index library based on the image-text pair dataset and a trained multimodal face representation model; a user request receiving module 740 for acquiring search requests initiated by users; a multimodal query encoding module 750 for encoding the user-initiated search requests into multimodal search request vectors; and a hierarchical face retrieval and matching module 760 for retrieving and matching the query vectors with the results based on the hierarchical face search index library to obtain a matching face sample set.

[0084] This application also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the multimodal face search method provided in the above embodiments.

[0085] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the multimodal face search method provided in the above embodiments.

[0086] In this application, the system, computer-readable storage medium, or computer program product provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0087] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments.

[0088] The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A multimodal face search method, characterized in that, include: Obtain a sample set of facial images; Image analysis and semantic encoding are performed on a sample set of face images to obtain an image-text pair dataset; Based on the image-text pair dataset and the trained multimodal face representation model, a hierarchical face search index library is constructed. Get the search request initiated by the user; Multimodal search request encoding is performed on user-initiated search requests to obtain query vectors; Based on a hierarchical face search index, query vectors are retrieved and results are matched to obtain a set of matching face samples.

2. The multimodal face search method according to claim 1, characterized in that, Image analysis and semantic encoding are performed on a sample set of face images to obtain an image-text pair dataset, including: Facial attribute features are extracted from facial image samples using a deep neural network to obtain facial attribute labels; Semantic text descriptions are obtained by generating semantic text based on a large language model from facial attribute labels. Cross-modal data mapping and binding are performed on face image samples and semantic text descriptions to obtain image-text pair data.

3. The multimodal face search method according to claim 2, characterized in that, Facial attribute tags include gender, age group, race / skin color, face shape, cheekbone features, hair color, hairstyle, beard, eye size, nose bridge, mouth condition, glasses, hat, mask, and expression.

4. The multimodal face search method according to claim 2, characterized in that, Based on image-text pair datasets and trained multimodal face representation models, a hierarchical face search index is constructed, including: Using the trained multimodal face representation model, multimodal feature extraction and fusion vector calculation are performed on the image-text pair dataset to obtain a set of fused feature vectors containing the fused features of all samples; Based on face attribute tags and preset index hierarchy configuration, a hierarchical index skeleton structure for unpopulated data is constructed. The set of fused feature vectors containing the fused features of all samples is filled into the hierarchical index skeleton structure to obtain the hierarchical face search index library.

5. The multimodal face search method according to claim 1, characterized in that, The user-initiated search request is encoded using a multimodal search request to obtain a query vector, including: Determine the data modality type of the search request; In response to the data modality type being image, the search request is input into the image encoder of the trained multimodal face representation model to extract features and obtain the query vector; In response to the data modality being text, the search request is input into the text encoder of the trained multimodal face representation model to extract features and obtain the query vector.

6. The multimodal face search method according to claim 1, characterized in that, Based on a hierarchical face search index, query vectors are retrieved and results are matched to obtain a set of matching face samples, including: Based on a hierarchical face search index, the query vectors are subjected to a tree-indexed search space coarse screening and candidate set location to obtain a subset of candidate vectors. Similarity calculation and scoring are performed on the query vector and the subset of candidate vectors to obtain a candidate score list; The candidate score list is sorted and the final sample output is obtained to obtain a set of matching face samples.

7. The multimodal face search method according to claim 6, characterized in that, Based on a hierarchical face search index, the query vectors are subjected to a tree-indexed search space coarse screening and candidate set localization to obtain a subset of candidate vectors, including: The query vector is parsed using semantic attributes of the query intent to obtain a set of attribute constraints; In the hierarchical face search index, constraint-based index node localization is performed on the attribute constraint set to obtain the target leaf node ID list; Based on the target leaf node ID list, a subset of candidate vectors is loaded from the hierarchical face search index.

8. The multimodal face search method according to claim 6, characterized in that, Similarity is calculated and scored between the query vector and a subset of candidate vectors to obtain a candidate score list, including: Construct a feature saliency mask for the query vector; Based on the feature saliency mask, the query vector and candidate vector are adaptively reweighted to obtain the reweighted query vector and the reweighted input vector; Calculate the weighted cosine similarity score between the reweighted query vector and the reweighted input vector to obtain candidate scores.

9. A multimodal face search system, characterized in that, include: The face image acquisition module is used to acquire a set of face image samples; The multimodal semantic coding module is used to perform image analysis and semantic coding on a set of face image samples to obtain an image-text pair dataset. The hierarchical index building module is used to build a hierarchical face search index library based on image-text pairs datasets and trained multimodal face representation models; The user request receiving module is used to obtain search requests initiated by users; The multimodal query encoding module is used to encode user-initiated search requests in a multimodal manner to obtain query vectors. The hierarchical face retrieval and matching module is used to retrieve and match query vectors with results based on a hierarchical face search index to obtain a set of matching face samples.

Images