Image tag identification method based on multi-modal feature fusion

Abstract

The invention relates to an image label identification method based on multi-modal feature fusion. Image label identification under multi-modal feature fusion is carried out on image visual features and EXIF parameter features. The method comprises the following steps: carrying out image visual feature extraction on a to-be-identified image by adopting a visual encoder in a target model to obtain image features; eXIF parameter extraction and preprocessing are carried out on a to-be-recognized image to obtain an EXIF feature vector; performing multi-modal feature fusion on the image features and the EXIF feature vectors to obtain fusion features; constructing a label set, and converting each label in the label set by using a text encoder in the target model to obtain a text feature vector; and calculating a similarity score of the fusion feature and the text feature vector to obtain a similarity score of the to-be-identified image and each label, and outputting the label with the highest similarity score.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and computer vision, in particular to an image label recognition method based on multi-modal feature fusion. BACKGROUND

[0002] With the rapid development of artificial intelligence and computer vision technology, image recognition and processing technology has been widely used in many fields. The current mainstream image recognition method mainly includes deep learning-based models (such as CNN, Transformer, ViT, etc.) and multi-modal fusion methods based on pre-trained models (such as CLIP, BERT, etc.). The CLIP (Contrastive Language-Image Pre-training) model has become an important basic model for image label recognition due to its strong zero-shot recognition ability and good cross-modal alignment characteristics. In specific implementation, the existing technology usually adopts a fixed structure model architecture, that is, the same model parameters and calculation path are used regardless of the characteristics of the input image and external constraints. For example, some systems uniformly use the ViT-L / 14 model to process images of all resolutions, while other systems fixedly use the ViT-B / 32 to pursue processing speed. Through literature retrieval of existing technologies, it is found that the patent with the application number 202510814911.6 and the patent name A multi-modal dynamic facial expression recognition method based on CLIP constructs a label enhancement module, generates positive-negative text supervision, obtains positive text features and negative text features, constructs a multi-modal data mining module to mine different levels of feature information from videos, uses an adaptive fusion strategy to complete the fusion of facial expression features, audio features, and fine-grained text description features, and obtains the fused feature representation. The cosine similarity of the fused feature representation, the positive text feature, and the negative text feature is calculated to obtain the final emotion classification. However, this patent lacks model selection for image feature extraction and lacks multi-modal information fusion. SUMMARY

[0003] Therefore, it is necessary to provide an image label recognition method based on multi-modal feature fusion to realize image label recognition under multi-modal feature fusion of image visual features and EXIF parameter features.

[0004] To solve the above technical problems, the present application provides the following technical solutions: The present application provides an image label recognition method based on multi-modal feature fusion, which comprises: S1: performing image visual feature extraction on the to-be-identified image using a visual encoder in the target model to obtain image features; S2: EXIF parameter extraction and preprocessing is performed on the image to be identified to obtain an EXIF feature vector; S3: The image features and the EXIF feature vector are fused to obtain a fusion feature; S4: A label set is constructed, and a text encoder in a target model is used to convert each label in the label set to obtain a text feature vector; S5: The similarity scores of the fusion feature and the text feature vector are calculated to obtain the similarity scores of the image to be identified and each label, and the label with the highest similarity score is output.

[0005] Preferably, dynamic routing decision analysis is performed on the image to be identified, and a target model is selected from a set of alternative models according to the analysis result.

[0006] Preferably, the dynamic routing decision analysis uses a multi-factor scoring strategy to calculate a selection score of the image to be identified selecting the target model, and the multi-factor scoring strategy calculation expression is as follows:

[0007]

[0008]

[0009]

[0010] wherein, represents the selection score of the image to be identified selecting the target model, represents a normalized resolution factor, represents a time delay constraint factor, represents an available computing resource factor, represents a first adjustable weight coefficient, represents a second adjustable weight coefficient, represents a third adjustable weight coefficient, .

[0011] Preferably, the EXIF parameter extraction and preprocessing includes: determining whether the image to be identified is missing metadata, if not, extracting the image to obtain metadata, and if missing, performing missing value processing on the image to obtain supplementary metadata; converting parameters in the metadata or the supplementary metadata into numerical parameters to obtain numerical metadata; z-score standardization is performed on all numerical parameters in the numerical metadata to obtain an EXIF feature vector, the z-score standardization makes the mean value of all numerical parameters 0 and the standard deviation 1.

[0012] Preferably, the missing value processing includes: using a trained neural network to predict the supplementary metadata or using a mean filling strategy to fill the supplementary metadata.

[0013] Preferably, the multi-modal feature fusion calculation expression is as follows:

[0014] Wherein, the multi-modal feature fusion is realized by a two-layer MLP network, represents the fusion feature, represents the activation function, represents the weight of the first layer MLP network, represents the weight of the second layer MLP network, represents the bias of the first layer MLP network, represents the bias of the second layer MLP network, represents the image feature and the EXIF feature vector is a splicing vector.

[0015] Preferably, the preset prompt template is embedded in the text encoder to enhance the description of each label.

[0016] Preferably, the similarity score calculation expression is as follows:

[0017] Wherein, represents the similarity score of the th label, represents the text feature vector of the th label.

[0018] Preferably, after step S5, it further includes: The similarity score of the to-be-identified image and each label is calculated by a Softmax function to obtain the confidence probability of the to-be-identified image and each label, and the confidence probability calculation expression is as follows:

[0019] Wherein, represents the confidence probability of the th label, represents the similarity score of the th label, represents the similarity score of the a cycle accumulates the similarity score of the label, indicates that there are labels in total; The label with the highest similarity score also has the highest confidence probability calculated, and it is judged whether the highest confidence probability in the confidence probability of the image to be identified and each label is higher than the preset threshold, if higher, the label with the highest similarity score is normally output, if lower, the closed loop learning process is executed.

[0020] Preferably, the closed loop learning process comprises: The image to be identified and the label with the highest confidence probability are recorded, and the image to be identified is manually audited, the correct label is labeled, and the correct label is added to the label set for updating; The image to be identified and the correct label are used as a new data set for image label identification, and then the text encoder is fine-tuned to obtain an updated text feature vector.

[0021] Compared with the prior art, the beneficial effects of the present application are: The present application provides an image label identification method based on multi-modal feature fusion, which fuses the extracted image features with the extracted EXIF parameters to improve the image recognition accuracy of the system, and obtains a text feature vector through the conversion of the constructed label set, and the image to be identified is based on the text feature vector to identify the label with the highest similarity score. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 It is a flowchart of the image label identification method based on multi-modal feature fusion in one embodiment; Figure 2 It is a complete frame diagram of the image label identification method based on multi-modal feature fusion in one embodiment. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0024] Embodiment 1 As shown in Figure 1 , the present embodiment proposes an image label identification method based on multi-modal feature fusion, which comprises: S1: image visual feature extraction is performed on the image to be identified using the visual encoder in the target model to obtain image features; S2: EXIF parameter extraction and preprocessing are performed on the image to be identified to obtain an EXIF feature vector; The implementation of this step is as follows: In order to significantly improve the accuracy of label recognition, the visual features extracted by the visual encoder are spliced with the standardized metadata features, and a multi-layer perception (MLP) is used for feature transformation and fusion to generate a unified multi-modal description vector. The process is as follows: EXIF parameter extraction and preprocessing is performed on the image to be recognized to obtain an EXIF feature vector, including: metadata analysis, EXIF metadata is extracted from the image file, focusing on parameters such as focal length, aperture value, ISO sensitivity, exposure time, and shooting device that are helpful for visual content understanding; missing value processing, for images that lack EXIF data (such as processed network images), a trained neural network is used to predict possible values, or a mean filling strategy is used; data type standardization, converting different formats of EXIF parameters into numerical vectors, such as converting aperture value "f / 2.8" to numerical value 2.8, focal length "50mm" to numerical value 50, and shooting device to one-hot encoding vector; numerical standardization, z-score standardization is performed on all numerical parameters to make their mean value 0 and standard deviation 1.

[0025] S3: Multi-modal feature fusion of the image features and the EXIF feature vector to obtain a fused feature; The implementation of this step is as follows: After obtaining the EXIF feature vector, the visual features extracted by the visual encoder in the CLIP model are effectively fused with the EXIF feature vector of the image. This process is mainly realized through a two-layer MLP network, with an intermediate layer dimension of 2048 and an output layer dimension the same as the CLIP feature dimension (512 dimensions), ensuring that the fused image features and the text feature vector output by the text encoder in the CLIP model are in the same space for similarity calculation.

[0026] S4: Construct a label set and convert each label in the label set using the text encoder in the target model to obtain a text feature vector; The implementation of this step is as follows: The label set is constructed according to industry needs, such as constructing a professional label set containing building elements, interior design, landscape features, etc. in the real estate industry, such as "modern style", "European architecture", "garden landscape", "indoor panorama", "living room", "bedroom", "kitchen", "bathroom", "balcony", "study", "swimming pool", "gym", "commercial facilities", "convenient transportation", "school district house", etc. After constructing the label set corresponding to the industry, all candidate labels are converted into text feature vectors using the text encoder in the CLIP model. In order to improve the description, each label is enhanced using a prompt template, such as "a photo of (label)", "(label) in a real estate scene", etc.

[0027] S5: Calculate the similarity score of the fusion feature and the text feature vector to obtain the similarity score of the to-be-identified image and each label, and output the label with the highest similarity score.

[0028] Embodiment 2 This embodiment further supplements the image label recognition method based on multi-modal feature fusion proposed in Embodiment 1.

[0029] A dynamic routing decision analysis is performed on the to-be-identified image, and a target model is selected from the set of alternative models according to the analysis result.

[0030] The specific implementation of this step is as follows: The resolution of the to-be-identified image, the real-time delay constraint of the system, and the available computing resources are analyzed, and the optimal CLIP model is intelligently selected according to the pre-defined multi-factor scoring strategy (considering the resolution of the image, the real-time delay constraint of the system, and the available computing resources). The visual encoder variant (such as ViT-L / 14, ViT-B / 16, ViT-B / 32) in the optimal CLIP model is obtained; the image visual features are extracted using the visual encoder in the optimal CLIP model to obtain the image features of the to-be-identified image.

[0031] The dynamic routing decision analysis uses a multi-factor scoring strategy to calculate the selection score of the to-be-identified image selecting the target model, and the calculation expression of the multi-factor scoring strategy is as follows:

[0032]

[0033]

[0034]

[0035] wherein, represents the selection score of the to-be-identified image selecting the target model, represents the normalized resolution factor, represents the delay constraint factor, represents the available computing resource factor, represents the first adjustable weight coefficient, represents the second adjustable weight coefficient, represents the third adjustable weight coefficient, .

[0036] The EXIF parameter extraction and preprocessing comprises: determining whether the to-be-identified image is missing metadata, if not, extracting the to-be-identified image to obtain metadata, and if missing, performing missing value processing on the to-be-identified image to obtain supplementary metadata; converting parameters in the metadata or the supplementary metadata into numerical parameters to obtain numerical metadata; performing z-score standardization on all numerical parameters in the numerical metadata to obtain an EXIF feature vector, the z-score standardization making the mean of all numerical parameters 0 and the standard deviation 1.

[0037] The missing value processing comprises: using a trained neural network to predict the supplementary metadata or using a mean filling strategy to fill the supplementary metadata.

[0038] The multi-modal feature fusion calculation expression is as follows:

[0039] The multi-modal feature fusion is realized by using a two-layer MLP network, denotes the fusion feature, denotes an activation function, denotes the weight of the first layer MLP network, denotes the weight of the second layer MLP network, denotes the bias of the first layer MLP network, denotes the bias of the second layer MLP network, denotes the image feature and the EXIF feature vector is a splicing vector.

[0040] The preset prompt template is embedded in the text encoder to enhance the description of each label.

[0041] The similarity score calculation expression is as follows:

[0042] wherein, denotes the similarity score of the i-th label, denotes the text feature vector of the i-th label.

[0043] ​​Taking the actual scene of the real estate industry as an example, a batch of newly shot house pictures need to be quickly processed and automatically labeled for subsequent material management and poster production. The images include indoor panoramic pictures, house type pictures and community environment pictures of different resolutions. It is hoped that appropriate processing speed and recognition accuracy can be obtained on different devices (from high-end workstations to mobile devices). The process is as follows: first, the system receives the house images to be processed, including 4K high-definition panoramic pictures (resolution of 3840x2160), 1080P indoor pictures (resolution of 1920x1080) and some low-resolution preview pictures (resolution lower than 720P); through dynamic routing decision analysis of images by using multi-factor scoring strategy, for 4K high-definition panoramic pictures, ViT-L / 14 visual encoder is selected for feature extraction, for 1080P indoor pictures, ViT-B / 16 visual encoder is selected for feature extraction, and for low-resolution preview pictures, ViT-B / 32 visual encoder is directly selected to realize millisecond-level response; when processing 4K high-definition panoramic pictures, EXIF metadata (focal length 28mm, aperture f / 2.8, ISO-200, and shooting device CanonEOS R5) is extracted from the image, and the extracted EXIF metadata is preprocessed and spliced with the visual features extracted by the ViT-L / 14 visual encoder, that is, 512-dimensional image features are generated through the MLP network; a preset real estate label set is converted into a text feature vector by using the text encoder in the CLIP model, the similarity between the image features and the text feature vector is calculated, and the most matched label and its confidence are output, for example: "luxury living room" (92%), "modern style" (88%) and "good lighting" (95%). The average time consumption of 4K image processing is 136ms, the label accuracy rate reaches 94.5%, the overall system throughput is increased by 42% compared with the fixed model strategy, and high-precision recognition is maintained.

[0044] Embodiment 3 This embodiment further supplements the image label recognition method based on multi-modal feature fusion proposed in embodiment 1.

[0045] When a new concept image to be recognized is encountered, there is no corresponding label matching the label set, and the similarity score calculated by the system is very low, as shown in Figure 2 To form a complete framework of image label recognition based on dynamic routing and multi-modal fusion, a closed-loop learning process needs to be added after step S5, which includes: The similarity score of the image to be recognized and each label is calculated by using the Softmax function to obtain the confidence probability of the image to be recognized and each label. The confidence probability calculation expression is as follows:

[0046] wherein, a confidence probability of a first label, a similarity score of a first label, a similarity score of a first accumulated label in a loop, a total of labels; The confidence probability calculated by the label with the highest similarity score is also the highest. It is judged whether the highest confidence probability of the confidence probability of the image to be identified and each label is higher than a preset threshold. If it is higher, the label with the highest similarity score is normally output. If it is lower, a closed loop learning process is performed.

[0047] The closed loop learning process comprises: The image to be identified and the label with the highest confidence probability are recorded, and the image to be identified is manually audited to obtain a correct label. The correct label is added to the label set for updating; The image to be identified and the correct label are used as a new data set for image label identification, and then the text encoder is fine-tuned to obtain an updated text feature vector.

[0048] The closed loop learning process enables the system to have continuous learning ability and adapt to new labels in the industry. It reduces mislabeling, improves long-term recognition accuracy, reduces manual auditing cost, and only processes low-confidence samples.

Claims

1. An image label recognition method based on multimodal feature fusion, characterized in that, include: S1: The visual encoder in the target model is used to extract the visual features of the image to be recognized, thus obtaining the image features; S2: Extract and preprocess the EXIF ​​parameters of the image to be recognized to obtain the EXIF ​​feature vector; S3: Perform multimodal feature fusion on the image features and the EXIF ​​feature vector to obtain fused features; S4: Construct a label set, and use the text encoder in the target model to transform each label in the label set to obtain a text feature vector; S5: Calculate the similarity score between the fused feature and the text feature vector to obtain the similarity score between the image to be identified and each label, and output the label with the highest similarity score.

2. The image tag recognition method based on multimodal feature fusion according to claim 1, characterized in that, Dynamic routing decision analysis is performed on the image to be recognized, and the target model is selected from the set of candidate models based on the analysis results.

3. The image tag recognition method based on multimodal feature fusion according to claim 2, characterized in that, The dynamic routing decision analysis employs a multi-factor scoring strategy to calculate the selection score of the target selection model for the image to be identified. The multi-factor scoring strategy calculation expression is as follows: in, This represents the selection score of the target model in the image to be identified. This represents the normalized resolution factor. Indicates the delay constraint factor. Indicates the available computational resource factors. This represents the first adjustable weighting coefficient. This represents the second adjustable weighting coefficient. This represents the third adjustable weighting coefficient. .

4. The image tag recognition method based on multimodal feature fusion according to claim 1, characterized in that, The EXIF ​​parameter extraction and preprocessing includes: Determine whether the image to be identified is missing metadata. If it is not missing, extract the metadata from the image to be identified. If it is missing, process the missing values ​​in the image to be identified to obtain supplementary metadata. Convert the parameters in the metadata or the supplementary metadata into numerical parameters to obtain numerical metadata; The z-score standardization is applied to all numerical parameters in the numerical metadata to obtain the EXIF ​​feature vector. The z-score standardization makes the mean of all numerical parameters 0 and the standard deviation 1.

5. The image tag recognition method based on multimodal feature fusion according to claim 4, characterized in that, The missing value processing includes: using a trained neural network to predict supplementary metadata or using a mean-filling strategy to fill in the missing metadata.

6. The image tag recognition method based on multimodal feature fusion according to claim 1, characterized in that, The multimodal feature fusion calculation expression is as follows: The multimodal feature fusion is implemented using a two-layer MLP network. Indicates fusion characteristics, This represents the activation function. This represents the weights of the first layer of the MLP network. This represents the weights of the second-layer MLP network. This represents the bias of the first-layer MLP network. This indicates the bias of the second-layer MLP network. Representing image features and EXIF ​​eigenvectors The concatenated vector.

7. The image tag recognition method based on multimodal feature fusion according to claim 1, characterized in that, The text encoder embeds a preset prompt template to enhance the descriptiveness of each tag.

8. The image tag recognition method based on multimodal feature fusion according to claim 6, characterized in that, The similarity score calculation expression is as follows: in, Indicates the first Similarity score of each tag Indicates the first The text feature vector of each label.

9. The image tag recognition method based on multimodal feature fusion according to claim 1, characterized in that, After step S5, the following also includes: The similarity score between the image to be identified and each label is used to calculate the confidence probability using the Softmax function, resulting in the confidence probability of the image to be identified and each label. The expression for calculating the confidence probability is as follows: in, Indicates the first The confidence probability of each label. Indicates the first Similarity score of each tag Indicates the first The similarity score of each tag is accumulated in a cycle. Indicates a total of One tag; The label with the highest similarity score also has the highest confidence probability. It is determined whether the highest confidence probability among the confidence probabilities of the image to be identified and each label is higher than a preset threshold. If it is higher, the label with the highest similarity score is output normally. If it is lower, a closed-loop learning process is executed.

10. The image tag recognition method based on multimodal feature fusion according to claim 9, characterized in that, The closed-loop learning process includes: Record the image to be identified and the label with the highest confidence probability, and manually review the image to be identified to obtain the correct label. Add the correct label to the label set for updating. The image to be identified and the correct label are used as a new dataset for image label recognition, and then the text encoder is fine-tuned to obtain an updated text feature vector.

Images