A user demand driven key data extraction method

Abstract

The invention discloses a key data extraction method driven by user demand, which includes sending the text of user demand into a preset text encoder model to extract text feature vectors; sending each image in the query image data set into an image encoder to generate image feature vectors, combining the obtained image feature vectors with image statistical features to obtain image quality scores; calculating the relevance score sim of the image and the text; multiplying the image quality score by the relevance score to obtain a quality-weighted relevance score; discarding image data with a relevance score less than a threshold; clustering the remaining image data, and taking the image closest to the cluster center as key data. The key data output by the invention is not only closely related to user demand, but also has higher quality itself, can better meet user demand, and reduces the adverse effects of low-quality data on user decision-making.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a key data extraction method driven by user demand. Background Art

[0002] In recent years, with the rapid development of image sensors, especially smartphone cameras, people have become more inclined to use photos to record various life events, such as work, entertainment, and travel. Furthermore, with the rise of various sharing websites such as WeChat Moments and Weibo, people are sharing the joy of life through photos. For example, the famous photo-sharing website Instagram is the seventh most visited website in the world. Due to these factors, the number of photos taken by individuals every day is exploding. While these massive amounts of photos capture every aspect of life in detail, this explosive growth not only consumes enormous storage space but also poses significant challenges in organizing, sharing, and retrieving these images. Finding key data from these massive images has important applications in photo organization and processing, and has become a hot research topic in image processing, computer vision, and deep learning.

[0003] Most existing methods for extracting key data from massive images rely on individual data quality and statistics, predicting image importance and then filtering out high-quality, highly representative key data. However, this approach ignores user needs. While the resulting key data may better represent the complete image dataset, it may deviate significantly from user needs and fail to meet the demands of actual tasks. Therefore, it is crucial to achieve demand-driven key data extraction by modeling and representing user needs. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, the present invention provides a user-demand-driven key data extraction method. The method includes: feeding the text of the user's requirements into a preset text encoder model to extract a text feature vector; feeding each image in the query image dataset into an image encoder to generate an image feature vector; combining the obtained image feature vector with the image statistical features to obtain an image quality score; calculating a correlation score sim between the image and the text; multiplying the image quality score by the correlation score to obtain a quality-weighted correlation score; discarding image data with a correlation score less than a threshold; clustering the remaining image data, and selecting the image closest to the cluster center as the key data. The key data output by the present invention is not only closely related to the user's requirements, but also has higher quality itself, can better meet user needs, and reduce the adverse impact of low-quality data on user decision-making.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: Feed the user's required text into the preset text encoder model to extract a d t dimensional text feature vector

[0007] Step 2: Feed each image in the query image dataset into the image encoder to generate a d i dimensional image feature vector The obtained image feature vector is combined with the image statistical features and input into a 5-layer MLP to obtain the image quality score q;

[0008] Step 3: Use a linear mapping to map the text feature vector obtained in step 1 and the image feature vector obtained in step 2 to the same dimension and perform L2 norm normalization; then use cosine similarity to calculate the relevance score sim between the image and text;

[0009] Step 4: Multiply the image quality score obtained in step 2 by the correlation score sim obtained in step 3 to obtain a quality-weighted correlation score s = q·sim;

[0010] Step 5: Based on the quality-weighted correlation scores obtained in step 4, discard image data with correlation scores less than a threshold value T = 0.3;

[0011] Step 6: Cluster the remaining image data from step 5 using the K-means algorithm based on sample weights. The resulting cluster centers are the key data.

[0012] Preferably, the text encoder is a Transformer model containing 12 8-heads and uses pre-trained model parameters on the large-scale dataset WIT.

[0013] Preferably, the step 2 is specifically as follows:

[0014] Step 2-1: The image encoder is a Transformer-based ViT model whose parameters are pre-trained on the WIT dataset;

[0015] Step 2-2: The statistical features of the image are calculated as follows: first convert the image into a grayscale image, and then calculate the average gradient G and image entropy E of all pixels in the grayscale image I ; Image entropy represents the amount of information an image has, and its calculation formula is:

[0016]

[0017] Where p(n) represents the probability that the pixel gray value is n;

[0018] Step 2-3: The obtained image feature vector fimg , average gradient G and image entropy E I The vectors are concatenated and fed into an MLP network with two hidden layers to predict the image quality score q. The hidden layer dimensions of the MLP network are 64 and 32 respectively, and the parameters of the network are trained using the image quality assessment dataset LIVE.

[0019] Preferably, the step 3 is specifically:

[0020] Step 3-1: Linear feature mapping and L2 norm normalization are uniformly expressed as:

[0021] f′ img =G(f img W img )

[0022] f′ text =G(f text W text )

[0023] where f′ img 、f′ text are the normalized image feature vector and text feature vector, W img 、W text Represents the weight of the corresponding linear mapping layer; G represents the L2 norm normalization operation;

[0024] Step 3-2: The calculation formula of the relevance score sim is:

[0025]

[0026] Preferably, the step 6 is specifically as follows:

[0027] Step 6-1: If the number of images remaining in step 5 is less than the number of cluster centers K, all remaining images are used as key data. Otherwise, K images are selected as key data using the K-means algorithm based on sample weights.

[0028] Step 6-2: Use the K-means algorithm based on sample weights to calculate the cluster center, using the correlation score generated in step 4 as the weight, specifically expressed as:

[0029]

[0030] Among them, c i represents the i-th cluster center, m i Indicates the number of samples; is the feature vector of the jth image sample, w j Represents the weight of the corresponding image, which is obtained by normalizing the metric index to a coarse class, namely:

[0031]

[0032] where s i represents the correlation score of the jth image;

[0033] Step 6-3: For each cluster center obtained in 6-2, select an image with the smallest Euclidean distance to its feature, thereby obtaining K images as key data;

[0034] Step 6-4: Based on the quality-weighted correlation scores obtained in step 4, the key data obtained in step 6-1 or step 6-3 are sorted and output from large to small.

[0035] Preferably, the number of cluster centers K is set to 0.1 of the number of remaining images after step 5, and is rounded up.

[0036] The beneficial effects of the present invention are as follows:

[0037] (1) The present invention comprehensively considers the similarity between images and user needs and the quality of individual images, so that the output key data is not only closely related to user needs, but also has higher quality itself, which can better meet user needs and reduce the adverse impact of low-quality data on user decision-making.

[0038] (2) The present invention designs a new key data selection method that can take into account the similarity between data and user needs, improve the diversity of the final output data through clustering operations, and better characterize the original samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below with reference to the accompanying drawings and examples.

[0041] This paper proposes a user-demand-driven key data extraction method. It designs a semantically consistent correlation model between user demands and image data, ensuring semantic consistency between the extracted key data and user demands. Specifically, the method uses text to represent user demands. Then, using a correlation metric model between text and image data, the method extracts data with high semantic consistency. Furthermore, through quality scores and diversity constraints, the image quality and diversity of the key data are further improved.

[0042] To achieve the above object, the specific steps of the present invention are:

[0043] Step 1: Feed the user's required text into the preset text encoder model to extract a d t dimensional text feature vector The text encoder used is a Transformer model with 12 8-heads, and uses pre-trained model parameters on the large-scale dataset WIT.

[0044] Step 2: Feed each image in the query image dataset into the image encoder to generate a d i dimensional image feature vector The obtained feature vector is combined with the image statistical features and input into a 5-layer MLP to obtain the image quality score q.

[0045] 1) The image encoder is a Transformer-based ViT model, whose parameters are pre-trained on the WIT dataset;

[0046] 2) The statistical feature calculation method of the image is as follows: first, the image is converted into a grayscale image, and then the average gradient G and image entropy E of all pixels in the grayscale image are calculated. I Image entropy represents the amount of information an image has, and its calculation formula is:

[0047]

[0048] Where p(n) represents the probability that the pixel gray value is n.

[0049] 3) The obtained image feature vector f img , average gradient G and image entropy E I The concatenation is then fed into a two-hidden-layer MLP network to predict the image quality score q. The hidden dimensions of the MLP network are 64 and 32, respectively. The parameters of the network are trained using the LIVE image quality assessment dataset.

[0050] Step 3: Use a linear mapping to map the text feature vector obtained in step 1 and the image feature vector obtained in step 2 to the same dimension and perform L2 norm normalization. Then use cosine similarity to calculate the relevance score sim between the image and text.

[0051] 1) The linear feature mapping and L2 norm normalization can be uniformly expressed as:

[0052] f′ img =G(f img W img )

[0053] f′ text =G(f text W text )

[0054] where f′ img 、f′ textare the normalized image feature vector and text feature vector, W img 、W text Represents the weights of the corresponding linear mapping layer. It is typically initialized to the model weights on the WIT dataset and then fine-tuned on the task dataset. G represents the L2 norm normalization operation.

[0055] 2) The calculation formula of cosine similarity is:

[0056]

[0057] sim represents the obtained image and text relevance score.

[0058] Step 4: Multiply the image quality score obtained in step 2 by the image and text similarity obtained in step 3 to obtain the quality-weighted correlation score s = q·sim

[0059] Step 5: Based on the quality-weighted correlation scores obtained in step 4, discard image data with scores less than a threshold value T=0.3.

[0060] Step 6: Cluster the image data generated in step 5 using the K-means algorithm based on sample weights. Then, for each cluster center, select an image with the closest Euclidean distance to its feature, and finally obtain K images as key data.

[0061] 1) In order to extract high-quality, highly relevant key data, the present invention uses a K-means algorithm based on sample weights. The key is to use the correlation scores generated in step 4 as weights when calculating cluster centers, which can be specifically expressed as

[0062]

[0063] Among them, c i represents the i-th cluster center, m i Indicates the number of samples. is the feature vector of the jth image sample, w j Represents the weight of the corresponding image, which is obtained by normalizing the metric index to a coarse class, that is,

[0064]

[0065] where s i represents the correlation score of the j-th image.

[0066] 2) The number of cluster centers, K, can be user-input. If not specified, it is set to 0.1 of the number of images remaining after step 5, rounded up. If the number of images obtained in step 5 is less than the user-specified number, the remaining data is used as the key data.

[0067] 3) When outputting key data, sort them from largest to smallest according to their quality-weighted relevance scores.

Claims

1. A user demand driven key data extraction method, characterized in that: The steps include: Step 1: Feed the text of the user's request into the preset text encoder model to extract a d t dimensional text feature vector Step 2: Feed each image in the query image dataset into the image encoder to generate a d i dimensional image feature vector The obtained image feature vector is combined with the image statistical features and input into a 5-layer MLP to obtain the image quality score q; Step 3: Use a linear mapping to map the text feature vector obtained in step 1 and the image feature vector obtained in step 2 to the same dimension and perform L2 norm normalization; then use cosine similarity to calculate the correlation score sim between the image and the text; Step 4: Multiply the image quality score obtained in step 2 by the correlation score sim obtained in step 3 to obtain a quality-weighted correlation score s = q·sim; Step 5: Based on the quality-weighted correlation scores obtained in step 4, discard image data with correlation scores less than a threshold value T=0.3; Step 6: For the remaining image data in step 5, use the K-means algorithm based on sample weights to cluster them, and the obtained cluster center is the key data.

2. The method for extracting key data driven by user demand according to claim 1, characterized in that: The text encoder is a Transformer model containing 12 8-heads and uses pre-trained model parameters on the large-scale dataset WIT.

3. The method for extracting key data driven by user demand according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2-1: The image encoder is a Transformer-based ViT model whose parameters are pre-trained on the WIT dataset; Step 2-2: The statistical features of the image are calculated as follows: first, the image is converted into a grayscale image, and then the average gradient G and image entropy E of all pixels in the grayscale image are calculated. I ; Image entropy represents the amount of information in an image, and its calculation formula is: Where p(n) represents the probability that the pixel gray value is n; Step 2-3: The obtained image feature vector f img , average gradient G and image entropy E I The images are concatenated into a vector and sent into an MLP network with two hidden layers to predict the image quality score q. The hidden layer dimensions of the MLP network are 64 and 32 respectively, and the parameters of the network are trained using the image quality assessment dataset LIVE.

4. The method for extracting key data driven by user demand according to claim 3, characterized in that: The step 3 is specifically as follows: Step 3-1: Linear feature mapping and L2 norm normalization are uniformly expressed as: f i ′ mg =G(f img W img ) f te ′ xt =G(f text W text ) where f i ' mg 、f te ' xt are the normalized image feature vector and text feature vector, W img , W text Represents the weight of the corresponding linear mapping layer; G represents the L2 norm normalization operation; Step 3-2: The calculation formula of the correlation score sim is:

5. The method for extracting key data driven by user demand according to claim 4, characterized in that: The step 6 is specifically as follows: Step 6-1: If the number of images remaining in step 5 is less than the number of cluster centers K, all remaining images are used as key data; otherwise, K images are selected as key data using the K-means algorithm based on sample weights; Step 6-2: Use the K-means algorithm based on sample weights to calculate the cluster center, using the correlation score generated in step 4 as the weight, specifically expressed as: Among them, c i represents the i-th cluster center, m i represents the number of samples; is the feature vector of the jth image sample, w j Represents the weight of the corresponding image, which is calculated by normalizing the metric index by coarse category, that is: where s i represents the relevant score of the jth image; Step 6-3: For each cluster center obtained in 6-2, select an image with the smallest characteristic Euclidean distance, thereby obtaining K images as key data; Step 6-4: Based on the quality-weighted correlation score obtained in step 4, the key data obtained in step 6-1 or step 6-3 are sorted and output from large to small.

6. The method for extracting key data driven by user demand according to claim 5, characterized in that: The number of cluster centers K is set to 0.1 of the number of remaining images after step 5, and is rounded up.

Images