Image retrieval method and device, computer device and storage medium

Abstract

The application relates to an image retrieval method and device, computer equipment and a storage medium, and relates to the technical field of big data. The method comprises the following steps: acquiring a retrieval text, and matching at least one target image category corresponding to the retrieval text and a target original image corresponding to the retrieval text; determining a target region corresponding to each target image category in the target original image; performing resolution enhancement processing on the target region in the target original image, so as to obtain a target image corresponding to the retrieval text. The method can improve the efficiency of retrieving images, and can improve the definition of the required region of a propagator in the target image.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to an image retrieval method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Currently, when companies create promotional materials, they typically overlay various materials onto a background panel. However, with the ever-growing size of material libraries, promotional staff often need to quickly retrieve the materials they require from the library when downloading them.

[0003] In existing technologies, material libraries can often only perform rough classification of materials. Publicity personnel usually need to manually search for the materials they need in the material library, resulting in low efficiency in material retrieval. Summary of the Invention

[0004] Therefore, it is necessary to provide an image retrieval method, apparatus, computer equipment, and storage medium to address the aforementioned technical problems.

[0005] Firstly, this application provides an image retrieval method. The method includes:

[0006] Obtain the search text and match it to obtain at least one target image category corresponding to the search text, as well as the original target image corresponding to the search text;

[0007] In the original target image, the target region corresponding to each target image category is determined respectively;

[0008] For any of the original target images, the target region in the original target image is subjected to resolution enhancement processing to obtain the target image corresponding to the searched text.

[0009] In one embodiment, the matching to obtain at least one target image category corresponding to the searched text and the original target image corresponding to the searched text includes:

[0010] The search text is processed by vector extraction to obtain the text vector corresponding to the search text;

[0011] Based on the text vector corresponding to the searched text and the text vector corresponding to each image category, determine the target image category corresponding to the searched text in each of the image categories;

[0012] Based on the text vector corresponding to the searched text and the feature vectors corresponding to each original image, the target original image corresponding to the searched text in each of the original images is determined.

[0013] In one embodiment, the feature vector corresponding to the original image is obtained based on the tag text vector corresponding to the image tag of the original image and the image vector of the original image.

[0014] In one embodiment, determining the target original image corresponding to the searched text in each of the original images based on the text vector corresponding to the searched text and the feature vector corresponding to each original image includes:

[0015] Determine the similarity between the text vector corresponding to the retrieved text and the feature vector corresponding to each of the original images;

[0016] The original image whose similarity meets the preset conditions is used as the target original image corresponding to the searched text.

[0017] In one embodiment, determining the target region corresponding to each target image classification in the original target image includes:

[0018] The original target image is convolved to obtain an image feature map of the original target image at the convolutional layer, and the image feature map includes at least one channel.

[0019] For any of the target image categories and any of the channels, determine the classification weight of the channel for the target image category;

[0020] For any of the target image classifications, the channels are superimposed according to each channel and the classification weight corresponding to each channel to obtain a target feature map, and the target region corresponding to the original image and the target image classification is determined from the target feature map.

[0021] In one embodiment, the step of performing resolution upscaling on each of the target regions to obtain the target image corresponding to the searched text includes:

[0022] Determine the matching degree between the searched text and each of the target image categories, and based on the matching degree, determine the resolution enhancement area of ​​the target region corresponding to each of the target image categories;

[0023] For any of the target regions, a resolution-enhancing region is determined from the target regions based on the resolution-enhancing area;

[0024] The resolution enhancement regions are processed to obtain the target image corresponding to the searched text.

[0025] In one embodiment, determining the resolution enhancement area of ​​the target region corresponding to each target image classification based on the matching degree includes:

[0026] Based on the matching degree of each target image classification, the area ratio of the region corresponding to each target image classification is determined, and the area ratio of the region is positively correlated with the matching degree;

[0027] For any of the target image classifications, the resolution enhancement area of ​​the target region corresponding to the target image classification is determined based on the ratio of the base resolution enhancement area to the area of ​​the region corresponding to the target image classification.

[0028] In one embodiment, the step of performing resolution upscaling on each of the target regions to obtain the target image corresponding to the searched text includes:

[0029] Interpolation processing is performed on each of the target regions to obtain the target image corresponding to the searched text.

[0030] Secondly, this application also provides an image retrieval device. The device includes:

[0031] The acquisition module is used to acquire the search text and match it to obtain at least one target image category corresponding to the search text, as well as the original target image corresponding to the search text;

[0032] The determination module is used to determine the target regions in the original target images that correspond to the classification of each target image;

[0033] The processing module is used to perform resolution enhancement processing on the target region in any of the original target images to obtain the target image corresponding to the searched text.

[0034] In one embodiment, the acquisition module is further configured to:

[0035] The search text is processed by vector extraction to obtain the text vector corresponding to the search text;

[0036] Based on the text vector corresponding to the searched text and the text vector corresponding to each image category, determine the target image category corresponding to the searched text in each of the image categories;

[0037] Based on the text vector corresponding to the searched text and the feature vectors corresponding to each original image, the target original image corresponding to the searched text in each of the original images is determined.

[0038] In one embodiment, the feature vector corresponding to the original image is obtained based on the tag text vector corresponding to the image tag of the original image and the image vector of the original image.

[0039] In one embodiment, the acquisition module is further configured to:

[0040] Determine the similarity between the text vector corresponding to the retrieved text and the feature vector corresponding to each of the original images;

[0041] The original image whose similarity meets the preset conditions is used as the target original image corresponding to the searched text.

[0042] In one embodiment, the determining module is further configured to:

[0043] The original target image is convolved to obtain an image feature map of the original target image at the convolutional layer, and the image feature map includes at least one channel.

[0044] For any of the target image categories and any of the channels, determine the classification weight of the channel for the target image category;

[0045] For any of the target image classifications, the channels are superimposed according to each channel and the classification weight corresponding to each channel to obtain a target feature map, and the target region corresponding to the original image and the target image classification is determined from the target feature map.

[0046] In one embodiment, the processing module is further configured to:

[0047] Determine the matching degree between the searched text and each of the target image categories, and based on the matching degree, determine the resolution enhancement area of ​​the target region corresponding to each of the target image categories;

[0048] For any of the target regions, a resolution-enhancing region is determined from the target regions based on the resolution-enhancing area;

[0049] The resolution enhancement regions are processed to obtain the target image corresponding to the searched text.

[0050] In one embodiment, the processing module is further configured to:

[0051] Based on the matching degree of each target image classification, the area ratio of the region corresponding to each target image classification is determined, and the area ratio of the region is positively correlated with the matching degree;

[0052] For any of the target image classifications, the resolution enhancement area of ​​the target region corresponding to the target image classification is determined based on the ratio of the base resolution enhancement area to the area of ​​the region corresponding to the target image classification.

[0053] In one embodiment, the processing module is further configured to:

[0054] Interpolation processing is performed on each of the target regions to obtain the target image corresponding to the searched text.

[0055] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any of the methods described above.

[0056] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements any of the above methods.

[0057] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements any of the above methods.

[0058] The aforementioned image retrieval method, apparatus, computer equipment, and storage medium match the retrieval text with the target image category and the original target image, and then perform resolution enhancement processing on the regions in the original target image that match the target image category to obtain the target image corresponding to the retrieval text. This application embodiment allows promotional personnel to match the required original target image simply by inputting text, improving the efficiency of retrieving the desired image; simultaneously, the resolution enhancement processing on the regions in the target image that match the target image category can also specifically improve the clarity of the regions in the target image that the promotional personnel need. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating an image retrieval method in one embodiment;

[0060] Figure 2 This is a flowchart illustrating step 102 in one embodiment;

[0061] Figure 3 This is a flowchart illustrating step 206 in one embodiment;

[0062] Figure 4 This is a flowchart illustrating step 104 in one embodiment;

[0063] Figure 5 This is a flowchart illustrating step 106 in one embodiment;

[0064] Figure 6 This is a flowchart illustrating step 502 in one embodiment;

[0065] Figure 7 This is a structural block diagram of an image retrieval device in one embodiment;

[0066] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] In one embodiment, such as Figure 1 As shown, an image retrieval method is provided. This embodiment illustrates the method applied to a server, but it is understood that the method can also be applied to a terminal, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0069] Step 102: Obtain the search text and match it to obtain at least one target image category corresponding to the search text, as well as the target original image corresponding to the search text.

[0070] In this embodiment, the search text is the text entered by the user when searching for images. It can be a keyword description of the image to be searched (e.g., "landscape," "architecture," "portrait," etc.) or a text description (e.g., "landscape with architecture," etc.). The image classification can be a classification pre-built by the server based on the common material types required. For example, if many publicity personnel need "cartoon" type materials, the server can build an image classification for "cartoon." After receiving the search text, the server can parse the search text, extract its corresponding text vector or keywords, and match the text vector or keywords with each pre-set image classification. The matched image classification is the target image classification.

[0071] The server can also match the target original image from the original images (i.e., images that can be used as source material) stored on the server based on the search text. The original images are all tagged. The server can match the search text with the tags of each original image and select the original images with the highest matching degree between the tags and the search text, or the original images with the matching degree between the tags and the search text exceeding a threshold, as the target original images. This application embodiment does not specifically limit this.

[0072] It's important to note that image classification and the labels of the original images do not necessarily correspond. For example, if the labels of the original images only include A, B, and C, the image classification could be one of A, B, and C, or it could be D, which is different from A, B, and C. This is because the original images may be downloaded from other databases and already tagged, while the image classification is used to characterize specific material categories that promotional personnel might need in the context of creating promotional materials, and it may not be one of the labels of the original images.

[0073] Step 104: Determine the target region corresponding to each target image classification in the original target image.

[0074] In this embodiment, since the searched text matches the target image category and also matches the original target image, there is a high probability that the original target image contains a target region that matches at least one target image category. Since the target image category is the image category that best matches the searched text and is likely the most needed by the advertising personnel, the resolution of the target region in the original target image can be increased to make the original target image better meet the needs of the advertising personnel.

[0075] A semantic segmentation model can be trained based on each image classification, and then the trained semantic segmentation model can be used to obtain the target region corresponding to the target image classification. Alternatively, algorithms such as visual class activation maps (CAM, or CAM heatmaps) can be used to determine the target region corresponding to the target image classification. This application does not specifically limit the algorithm for determining the target region in its embodiments.

[0076] A target region can be determined for each target image classification. The target regions determined for different target image classifications may overlap. In this case, the overlapping target regions can be merged into one target region, or the overlapping parts can be removed and each overlapping target region can be used as a separate target region. This application does not specifically limit this approach.

[0077] Step 106: For any target original image, perform resolution enhancement processing on the target region in the target original image to obtain the target image corresponding to the searched text.

[0078] In this embodiment, resolution enhancement processing can be performed on the determined target area to make the processed original image better meet the needs of the publicity personnel. Resolution enhancement processing can be performed on the entire target area, or only on a portion of the target area, or a portion of the target area; this embodiment does not specifically limit this.

[0079] The specific target regions and their corresponding parts can be upgraded in resolution based on one or more factors, such as the size of each target region, the degree of matching between each target region and the target image category, and the contour of each target region. For example, if a target region is too small, even after resolution upgrades, it may not meet the requirements for use as source material. Therefore, a region size threshold can be set, and resolution upgrades can only be performed on target regions whose size exceeds this threshold. When there are many target image categories matched by the search text, it may not be necessary to upgrade the resolution of all target regions corresponding to all target image categories. In this case, the resolution of target regions with a high degree of matching between each target region and the target image category can be selected for resolution upgrades. When the contour of a target region does not match the contour of an object within the target region (for example, a target region detected by CAM heatmaps usually covers a region of the object within the target region, and the contour of this region does not match the contour of the object), since resolution upgrades are generally only needed for the objects within the target region, edge detection algorithms can be used to detect the objects within the target region, and then resolution upgrades can be performed on the objects. In cases where there are other resolution improvement requirements, the specific requirements for how to improve the resolution of the target area can be determined according to actual needs. This application does not impose specific limitations on this.

[0080] Any resolution enhancement algorithm can be used to enhance the resolution of the target region (such as super-resolution reconstruction algorithms or interpolation algorithms, etc.). This application embodiment does not specifically limit the algorithm used. After performing resolution enhancement processing on each target region, the server can use the obtained original target image as the target image and return the target image to the user.

[0081] The image retrieval method provided in this application matches the search text with the target image category and the original target image. Then, it performs resolution enhancement processing on the regions in the original target image that match the target image category to obtain the target image corresponding to the search text. This application allows promotional personnel to match the required original target image simply by inputting text, improving the efficiency of image retrieval. Furthermore, the resolution enhancement processing on the regions in the target image that match the target image category can specifically improve the clarity of the areas in the target image that promotional personnel need.

[0082] In one embodiment, such as Figure 2 As shown, in step 102, matching at least one target image category corresponding to the searched text and the original target image corresponding to the searched text includes:

[0083] Step 202: Perform vector extraction processing on the searched text to obtain the text vector corresponding to the searched text.

[0084] Step 204: Based on the text vector corresponding to the retrieved text and the text vector corresponding to each image category, determine the target image category corresponding to the retrieved text in each image category.

[0085] Step 206: Based on the text vector corresponding to the searched text and the feature vectors corresponding to each original image, determine the target original image corresponding to the searched text in each original image.

[0086] In this embodiment, the target image classification and the original target image can be matched based on the corresponding text vector in the retrieved text. For example, the Transformer model (a semantic extraction model) can be used to extract vectors from the retrieved text. In the Transformer model, each word has three different vectors: the Query vector Q, the Key vector K, and the Value vector V. These three vectors are obtained by converting each word in the retrieved text into a word vector X, and then multiplying each word vector X by three different weight matrices W. Q W K W V The result is obtained by multiplication. The self-attention vector of the word can be calculated using vectors Q, K, and V (see formula (I)).

[0087]

[0088] Here, attention(Q,K,V) is the word's self-attention vector, and softmax is a mapping function used to map the Q, K, and V vectors to the (0,1) interval. T This refers to finding the transpose of vector K. K This is the dimension of vector K. Based on the self-attention vector of each word, the text vector of the retrieved text can then be obtained.

[0089] Each image category also has its corresponding text vector, which can be extracted using the Transformer model. By calculating the similarity between the text vector of the retrieved text and the text vectors of each image category, the target image category corresponding to the retrieved text can be obtained.

[0090] The feature vector corresponding to the original image can be the text vector of the label of the original image, or it can be the image feature vector obtained by convolution of the original image. This application does not specifically limit the convolutional neural network used. The image feature vector obtained by convolution of the original image I can be shown in formula (II):

[0091] O(i,j)=(H×I)(i,j)=∑ m ∑ n Formula (II) for I(i+m,i+n)H(m,n)

[0092] Where i and j represent the total number of pixels in the original image I in terms of width and height, respectively, O(i,j) represents the obtained image feature vector, H is the convolution kernel, and m and n represent the width and height of the convolution kernel, respectively. I(i+m,i+n)H(m,n) represents the multiplication of a region I(i+m,i+n) of the original image I with width m and height n by the convolution kernel H(m,n), ∑ m and ∑ n This indicates that the summation is performed horizontally and vertically within the matrix.

[0093] When the feature vector is an image feature vector, a matching model can be constructed and trained based on the text vector of the label of the original image and the corresponding image feature vector of the original image. Using the trained matching model, the target original image of the retrieved text can be obtained by matching the text vector of the retrieved text.

[0094] The image retrieval method provided in this application extracts the text vector of the retrieval text and matches the target image classification and the original target image based on the text vector. By performing resolution enhancement processing on the regions in the target image that match the target image classification, the clarity of the areas in the target image that require promotional materials can be specifically improved.

[0095] In one embodiment, the feature vector corresponding to the original image is obtained based on the label text vector corresponding to the image label of the original image and the image vector of the original image.

[0096] In this embodiment, the label text vector corresponding to the image label of the original image can be concatenated with the image vector of the original image to obtain the cross-modal feature vector of the original image, thereby improving the effect of retrieving the target original image based on the text vector. The label text vector can be extracted based on the image label and the Transformer model, and the image vector can be extracted by any convolutional neural network. After unifying the dimensions of the label text vector and the image vector (that is, transforming the label text vector into a vector of size 1×d and the image vector into a vector of size N×d), the feature vector corresponding to the original image can be obtained by adding each row of the label text vector and the image vector separately, or by concatenating the label text vector to the image vector (that is, obtaining a vector of size (N+1)×d).

[0097] The image retrieval method provided in this application concatenates the tag text vector corresponding to the image tag of the original image with the image vector of the original image to obtain the feature vector of the original image. This allows the feature vector to include cross-modal information, improves the accuracy of the feature vector, and thus enhances the accuracy of retrieving the target original image based on the retrieval text.

[0098] In one embodiment, such as Figure 3 As shown, in step 206, based on the text vector corresponding to the retrieved text and the feature vectors corresponding to each original image, the target original image corresponding to the retrieved text in each original image is determined, including:

[0099] Step 302: Determine the similarity between the text vector corresponding to the retrieved text and the feature vector corresponding to each original image.

[0100] Step 304: The original image whose similarity meets the preset conditions is used as the target original image corresponding to the search text.

[0101] In this embodiment, the target original image can be determined based on the similarity between the text vector and the feature vectors of each original image. This similarity can be calculated using a trained matching model: the matching model can be trained using the label text vectors and feature vectors of the original images, such that the similarity calculated by the matching model is as large as possible for label text vectors and feature vectors from the same original image, and as small as possible for label text vectors and feature vectors from different original images, until the loss is less than a preset loss threshold, thus obtaining a trained matching model.

[0102] After calculating the similarity between the searched text and each original image based on the text vector of the searched text and the feature vectors of each original image, the original image whose similarity meets the preset conditions can be used as the target original image. The preset conditions can be set by those skilled in the art according to actual needs, such as similarity greater than a similarity threshold, or after sorting the similarities from largest to smallest, the similarity is placed in the first preset position, etc.

[0103] The image retrieval method provided in this application determines the target original image based on the similarity between text vectors and feature vectors. It can match the target original image that is similar to the text vector from each original image, thereby improving the accuracy of retrieving the target original image based on the retrieved text.

[0104] In one embodiment, such as Figure 4 As shown, in step 104, the target regions corresponding to each target image classification are determined in the original target image, including:

[0105] Step 402: Perform convolution processing on the original target image to obtain the image feature map corresponding to the original target image in the convolution layer. The image feature map includes at least one channel.

[0106] Step 404: For any target image classification and any channel, determine the classification weight of the channel for the target image classification.

[0107] Step 406: For any target image classification, according to each channel and the classification weight corresponding to each channel, the channels are superimposed to obtain the target feature map, and the target region corresponding to the classification of the original image and the target image is determined from the target feature map.

[0108] In this embodiment, a CAM heatmap can be used to determine the target region corresponding to the target image classification. The principle of the CAM heatmap is to extract the image feature map of the image in the convolutional layer of the neural network, and based on the classification weight assigned to each channel in the image feature map when the neural network classifies the image according to the image feature map, determine the contribution of each channel to classifying the image into the target image category. Then, multiply each channel with its corresponding classification weight and perform superposition processing. From the target feature map obtained after superposition, determine the region that contributes the most to classifying the original target image into the target image category, which is the target region.

[0109] To obtain classification weights, an image classification neural network can be pre-trained to classify images into different categories based on the image feature maps output by the convolutional layers of the convolutional neural network. Since image feature maps have multiple channels, the image classification neural network assigns a classification weight to each channel when classifying images. The probability of an image belonging to a particular category is obtained by weighting and summing the values ​​of each channel after average pooling according to their classification weights. After training the image classification neural network, the classification weights corresponding to each target image category can be obtained.

[0110] When determining the target region of a target image for a specific target image classification, the classification weights of each channel corresponding to the target image classification can be multiplied by each channel before average pooling, and then the channels multiplied by the classification weights are superimposed. Since a larger classification weight indicates a higher contribution of that channel to classifying the target image to the target image classification, in the target feature map obtained by superimposing the channels multiplied by their respective classification weights, a larger pixel value indicates a higher probability that the pixel belongs to one or more non-zero pixels in channels with larger classification weights. In other words, the greater the pixel's contribution to classifying the target image to the target image classification, the more likely the pixel belongs to the target region. Because the image feature map of the convolutional layer is different in size from the target image, the target feature map can first be interpolated to restore it to the size of the target image. Then, based on a preset pixel threshold, all pixels with pixel values ​​greater than the threshold are obtained, and the region represented by these pixels in the restored target feature map is taken as the target region.

[0111] The image retrieval method provided in this application determines the target region for classifying the target image in the original target image based on the CAM heatmap. It can obtain the region that contributes the most to classifying the original target image into the target image classification, thereby improving the accuracy of target region determination.

[0112] In one embodiment, such as Figure 5 As shown, in step 106, resolution enhancement processing is performed on each target region to obtain the target image corresponding to the retrieved text, including:

[0113] Step 502: Determine the matching degree between the retrieved text and each target image category, and based on the matching degree, determine the resolution enhancement area of ​​the target region corresponding to each target image category.

[0114] Step 504: For any target region, determine the resolution enhancement region from the target region based on the resolution enhancement area.

[0115] Step 506: Perform resolution enhancement processing on each resolution enhancement region to obtain the target image corresponding to the searched text.

[0116] In this embodiment, resolution enhancement can be performed on only a portion of each target region. Since a higher matching degree between the search text and the target image category indicates that the target region corresponding to that image category is more likely to be the region needed by the advertising personnel, the area for resolution enhancement can be larger. When determining the target image category based on keywords in the search text, the matching degree between the keywords and the target image category (e.g., the overlap between the characters of the keywords and the characters of the target image category, the similarity between the word vectors of the keywords and the word vectors of the target image category, etc.) can be used as the matching degree; when determining the target image category based on the text vector of the search text, the similarity between the text vector of the search text and the word vectors of the target image category can be used as the matching degree. This embodiment does not specifically limit this.

[0117] This approach allows for a positive correlation between the resolution enhancement area of ​​the target region corresponding to the target image category and the matching degree between the searched text and the target image category. For example, a minimum resolution enhancement area (which could be the minimum image size required for effective resolution enhancement) can be pre-set and used as the resolution enhancement area of ​​the target region for the target image category with the lowest matching degree with the searched text. Then, the resolution enhancement areas of the target regions for other target image categories are set based on the ratio of the matching degree of other target image categories to that of the target image category. Alternatively, other methods can be used to set the resolution enhancement area of ​​each target region so that the resolution enhancement area is positively correlated with the matching degree between the target image category and the searched text corresponding to the target region. This application does not specifically limit this approach.

[0118] After determining the resolution enhancement area for each target region, the actual resolution enhancement region to be processed can be determined from the target region based on the resolution enhancement area. For example, edge detection algorithms or similar algorithms can be used to detect the areas in the target region where objects actually exist, and these areas can be prioritized as resolution enhancement regions. If there is still a remainder after subtracting the resolution enhancement region from the resolution enhancement area, other parts of the target region (such as areas adjacent to the areas where objects actually exist) can also be included as resolution enhancement regions as needed. When using CAM heatmaps to detect target regions, starting from the pixel with the highest pixel value in the resized target feature map, pixels with decreasing pixel values ​​can be included as resolution enhancement regions one by one until the size of the resolution enhancement region and the resolution enhancement area are equal.

[0119] When the area of ​​resolution enhancement is greater than the area of ​​the target region, the entire target region can be considered as the resolution enhancement area. In this case, either the remaining resolution enhancement area (which is the difference between the resolution enhancement area and the area of ​​the target region) can be ignored, or the remaining resolution enhancement area can be allocated to the target image category with the highest matching degree with the searched text among the target image categories whose resolution enhancement area is smaller than the area of ​​the target region. This application does not specifically limit this approach.

[0120] The image retrieval method provided in this application determines the resolution enhancement regions that need to be improved in each target region based on the matching degree between the retrieval text and the target image classification. This can improve the accuracy of the resolution enhancement regions and thus improve the accuracy of the processed target image.

[0121] In one embodiment, such as Figure 6 As shown, in step 502, based on the matching degree, the resolution enhancement area of ​​the target region corresponding to each target image classification is determined, including:

[0122] Step 602: Based on the matching degree of each target image classification, determine the area ratio of the region corresponding to each target image classification. The area ratio is positively correlated with the matching degree.

[0123] Step 604: For any target image classification, determine the resolution enhancement area of ​​the target region corresponding to the target image classification based on the ratio of the base resolution enhancement area to the area of ​​the region corresponding to the target image classification.

[0124] In this embodiment, the area ratio of each target image category can be determined based on the matching degree of each target image category. The area ratio is positively correlated with the matching degree, and the sum of the area ratios of each category should be 100%. For example, if the matching degree ratios of each target image category are 85:90:75, then the area ratio of each target image category can be 34:36:30.

[0125] The baseline resolution upscaling area characterizes the total area of ​​the region in the original target image where resolution upscaling is performed. The baseline resolution upscaling area can be a fixed value or a value positively correlated with the number of identified target image categories. For any target region, the resolution upscaling area can be obtained by multiplying the area ratio of that target region by the baseline resolution upscaling area.

[0126] The image retrieval method provided in this application determines the region area ratio based on the matching degree, and then determines the specific resolution enhancement region that needs to be enhanced in each target region based on the region area ratio and the base resolution enhancement area. This can control the total area of ​​resolution enhancement processing in each target original image and improve the efficiency of resolution enhancement processing of the target original image.

[0127] In one embodiment, step 106 involves performing resolution upscaling on each target region to obtain the target image corresponding to the retrieved text, including:

[0128] Interpolation is performed on each target region to obtain the target image corresponding to the retrieved text.

[0129] In this embodiment, interpolation can be used for resolution enhancement. Interpolation involves filling the gap between two adjacent pixels with a new pixel based on their pixel values. The formula for interpolation using quadratic linear interpolation can be found in formula (III):

[0130]

[0131] Among them, Q 11 Q 21 Q 12 and Q 22 It consists of four adjacent pixels arranged in a square pattern, Q. 11 Located in the bottom left corner of the square, Q 21 and Q 11 Adjacent and located at the bottom right corner of the square, Q 12 and Q 11 Adjacent to and located at the top left corner of the square, Q 22 and Q 12 and Q 21 Adjacent to and located at the top right corner of the square. (x1, y1) is Q. 11 The coordinates (x2, y1) are Q. 21 The coordinates (x1, y2) are Q. 12 The coordinates (x2, y2) are Q. 22 The coordinates of the coordinates. For ease of calculation, x1 = y1 = 0, x2 = y2 = 1 are generally taken. f(Q 11 f(Q) 21 f(Q) 12 ) and f(Q 22 ) respectively refer to Q 11 Q 21 Q 12 and Q 22 The pixel values ​​are f(x,y1) and f(x,y2), which are the two intermediate pixels obtained by the first linear interpolation, and f(x,y) is the final target pixel obtained by the second linear interpolation.

[0132] The image retrieval method provided in this application improves the resolution of the target region through interpolation. Since the interpolation algorithm is relatively simple, it can improve the efficiency of improving the resolution of the target region, thereby improving the efficiency of returning the target image to the promotional personnel.

[0133] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0134] Based on the same inventive concept, this application also provides an image retrieval device for implementing the image retrieval method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more image retrieval device embodiments provided below can be found in the limitations of the image retrieval method described above, and will not be repeated here.

[0135] In one embodiment, such as Figure 7 As shown, an image retrieval device 700 is provided, including an acquisition module 702, a determination module 704, and a processing module 706, wherein:

[0136] The acquisition module 702 is used to acquire the search text and match it to obtain at least one target image category corresponding to the search text and the target original image corresponding to the search text;

[0137] The determination module 704 is used to determine the target regions in the original target images that correspond to the classification of each target image;

[0138] The processing module 706 is used to perform resolution enhancement processing on the target region in any of the original target images to obtain the target image corresponding to the searched text.

[0139] The image retrieval device provided in this application matches the target image category and the original target image with the search text, and then performs resolution enhancement processing on the region in the original target image that matches the target image category to obtain the target image corresponding to the search text. This application embodiment enables publicity personnel to match the required original target image by inputting text, improving the efficiency of retrieving the required image; at the same time, the resolution enhancement processing on the region in the target image that matches the target image category can also specifically improve the clarity of the region in the target image that the publicity personnel need.

[0140] In one embodiment, the acquisition module 702 is further configured to:

[0141] The search text is processed by vector extraction to obtain the text vector corresponding to the search text;

[0142] Based on the text vector corresponding to the searched text and the text vector corresponding to each image category, determine the target image category corresponding to the searched text in each of the image categories;

[0143] Based on the text vector corresponding to the searched text and the feature vectors corresponding to each original image, the target original image corresponding to the searched text in each of the original images is determined.

[0144] In one embodiment, the feature vector corresponding to the original image is obtained based on the tag text vector corresponding to the image tag of the original image and the image vector of the original image.

[0145] In one embodiment, the acquisition module 702 is further configured to:

[0146] Determine the similarity between the text vector corresponding to the retrieved text and the feature vector corresponding to each of the original images;

[0147] The original image whose similarity meets the preset conditions is used as the target original image corresponding to the searched text.

[0148] In one embodiment, the determining module 704 is further configured to:

[0149] The original target image is convolved to obtain an image feature map of the original target image at the convolutional layer, and the image feature map includes at least one channel.

[0150] For any of the target image categories and any of the channels, determine the classification weight of the channel for the target image category;

[0151] For any of the target image classifications, the channels are superimposed according to each channel and the classification weight corresponding to each channel to obtain a target feature map, and the target region corresponding to the original image and the target image classification is determined from the target feature map.

[0152] In one embodiment, the processing module 706 is further configured to:

[0153] Determine the matching degree between the searched text and each of the target image categories, and based on the matching degree, determine the resolution enhancement area of ​​the target region corresponding to each of the target image categories;

[0154] For any of the target regions, a resolution-enhancing region is determined from the target regions based on the resolution-enhancing area;

[0155] The resolution enhancement regions are processed to obtain the target image corresponding to the searched text.

[0156] In one embodiment, the processing module 706 is further configured to:

[0157] Based on the matching degree of each target image classification, the area ratio of the region corresponding to each target image classification is determined, and the area ratio of the region is positively correlated with the matching degree;

[0158] For any of the target image classifications, the resolution enhancement area of ​​the target region corresponding to the target image classification is determined based on the ratio of the base resolution enhancement area to the area of ​​the region corresponding to the target image classification.

[0159] In one embodiment, the processing module 706 is further configured to:

[0160] Interpolation processing is performed on each of the target regions to obtain the target image corresponding to the searched text.

[0161] Each module in the aforementioned image retrieval device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0162] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image retrieval method.

[0163] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0164] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0165] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0166] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0167] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0168] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image retrieval method, characterized in that, The method includes: Obtain the search text and match it to obtain at least one target image category corresponding to the search text, as well as the original target image corresponding to the search text; In the original target image, the target region corresponding to each target image category is determined respectively; For any of the original target images, the target region in the original target image is subjected to resolution enhancement processing to obtain the target image corresponding to the searched text; The resolution enhancement process performed on each of the target regions to obtain the target image corresponding to the searched text includes: Determine the matching degree between the searched text and each of the target image categories, and based on the matching degree, determine the resolution enhancement area of ​​the target region corresponding to each of the target image categories; For any of the target regions, a resolution-enhancing region is determined from the target regions based on the resolution-enhancing area; The resolution enhancement regions are subjected to resolution enhancement processing to obtain the target image corresponding to the searched text. The method further includes: when the outline of the target region does not match the outline of the object in the target region, using an algorithm to detect the region in the target region where the object exists, and using it as the resolution enhancement region.

2. The method according to claim 1, characterized in that, The matching process obtains at least one target image category corresponding to the searched text, and the original target image corresponding to the searched text, including: The search text is processed by vector extraction to obtain the text vector corresponding to the search text; Based on the text vector corresponding to the searched text and the text vector corresponding to each image category, determine the target image category corresponding to the searched text in each of the image categories; Based on the text vector corresponding to the searched text and the feature vectors corresponding to each original image, the target original image corresponding to the searched text in each of the original images is determined.

3. The method according to claim 2, characterized in that, The feature vector corresponding to the original image is obtained based on the tag text vector corresponding to the image tag of the original image and the image vector of the original image.

4. The method according to claim 2 or 3, characterized in that, The step of determining the target original image corresponding to the searched text in each of the original images based on the text vector corresponding to the searched text and the feature vector corresponding to each original image includes: Determine the similarity between the text vector corresponding to the retrieved text and the feature vector corresponding to each of the original images; The original image whose similarity meets the preset conditions is used as the target original image corresponding to the searched text.

5. The method according to claim 1, characterized in that, The step of determining the target region in the original target image corresponding to each target image category includes: The original target image is convolved to obtain an image feature map of the original target image at the convolutional layer, and the image feature map includes at least one channel. For any of the target image categories and any of the channels, determine the classification weight of the channel for the target image category; For any of the target image classifications, the channels are superimposed according to each channel and the classification weight corresponding to each channel to obtain a target feature map, and the target region corresponding to the original image and the target image classification is determined from the target feature map.

6. The method according to claim 1, characterized in that, The step of determining the resolution enhancement area of ​​the target region corresponding to each target image classification based on the matching degree includes: Based on the matching degree of each target image classification, the area ratio of the region corresponding to each target image classification is determined, and the area ratio of the region is positively correlated with the matching degree; For any of the target image classifications, the resolution enhancement area of ​​the target region corresponding to the target image classification is determined based on the ratio of the base resolution enhancement area to the area of ​​the region corresponding to the target image classification.

7. The method according to claim 1, characterized in that, The step of performing resolution upscaling on each of the target regions to obtain the target image corresponding to the searched text includes: Interpolation processing is performed on each of the target regions to obtain the target image corresponding to the searched text.

8. An image retrieval device, characterized in that, The device includes: The acquisition module is used to acquire the search text and match it to obtain at least one target image category corresponding to the search text, as well as the original target image corresponding to the search text; The determination module is used to determine the target regions in the original target images that correspond to the classification of each target image; The processing module is used to perform resolution enhancement processing on the target region in any of the original target images to obtain the target image corresponding to the searched text. The processing module is further configured to: determine the matching degree between the searched text and each of the target image categories, and based on the matching degree, determine the resolution enhancement area of ​​the target region corresponding to each of the target image categories; for any target region, determine a resolution enhancement region from the target region according to the resolution enhancement area; perform resolution enhancement processing on each of the resolution enhancement regions to obtain the target image corresponding to the searched text; The device is also used to detect, using an algorithm, areas in the target region where objects exist when the outline of the target region does not match the outline of objects in the target region, and to designate these areas as the resolution-enhancing regions.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images