Image searching method and device, equipment, storage medium and product

Abstract

Embodiments of the present application provide a kind of image search method, device, equipment, storage medium and product with graph. Including: the feature extraction and feature dimension reduction processing of the image to be searched are carried out, obtain the first low-dimensional feature vector of the image to be searched;Any preprocessed library image and descriptive text are input into open vocabulary target detection model to output confidence;If confidence is greater than target detection confidence threshold, then the target area of any preprocessed library image is cropped, obtain multiple target small graphs and its positioning information;The feature extraction and feature dimension reduction processing of each target small graph are carried out, obtain the second low-dimensional feature vector corresponding to each target small graph;First low-dimensional feature vector and second low-dimensional feature vector are compared in similarity, calculate similarity ratio result and determine multiple target small graphs meeting preset condition;Multiple target small graphs and corresponding image database big graph are output. The method is used to achieve the effect of fine search to the image to be searched.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium and product for image search. Background Technology

[0002] With the rapid development of internet technology and various camera devices, users' requirements for image search functions are gradually increasing. How to quickly and accurately retrieve images similar to the query image from a massive image database has become a research hotspot.

[0003] In existing technologies, deep learning-based object detection and feature extraction methods have provided a breakthrough tool for improving image search technology, but these methods have limitations such as limited detection categories and inability to perform fine-grained searches. Summary of the Invention

[0004] This application provides image search methods, apparatus, devices, storage media, and products to achieve the effect of fine-grained retrieval of images to be searched.

[0005] In a first aspect, embodiments of this application provide an image search method applied to a computer device, comprising: acquiring an image to be searched and descriptive text of the image to be searched, and preprocessing the image to be searched; performing feature extraction and feature dimensionality reduction on the preprocessed image to be searched to obtain a low-dimensional feature vector of the image to be searched; acquiring multiple large images from an image database, and preprocessing the multiple large images from the image database to obtain a preprocessed database image; for any preprocessed database image, inputting any preprocessed database image and descriptive text into a pre-trained open vocabulary object detection network model to output target region location information and confidence level; if the confidence level is greater than the target detection confidence threshold, then based on the target region location information... The target region is cropped to obtain multiple target thumbnails corresponding to the descriptive text and the location information of each target thumbnail. Feature extraction and dimensionality reduction are performed on each target thumbnail to obtain a low-dimensional feature vector corresponding to each target thumbnail. The low-dimensional feature vector of the image to be retrieved is compared with the low-dimensional feature vector of each target thumbnail in the retrieval information repository to calculate the similarity comparison result. The retrieval information repository is used to store the target region location information, target thumbnail location information, and low-dimensional feature vectors corresponding to each target thumbnail in each preprocessed image. Based on the similarity comparison result, multiple target thumbnails that meet the preset conditions are determined. The multiple target thumbnails and the corresponding large image database are output.

[0006] In one possible implementation, feature extraction and dimensionality reduction are performed on the preprocessed image to be retrieved to obtain a low-dimensional feature vector of the image to be retrieved. This includes: inputting the preprocessed image to be retrieved into a pre-trained open vocabulary object detection network model, so that the detector in the open vocabulary object detection network model can extract features from the preprocessed image to obtain 1024-dimensional fused features; and using a preset dimensionality reduction method to reduce the 1024-dimensional fused features to 32 dimensions to obtain a low-dimensional feature vector of the image to be retrieved.

[0007] In one possible implementation, cropping the target region includes: expanding outward by a preset percentage of pixels based on the upper boundary of the target region to determine the target upper bound of the target small image corresponding to the expanded target region; if the target upper bound exceeds the upper boundary of any preprocessed library image, then the target upper bound of the target small image corresponding to the expanded target region is determined as the upper boundary of any preprocessed library image; expanding outward by a preset percentage of pixels based on the lower boundary of the target region to determine the target lower bound of the target small image corresponding to the expanded target region; if the target lower bound exceeds the lower boundary of any preprocessed library image, then the target lower bound of the target small image corresponding to the expanded target region is determined as the lower boundary of any preprocessed library image; based on the target... The left boundary of the target region is expanded outward by a preset percentage of pixels to determine the target left bounding box of the target thumbnail corresponding to the expanded target region. If the target left bounding box exceeds the left boundary of any preprocessed library image, then the target left bounding box of the target thumbnail corresponding to the expanded target region is determined as the left boundary of any preprocessed library image. Based on the right boundary of the target region, the right bounding box of the target thumbnail is expanded outward by a preset percentage of pixels to determine the target right bounding box of the target thumbnail corresponding to the expanded target region. If the target right bounding box exceeds the right boundary of any preprocessed library image, then the target right bounding box of the target thumbnail corresponding to the expanded target region is determined as the right boundary of any preprocessed library image. The target region is then cropped based on the target top bounding box, target bottom bounding box, target left bounding box, and target right bounding box.

[0008] In one possible implementation, multiple target small images and corresponding large image database images are output, including: if the number of target small images is 0, a prompt indicating no results were found is output; if the number of target small images is not greater than a preset number, the target small images and their corresponding large image database images are displayed sequentially from smallest to largest according to the similarity comparison results, wherein the target small images are highlighted in the large image database images based on their location information; if the number of target small images is greater than a preset number, the first preset number of target small images and their corresponding large image database images are displayed sequentially from smallest to largest according to the similarity comparison results, wherein the target small images are highlighted in the large image database images based on their location information.

[0009] In one possible implementation, the low-dimensional feature vector of the image to be retrieved is compared with the low-dimensional feature vectors of each target small image in the retrieval information repository to calculate the similarity comparison result. This includes: calculating the cosine distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; calculating the Euclidean distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; and adding half of the cosine distance and half of the Euclidean distance to obtain the similarity comparison result between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository.

[0010] In one possible implementation, the formula for calculating the cosine distance is:

[0011]

[0012] In the formula, d1 represents the cosine distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; x 1k x represents the low-dimensional feature vector of the image to be retrieved; 2k This represents the low-dimensional feature vector of each target small image in the retrieval information database; n takes the value of 32.

[0013] In one possible implementation, the formula for calculating Euclidean distance is:

[0014]

[0015] In the formula, d2 represents the Euclidean distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; x 1k x represents the low-dimensional feature vector of the image to be retrieved; 2k This represents the low-dimensional feature vector of each target small image in the retrieval information database; n takes the value of 32.

[0016] Secondly, embodiments of this application provide an image search device applied to a computer device, comprising:

[0017] The image acquisition module is used to acquire the image to be retrieved and its descriptive text, and to preprocess the image to be retrieved.

[0018] The first feature extraction and feature dimensionality reduction module performs feature extraction and feature dimensionality reduction on the preprocessed image to be retrieved, and obtains the low-dimensional feature vector of the image to be retrieved.

[0019] The database large image acquisition module is used to acquire multiple large images from the image database and preprocess them to obtain preprocessed database images.

[0020] The object detection module is used to take any preprocessed image from the library and descriptive text as input to a pre-trained open vocabulary object detection network model, and output the location information and confidence score of the target region.

[0021] The cropping module is used to crop the target region based on the target region location information if the confidence level is greater than the target detection confidence threshold, so as to obtain multiple target mini-images corresponding to the descriptive text and the location information of each target mini-image.

[0022] The second feature extraction and feature dimensionality reduction module is used to extract features and reduce the dimensionality of each target small image to obtain the low-dimensional feature vector corresponding to each target small image.

[0023] The similarity matching module is used to compare the low-dimensional feature vector of the image to be retrieved with the low-dimensional feature vector of each target small image in the retrieval information repository, and calculate the similarity comparison result. The retrieval information repository is used to store the target region location information, target small image positioning information, and low-dimensional feature vectors corresponding to each target small image in each preprocessed library image.

[0024] The target image identification module determines multiple target images that meet preset conditions based on similarity comparison results.

[0025] The output module is used to output multiple target thumbnails and the corresponding large image database image.

[0026] Thirdly, embodiments of this application provide a computer device, including: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0027] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0028] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0029] The image search method, apparatus, device, storage medium, and product provided in this application's embodiments input descriptive text of the image to be searched and multiple large images from an image database into a pre-trained open vocabulary target network detection model. The model outputs target region location information with a confidence level greater than a target detection confidence threshold. Based on this location information, the target region is cropped to obtain multiple small target images corresponding to the descriptive text. These small target images and their corresponding large image database images provide more refined retrieval target data for the image to be searched, achieving real-time open target detection. Furthermore, the LargeVis dimensionality reduction module reduces the high-dimensional feature vector to 32 dimensions, reducing complexity and storage costs. When cropping the target image, adaptive boundary expansion helps preserve background information around the target, thereby improving the robustness of feature extraction. When outputting the large image database image, using highlighted bounding boxes to select the position of the small target image within the large image database image further improves retrieval efficiency. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0031] Figure 1 A schematic diagram illustrating a scenario for the image search method provided in this application embodiment;

[0032] Figure 2 A schematic flowchart illustrating the image search method provided in this application embodiment;

[0033] Figure 3 This is a schematic diagram of the image search device provided in the embodiments of this application;

[0034] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application.

[0035] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0037] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0038] To clearly understand the technical solution of this application, the existing technical solutions are first described in detail. Image search technology occupies an important position in the field of modern information processing and is widely used in e-commerce, social media, security monitoring, medical imaging, and many other fields. Currently, with the rapid development of Internet technology and various camera devices, the amount of image data in social media, shopping platforms, or security systems is growing exponentially, and users' requirements for image search functions are gradually increasing. How to quickly and accurately retrieve images similar to the query image from massive image databases has become a research hotspot. With the rapid development of computer vision technology, image content-based retrieval has been widely used in many fields, such as security monitoring, face recognition, and product retrieval. Traditional image retrieval methods mostly rely on manually extracted features; however, these methods suffer from high computational complexity and low accuracy when processing large-scale image databases. With the rapid development of deep learning, deep learning-based object detection and feature extraction methods have provided breakthrough tools for improving image search technology, but these methods have limitations such as limited detection categories and inability to perform fine-grained retrieval.

[0039] To address the aforementioned technical issues, the inventors devised a method to achieve real-time open target detection of large images in an image database using a pre-trained open vocabulary target detection network model, thereby obtaining more refined target mini-images. Secondly, by performing feature extraction and dimensionality reduction on each target mini-image and the image to be retrieved, image features with lower dimensionality and stronger expressive power can be efficiently obtained, reducing subsequent computing power requirements while improving the processing efficiency of the algorithm. Finally, through similarity comparison, efficient measurement and matching between the features of the target mini-image and the features of the image to be retrieved can be achieved.

[0040] Based on the above-mentioned inventive discovery, the inventor has proposed the technical solution of this application.

[0041] Figure 1 This is a schematic diagram illustrating a scenario of the image search method provided in the embodiments of this application, such as... Figure 1 As shown, the specific application scenarios of this application include: receiving device 101, processor 102 and display device 103.

[0042] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the image search method. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0043] In the specific implementation process, the receiving device 101 can be an input / output interface or a communication interface, used to acquire the image to be retrieved, its descriptive text, and multiple large images from the image database.

[0044] The processor 102 can preprocess the image to be retrieved, perform feature extraction and dimensionality reduction on the preprocessed image to obtain a low-dimensional feature vector; preprocess large images from multiple image databases to obtain preprocessed library images; for any preprocessed library image, input any preprocessed library image and descriptive text into a pre-trained open vocabulary object detection network model to output target region location information and confidence score; if the confidence score is greater than the target detection confidence threshold, the target region is cropped according to the target region location information to obtain the descriptive text. This involves identifying multiple target images and their location information; performing feature extraction and dimensionality reduction on each target image to obtain a low-dimensional feature vector; comparing the low-dimensional feature vector of the image to be retrieved with the low-dimensional feature vectors of each target image in the retrieval information repository to calculate the similarity comparison result; the retrieval information repository stores the target region location information, target image location information, and low-dimensional feature vectors corresponding to each target image in each preprocessed image; and determining multiple target images that meet preset conditions based on the similarity comparison result.

[0045] Display device 103 can be used to display multiple target thumbnails and corresponding large image database images. The display device can also be a touch screen, used to receive user commands while displaying the above content, to achieve user interaction.

[0046] It should be understood that the aforementioned processor can be implemented by reading instructions from memory and executing those instructions, or it can be implemented through chip circuitry.

[0047] Figure 2 This is a flowchart illustrating the image search method provided in the embodiments of this application, such as... Figure 2 As shown, the method includes:

[0048] S201: Obtain the image to be retrieved and its descriptive text, and preprocess the image to be retrieved.

[0049] In this context, the descriptive text of the image to be retrieved refers to the target category to be detected, which can be the category to which the target object belongs in various daily life or work scenarios. The descriptive text can include one or more categories.

[0050] Specifically, an image can be selected from the network or local storage as the search image, or an image from a preset image library can be selected as the search image. The search image should ideally contain only the target image. Preprocessing operations include image denoising and image contrast enhancement.

[0051] S202: Perform feature extraction and feature dimensionality reduction on the preprocessed image to be retrieved to obtain a low-dimensional feature vector of the image to be retrieved.

[0052] Specifically, it includes steps Sa1 to Sa2:

[0053] Sa1: Input the preprocessed image to be retrieved into the pre-trained open vocabulary object detection network model so that the detector in the open vocabulary object detection network model can extract features from the preprocessed image to obtain 1024-dimensional fused features.

[0054] The open vocabulary object detection network model is the YOLO-World network model.

[0055] In this context, the detector in the open vocabulary object detection network model refers to the YOLO detector. The YOLO detector is based on YOLOv8 and uses Darknet as its image encoder. Darknet is a deep convolutional neural network used to extract multi-scale features from images. Furthermore, through a path aggregation network, the YOLO detector constructs a feature pyramid, fusing feature maps from different layers or scales along the channel dimension to achieve the detection of objects of different sizes.

[0056] Specifically, the preprocessed image to be retrieved is input into YOLO-World, and after feature extraction by the YOLO detector in YOLO-World, 1024-dimensional fused features are obtained.

[0057] Sa2: The 1024-dimensional fused features are reduced to 32 dimensions using a preset dimensionality reduction method to obtain the low-dimensional feature vector of the image to be retrieved.

[0058] The preset dimensionality reduction method is LargeVis dimensionality reduction. LargeVis dimensionality reduction is an improved visualization algorithm for large-scale high-dimensional data, based on the Line (Large-scale Information Network Embedding) visualization algorithm and the t-SNE (t-Distributed Stochastic Neighbor Embedding) visualization algorithm. Its basic idea is to preserve the original structure between high-dimensional data points as much as possible in the low-dimensional space. LargeVis first constructs a KNN (K-Nearest Neighbors) graph, and then constructs an objective function for optimization to obtain a low-dimensional representation of the data. Specifically, LargeVis first uses a combination of spatial segmentation tree algorithm and neighbor search algorithm to construct an accurate KNN graph; at the same time, it uses negative sampling technology and edge sampling technology in edge algorithms to optimize the objective function; finally, it is trained based on asynchronous stochastic gradient descent to obtain the low-dimensional data after dimensionality reduction using the LargeVis dimensionality reduction method.

[0059] Specifically, the dimension of the low-dimensional feature vector obtained after LargeVis processing is pre-defined, with a dimension value of 32. The LargeVis dimensionality reduction method is used to reduce the dimension of the fused features to 32, resulting in the low-dimensional feature vector of the image to be retrieved.

[0060] S203: Obtain large images from multiple image databases and preprocess them to obtain preprocessed images.

[0061] The large image in the image database refers to an image containing one or more targets.

[0062] Specifically, preprocessing is performed on large images from multiple image databases, such as image denoising and image contrast enhancement, to obtain images from each preprocessed database.

[0063] S204: For any preprocessed image from the library, input the image and descriptive text into a pre-trained open vocabulary object detection network model to output the target region location information and confidence level.

[0064] Specifically, for any preprocessed library image, the image and the descriptive text of the image to be detected are input into the YOLO-World network model to simultaneously identify and locate all categories of target objects specified in the descriptive text. The YOLO-World network model outputs the location information and confidence score of the target region that matches any preprocessed library image and descriptive text, where the confidence score represents the degree of matching between the detected target region and the descriptive text.

[0065] S205: If the confidence level is greater than the target detection confidence threshold, the target region is cropped according to the target region location information to obtain multiple target mini-images corresponding to the descriptive text and the location information of each target mini-image.

[0066] Specifically, a target detection confidence threshold, such as 70% or 85%, is first preset. If the detection confidence of any preprocessed image in the library and the descriptive text is greater than the target detection confidence threshold, then that preprocessed image is deemed valid and retained. Image extraction is then performed on the target objects identified in any preprocessed image. Based on the target region output by the YOLO-World network model, the target region location information of any preprocessed image is obtained. The target region is then cropped based on this location information to obtain multiple corresponding target mini-images and their location information.

[0067] The location information of the target thumbnail refers to the position coordinates of the region corresponding to the target thumbnail in any preprocessed library image, and also the position coordinates of the region corresponding to the target thumbnail in the large image of the image database.

[0068] The target region location information is used to confirm the location of the target region in any preprocessed library image, for example, the top left corner (x). lt Coordinates, top left y lt Coordinates, bottom right x rb Coordinates and bottom right y rb A combination of coordinates or the coordinates of the top left corner of the target box (x lt ,y lt The combination of the target bounding box width and height (w, h), etc.

[0069] Specifically, cropping the target area includes steps Sb1 to Sb9:

[0070] Sb1: Based on the upper boundary of the target area, expand outward by a preset percentage of pixels to determine the target upper bound of the target thumbnail corresponding to the expanded target area.

[0071] Sb2: If the target bounding box extends beyond the upper boundary of any preprocessed library image, then the target bounding box of the target thumbnail corresponding to the expanded target region is determined as the upper boundary of any preprocessed library image.

[0072] Sb3: Based on the lower boundary of the target region, expand outward by a preset percentage of pixels to determine the target lower bound of the target thumbnail corresponding to the expanded target region.

[0073] Sb4: If the target lower bound exceeds the lower boundary of any preprocessed library image, then the target lower bound of the target thumbnail corresponding to the expanded target region is determined as the lower boundary of any preprocessed library image.

[0074] Sb5: Based on the left boundary of the target region, expand outward by a preset percentage of pixels to determine the target left bounding box of the target thumbnail corresponding to the expanded target region.

[0075] Sb6: If the left bounding box of the target exceeds the left boundary of any preprocessed library image, then the left bounding box of the target thumbnail corresponding to the expanded target region is determined as the left boundary of any preprocessed library image.

[0076] Sb7: Based on the right boundary of the target region, expand outward by a preset percentage of pixels to determine the target right bounding box of the target thumbnail corresponding to the expanded target region.

[0077] Sb8: If the right bounding box of the target exceeds the right boundary of any preprocessed library image, then the right bounding box of the target thumbnail corresponding to the expanded target region is determined as the right boundary of any preprocessed library image.

[0078] Sb9: Crops the target area based on the target's top, bottom, left, and right bounding boxes.

[0079] Specifically, target areas of different sizes are enlarged using an adaptive adjustment ratio, increasing the border length of the target area to several times its original size. The adaptive adjustment ratio is preset, such as 5% or 10%, correspondingly increasing the border length of the target area to 105% or 110% of its original size.

[0080] S206: Perform feature extraction and feature dimensionality reduction on each target mini-image to obtain the low-dimensional feature vector corresponding to each target mini-image.

[0081] Specifically, each target image is input into the YOLO-World network model. After feature extraction by the YOLO detector in YOLO-World, a 1024-dimensional fused feature is obtained. The LargeVis dimensionality reduction method is used to reduce the dimension of the fused feature to 32 dimensions, resulting in a low-dimensional feature vector corresponding to each target image.

[0082] S207: Compare the low-dimensional feature vector of the image to be retrieved with the low-dimensional feature vector of each target small image in the retrieval information repository, and calculate the similarity comparison result; wherein the retrieval information repository is used to store the target region location information, target small image positioning information and low-dimensional feature vector corresponding to each target small image of each preprocessed library image.

[0083] Specifically, the low-dimensional feature vector of the image to be retrieved is compared with the low-dimensional feature vectors of each target small image in the retrieval information repository in turn. The smaller the similarity comparison result, the more similar the two feature vectors are. The low-dimensional feature vector is 32-dimensional.

[0084] Specifically, the steps for calculating the similarity comparison results include Sc1~Sc3:

[0085] Sc1: Calculate the cosine distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository.

[0086] Specifically, the formula for calculating the cosine distance is:

[0087]

[0088] In the formula, d1 represents the cosine distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; x 1k x represents the low-dimensional feature vector of the image to be retrieved; 2k This represents the low-dimensional feature vector of each target small image in the retrieval information database; n takes the value of 32.

[0089] Sc2: Calculate the Euclidean distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository.

[0090] Specifically, the formula for calculating Euclidean distance is:

[0091]

[0092] In the formula, d2 represents the Euclidean distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; x 1k x represents the low-dimensional feature vector of the image to be retrieved; 2k This represents the low-dimensional feature vector of each target small image in the retrieval information database; n takes the value of 32.

[0093] Sc3: Add half the cosine distance value and half the Euclidean distance value to obtain the similarity comparison result between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository.

[0094] Specifically, the formula for calculating the similarity comparison result is as follows:

[0095]

[0096] In the formula, d1 represents the cosine distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository; d2 represents the Euclidean distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository.

[0097] S208: Based on the similarity comparison results, determine multiple target small images that meet the preset conditions.

[0098] Specifically, a similarity threshold z is set as the criterion for judging whether features are similar. When the similarity comparison result between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository is greater than the similarity threshold z, it is determined that there is no target image feature vector in the retrieval information repository that is similar to the image to be retrieved. If there are low-dimensional feature vectors in the retrieval information repository with similarity comparison results less than the threshold z, then all feature vectors in the retrieval information repository with similarity comparison results less than the threshold z are selected as similar image feature vectors, and the corresponding target small image in the retrieval information repository is returned.

[0099] S209: Output multiple target thumbnails and the corresponding large image database of the target thumbnails.

[0100] Specifically, the output strategy includes steps Sd1~Sd3:

[0101] Sd1: If the number of target thumbnails is 0, output a message indicating that no results were found.

[0102] For example, if the number of similar target images is 0, it will display that no relevant images were found.

[0103] Sd2: If the number of target small images is not greater than the preset number, then the target small images and the corresponding large images in the image database will be displayed in order of increasing similarity comparison results. In the large images in the image database, the target small images are highlighted and selected based on their location information.

[0104] For example, if the number of similar images is no more than 5, all result images are displayed in ascending order of similarity comparison results. Each result image includes the target small image and the corresponding large image in the image database. The large image database image can also highlight and select the corresponding target cell location area based on the location information of the target small image.

[0105] Sd3: If the number of large images in the image database corresponding to the target small image is greater than the preset number, then the first preset number of target small images and the large images in the image database corresponding to the target small images will be displayed in order of increasing similarity comparison results. In the large images in the image database, the target small image will be highlighted and selected based on the positioning information of the target small image.

[0106] For example, if the number of similar images is greater than 5, the first 5 result images are displayed in ascending order of similarity comparison results. Each result image includes the target small image and the corresponding large image in the image database. The large image database image can also highlight and select the corresponding target cell location area based on the location information of the target small image.

[0107] In summary, by inputting the descriptive text of the image to be retrieved and multiple large images from the image database into a pre-trained open-vocabulary target network detection model, the model outputs target region location information with a confidence score greater than the target detection confidence threshold. Based on this location information, the target region is cropped to obtain multiple small target images corresponding to the descriptive text. These small target images and their corresponding large image database images provide more refined target data for the image to be retrieved, enabling real-time open target detection. Furthermore, the LargeVis dimensionality reduction module reduces the high-dimensional feature vector to 32 dimensions, reducing complexity and storage costs. When cropping the target image, adaptive boundary expansion helps preserve background information around the target, thereby improving the robustness of feature extraction. Finally, when outputting the large image database image, using highlighted bounding boxes to select the position of the small target image within the large image database image further improves retrieval efficiency.

[0108] In another embodiment provided in this application, the retrieval information repository is used to store the location information of the large image in the image database, the target region location information of each preprocessed image, the location information of the target small image, and the low-dimensional feature vector corresponding to each target small image. The location information of the large image in the image database refers to the storage location information of the original image in the image database; the target region location information of each preprocessed image refers to the storage location information of each preprocessed image in the retrieval information repository; the location information of the target small image is the position coordinate information of the target small image in the large image database; and the low-dimensional feature vector is the feature vector with low-dimensional feature values ​​obtained after feature extraction and feature dimensionality reduction of the target small image. The retrieval information repository can be created in the form of a CSV table or an XML table, and its creation process is as follows:

[0109] S301: Enter information sequentially using the table column names as “Location information of large images in the image database”, “Location information of target regions of images in each preprocessed library”, “Location information of target small images”, and “Low-dimensional feature vectors corresponding to each target small image”.

[0110] S302: If there is only one target small image in the current image database, then the corresponding information is written into the corresponding column of the same row in the order of S301.

[0111] S303: If there are multiple target small images in the current large image database, the positioning information of each target small image and the low-dimensional feature vector corresponding to each target small image are matched, and the corresponding information is written into the corresponding column of the same row in the order of step S301. The information in the position information of the large image database is the position information of the large image database.

[0112] In summary, storing the location information of the target small image corresponding to the large image in the image database, along with the low-dimensional feature vector of the target small image, together in the retrieval information repository can improve the processing efficiency of similarity comparison between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository.

[0113] Figure 3 This is a schematic diagram of the image search device provided in an embodiment of this application. Figure 3 As shown, the device includes: an image acquisition module 301, a first feature extraction and feature dimensionality reduction module 302, a database large image acquisition module 303, a target detection module 304, a cropping module 305, a second feature extraction and feature dimensionality reduction module 306, a similarity matching module 307, a target small image determination module 308, and an output module 309.

[0114] The image acquisition module 301 is used to acquire the image to be retrieved and its descriptive text, and to preprocess the image to be retrieved.

[0115] The first feature extraction and feature dimensionality reduction module 302 performs feature extraction and feature dimensionality reduction on the preprocessed image to be retrieved to obtain a low-dimensional feature vector of the image to be retrieved.

[0116] The database large image acquisition module 303 is used to acquire multiple large images from the image database and preprocess them to obtain preprocessed database images.

[0117] The object detection module 304 is used to input any preprocessed library image and descriptive text into a pre-trained open vocabulary object detection network model for any preprocessed library image, so as to output the target region location information and confidence score.

[0118] The cropping module 305 is used to crop the target region according to the target region location information if the confidence level is greater than the target detection confidence level threshold, so as to obtain multiple target small images corresponding to the descriptive text and the location information of each target small image.

[0119] The second feature extraction and feature dimensionality reduction module 306 is used to perform feature extraction and feature dimensionality reduction on each target small image to obtain the low-dimensional feature vector corresponding to each target small image.

[0120] The similarity matching module 307 is used to compare the low-dimensional feature vector of the image to be retrieved with the low-dimensional feature vector of each target small image in the retrieval information repository, and calculate the similarity comparison result; wherein the retrieval information repository is used to store the target region location information, the target small image positioning information, and the low-dimensional feature vector corresponding to each target small image of each preprocessed library image.

[0121] The target small image determination module 308 determines multiple target small images that meet preset conditions based on the similarity comparison results.

[0122] Output module 309 is used to output multiple target small images and the corresponding large image database of the multiple target small images.

[0123] In one possible implementation, the first feature extraction and feature dimensionality reduction module 302 is specifically used to input the preprocessed image to be retrieved into a pre-trained open vocabulary object detection network model, so that the detector in the open vocabulary object detection network model can extract features from the preprocessed image to be retrieved to obtain 1024-dimensional fused features; and to use a preset dimensionality reduction method to reduce the 1024-dimensional fused features to 32 dimensions to obtain a low-dimensional feature vector of the image to be retrieved.

[0124] In one possible implementation, the cropping module 305 is specifically used to expand outward by a preset percentage of pixels based on the upper boundary of the target region to determine the target upper bound of the target small image corresponding to the expanded target region; if the target upper bound exceeds the upper boundary of any preprocessed library image, then the target upper bound of the target small image corresponding to the expanded target region is determined as the upper boundary of any preprocessed library image; based on the lower boundary of the target region, it expands outward by a preset percentage of pixels to determine the target lower bound of the target small image corresponding to the expanded target region; if the target lower bound exceeds the lower boundary of any preprocessed library image, then the target lower bound of the target small image corresponding to the expanded target region is determined as the lower boundary of any preprocessed library image; based on the target region... The left boundary of the target region is expanded outward by a preset percentage of pixels to determine the target left bounding box of the target thumbnail corresponding to the expanded target region. If the target left bounding box exceeds the left boundary of any preprocessed library image, then the target left bounding box of the target thumbnail corresponding to the expanded target region is determined as the left boundary of any preprocessed library image. Based on the right boundary of the target region, the right bounding box of the target thumbnail is expanded outward by a preset percentage of pixels to determine the target right bounding box of the target thumbnail corresponding to the expanded target region. If the target right bounding box exceeds the right boundary of any preprocessed library image, then the target right bounding box of the target thumbnail corresponding to the expanded target region is determined as the right boundary of any preprocessed library image. The target region is then cropped based on the target top bounding box, target bottom bounding box, target left bounding box, and target right bounding box.

[0125] In one possible implementation, the output module 309 is specifically used to output a prompt that no results were found if the number of target small images is 0; if the number of target small images is not greater than a preset number, then the target small images and the corresponding large images in the image database are displayed in ascending order of similarity comparison results, wherein the target small images are highlighted in the large images in the image database based on their location information; if the number of target small images is greater than the preset number, then the first preset number of target small images and the corresponding large images in the image database are displayed in ascending order of similarity comparison results, wherein the target small images are highlighted in the large images in the image database based on their location information.

[0126] In one possible implementation, the similarity matching module 307 is specifically used to calculate the cosine distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository; calculate the Euclidean distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository; and add half of the cosine distance and half of the Euclidean distance to obtain the similarity comparison result between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository.

[0127] In one possible implementation, the formula for calculating the cosine distance in the similarity matching module 307 is:

[0128]

[0129] In the formula, d1 represents the cosine distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; x 1k x represents the low-dimensional feature vector of the image to be retrieved; 2k This represents the low-dimensional feature vector of each target small image in the retrieval information database; n takes the value of 32.

[0130] In one possible implementation, the formula for calculating the Euclidean distance in the similarity matching module 307 is:

[0131]

[0132] In the formula, d2 represents the Euclidean distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; x 1k x represents the low-dimensional feature vector of the image to be retrieved; 2k This represents the low-dimensional feature vector of each target small image in the retrieval information database; n takes the value of 32.

[0133] The image search device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0134] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application. For example... Figure 4 As shown, the computer device provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0135] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0136] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0137] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0138] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0139] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0140] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0141] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0142] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0143] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0144] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0147] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0149] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for image search, characterized in that, Applied to computer equipment, including: Obtain the image to be retrieved and its descriptive text, and preprocess the image to be retrieved; Feature extraction and feature dimensionality reduction are performed on the preprocessed image to be retrieved to obtain a low-dimensional feature vector of the image to be retrieved. Multiple large images from an image database are acquired, and the multiple large images from the image database are preprocessed to obtain preprocessed database images; For any preprocessed image from the library, the image and the descriptive text are input into a pre-trained open vocabulary object detection network model to output the target region location information and confidence level. If the confidence level is greater than the target detection confidence level threshold, the target region is cropped according to the target region location information to obtain multiple target small images corresponding to the descriptive text and the location information of each target small image; Feature extraction and feature dimensionality reduction are performed on each of the target small images to obtain the low-dimensional feature vectors corresponding to each target small image; The low-dimensional feature vector of the image to be retrieved is compared with the low-dimensional feature vector of each target small image in the retrieval information repository to calculate the similarity comparison result; wherein the retrieval information repository is used to store the target region location information, target small image positioning information and low-dimensional feature vector corresponding to each target small image of each preprocessed library image; Based on the similarity comparison results, multiple target small images that meet the preset conditions are determined; Output the multiple target small images and the corresponding large image database of the multiple target small images.

2. The method according to claim 1, characterized in that, The step of performing feature extraction and dimensionality reduction on the preprocessed image to be retrieved to obtain a low-dimensional feature vector of the image to be retrieved includes: The preprocessed image to be retrieved is input into the pre-trained open vocabulary object detection network model, so that the detector in the open vocabulary object detection network model can extract features from the preprocessed image to obtain 1024-dimensional fused features. The 1024-dimensional fused features are reduced to 32 dimensions using a preset dimensionality reduction method to obtain the low-dimensional feature vector of the image to be retrieved.

3. The method according to claim 1, characterized in that, The cropping of the target region includes: Based on the upper boundary of the target area, a preset percentage of pixels are extended outward to determine the target upper frame of the target thumbnail corresponding to the expanded target area; If the target bounding box extends beyond the upper boundary of any preprocessed library image, then the target bounding box of the target thumbnail corresponding to the expanded target region is determined as the upper boundary of any preprocessed library image. Based on the lower boundary of the target region, expand outward by a preset percentage of pixels to determine the target lower bound of the target thumbnail corresponding to the expanded target region; If the target lower bound exceeds the lower boundary of any preprocessed library image, then the target lower bound of the target thumbnail corresponding to the expanded target region is determined as the lower boundary of any preprocessed library image. Based on the left boundary of the target region, expand outward by a preset percentage of pixels to determine the target left frame of the target thumbnail corresponding to the expanded target region; If the left bounding box of the target exceeds the left boundary of any preprocessed library image, then the left bounding box of the target thumbnail corresponding to the expanded target region is determined as the left boundary of any preprocessed library image. Based on the right boundary of the target region, expand outward by a preset percentage of pixels to determine the target right frame of the target thumbnail corresponding to the expanded target region; If the target right bounding box extends beyond the right boundary of any preprocessed library image, then the target right bounding box of the target thumbnail corresponding to the expanded target region is determined as the right boundary of any preprocessed library image. The target region is cropped based on the target's top frame, bottom frame, left frame, and right frame.

4. The method according to claim 1, characterized in that, The step of outputting the plurality of target thumbnails and the corresponding large image database image includes: If the number of target small images is 0, a message indicating that no results were found will be output. If the number of target small images is not greater than a preset number, the target small images and the corresponding large images in the image database are displayed in order of increasing similarity comparison results, wherein the target small images are highlighted in the large images in the image database based on the positioning information of the target small images; If the number of target small images is greater than the preset number, then according to the similarity comparison results from small to large, the preset number of target small images and the corresponding large image in the image database are displayed, wherein the target small images are highlighted and selected in the large image database based on the positioning information of the target small images.

5. The method according to claim 1, characterized in that, The step of comparing the low-dimensional feature vector of the image to be retrieved with the low-dimensional feature vectors of each target small image in the retrieval information repository, and calculating the similarity comparison result, includes: Calculate the cosine distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository; Calculate the Euclidean distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository; The half-value of the cosine distance is added to the half-value of the Euclidean distance to obtain the similarity comparison result between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vector of each target small image in the retrieval information repository.

6. The method according to claim 5, characterized in that, The formula for calculating the cosine distance is: In the formula, d1 represents the cosine distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; x 1k x represents the low-dimensional feature vector of the image to be retrieved; 2k This represents the low-dimensional feature vector of each target small image in the retrieval information database; n takes the value of 32.

7. The method according to claim 5, characterized in that, The formula for calculating the Euclidean distance is: In the formula, d2 represents the Euclidean distance between the low-dimensional feature vector of the image to be retrieved and the low-dimensional feature vectors of each target small image in the retrieval information repository; x 1k x represents the low-dimensional feature vector of the image to be retrieved; 2k This represents the low-dimensional feature vector of each target small image in the retrieval information database; n takes the value of 32.

8. An image search device, characterized in that, Applied to computer equipment, including: The image acquisition module is used to acquire the image to be retrieved and the descriptive text of the image to be retrieved, and to preprocess the image to be retrieved; The first feature extraction and feature dimensionality reduction module performs feature extraction and feature dimensionality reduction on the preprocessed image to be retrieved to obtain a low-dimensional feature vector of the image to be retrieved. The database large image acquisition module is used to acquire multiple large images from the image database and preprocess the multiple large images from the image database to obtain preprocessed database images; The target detection module is used to input any preprocessed library image and the descriptive text into a pre-trained open vocabulary target detection network model to output the target region location information and confidence level. The cropping module is used to crop the target region based on the target region location information if the confidence level is greater than the target detection confidence level threshold, so as to obtain multiple target small images corresponding to the descriptive text and the positioning information of each target small image; The second feature extraction and feature dimensionality reduction module is used to perform feature extraction and feature dimensionality reduction on each target small image to obtain the low-dimensional feature vector corresponding to each target small image. The similarity matching module is used to compare the low-dimensional feature vector of the image to be retrieved with the low-dimensional feature vector of each target small image in the retrieval information repository, and calculate the similarity comparison result; wherein the retrieval information repository is used to store the target region location information, the target small image positioning information and the low-dimensional feature vector corresponding to each target small image of each preprocessed library image. The target small image determination module determines multiple target small images that meet preset conditions based on the similarity comparison results. The output module is used to output the plurality of target small images and the corresponding large image database image of the plurality of target small images.

9. A computer device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images