Image retrieval method and device, electronic equipment and storage medium

Abstract

The invention provides an image retrieval method, which comprises the following steps of: performing target detection processing on a bottom library image to obtain a target image in the bottom library image; encoding the target image into a unified vector space to obtain a first vector feature corresponding to the target image, and constructing the unified vector space according to the vector features of the sample image and the sample text; constructing a target database based on the first vector feature; when a query text is received, encoding the query text into the uniform vector space to obtain a second vector feature corresponding to the query text; and based on the second vector feature, performing retrieval in a target database to obtain an image retrieval result. According to the cross-modal image-text retrieval method, the target image is extracted from the base library image, then the target image is coded into the unified vector space to be subjected to vector alignment with the query text, image compression is not needed, detail loss of a target object is avoided, only the target image is coded, interference of the background is reduced, and the accuracy of cross-modal image-text retrieval is improved.

Description

Technical Field

[0001] This invention relates to the field of image retrieval technology, and in particular to an image retrieval method, apparatus, electronic device, and storage medium. Background Technology

[0002] Cross-modal image-text retrieval retrieves corresponding images based on text input. For example, a user inputs text, and the system searches for the corresponding image. However, directly processing high-resolution images consumes significant computational resources, impacting real-time performance. Current mainstream methods for cross-modal image-text retrieval typically involve global image compression (e.g., to 224*224 pixels), extracting multi-granularity features (e.g., ResNet for global features, Region Proposal Networks for local features), and then aligning the images with the text using a cross-attention mechanism. However, global image compression reduces the resolution of small objects (e.g., a watch on a person's wrist, a tissue box on a car's windshield). For instance, a 2-megapixel image compressed to only 20×20 pixels may not retain details like clothing patterns or vehicle decorations. Furthermore, since objects typically occupy a small portion of an image, global features are easily affected by irrelevant background elements (e.g., trees, buildings, ground), leading to decreased accuracy in searching for key targets. Therefore, current cross-modal image-text retrieval methods suffer from detail loss due to global image compression and interference from irrelevant backgrounds, resulting in lower accuracy. Summary of the Invention

[0003] This invention provides an image retrieval method aimed at addressing the low accuracy issues in existing cross-modal image-text retrieval methods, which suffer from detail loss due to global image compression and interference from irrelevant backgrounds. This invention extracts the target image from a base image database and then encodes it into a unified vector space, aligning it with the query text. Since the target image has a lower resolution than the global image, image compression is unnecessary under limited computational resources, avoiding detail loss of the target object. Furthermore, encoding only the target image reduces interference from irrelevant backgrounds, thus improving the accuracy of cross-modal image-text retrieval.

[0004] In a first aspect, embodiments of the present invention provide an image retrieval method, the method comprising the following steps:

[0005] Target detection processing is performed on the base database image to obtain the target image in the base database image;

[0006] The target image is encoded into a unified vector space to obtain the first vector feature corresponding to the target image. The unified vector space is constructed based on the vector features of the sample image and the sample text.

[0007] Construct a target database based on the features of the first vector;

[0008] Upon receiving a query text, the query text is encoded into the unified vector space to obtain the second vector feature corresponding to the query text;

[0009] Based on the second vector feature, a search is performed in the target database to obtain image retrieval results.

[0010] Optionally, before performing target detection processing on the base image, the method further includes:

[0011] Get the video stream file;

[0012] The video stream file is subjected to target tracking and detection processing to obtain a tracking image containing the target;

[0013] Based on the image quality of the tracked images, the image with the highest image quality is selected as the base image.

[0014] Optionally, the step of performing target detection processing on the base database images to obtain target images in the base database images includes:

[0015] The target detection image is processed by a preset target detection model to obtain the coordinate information of the target detection box;

[0016] Based on the coordinate information of the target detection box, the target image corresponding to the target detection box is extracted from the base image to obtain the target image in the base image.

[0017] Optionally, the unified vector space is constructed based on the vector features of the sample image data and sample text, including:

[0018] Acquire sample images and sample text, where each sample image includes at least one sample target image and each sample target image corresponds to at least one sample text;

[0019] The sample target image is subjected to feature encoding processing by a pre-trained image encoder to obtain the first image vector feature corresponding to the sample target image;

[0020] Furthermore, the sample text is processed by a pre-trained text encoder to obtain the first text vector feature corresponding to the sample text;

[0021] With the goal of maximizing the similarity between the first image vector features and the first text vector features, the parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted, and the parameter adjustment process is iterated. When the preset first optimization condition is reached, the iteration is stopped, and the adjusted image encoder and the adjusted text encoder are obtained.

[0022] Based on the adjusted image encoder and the adjusted text encoder, the unified vector space is constructed.

[0023] Optionally, the unified vector space is constructed based on the vector features of the sample image data and sample text, and further includes:

[0024] The sample image is processed by feature encoding using an initial image encoder to obtain the second image vector feature corresponding to the sample image;

[0025] The sample text is processed by feature encoding through an initial text encoder to obtain the second text vector feature corresponding to the sample text;

[0026] With the goal of maximizing the similarity between the second image vector features and the second text vector features, the parameters of the initial image encoder and the initial text encoder are adjusted, and the parameter adjustment process is iterated. When the preset second optimization condition is reached, the iteration stops, and the pre-trained image encoder and the pre-trained text encoder are obtained.

[0027] Optionally, adjusting the parameters of the pre-trained image encoder and the pre-trained text encoder with the optimization objective of maximizing the similarity between the first image vector features and the first text vector features includes:

[0028] If the sample target image corresponds to a sample text, then calculate the first cosine similarity between the first image vector features and the first text vector features;

[0029] The parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted with the first cosine similarity as the optimization objective.

[0030] If the sample target image corresponds to multiple sample texts, then the center vector of the multiple first text vector features is calculated, and the second cosine similarity between the first image vector features and the center vector is calculated;

[0031] The parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted with the second cosine similarity as the optimization objective.

[0032] Optionally, the step of retrieving images from the target database based on the second vector features to obtain image retrieval results includes:

[0033] The similarity between the second vector feature and the first vector feature in the target database is calculated to obtain the similarity between the second vector feature and the first vector feature.

[0034] The target images corresponding to the N first vector features with similarity greater than the similarity threshold are determined as the retrieval target images;

[0035] The images in the base database to which the target image belongs are determined as the image retrieval results.

[0036] Secondly, embodiments of the present invention provide an image retrieval device, the image retrieval device comprising:

[0037] The target detection module is used to perform target detection processing on the base database image to obtain the target image in the base database image;

[0038] The first encoding module is used to encode the target image into a unified vector space to obtain the first vector feature corresponding to the target image. The unified vector space is constructed based on the vector features of the sample image and the sample text.

[0039] A construction module is used to construct a target database based on the features of the first vector.

[0040] The second encoding module is used to encode the query text into the unified vector space when the query text is received, so as to obtain the second vector feature corresponding to the query text;

[0041] The retrieval module is used to perform a retrieval in the target database based on the second vector features to obtain image retrieval results.

[0042] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the image retrieval method provided in embodiments of the present invention.

[0043] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image retrieval method provided in the embodiments of the invention.

[0044] In this embodiment of the invention, target detection processing is performed on the base database image to obtain the target image in the base database image; the target image is encoded into a unified vector space to obtain the first vector feature corresponding to the target image, and the unified vector space is constructed based on the vector features of sample images and sample text; a target database is constructed based on the first vector feature; when a query text is received, the query text is encoded into the unified vector space to obtain the second vector feature corresponding to the query text; based on the second vector feature, a search is performed in the target database to obtain the image retrieval result. This invention extracts the target image from the base database image and then encodes the target image into a unified vector space and aligns it with the query text. Since the resolution of the target image is lower than that of the global image, image compression is not required under limited computing resources, avoiding the loss of detail of the target object. Furthermore, encoding only the target image reduces interference from irrelevant backgrounds, thus improving the accuracy of cross-modal image-text retrieval. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of an image retrieval method provided in an embodiment of the present invention;

[0047] Figure 2 This is a comparative diagram of the search results provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the structure of an image retrieval device provided in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, Figure 1This is a flowchart of an image retrieval method provided in an embodiment of the present invention. The image retrieval method includes the following steps:

[0052] 101. Perform target detection processing on the base database image to obtain the target image in the base database image.

[0053] In this embodiment of the invention, the image retrieval method described above can be applied to an image retrieval platform. This platform can be built on a server-based or distributed architecture and includes a data interface (for sensor or user uploads), an image database, and an image database construction program. The data interface can be used to obtain a first feature vector to be searched, and the image database construction program can be used to construct the image database. The image database is specifically used to store base images and vector feature data of each target image within the base images.

[0054] The aforementioned base images can be used to construct an image database, which stores a large number of base images. These base images can be images acquired by image devices, such as video surveillance equipment, for example, traffic video surveillance equipment. Traffic video surveillance equipment can be based on video surveillance equipment networking standards formulated by relevant departments (such as GB / T 28181-2016 "Technical Requirements for Information Transmission, Exchange and Control of Public Security Video Surveillance Network System"), defining signaling interaction protocols for device registration, real-time audio and video on-demand, and historical playback, and supporting national cryptographic SM2 / SM3 encryption.

[0055] The target image is a subset of images from the base database. For example, the base database may contain images of people, vehicles, and non-motorized vehicles (including people, motorized vehicles, and non-motorized vehicles), and the target image may be an image of a person, a motorized vehicle, or a non-motorized vehicle from the base database. It should be noted that the aforementioned images of people, vehicles, and non-motorized vehicles are exemplary image types, and other image types that need to be searched may also be used.

[0056] For each base image, object detection processing can be performed to obtain the target image from the base image. This object detection processing can be implemented based on object detection algorithms, such as Fast R-CNN, YOLO series, RetinaNet, etc. The target image can be understood as a local image within the base image, or the base image can be understood as a large image containing both the target and the background, while the target image can be understood as a small image containing the target.

[0057] 102. Encode the target image into a unified vector space to obtain the first vector feature corresponding to the target image.

[0058] In this embodiment of the invention, the aforementioned unified vector space can be understood as a constraint that encodes images and text into a vector space with the same dimension, thereby enabling cross-modal image-text comparison through vector features of the same dimension.

[0059] The unified vector space is constructed based on the vector features of sample images and sample text. This unified vector space can be built upon the dual-tower model structure proposed by OpenAI, which includes an image encoder (VisualEncoder) and a text encoder (TextEncoder). Cross-modal semantic alignment is achieved through comparative learning of 400 million image-text pairs, thus constructing the unified vector space. For example, using the unified vector space, the similarity between images and natural language descriptions can be compared within the 512-dimensional semantic space constructed by CLIP.

[0060] For the target image, no compression is required. The target image is encoded into a unified vector space by an image encoder, thereby obtaining the first vector feature corresponding to the target image. Each target image corresponds to one first vector feature. Since there is a one-to-one correspondence between the first vector feature and the target image, the target image corresponding to the successfully matched first vector feature can be found by comparing the similarity with the first vector feature. In addition, the base image database to which the target image belongs can be found.

[0061] 103. Construct a target database based on the first vector features.

[0062] In this embodiment of the invention, the target database may include a base image, a target image, and a first vector feature corresponding to the target image. The target image has an identifier of the base image, and the base image to which the target image belongs can be found based on the identifier. The first vector feature has a corresponding relationship with the target image, so the corresponding target image can be found based on the first vector feature.

[0063] 104. Upon receiving the query text, encode the query text into a unified vector space to obtain the second vector feature corresponding to the query text.

[0064] In this embodiment of the invention, the query text can be input by the user through a terminal device, which can be a voice input terminal or a text input terminal. If the terminal device is a voice input terminal, the voice data is converted into text data through voice-to-text conversion, and this text data is the query text; if the terminal device is a text input terminal, the input text is the query text.

[0065] After obtaining the query text, it can be encoded into a unified vector space using the text encoder described above, thereby obtaining the second vector feature corresponding to the target image. Each query text corresponds to one second vector feature. It should be noted that the first and second vector features mentioned above have the same vector dimension, for example, both are 512 vector dimensions.

[0066] 105. Based on the second vector feature, a search is performed in the target database to obtain image retrieval results.

[0067] In this embodiment of the invention, after obtaining the second vector feature corresponding to the query text, the similarity between the second vector feature and the first vector feature in the target database is calculated to obtain the similarity between the second vector feature and the first vector feature. The base image of the database to which the target image corresponding to the first vector feature with the highest similarity is belonged is returned to the user as the image retrieval result. Alternatively, the target image and its base image in the database are returned to the user as the image retrieval result, so that the user can search for both large and small images simultaneously and compare more details in the small image.

[0068] In this embodiment of the invention, target detection processing is performed on the base database image to obtain the target image in the base database image; the target image is encoded into a unified vector space to obtain the first vector feature corresponding to the target image, and the unified vector space is constructed based on the vector features of sample images and sample text; a target database is constructed based on the first vector feature; when a query text is received, the query text is encoded into the unified vector space to obtain the second vector feature corresponding to the query text; based on the second vector feature, a search is performed in the target database to obtain the image retrieval result. This invention extracts the target image from the base database image and then encodes the target image into a unified vector space and aligns it with the query text. Since the resolution of the target image is lower than that of the global image, image compression is not required under limited computing resources, avoiding the loss of detail of the target object. Furthermore, encoding only the target image reduces interference from irrelevant backgrounds, thus improving the accuracy of cross-modal image-text retrieval.

[0069] It is understood that in the specific implementation of this application, data such as image data, text data, and user data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with relevant laws, regulations, and standards.

[0070] Optionally, before performing target detection processing on the base database images, a video stream file can be acquired; target tracking and detection processing can be performed on the video stream file to obtain a tracking image containing the target; based on the image quality of the tracking image, the image with the highest image quality is selected from the tracking images and determined as the base database image.

[0071] In this embodiment of the invention, a specified video surveillance camera (video surveillance device) can be associated and bound via the GB28181 protocol to obtain the real-time video stream file from the video surveillance camera. Alternatively, the corresponding video stream file can be obtained through user uploads.

[0072] After obtaining the video stream file, a target tracking and detection algorithm can be used to analyze it. This algorithm can be a combination of target detection algorithms based on models such as Fast R-CNN, YOLO series, and RetinaNet, and tracking algorithms based on Kalman filtering. The target detection algorithm performs target detection on each frame of the video stream file to determine if a target exists. If a target is found, the Kalman filtering-based tracking algorithm tracks it in subsequent frames until the target disappears, resulting in a tracked video segment containing the target. Image quality is then evaluated for each frame in the tracked video segment, resulting in a quality score for each frame. The frame with the highest quality score is designated as the base image, and each tracked video segment corresponds to one base image. This image quality evaluation can be implemented using image quality assessment algorithms such as mean squared error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM).

[0073] In non-person / vehicle detection scenarios, the aforementioned targets can be people, motor vehicles, or non-motor vehicles. For different detection scenarios, the targets can be those related to that scenario.

[0074] Optionally, in the step of performing target detection processing on the base image to obtain the target image in the base image, the base image can be processed by a preset target detection model to obtain the coordinate information of the target detection box; based on the coordinate information of the target detection box, the target image corresponding to the target detection box is extracted from the base image to obtain the target image in the base image.

[0075] In this embodiment of the invention, considering the improvement of target detection accuracy, the coordinate information of the target box can be output by a separately trained target detection algorithm.

[0076] In one possible embodiment, considering the overall improvement in the construction speed of the target database, target detection processing can also be performed using the target detection algorithm in the target tracking and detection process, thus eliminating the need for separate target detection processing.

[0077] The coordinate information mentioned above can be (x, y, w, h), where x and y are the coordinates of the center point of the target box, w is the width of the target box, and h is the height of the target box. By mapping the coordinate information of the target box to the image in the library, a bounding box containing the target is obtained. The image within the bounding box is then cropped to obtain the target image containing the target.

[0078] Optionally, in the step of constructing a unified vector space based on the vector features of sample image data and sample text, sample images and sample text can be obtained, each sample image including at least one sample target image, and each sample target image corresponding to at least one sample text; the sample target image is processed by feature encoding through a pre-trained image encoder to obtain the first image vector feature corresponding to the sample target image; and the sample text is processed by feature encoding through a pre-trained text encoder to obtain the first text vector feature corresponding to the sample text; the parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted with the optimization objective of maximizing the similarity between the first image vector feature and the first text vector feature, and the parameter adjustment process is iterated until a preset first optimization condition is reached, at which point the iteration stops, resulting in the adjusted image encoder and the adjusted text encoder; and a unified vector space is constructed based on the adjusted image encoder and the adjusted text encoder.

[0079] In this embodiment of the invention, the above-mentioned sample image is an image containing a target, and the above-mentioned sample text is text containing target description information. Each sample image contains one or more target images, and one target image corresponds to one or more sample texts. That is, one target image can be described by multiple types of description information.

[0080] The pre-trained image encoder mentioned above can be the image encoder (VisualEncoder) in a dual-tower model structure, and the pre-trained text encoder mentioned above can be the text encoder (TextEncoder) in a dual-tower model structure. The image encoder is used to encode the image into a vector, and the text encoder is used to encode the text into a vector. Through the image encoder and the text encoder, the image and text are encoded into a unified vector space. The parameters of the image encoder and the text encoder need to be adjusted so that images and texts with the same semantics can be encoded into the same or highly similar vectors.

[0081] The sample target image is feature-encoded using a pre-trained image encoder to obtain the first image vector feature corresponding to the sample target image. Similarly, the sample text is feature-encoded using a pre-trained text encoder to obtain the first text vector feature corresponding to the sample text. For sample target images and sample text with the same semantics, the similarity between the corresponding first image vector feature and the first text vector feature is maximized. In this embodiment, cosine similarity is used, which makes the similarity between the first image vector feature and the first text vector feature close to 1.

[0082] Specifically, the error value between the first image vector feature and the first text vector feature can be calculated. The smaller the error value, the higher the error between the first image vector feature and the first text vector feature. That is, the parameters of the image encoder and the text encoder can be adjusted by backpropagation of the error value. By iterating the above parameter adjustment process, the error value between the first image vector feature and the first text vector feature becomes smaller and smaller. When the error value reaches the preset error value threshold, or when the number of iterations reaches the preset number, the iteration stops, and the adjusted image encoder and the adjusted text encoder are obtained.

[0083] By adjusting the image encoder and the text encoder, a unified vector space is constructed. That is, the adjusted image encoder can encode the target image into the unified vector space, and the adjusted text encoder can encode the query text into the unified vector space, so that the target image and the query text can be compared in the unified vector space.

[0084] Optionally, in the step of constructing a unified vector space based on the vector features of sample image data and sample text, the sample image can be feature-encoded by an initial image encoder to obtain the second image vector feature corresponding to the sample image; the sample text can be feature-encoded by an initial text encoder to obtain the second text vector feature corresponding to the sample text; the parameters of the initial image encoder and the initial text encoder are adjusted with the optimization objective of maximizing the similarity between the second image vector feature and the second text vector feature, and the parameter adjustment process is iterated. When the preset second optimization condition is reached, the iteration is stopped, and the pre-trained image encoder and the pre-trained text encoder are obtained.

[0085] In this embodiment of the invention, to accelerate the convergence rate of the image encoder and the text encoder, the initial image encoder and the initial text encoder can be pre-trained. The initial image encoder can be the image encoder (Visual Encoder) in a dual-tower model structure, and the initial text encoder can be the text encoder (Text Encoder) in a dual-tower model structure.

[0086] The sample images are feature-encoded using an initial image encoder to obtain the second image vector features corresponding to the sample images. Similarly, the sample text is feature-encoded using an initial text encoder to obtain the second text vector features corresponding to the sample text. For sample images and sample text with the same semantics, the similarity between the corresponding second image vector features and the second text vector features is maximized. In this embodiment, cosine similarity is used, which makes the similarity between the second image vector features and the second text vector features close to 1.

[0087] Specifically, the error value between the second image vector feature and the second text vector feature can be calculated. The smaller the error value, the higher the error between the second image vector feature and the second text vector feature. That is, the parameters of the image encoder and the text encoder can be adjusted by backpropagation of the error value. By iterating the above parameter adjustment process, the error value between the second image vector feature and the second text vector feature becomes smaller and smaller. When the error value reaches the preset error value threshold, or when the number of iterations reaches the preset number, the iteration stops, and the pre-trained image encoder and the pre-trained text encoder are obtained.

[0088] After obtaining the pre-trained image encoder and the pre-trained text encoder, the parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted by using sample target images and sample text. Since the sample images include sample target images, the pre-training process can extract the global features of the sample images. Using the global features and sample text for pre-training can make the training process of the local features corresponding to the sample target images and the sample text converge quickly.

[0089] Optionally, in the step of adjusting the parameters of the pre-trained image encoder and the pre-trained text encoder with the optimization objective of maximizing the similarity between the first image vector features and the first text vector features, if the sample target image corresponds to one sample text, then the first cosine similarity between the first image vector features and the first text vector features is calculated; the parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted with the optimization objective of maximizing the first cosine similarity; if the sample target image corresponds to multiple sample texts, then the center vectors of the multiple first text vector features are calculated, and the second cosine similarity between the first image vector features and the center vectors is calculated; the parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted with the optimization objective of maximizing the second cosine similarity.

[0090] In this embodiment of the invention, a single sample target image can have multiple descriptive information entries. For example, if the sample target image is a seat belt, the descriptive information would be "seat belt"; if the sample target image is a tissue box, the descriptive information could be "tissue box at the right windshield" and "tissue box in front of the passenger seat." Therefore, when considering the diversity of descriptive information, the parameter adjustment process can be determined based on whether there are multiple sample text entries corresponding to the sample target image.

[0091] Specifically, if a target image corresponds to one sample text, it means the target image does not contain any other descriptive information. Therefore, the first cosine similarity between the first image vector features and the first text vector features can be calculated. The optimization objective is to maximize the first cosine similarity, and the parameters of both the pre-trained image encoder and the pre-trained text encoder are then adjusted accordingly. If a target image corresponds to multiple sample texts, it means the target image contains other descriptive information. Therefore, the center vectors of the multiple first text vector features can be calculated, and the second cosine similarity between the first image vector features and the center vectors can be calculated. The optimization objective is to maximize the second cosine similarity, and the parameters of both the pre-trained image encoder and the pre-trained text encoder are then adjusted accordingly. The aforementioned center vector can be the average of the multiple first text vector features or the cluster center of the multiple first text vector features.

[0092] Optionally, in the step of retrieving images from the target database based on the second vector feature and obtaining image retrieval results, the similarity between the second vector feature and the first vector feature in the target database can be calculated to obtain the similarity between the second vector feature and the first vector feature; the target images corresponding to the N first vector features with similarity greater than the similarity threshold are determined as the retrieval target images; and the base database images to which the retrieval target images belong are determined as the image retrieval results.

[0093] In this embodiment of the invention, after obtaining the second vector feature corresponding to the query text, the cosine similarity between the second vector feature and the first vector feature in the target database is calculated to obtain the cosine similarity between the second vector feature and the first vector feature. The N first vector features with the largest cosine similarity are determined as the retrieved first vector features. Then, based on the correspondence between the first vector features and the target images, the target images corresponding to the retrieved first vector features are found, resulting in N retrieved target images. For each target image, the base image to which the target image belongs is found based on the target image's identifier, and the corresponding base image is returned to the user as the image retrieval result. N is an integer greater than or equal to 1.

[0094] In this embodiment of the invention, the search accuracy is improved by more than 10% through lossless compression parsing;

[0095] In tests of 65 common sentence search scenarios, multiple algorithm versions demonstrated that the search for small target samples after lossless compressed feature extraction can effectively improve the accuracy of Top 1 and Top 20. The improved search accuracy can effectively support the search for key targets in criminal investigation and case handling, and obtain the trajectory information of the required targets more quickly and accurately.

[0096] Dataset:

[0097] The traditional method involves capturing 1 million complete large images from a 2-megapixel video surveillance camera, compressing each large image (1920*1080 pixels) globally to a maximum of 384*384 pixels, and then inputting it into a semantic search engine for feature extraction. The results show the search accuracy statistics when performing cross-modal text-to-image search.

[0098] This invention involves processing 1 million complete large-scale images using a human, vehicle, and non-motorized vehicle (PMV) target tracking and detection algorithm to detect a total of 6.11 million PMV images. These images are then losslessly compressed and input into a semantic search engine for feature extraction. The search accuracy statistics for cross-modal text-to-image search are as follows: Figure 2 As shown.

[0099] Hardware environment:

[0100] CPU: 2X Intel Xeon Gold 5318Y Processor / 2.10GHz / 36MB / 24C / 48T / 165W / 3UPI / 2933MHz;

[0101] Memory: 32x 32GB / DDR4 / 2666MHz or 2933MHz or 3200MHz / ECC / REG;

[0102] Hard Drives: 4 x 6TB / SATA / 7200RPM / 3.5-inch / Enterprise-grade SSDs: 2 x SSDs / 960GB / SATA 6Gb / 2.5-inch / Read speed <1DWPD;

[0103] Graphics cards: 8 NVIDIA 4090 graphics cards.

[0104] like Figure 3 As shown, an embodiment of the present invention provides an image retrieval device, which includes:

[0105] The target detection module 201 is used to perform target detection processing on the base database image to obtain the target image in the base database image;

[0106] The first encoding module 202 is used to encode the target image into a unified vector space to obtain the first vector feature corresponding to the target image. The unified vector space is constructed based on the vector features of the sample image and the sample text.

[0107] Construction module 203 is used to construct a target database based on the features of the first vector;

[0108] The second encoding module 204 is used to encode the query text into the unified vector space when the query text is received, so as to obtain the second vector feature corresponding to the query text;

[0109] The retrieval module 205 is used to perform a retrieval in the target database based on the second vector features to obtain image retrieval results.

[0110] Optionally, the device further includes:

[0111] The first acquisition module is used to acquire video stream files;

[0112] The tracking module is used to perform target tracking and detection processing on the video stream file to obtain a tracking image containing the target;

[0113] The processing module is used to select the image with the highest image quality from the tracked images and determine it as the base image based on the image quality of the tracked images.

[0114] Optionally, the target detection module 201 is further configured to perform target detection processing on the base image using a preset target detection model to obtain the coordinate information of the target detection box; and, based on the coordinate information of the target detection box, extract the target image corresponding to the target detection box from the base image to obtain the target image in the base image.

[0115] Optionally, the device further includes:

[0116] The second acquisition module is used to acquire sample images and sample text. Each sample image includes at least one sample target image, and each sample target image corresponds to at least one sample text.

[0117] The third encoding module is used to perform feature encoding processing on the sample target image through a pre-trained image encoder to obtain the first image vector feature corresponding to the sample target image;

[0118] The fourth encoding module performs feature encoding on the sample text using a pre-trained text encoder to obtain the first text vector feature corresponding to the sample text.

[0119] The first adjustment module is used to adjust the parameters of the pre-trained image encoder and the pre-trained text encoder with the optimization objective of maximizing the similarity between the first image vector features and the first text vector features, and to iterate the parameter adjustment process. When the preset first optimization condition is reached, the iteration stops, and the adjusted image encoder and the adjusted text encoder are obtained.

[0120] A construction module is used to construct the unified vector space based on the adjusted image encoder and the adjusted text encoding.

[0121] Optionally, the device further includes:

[0122] The fifth encoding module is used to perform feature encoding processing on the sample image through the initial image encoder to obtain the second image vector feature corresponding to the sample image;

[0123] The sixth encoding module is used to perform feature encoding processing on the sample text through the initial text encoder to obtain the second text vector feature corresponding to the sample text;

[0124] The second adjustment module is used to adjust the parameters of the initial image encoder and the initial text encoder with the optimization objective of maximizing the similarity between the second image vector features and the second text vector features, and to iterate the parameter adjustment process. When the preset second optimization condition is reached, the iteration stops, and a pre-trained image encoder and a pre-trained text encoder are obtained.

[0125] Optionally, the first adjustment module is further configured to: if the sample target image corresponds to a sample text, calculate a first cosine similarity between the first image vector feature and the first text vector feature; adjust the parameters of the pre-trained image encoder and the pre-trained text encoder with the first cosine similarity being maximized as the optimization objective; if the sample target image corresponds to multiple sample texts, calculate the center vectors of multiple first text vector features, and calculate a second cosine similarity between the first image vector feature and the center vector; adjust the parameters of the pre-trained image encoder and the pre-trained text encoder with the second cosine similarity being maximized as the optimization objective.

[0126] Optionally, the retrieval module 205 is further configured to perform similarity calculation between the second vector feature and the first vector feature in the target database to obtain the similarity between the second vector feature and the first vector feature; determine the target images corresponding to the N first vector features whose similarity is greater than the similarity threshold as the retrieval target images; and determine the base database images to which the retrieval target images belong as the image retrieval results.

[0127] like Figure 4 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described image retrieval methods.

[0128] Specifically, it includes a processor 301 and a memory 302, as well as a computer program stored in the memory 302 and capable of running on the processor 301 to execute the image retrieval method, wherein:

[0129] The processor 301 executes the calculator program for the image retrieval method stored in the memory 302, performing the following steps:

[0130] Target detection processing is performed on the base database image to obtain the target image in the base database image;

[0131] The target image is encoded into a unified vector space to obtain the first vector feature corresponding to the target image. The unified vector space is constructed based on the vector features of the sample image and the sample text.

[0132] Construct a target database based on the features of the first vector;

[0133] Upon receiving a query text, the query text is encoded into the unified vector space to obtain the second vector feature corresponding to the query text;

[0134] Based on the second vector feature, a search is performed in the target database to obtain image retrieval results.

[0135] Optionally, before performing target detection processing on the base image, the method executed by the processor 301 further includes:

[0136] Get the video stream file;

[0137] The video stream file is subjected to target tracking and detection processing to obtain a tracking image containing the target;

[0138] Based on the image quality of the tracked images, the image with the highest image quality is selected as the base image.

[0139] Optionally, the target detection processing of the base image performed by the processor 301 to obtain the target image in the base image includes:

[0140] The target detection image is processed by a preset target detection model to obtain the coordinate information of the target detection box;

[0141] Based on the coordinate information of the target detection box, the target image corresponding to the target detection box is extracted from the base image to obtain the target image in the base image.

[0142] Optionally, the unified vector space executed by processor 301 is constructed based on the vector features of sample image data and sample text, including:

[0143] Acquire sample images and sample text, where each sample image includes at least one sample target image and each sample target image corresponds to at least one sample text;

[0144] The sample target image is subjected to feature encoding processing by a pre-trained image encoder to obtain the first image vector feature corresponding to the sample target image;

[0145] Furthermore, the sample text is processed by a pre-trained text encoder to obtain the first text vector feature corresponding to the sample text;

[0146] With the goal of maximizing the similarity between the first image vector features and the first text vector features, the parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted, and the parameter adjustment process is iterated. When the preset first optimization condition is reached, the iteration is stopped, and the adjusted image encoder and the adjusted text encoder are obtained.

[0147] Based on the adjusted image encoder and the adjusted text encoder, the unified vector space is constructed.

[0148] Optionally, the unified vector space executed by processor 301, constructed based on the vector features of sample image data and sample text, further includes:

[0149] The sample image is processed by feature encoding using an initial image encoder to obtain the second image vector feature corresponding to the sample image;

[0150] The sample text is processed by feature encoding through an initial text encoder to obtain the second text vector feature corresponding to the sample text;

[0151] With the goal of maximizing the similarity between the second image vector features and the second text vector features, the parameters of the initial image encoder and the initial text encoder are adjusted, and the parameter adjustment process is iterated. When the preset second optimization condition is reached, the iteration stops, and the pre-trained image encoder and the pre-trained text encoder are obtained.

[0152] Optionally, the process performed by processor 301 to adjust the parameters of the pre-trained image encoder and the pre-trained text encoder, with the optimization objective of maximizing the similarity between the first image vector features and the first text vector features, includes:

[0153] If the sample target image corresponds to a sample text, then calculate the first cosine similarity between the first image vector features and the first text vector features;

[0154] The parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted with the first cosine similarity as the optimization objective.

[0155] If the sample target image corresponds to multiple sample texts, then the center vector of the multiple first text vector features is calculated, and the second cosine similarity between the first image vector features and the center vector is calculated;

[0156] The parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted with the second cosine similarity as the optimization objective.

[0157] Optionally, the process executed by processor 301 to perform a search in the target database based on the second vector features to obtain image retrieval results includes:

[0158] The similarity between the second vector feature and the first vector feature in the target database is calculated to obtain the similarity between the second vector feature and the first vector feature.

[0159] The target images corresponding to the N first vector features with similarity greater than the similarity threshold are determined as the retrieval target images;

[0160] The images in the base database to which the target image belongs are determined as the image retrieval results.

[0161] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the image retrieval method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0162] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0163] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. An image retrieval method, characterized in that, The method includes the following steps: Target detection processing is performed on the base database image to obtain the target image in the base database image; The target image is encoded into a unified vector space to obtain the first vector feature corresponding to the target image. The unified vector space is constructed based on the vector features of the sample image and the sample text. Construct a target database based on the features of the first vector; Upon receiving a query text, the query text is encoded into the unified vector space to obtain the second vector feature corresponding to the query text; Based on the second vector feature, a search is performed in the target database to obtain image retrieval results.

2. The image retrieval method as described in claim 1, characterized in that, Before performing target detection processing on the base image, the method further includes: Get the video stream file; The video stream file is subjected to target tracking and detection processing to obtain a tracking image containing the target; Based on the image quality of the tracked images, the image with the highest image quality is selected as the base image.

3. The image retrieval method as described in claim 1, characterized in that, The step of performing target detection processing on the base database images to obtain target images from the base database images includes: The target detection image is processed by a preset target detection model to obtain the coordinate information of the target detection box; Based on the coordinate information of the target detection box, the target image corresponding to the target detection box is extracted from the base image to obtain the target image in the base image.

4. The image retrieval method as described in claim 1, characterized in that, The unified vector space is constructed based on the vector features of sample image data and sample text, including: Acquire sample images and sample text, where each sample image includes at least one sample target image and each sample target image corresponds to at least one sample text; The sample target image is subjected to feature encoding processing by a pre-trained image encoder to obtain the first image vector feature corresponding to the sample target image; Furthermore, the sample text is processed by a pre-trained text encoder to obtain the first text vector feature corresponding to the sample text; With the goal of maximizing the similarity between the first image vector features and the first text vector features, the parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted, and the parameter adjustment process is iterated. When the preset first optimization condition is reached, the iteration is stopped, and the adjusted image encoder and the adjusted text encoder are obtained. Based on the adjusted image encoder and the adjusted text encoder, the unified vector space is constructed.

5. The image retrieval method as described in claim 4, characterized in that, The unified vector space is constructed based on the vector features of sample image data and sample text, and also includes: The sample image is processed by feature encoding using an initial image encoder to obtain the second image vector feature corresponding to the sample image; The sample text is processed by feature encoding through an initial text encoder to obtain the second text vector feature corresponding to the sample text; With the goal of maximizing the similarity between the second image vector features and the second text vector features, the parameters of the initial image encoder and the initial text encoder are adjusted, and the parameter adjustment process is iterated. When the preset second optimization condition is reached, the iteration stops, and the pre-trained image encoder and the pre-trained text encoder are obtained.

6. The image retrieval method as described in claim 4, characterized in that, The step of adjusting the parameters of the pre-trained image encoder and the pre-trained text encoder, with the optimization objective of maximizing the similarity between the first image vector features and the first text vector features, includes: If the sample target image corresponds to a sample text, then calculate the first cosine similarity between the first image vector features and the first text vector features; The parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted with the first cosine similarity as the optimization objective. If the sample target image corresponds to multiple sample texts, then the center vector of the multiple first text vector features is calculated, and the second cosine similarity between the first image vector features and the center vector is calculated; The parameters of the pre-trained image encoder and the pre-trained text encoder are adjusted with the second cosine similarity as the optimization objective.

7. The image retrieval method according to any one of claims 1-6, characterized in that, The step of retrieving images from the target database based on the second vector feature to obtain image retrieval results includes: The similarity between the second vector feature and the first vector feature in the target database is calculated to obtain the similarity between the second vector feature and the first vector feature. The target images corresponding to the N first vector features with similarity greater than the similarity threshold are determined as the retrieval target images; The images in the base database to which the target image belongs are determined as the image retrieval results.

8. An image retrieval device, characterized in that, The image retrieval device includes: The target detection module is used to perform target detection processing on the base database image to obtain the target image in the base database image; The first encoding module is used to encode the target image into a unified vector space to obtain the first vector feature corresponding to the target image. The unified vector space is constructed based on the vector features of the sample image and the sample text. A construction module is used to construct a target database based on the features of the first vector. The second encoding module is used to encode the query text into the unified vector space when the query text is received, so as to obtain the second vector feature corresponding to the query text; The retrieval module is used to perform a retrieval in the target database based on the second vector features to obtain image retrieval results.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the image retrieval method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image retrieval method as described in any one of claims 1 to 7.

Images