Image retrieval method and device, electronic equipment and storage medium

Abstract

The application provides an image retrieval method and device, electronic equipment and storage medium, the method comprises the following steps: acquiring an arbitrary text prompt word input by a user, and extracting a text feature of the arbitrary text prompt word by using a preset image-text matching model to obtain a first text feature; performing image retrieval in a preset base library according to the first text feature to determine a plurality of first candidate images of the arbitrary text prompt word; matching the first text feature with a preset standard image corpus to determine a target standard small image corresponding to the first text feature; extracting contour features of the target standard small image and contour features of each first candidate image, and adjusting the sorting of the plurality of first candidate images according to the contour features of the target standard small image and the contour features of each first candidate image to determine a plurality of first candidate images sorted according to a target sorting. The technical scheme of the application can be applied to the image retrieval scene of an arbitrary text prompt word.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an image retrieval method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the security industry, surveillance cameras in monitoring systems generally only provide functions such as video recording and real-time monitoring / surveillance. However, the number of surveillance cameras and the amount of video they capture is enormous, making it impossible to monitor and analyze every detail in the surveillance video in real time. Therefore, it is crucial to find effective video / image data from massive amounts of surveillance video data.

[0003] In related technologies, when retrieving the required images from a massive amount of images, the user's needs are usually divided into different functional modules in advance by manual intervention, such as object recognition, behavior recognition, and object association. Then, different recognition algorithms are developed according to different functions, and a large amount of algorithmic logic is used to achieve the retrieval of the required images from a massive amount of images.

[0004] However, the above technologies require the development of different functional modules to achieve image retrieval functions for different user needs, which limits their applicability. Summary of the Invention

[0005] This invention provides an image retrieval method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies that require the development of different functional modules to achieve image retrieval functions for different user needs, resulting in limited applicability. This invention enables image retrieval of arbitrary text prompts through an open image retrieval logic.

[0006] This invention provides an image retrieval method, comprising: Get any text prompt words input by the user, and use a preset image-text matching model to extract the text features of the arbitrary text prompt words to obtain the first text feature; Based on the first text feature, an image retrieval is performed in a preset database to determine multiple first candidate images corresponding to any text prompt word; the aforementioned database includes multiple candidate small images to be retrieved and / or a monitoring large image corresponding to each candidate small image; The first text feature is matched with a preset standard image corpus to determine the target standard small image corresponding to the first text feature; the standard image corpus includes a standard text prompt word library and multiple standard small images, and each standard text prompt word in the standard text prompt word library has a corresponding complete standard small image; Extract the contour features of the target standard thumbnail and the contour features of each first candidate image. Based on the contour features of the target standard thumbnail and the contour features of each first candidate image, adjust the order of the multiple first candidate images to determine the multiple first candidate images sorted according to the target order.

[0007] According to an image retrieval method provided by the present invention, the above-mentioned matching of a first text feature with a preset standard image corpus to determine the target standard thumbnail corresponding to the first text feature includes: Text features are extracted from each standard text prompt word in the standard text prompt word library to obtain the second text feature corresponding to each standard text prompt word; Calculate the text similarity between the first text feature and each of the second text features, and obtain the maximum text similarity among all text similarities; If the maximum text similarity is greater than a preset threshold, the standard small image corresponding to the maximum text similarity is obtained from the standard image corpus, and the standard small image corresponding to the maximum text similarity is determined as the target standard small image corresponding to the first text feature.

[0008] According to an image retrieval method provided by the present invention, the above-mentioned adjustment of the ranking of multiple first candidate images based on the contour features of the target standard thumbnail and the contour features of each first candidate image to determine multiple first candidate images ranked according to the target ranking includes: Calculate the feature similarity between the contour features of the target standard thumbnail and the contour features of each first candidate image; Obtain the first text features and the first image-text similarity between each first candidate image; The similarity of each feature and the corresponding first image-text similarity are averaged to obtain multiple averages, and the averages are sorted from largest to smallest to obtain the target ranking. The multiple first candidate images are sorted and adjusted according to the target sorting to determine the multiple first candidate images sorted according to the target sorting.

[0009] According to an image retrieval method provided by the present invention, the extraction of contour features of the target standard thumbnail and the extraction of contour features of each first candidate image include: The contour features of the target standard small image and the contour features of each first candidate image are extracted using a preset contour feature extraction model, so as to obtain the contour features corresponding to the target standard small image and the contour features corresponding to each first candidate image. The above contour feature extraction model is trained based on multiple sample images and the sample contour features of each sample image. The contour features of each sample image are obtained by performing image feature extraction, clustering and contour extraction on each sample image, and then performing contour feature extraction on each sample contour image based on the GEI gait recognition model.

[0010] According to an image retrieval method provided by the present invention, the establishment of the aforementioned base database includes: Obtain multiple large monitoring images from the monitoring scene; each of the monitoring images includes at least one target. The target detection model is used to detect targets in each monitoring image, determine the detection box of the target in each monitoring image, and then crop the corresponding monitoring image according to the detection box of each target to obtain the candidate small image corresponding to each target. Establish a binding relationship between each candidate small image and its corresponding monitoring large image to obtain the base database.

[0011] According to an image retrieval method provided by the present invention, the method further includes: The first text feature is input into a preset classification model for classification to determine the target category corresponding to any text prompt word; the target category includes any text prompt word being a negative prompt word or any text prompt word not being a negative prompt word, and the classification model is trained based on a negative prompt word dataset and a non-negative prompt word dataset in the monitoring scenario; If the target category is any of the above text prompt words as negative prompt words, then obtain the text similarity between the first text feature and the second text feature of each standard text prompt word in the standard text prompt word library, and according to the ranking result of each text similarity, select multiple candidate standard text prompt words with smaller text similarity from each of the above standard text prompt words; Calculate the second textual similarity between the second textual features of each candidate standard textual prompt and the image features corresponding to each candidate small image in the base database, and retrieve and determine multiple second candidate images corresponding to each candidate standard textual prompt based on the second textual similarity. Based on the second image-text similarity of each of the aforementioned second candidate images, multiple second candidate images with higher second image-text similarity are selected and output.

[0012] According to an image retrieval method provided by the present invention, the method further includes: If the target category is any text prompt word that is not a negative prompt word, then return to the above steps of performing image retrieval in the preset base database based on the first text features to determine multiple first candidate images corresponding to the text prompt word.

[0013] The present invention also provides an image retrieval device, comprising the following modules: The text feature extraction module is used to obtain any text prompt words input by the user, and to extract the text features of any text prompt words using a preset image-text matching model to obtain the first text feature; The first candidate image retrieval module is used to perform image retrieval in a preset base database based on the first text features, and determine multiple first candidate images corresponding to any text prompt word; the base database includes multiple candidate small images to be retrieved and / or a monitoring large image corresponding to each candidate small image; The standard corpus matching module is used to match the first text feature with a preset standard image corpus to determine the target standard small image corresponding to the first text feature; the standard image corpus includes a standard text prompt word library and multiple standard small images, and each standard text prompt word in the standard text prompt word library has a corresponding complete standard small image; The contour matching module is used to extract the contour features of the target standard small image and the contour features of each first candidate image. Based on the contour features of the target standard small image and the contour features of each first candidate image, the module adjusts the order of the multiple first candidate images to determine the multiple first candidate images sorted according to the target order.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image retrieval method as described above.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image retrieval method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image retrieval method as described above.

[0017] The image retrieval method, apparatus, electronic device, and storage medium provided by this invention obtains arbitrary text prompts input by the user and extracts the question features of the arbitrary text prompts using an image-text matching model to obtain first text features. Then, based on the first text features, image retrieval is performed in a base database to determine multiple first candidate images corresponding to the arbitrary text prompts. Subsequently, the first text features are matched with a standard image corpus to determine the target standard small image corresponding to the first text features. The contour features of the target standard small image and the contour features of each first candidate image are extracted. The order of the multiple first candidate images is adjusted based on the contour features of the target standard small image and the contour features of each first candidate image to determine multiple first candidate images sorted according to the target order. The base database includes multiple candidate small images to be retrieved and a monitoring large image corresponding to each candidate small image. The standard image corpus includes a standard text prompt word database and multiple standard small images, and each standard text prompt word in the standard text prompt word database has a corresponding complete standard small image. In this method, the order of multiple images retrieved from the base database can be adjusted based on the corresponding standard thumbnails retrieved from the standard image corpus for any text prompt word input by the user. This makes the final image retrieval results more accurate, and the method does not require the development of corresponding functional modules for different functions, thus enabling accurate image retrieval for any text prompt word. At the same time, since the order of multiple images retrieved from the base database can be adjusted by comparing the contour features of the corresponding complete standard thumbnails retrieved from the standard image corpus with the contour features of multiple images retrieved from the base database, the perception of the completeness of the target can be improved, making the final sorted multiple retrieved images more consistent with the actual situation and avoiding retrieval ambiguity. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the image retrieval method provided by the present invention.

[0020] Figure 2 This is the second flowchart of the image retrieval method provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the image retrieval device provided by the present invention.

[0022] Figure 4This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] In the security industry, behavioral alerting is a key technology used to detect and identify behaviors that may indicate potential dangers or specific situations. With the continuous development and increasing intelligence of video surveillance technology, behavioral alerting has become an important component of security systems. Currently, in the security field, surveillance cameras often only provide video recording and real-time monitoring functions. However, due to the vast monitoring range, numerous cameras, and limited human resources, monitoring systems cannot monitor and analyze every detail in real time. Meanwhile, with the promotion of intelligent terminal devices, especially in the security field, multimedia data, particularly visual data, is experiencing explosive growth. Therefore, it is crucial for users to quickly retrieve useful information or data of any need from massive amounts of data. For retrieval tasks, users typically input the content to be searched, and the system searches the database for images, text, or video data based on the user's input. Commonly used algorithms in retrieval tasks mainly use object detection and recognition algorithms to search for a specific behavior or target. Commonly used object detection and recognition algorithms include: YOLO (You Only Look Once, object detection) series models, ResNet (residual) series models, and lightweight VIT (Vision Transformer, a visual model mainly used for extracting image features) series models, etc.

[0025] Currently, a common approach to image retrieval is to combine general-purpose large models. However, these models still suffer from various problems during the retrieval process, resulting in less than ideal search results. This is particularly problematic in security scenarios where performance requirements are high, and the variety of large models also presents limitations. Another technique involves developing different recognition algorithms and employing extensive algorithmic logic for different tasks to retrieve desired images from massive datasets. However, this technique requires developing different functional modules to meet diverse user needs, limiting its applicability.

[0026] Based on this, the present invention provides an image retrieval method, apparatus, electronic device, and storage medium to solve this technical problem.

[0027] The following is combined with Figures 1-2 This invention describes an image retrieval method according to an embodiment of the present invention.

[0028] It should be noted that the execution subject of the embodiments of the present invention may be an image retrieval device, an electronic device including an image retrieval device, or other devices, equipment or systems, etc. This embodiment does not specifically limit it. The following embodiments will use an electronic device as an example to illustrate the execution subject.

[0029] Figure 1 This is one of the flowcharts illustrating the image retrieval method provided by the present invention, such as... Figure 1 As shown, the method includes the following steps: S102, obtain any text prompt word input by the user, and use a preset image-text matching model to extract the text features of the arbitrary text prompt word to obtain the first text feature.

[0030] If a user needs to retrieve a desired image from the database using text, they can first input the corresponding prompt text into their electronic device. The user can input any desired prompt text, referred to as "arbitrary prompt text." It is understood that this arbitrary prompt text can be English or Chinese text; this embodiment primarily uses Chinese text as an example for explanation.

[0031] After obtaining any text prompt, the text features of the text within that prompt can be extracted using a pre-defined image-text matching model. This image-text matching model can be the Chinese CLIP (Contrastive Language-Image Pre-training) model. The Chinese CLIP model is a multimodal pre-training model based on contrastive learning, where the core principle is to extract features from both text and image using encoders, mapping them to the same representation space. The model is trained by calculating the similarity and difference between text-image pairs, thus generating an image that matches the given text. This image-text matching model can include an image encoder and a text encoder, and may also include other network structures. After the image encoder and text encoder encode the corresponding image and text, they can map the text and visual embeddings to the same space. Using the idea of ​​contrastive learning, the distance between matching image-text embeddings is shortened, while the distance between mismatched embeddings is widened. Optionally, the preset image-text matching model here can be the Chinese CLIP-L-14-336 basic visual model, or it can be a Chinese CLIP-L-14, Chinese CLIP-B-32, M2-Decode series, or other Chinese image-text matching models.

[0032] Based on this, after obtaining the arbitrary prompt text input by the user, the text encoder of the image-text matching model can be used to extract text features from the arbitrary prompt text to obtain the text features corresponding to the arbitrary prompt text, which are denoted as the first text features.

[0033] S104, based on the first text features, perform image retrieval in a preset base database to determine multiple first candidate images corresponding to any text prompt word; the base database includes multiple candidate small images to be retrieved and / or a monitoring large image corresponding to each candidate small image.

[0034] In this step, after obtaining the first text feature corresponding to any prompt word, a predetermined number of relevant images can be roughly retrieved from the massive data in the base database based on this first text feature. Subsequent fine-grained searches and order adjustments are then performed to make the image retrieval results more accurate and improve the sorting precision of the output retrieval results, thereby enhancing user satisfaction with the retrieval results. The following explains how the base database is established.

[0035] As an optional embodiment, the above-mentioned method for establishing the base library may include the following steps: Obtain multiple large monitoring images from the monitoring scene; each of these monitoring images includes at least one target.

[0036] The target detection model is used to detect targets in each monitoring image, determine the detection box of each target in each monitoring image, and then crop the corresponding monitoring image based on the detection box of each target to obtain the candidate small image corresponding to each target.

[0037] Establish a binding relationship between each candidate small image and its corresponding monitoring large image to obtain the base database.

[0038] This embodiment mainly describes the process of building a base library for massive images in a monitoring scenario. Of course, in actual practice, base libraries can also be built in this way for other scenarios, and no specific limitation is made here.

[0039] Specifically, multiple images or videos can be obtained by capturing images of the monitored scene using various cameras. Videos typically consist of multiple frames, meaning multiple monitoring images are obtained. The monitoring images acquired through cameras are generally large-scale images, potentially including one or more targets. These large-scale images are usually quite large. Targets in the large-scale monitoring images can be people, animals, plants, trees, buildings, vehicles, etc.

[0040] After obtaining multiple large-scale surveillance images, a target detection model can be used to perform target detection on each large-scale image, obtaining the detection results for the targets included in each image. These results can include the detection bounding box corresponding to the target in the large-scale image, its location information, and its identifier. The detection bounding box can include the center point position and size information (e.g., length, width, and height). After obtaining the detection bounding boxes for the targets in each large-scale image, the corresponding region in the image can be cropped using the location information of the detection bounding boxes to obtain the image corresponding to the region of the detection bounding box, i.e., the image corresponding to a single target, denoted as the target-corresponding candidate small image. This method can obtain candidate small images corresponding to each target in a massive number of large-scale surveillance images. Then, each candidate small image can be directly bound to its corresponding large-scale surveillance image to establish a base database. Alternatively, after obtaining the candidate small images, an image encoder using an image-text matching model can be used to extract image features from each candidate small image and each monitoring large image, determining the image features corresponding to each candidate small image and each monitoring large image. Then, a binding relationship is established between the image features corresponding to each candidate small image, the image features corresponding to each monitoring large image, and each candidate small image and its corresponding monitoring large image, to obtain a base library. That is, the established base library can include each candidate small image and its corresponding monitoring large image, or it can include the image features of each candidate small image and the image features of each monitoring large image. Furthermore, the above object detection model can use the open detection model YOLO-World, or other models; no specific limitations are made here.

[0041] After obtaining the first text feature corresponding to any text prompt word, the image-text similarity is calculated together with the first text feature, the image features of each candidate small image in the database, and the image features of each monitoring large image. Each obtained image-text similarity can be recorded as the first image-text similarity. Then, multiple images with high first image-text similarity from the database are output. These multiple output images can be multiple candidate small images with high first image-text similarity and / or their corresponding monitoring large images, etc., and are all recorded as first candidate images. It can be understood that these multiple first candidate images are output according to the image-text similarity between image features (including the image features of candidate small images and the image features of monitoring large images). Therefore, these multiple first candidate images can include candidate small images or monitoring large images. Furthermore, the number of multiple first candidate images output can be, for example, 100, or it can be multiple images from the top 100 (i.e., Top 100) of the first image-text similarity in the database.

[0042] It should be noted that in the retrieval task, the open set / open object detection of the YOLO-World model does not completely pursue the edge detection of targets in the subsequent open retrieval of the image-text matching model. Instead, it is based on the security / surveillance scenario to distinguish targets as much as possible and cover the entire image, thereby reducing the image area based on the target. This solves the problem of inaccurate matching caused by multiple and small targets when using the image-text matching model to retrieve large surveillance images in security / surveillance scenarios.

[0043] Furthermore, when building the base database, the YOLO-World model (i.e., the object detection model) is used to perform object detection and image cropping on the large surveillance images. This narrows the scope and improves the accuracy of the Chinese CLIP model (i.e., the image-text matching model). Additionally, since CLIP is a basic large-scale visual model, it has stronger image feature extraction capabilities than YOLO-World. Therefore, by using the high-performance visual / image encoder in the Chinese CLIP model, it is possible to achieve retrieval of any target, behavior, or event based on any input text. Further, while the YOLO-World open-source detection model performs well in object-level detection, events and behaviors still require retrieval using the CLIP model. For example, the YOLO-World model can detect and expand the edges of a human body, while the Chinese CLIP model can retrieve targets such as "someone is smoking," "someone is fighting," and "someone is wearing a helmet." Therefore, combining the YOLO-World model and the Chinese CLIP model for image retrieval improves both retrieval effectiveness and efficiency.

[0044] S106, the first text feature is matched with the preset standard image corpus to determine the target standard small image corresponding to the first text feature; the standard image corpus includes a standard text prompt word library and multiple standard small images, and each standard text prompt word in the standard text prompt word library has a corresponding complete standard small image.

[0045] In this step, standard text prompts and corresponding standard images that are similar to any prompt text input by the user can be found in a pre-established standard image corpus, so as to carry out the next steps such as contour feature extraction and similarity comparison.

[0046] The steps for establishing a standard image corpus may include: analyzing the data used in security / surveillance scenarios and combining it with the 133 categories of the Cocoa open-source dataset, expanding the text of the 133 categories to 280 categories to obtain a text prompt dictionary A. This text prompt dictionary A is generally an English prompt dictionary. To accommodate Chinese prompts, it can be converted to Chinese to obtain a Chinese prompt dictionary B, denoted as the standard text prompt dictionary. In this embodiment, the standard text prompt dictionary is a Chinese prompt dictionary, including multiple Chinese standard text prompts (i.e., the text corresponding to the aforementioned categories). Then, various complete standard small images can be found among the candidate small images obtained during the establishment of the base corpus. Each standard text prompt is then bound to its corresponding complete standard small image to obtain the established standard image corpus. This means that each standard text prompt can correspond to one or more complete standard small images, and each standard text prompt has a corresponding complete standard small image. That is, the complete standard small image of the target can be selected from the candidate small images obtained in the surveillance scenario as the complete standard small image corresponding to the standard text prompt.

[0047] Specifically, after establishing the standard image corpus, the first text feature of any text prompt word input by the user can be entered into the standard image corpus for matching and retrieval. The standard small image matching the first text feature is then obtained and denoted as the target standard small image. The matching and retrieval method can be, for example, direct image-text matching between the first text feature and each standard small image in the standard image corpus; or it can be that the first text feature is first matched with the standard text prompt word library, and then matched with each standard small image; or other matching and retrieval methods can be used. In short, the target standard small image matching the first text feature can be retrieved.

[0048] S108, extract the contour features of the target standard small image and extract the contour features of each first candidate image, and adjust the sorting of the multiple first candidate images according to the contour features of the target standard small image and the contour features of each first candidate image to determine the multiple first candidate images sorted according to the target sorting.

[0049] In this step, after obtaining the first candidate images through the coarse search and retrieving the target standard small image from the standard image corpus, the contour features of the target in each first candidate image and the target standard small image can be extracted. This yields the contour features of the target in each first candidate image and the target in the target standard small image. Then, by comparing the similarity between the contour features of the target in the target standard small image and the contour features of the target in each first candidate image, and reordering the previously coarsely retrieved first candidate images according to the new similarity ranking (i.e., target ranking), a newly ranked (i.e., target-ranked) set of first candidate images is obtained.

[0050] As described above, the existing CLIP general model is unable to perceive the completeness of the target, which easily leads to retrieval errors due to ambiguity during image retrieval. For example, when searching for an image of only a dog's leg and an image of a complete dog using the Chinese CLIP model with the text prompt "dog," the image of only the dog's leg is likely to be ranked higher than the image of the complete dog, thus causing retrieval ambiguity. This embodiment, based on an open retrieval scheme for security / surveillance scenarios, after performing open image-text retrieval in the base database using text features, introduces standard image corpus queries and contour feature extraction and comparison, etc., which can optimize and improve the retrieval ambiguity problem caused by the lack of sensitivity to target completeness in the Chinese CLIP model during image-text retrieval, thereby improving retrieval accuracy.

[0051] In this embodiment, arbitrary text prompts input by the user are obtained, and the question features of the arbitrary text prompts are extracted using an image-text matching model to obtain the first text features. Then, based on the first text features, image retrieval is performed in the base database to determine multiple first candidate images corresponding to the arbitrary text prompts. Subsequently, the first text features are matched with a standard image corpus to determine the target standard small image corresponding to the first text features. The contour features of the target standard small image and the contour features of each first candidate image are extracted. The order of the multiple first candidate images is adjusted based on the contour features of the target standard small image and the contour features of each first candidate image to determine multiple first candidate images sorted according to the target order. The base database includes multiple candidate small images to be retrieved and a monitoring large image corresponding to each candidate small image. The standard image corpus includes a standard text prompt word database and multiple standard small images, and each standard text prompt word in the standard text prompt word database has a corresponding complete standard small image. In this method, the order of multiple images retrieved from the base database can be adjusted based on the corresponding standard thumbnails retrieved from the standard image corpus for any text prompt word input by the user. This makes the final image retrieval results more accurate, and the method does not require the development of corresponding functional modules for different functions, thus enabling accurate image retrieval for any text prompt word. At the same time, since the order of multiple images retrieved from the base database can be adjusted by comparing the contour features of the corresponding complete standard thumbnails retrieved from the standard image corpus with the contour features of multiple images retrieved from the base database, the perception of the completeness of the target can be improved, making the final sorted multiple retrieved images more consistent with the actual situation and avoiding retrieval ambiguity.

[0052] The following examples illustrate the specific process of image-text matching retrieval in a standard image corpus using first text features.

[0053] In some embodiments, S106 may include the following steps: Step A1: Extract text features from each standard text prompt word in the standard text prompt word library to obtain the second text feature corresponding to each standard text prompt word.

[0054] In this step, after establishing the standard image corpus as described above, a standard text prompt word library containing multiple standard prompt words can be obtained. Then, the text encoder of the Chinese CLIP model (i.e., image-text matching model) can be used to extract text features from each standard text prompt word to obtain the text features corresponding to each standard text prompt word, which are all recorded as the second text features.

[0055] Step A2: Calculate the text similarity between the first text feature and each of the second text features, and obtain the maximum text similarity among all text similarities.

[0056] In this step, after obtaining the first text feature corresponding to any text prompt word input by the user and the second text feature of each standard text prompt word in the standard text prompt word library, the similarity between the first text feature and each second text feature can be calculated, and each obtained similarity can be recorded as text similarity.

[0057] Then, the text similarities can be sorted from largest to smallest or smallest to largest, and the largest text similarity can be found from the sorting results and recorded as the maximum text similarity. For example, the first text similarity sorted from largest to smallest can be the maximum text similarity, or the last text similarity sorted from smallest to largest can be the maximum text similarity.

[0058] After obtaining the maximum text similarity among all text similarities, the maximum text similarity can be compared with a preset threshold. Specifically, it can be determined whether the maximum text similarity is greater than the preset threshold to obtain a judgment result. The size of the preset threshold can be set according to the actual situation.

[0059] Step A3: If the maximum text similarity is greater than a preset threshold, then obtain the standard small image corresponding to the maximum text similarity in the standard image corpus, and determine the standard small image corresponding to the maximum text similarity as the target standard small image corresponding to the first text feature.

[0060] In this step, if the judgment result is that the maximum text similarity is greater than a preset threshold, it indicates that there is text in each standard prompt word text that is quite similar to the input arbitrary prompt word text. That is, the input arbitrary prompt word text has a complete standard small image. This complete standard small image can be used to adjust the multiple first candidate images initially retrieved to avoid ambiguity caused by the incompleteness of the target. Specifically, if the maximum text similarity is greater than the preset threshold, the complete standard small image corresponding to the maximum text similarity can be found in the standard image corpus, and then it can be used as the first text feature or the target standard small image corresponding to any text prompt word.

[0061] Furthermore, if the above judgment result is that the maximum text similarity is not greater than (i.e., less than or equal to) the preset threshold, it means that there is no text in each standard prompt word text that is significantly similar to the input prompt word text. In other words, the input prompt word text may not have a complete standard small image. Therefore, no further processing is required, and the multiple first candidate images retrieved from the base database and sorted by similarity can be directly output. For example, when the user inputs a scene-related prompt word (such as "sky"), since the search content corresponding to scene-related text generally only exists in the monitoring large image and does not have a complete standard small image, after searching the base database using the first text feature corresponding to the prompt word, the result is the multiple monitoring large images (i.e., multiple first candidate images) sorted by similarity. These multiple monitoring large images can be directly output here.

[0062] In this embodiment, the similarity of text features between arbitrary prompt words input by the user and standard text prompt words in the standard image corpus is calculated, and a complete standard small image that is more similar to the arbitrary text prompt word is selected accordingly. This provides a more accurate reference image basis for subsequent contour feature matching and comparison, thereby improving the accuracy of subsequent contour feature matching and comparison.

[0063] The following examples illustrate the process of contour feature extraction and contour feature matching comparison of the target standard thumbnail and each first candidate image.

[0064] In some embodiments, the "extracting contour features of the target standard thumbnail and extracting contour features of each first candidate image" in step 108 above may include the following steps: Step B involves using a pre-defined contour feature extraction model to extract the contour features of the target standard small image and the contour features of each first candidate image, thereby obtaining the contour features corresponding to the target standard small image and the contour features corresponding to each first candidate image.

[0065] The aforementioned contour feature extraction model is trained based on multiple sample images and the contour features of each sample image. The contour features of each sample image are obtained by performing image feature extraction, clustering, and contour extraction on each sample image, and then performing contour feature extraction on each sample contour image based on the GEI gait recognition model.

[0066] First, let me explain a few technical terms: GEI, Gait Energy Image, is a data format used in gait recognition. It is obtained by averaging the binary mask images of multiple gait sequences.

[0067] HED, Holistically-Nested Edge Detection, is a deep learning-based edge detection algorithm that takes an RGB color image as input and outputs a contour image.

[0068] Secondly, the aforementioned contour feature extraction model can be a neural network model or other network models, without specific limitations. The training process of the aforementioned contour feature extraction model can include: processing existing data and open-source data in security / surveillance scenarios (such as size processing, expansion processing, etc.) to obtain multiple sample images; performing image feature extraction processing on each sample image using the image encoder of the Chinese image-text matching model to obtain the image features corresponding to each sample image; then using the InfoMap (a community detection algorithm based on information theory) clustering algorithm and parameter settings, and combining each image feature to cluster each sample image, so that sample images with high similarity are clustered into the same ID (Identity) cluster; then using the HED model to perform edge detection processing on the sample images in each category to obtain the contour images corresponding to each sample image, all of which are denoted as sample contour images; then using the GEI gait recognition model to perform contour feature extraction processing on each sample contour image to obtain the sample contour features of each sample image; finally, using each sample image and the corresponding sample contour features to train the initial contour feature extraction model to obtain the trained contour feature extraction model. It should be noted that the GEI gait recognition model is used here for contour feature extraction, which can shorten the distance between similar sample contour images, making the final trained contour feature extraction model more accurate in extracting contour features.

[0069] After the contour feature model is trained, the target standard small image and each first candidate image can be input into the trained contour feature extraction model for processing to obtain the contour features corresponding to the target standard small image and the contour features corresponding to each first candidate image.

[0070] Additionally, it should be noted that the CLIP model (i.e., image-text matching model) does not encounter the problem of incompleteness when performing image feature matching, while the problem only arises in image-text feature matching. Based on this, this embodiment can alleviate the problem of differences caused by target attributes by extracting the contour features of the target in the image, so that the standard image corpus does not need to consider all target attributes. For example, the contour features of people in white clothes and people in red clothes are the same, thereby improving the completeness of the image obtained by image-text matching.

[0071] Furthermore, after extracting the contour features, the new order of each first candidate image retrieved from the base database can be determined, and the reordering can be performed accordingly. As an optional embodiment, the above-mentioned step S108, "adjusting the order of multiple first candidate images according to the contour features of the target standard thumbnail and the contour features of each first candidate image, and determining multiple first candidate images ordered according to the target order," may include the following steps: Step C1: Calculate the feature similarity between the contour features of the target standard small image and the contour features of each first candidate image.

[0072] This involves calculating the similarity between the contour features of the target standard small image and the contour features of each first candidate image, with each similarity obtained being denoted as feature similarity.

[0073] Step C2: Obtain the first text feature and the first image-text similarity between each first candidate image.

[0074] In this step, as mentioned above, after the user inputs any prompt text, multiple first candidate images matching that prompt text can be initially filtered from the base database. Specifically, this filtering can involve calculating the first text feature of the prompt text and the first image-text similarity between each candidate image in the base database. This calculation yields the first image-text similarity between the first text feature and all candidate images in the base database. Then, these first image-text similarities can be sorted from largest to smallest. After sorting, the top-ranked candidate images or their corresponding monitoring images can be filtered out. These top-ranked candidate images or their corresponding monitoring images are the first candidate images. Simultaneously, during the filtering process, the first image-text similarity between each first candidate image and the prompt text can also be obtained.

[0075] Step C3: Average the similarity of each feature and the corresponding first image-text similarity to obtain multiple averages, and sort the averages from largest to smallest to obtain the target ranking.

[0076] In this step, the feature similarity of the contour features between any prompt text and each first candidate image, as well as the corresponding first image-text similarity, can be summed and averaged to obtain the average similarity between any prompt text and each first candidate image. Then, the average similarity values ​​can be sorted from largest to smallest to obtain the target ranking result.

[0077] Step C4: Adjust the sorting of multiple first candidate images according to the target sorting to determine the multiple first candidate images sorted according to the target sorting.

[0078] In this step, after obtaining the target sorting result, the order of each first candidate image can be adjusted according to the target sorting result, ultimately obtaining multiple first candidate images sorted according to the target sorting. For example, the top 100 search results can be re-sorted and returned, realizing the adjustment of the order of the top 100 image search results through contour information.

[0079] In this embodiment, a pre-trained contour feature extraction model is used to extract contour features from the target standard small image and each first candidate image. This alleviates the discrepancy problem caused by target attributes, ensuring that the final extracted contour features are unaffected by target attributes. Furthermore, by calculating the similarity of contour features between the target standard small image and each first candidate image and determining the new ranking, the retrieval ambiguity caused by the lack of sensitivity to target integrity in image-text matching models during image-text retrieval can be improved, thereby enhancing the accuracy and effectiveness of the final image retrieval results.

[0080] Currently, in the process of image-text retrieval, it has been found that the image retrieval results obtained by using related technologies to search for negative prompts are not good. For example, searching for images of "no fighting" often yields images of fighting, and searching for images of "no fire" often yields images of fire. Some technologies use learnable negative prompts and negative text encoders in the CLIPN (CLIP say no) model to add negative logic to CLIPN, and train the model using prompt image-text binary inverse loss and text semantic inverse loss to teach CLIPN to match images with negative prompts. However, this technology increases the negative ability of the original CLIPN model by training it, which inevitably weakens the original CLIPN model's retrieval accuracy for positive prompts. In addition, the trained CLIPN currently only supports English, while Chinese prompts are required in open retrieval. Based on this, this embodiment proposes to use a classification model and text matching to solve this problem, and the following embodiments will illustrate this.

[0081] Figure 2 This is a second schematic flowchart of the image retrieval method provided by the present invention. The method further includes the following steps: S202, input the first text feature into the preset classification model for classification to determine the target category corresponding to any text prompt word; the target category includes any text prompt word being a negative prompt word or any text prompt word not being a negative prompt word, and the classification model is trained based on the negative prompt word dataset and the non-negative prompt word dataset in the monitoring scenario.

[0082] This involves pre-training a classification model to distinguish whether input prompts are negative. This model can be a binary classification model, and its training process can include: acquiring existing text prompts and open-source text prompt datasets from current security / surveillance scenarios; selecting some prompts and fitting them into negative word templates to form a negative prompt dataset (for example, for fire prompts, adding negative word templates such as "none," "no," or "not") will yield corresponding negative prompts, such as "no fire," "no fire," or "not a fire"); and combining the constructed negative prompt dataset with the existing text prompts and open-source text prompt datasets (referred to as the non-negative prompt dataset, i.e., the positive prompt dataset) to form the training dataset for the classification model. The initial classification model is then trained to obtain the trained classification model. It's understood that the datasets participating in model training have corresponding category labels, and training can be completed using these labels. There are two categories of category labels: one for negative prompts and the other for non-negative prompts.

[0083] Additionally, it should be noted that the classification model in this embodiment is input with text features during both training and testing. This is because classification models generally process features and cannot directly identify or classify text content.

[0084] After obtaining any text prompt word input by the user and its corresponding first text feature, the first text feature can be input into the trained classification model for semantic classification processing to determine whether the arbitrary text prompt word is a negative word.

[0085] S204, if the target category is any of the above text prompt words as negative prompt words, then obtain the text similarity between the first text feature and the second text feature of each standard text prompt word in the standard text prompt word library, and according to the ranking result of each text similarity, select multiple candidate standard text prompt words with smaller text similarity from each of the above standard text prompt words.

[0086] In this step, as an optional embodiment, if the target category is any text prompt word that is not a negative prompt word, then the process returns to step 104 above, which involves "performing image retrieval in a preset database based on the first text feature to determine multiple first candidate images corresponding to the text prompt word." That is, if semantic classification determines that any text prompt word is not a negative prompt word, then steps S104-S108 above can be directly executed to obtain multiple first candidate images finally sorted according to the target order.

[0087] If the semantic classification results in a negative text prompt, then step A2 above can be used to obtain the text similarity between the first text feature and each of the second text features. The text similarities are then sorted from largest to smallest. One or more text similarities with the smallest similarity can be selected from the sorted list, for example, the last N text similarities in the sort. Based on these N smaller similarities, their corresponding standard text prompts are obtained. It can be understood that the sorting of text similarities from largest to smallest represents the degree of relevance between each standard text prompt in the standard text prompt library and any prompt text entered by the user, from highest to lowest. Based on the text similarity, N sets of standard text prompts with the lowest similarity are selected from the standard text prompt library as the matching results. These matching results can be understood as the prompts that best match / are most relevant to any text prompt entered by the user. It should be noted that although the input text prompt is a negative prompt, its meaning is actually the prompt without the negative meaning. When searching in the standard text prompt database, it will also be searched according to the semantics of the positive word with the negative meaning removed. Therefore, the N groups of standard text prompts with the smallest / smallest text similarity are selected here. In fact, the prompts selected are the prompts with the closest semantics to the input negative prompt.

[0088] The N sets of standard text prompts selected from the standard text prompt word library can all be denoted as candidate standard text prompts, where N is an integer greater than or equal to 1, such as 3, 5, 10, 20, etc. It should be noted that the parameter N can be set as large as possible to enrich the search results for negative words.

[0089] S206, calculate the second image-text similarity between the second text features of each candidate standard text prompt word and the image features corresponding to each candidate small image in the base database, and retrieve and determine multiple second candidate images corresponding to each candidate standard text prompt word in the base database based on each second image-text similarity.

[0090] In this step, for each candidate standard text prompt, it can be used as the input text prompt. Image retrieval (i.e., reverse retrieval) is performed in the base database by calculating image-text similarity. The specific method for calculating image-text similarity can be found in the retrieval process in the base database in S104 above, and will not be repeated here. In short, multiple images corresponding to each candidate standard text prompt can be retrieved from the base database. These multiple images corresponding to each candidate standard text prompt can be recorded as second candidate images. For example, the top 100 matching images corresponding to each candidate standard text prompt can be retrieved from the base database.

[0091] S208, based on the second image-text similarity of each of the above-mentioned second candidate images, select a number of second candidate images with greater second image-text similarity from the above-mentioned second candidate images and output them.

[0092] In this step, when performing image retrieval in the base database by calculating image-text similarity, the image-text similarity between each second candidate image and its corresponding input candidate standard text prompt can be calculated and denoted as the second image-text similarity. Then, the second image-text similarity of each second candidate image is used as a confidence score. The second image-text similarities are sorted from high to low / small to large. From the sorted results, the top 100 second candidate images with the highest second image-text similarity are selected for output. It can be understood that the final output of multiple second candidate images consists of the images in the base database that best match any prompt text input by the user.

[0093] For example, 1. Suppose the user inputs the text prompt "no fire". 2. The text prompt "no fire" can be input into the text encoder of the image-text matching model to obtain text features. Then, the text features are input into the classification model to determine whether the user input text prompt is a "negative" word. If not, the normal retrieval process is followed; if so, the negative word retrieval process continues. 3. Since "no fire" is a negative word, the text features of the user input text prompt can be compared with the text features of each standard text prompt in the YOLO-world Chinese standard text prompt library to calculate text similarity. The ranking of text similarity from high to low can be regarded as the standard. 4. Rank the relevance of each standard text prompt word in the quasi-text prompt word library to "fire" from high to low (currently, image-text matching models or large models cannot parse negative words such as "none" or "nothing"); 5. Select N sets of standard text prompt words with the lowest similarity from the standard text prompt words in the standard text prompt word library based on text similarity, representing those least related to "fire," such as water, sky, river, etc.; 6. Search these N sets of text prompt words in the base library to obtain results related to water, sky, river, etc.; 7. Combine all search results, sort them according to confidence, and finally return the results as the search results for the user's input "no fire."

[0094] It should be noted that all the execution results here are imperceptible to the user. The user can get search results unrelated to fire by entering "no fire".

[0095] In this embodiment, a classification model trained using a negative prompt word dataset is first used to semantically classify the prompt words input by the user. When the user input prompt words are classified as negative, the model filters out the few prompt words that do not match the standard text prompt word library the most. The model then retrieves relevant images from the base library using the few least matching prompt words and outputs them in a comprehensive manner. This can improve and solve the problems of ambiguity and poor performance of the CLIP large model (i.e., image-text matching model) in security monitoring scenarios due to negative prompt words, effectively improving the search results and enhancing their accuracy and effectiveness.

[0096] The image retrieval device provided by the present invention is described below. The image retrieval device described below can be referred to in correspondence with the image retrieval method described above.

[0097] Figure 3 This is a schematic diagram of the image retrieval device provided by the present invention. See also: Figure 3 As shown, the device may include: The text feature extraction module 310 is used to obtain any text prompt words input by the user, and to extract the text features of the arbitrary text prompt words using a preset image-text matching model to obtain the first text feature; The first candidate image retrieval module 320 is used to perform image retrieval in a preset base database based on the first text features to determine multiple first candidate images corresponding to any text prompt word; the base database includes multiple candidate small images to be retrieved and / or a monitoring large image corresponding to each candidate small image; The standard corpus matching module 330 is used to match the first text feature with a preset standard image corpus to determine the target standard small image corresponding to the first text feature; the standard image corpus includes a standard text prompt word library and multiple standard small images, and each standard text prompt word in the standard text prompt word library has a corresponding complete standard small image; The contour matching module 340 is used to extract the contour features of the target standard small image and the contour features of each first candidate image, and adjust the order of multiple first candidate images according to the contour features of the target standard small image and the contour features of each first candidate image to determine multiple first candidate images sorted according to the target order.

[0098] In some embodiments, the standard corpus matching module 330 described above is specifically used for Text features are extracted from each standard text prompt word in the standard text prompt word library to obtain the second text feature corresponding to each standard text prompt word; the text similarity between the first text feature and each second text feature is calculated, and the maximum text similarity is obtained among the text similarities; if the maximum text similarity is greater than a preset threshold, the standard small image corresponding to the maximum text similarity is obtained from the standard image corpus, and the standard small image corresponding to the maximum text similarity is determined as the target standard small image corresponding to the first text feature.

[0099] In some embodiments, the contour matching module 340 described above is specifically used for Calculate the feature similarity between the contour features of the target standard small image and the contour features of each first candidate image; obtain the first text feature and the first image-text similarity between each first candidate image; perform mean processing on each feature similarity and the corresponding first image-text similarity to obtain multiple mean values, and sort each mean value from largest to smallest to obtain the target ranking; perform corresponding sorting adjustments on multiple first candidate images according to the target ranking to determine multiple first candidate images sorted according to the target ranking.

[0100] In some embodiments, the contour matching module 340 described above is specifically used for A preset contour feature extraction model is used to extract the contour features of the target standard small image and the contour features of each first candidate image, respectively, to obtain the contour features corresponding to the target standard small image and the contour features corresponding to each first candidate image. The above contour feature extraction model is trained based on multiple sample images and the sample contour features of each sample image. The contour features of each sample image are obtained by performing image feature extraction, clustering and contour extraction on each sample image, and then performing contour feature extraction on each sample contour image based on the GEI gait recognition model.

[0101] In some embodiments, the above-described apparatus may further include a base library creation module, which is used for... Obtain multiple large monitoring images from the monitoring scene; each large monitoring image contains at least one target; use a target detection model to detect the target in each large monitoring image, determine the detection box of the target in each large monitoring image, and crop the corresponding large monitoring image according to the detection box of each target to obtain candidate small images corresponding to each target; establish a binding relationship between each candidate small image and its corresponding large monitoring image to obtain the base database.

[0102] In some embodiments, the above-described apparatus may further include: The classification module is used to input the first text features into a preset classification model for classification, and determine the target category corresponding to any text prompt word; the target category includes any text prompt word being a negative prompt word or any text prompt word not being a negative prompt word, and the classification model is trained based on a negative prompt word dataset and a non-negative prompt word dataset in the monitoring scenario; The text filtering module is used to obtain the text similarity between the first text feature and the second text feature of each standard text prompt in the standard text prompt library if the target category is any of the above text prompt words is a negative prompt word, and to filter out multiple candidate standard text prompt words with smaller text similarity from each of the above standard text prompt words according to the ranking result of each text similarity. The second candidate image retrieval module is used to calculate the second image-text similarity between the second text features of each candidate standard text prompt word and the image features corresponding to each candidate small image in the base database, and to retrieve and determine multiple second candidate images corresponding to each candidate standard text prompt word in the base database based on the second image-text similarity. The output module is used to select and output multiple second candidate images with high second image-text similarity from the aforementioned second candidate images based on the second image-text similarity of each of the aforementioned second candidate images.

[0103] Optionally, the above-mentioned device may further include a return module, which is used to return to the above-mentioned step of performing image retrieval in a preset base library based on the first text features and determining multiple first candidate images corresponding to the text prompt if the target category is any text prompt word that is not a negative prompt word.

[0104] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0105] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can call logical instructions in the memory 430 to execute an image retrieval method. This method includes: acquiring any text prompt word input by the user, and extracting text features of the arbitrary text prompt word using a preset image-text matching model to obtain a first text feature; performing image retrieval in a preset base database based on the first text feature to determine multiple first candidate images corresponding to the arbitrary text prompt word; the base database includes multiple candidate small images to be retrieved and / or a monitoring large image corresponding to each candidate small image; matching the first text feature with a preset standard image corpus to determine a target standard small image corresponding to the first text feature; the standard image corpus includes a standard text prompt word database and multiple standard small images, and each standard text prompt word in the standard text prompt word database has a corresponding complete standard small image; extracting the contour features of the target standard small image and the contour features of each first candidate image, and adjusting the order of the multiple first candidate images based on the contour features of the target standard small image and the contour features of each first candidate image to determine multiple first candidate images sorted according to the target order.

[0106] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image retrieval method provided by the above methods. The method includes: acquiring any text prompt word input by a user, and extracting the text features of the arbitrary text prompt word using a preset image-text matching model to obtain a first text feature; performing image retrieval in a preset base database based on the first text feature to determine multiple first candidate images corresponding to the arbitrary text prompt word; the base database includes multiple candidate small images to be retrieved and / or the monitoring large image corresponding to each candidate small image; match the first text feature with the preset standard image corpus to determine the target standard small image corresponding to the first text feature; the above standard image corpus includes a standard text prompt word library and multiple standard small images, and each standard text prompt word in the standard text prompt word library has a corresponding complete standard small image; extract the contour features of the target standard small image and extract the contour features of each first candidate image, and adjust the sorting of multiple first candidate images according to the contour features of the target standard small image and the contour features of each first candidate image to determine multiple first candidate images sorted according to the target sorting.

[0108] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image retrieval method provided by the above methods. The method includes: acquiring any text prompt word input by a user, and extracting text features of the arbitrary text prompt word using a preset image-text matching model to obtain a first text feature; performing image retrieval in a preset base database based on the first text feature to determine multiple first candidate images corresponding to the arbitrary text prompt word; the base database includes multiple candidate small images to be retrieved and / or a monitoring large image corresponding to each candidate small image; matching the first text feature with a preset standard image corpus to determine a target standard small image corresponding to the first text feature; the standard image corpus includes a standard text prompt word database and multiple standard small images, and each standard text prompt word in the standard text prompt word database has a corresponding complete standard small image; extracting the contour features of the target standard small image and extracting the contour features of each first candidate image, and adjusting the order of the multiple first candidate images based on the contour features of the target standard small image and the contour features of each first candidate image to determine multiple first candidate images sorted according to the target order.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image retrieval method, characterized in that, include: Get any text prompt word input by the user, and use a preset image-text matching model to extract the text features of the arbitrary text prompt word to obtain the first text feature; Based on the first text features, an image retrieval is performed in a preset database to determine multiple first candidate images corresponding to the arbitrary text prompt word; the database includes multiple candidate small images to be retrieved and / or a monitoring large image corresponding to each candidate small image; The first text feature is matched with a preset standard image corpus to determine the target standard small image corresponding to the first text feature; the standard image corpus includes a standard text prompt word library and multiple standard small images, and each standard text prompt word in the standard text prompt word library has a corresponding complete standard small image; The contour features of the target standard thumbnail and the contour features of each of the first candidate images are extracted. Based on the contour features of the target standard thumbnail and the contour features of each of the first candidate images, the order of the multiple first candidate images is adjusted to determine the multiple first candidate images sorted according to the target order.

2. The image retrieval method according to claim 1, characterized in that, The step of matching the first text feature with a preset standard image corpus to determine the target standard thumbnail corresponding to the first text feature includes: Text features are extracted from each standard text prompt word in the standard text prompt word library to obtain the second text feature corresponding to each standard text prompt word; Calculate the text similarity between the first text feature and each of the second text features, and obtain the maximum text similarity among the text similarities; If the maximum text similarity is greater than a preset threshold, then the standard small image corresponding to the maximum text similarity is obtained from the standard image corpus, and the standard small image corresponding to the maximum text similarity is determined as the target standard small image corresponding to the first text feature.

3. The image retrieval method according to claim 1, characterized in that, The step of adjusting the order of the plurality of first candidate images based on the contour features of the target standard thumbnail and the contour features of each of the first candidate images, and determining the plurality of first candidate images sorted according to the target order, includes: Calculate the feature similarity between the contour features of the target standard thumbnail and the contour features of each of the first candidate images; Obtain the first text feature and the first image-text similarity between each of the first candidate images; The similarity of each feature and the corresponding first image-text similarity are averaged to obtain multiple averages, and the averages are sorted from largest to smallest to obtain the target ranking. The plurality of first candidate images are sorted and adjusted according to the target sorting to determine the plurality of first candidate images sorted according to the target sorting.

4. The image retrieval method according to claim 1, characterized in that, Before determining the multiple first candidate images corresponding to the arbitrary text prompt word by performing image retrieval in a preset database based on the first text features, the method further includes: The first text feature is input into a preset classification model for classification to determine the target category corresponding to the arbitrary text prompt word; the target category includes whether the arbitrary text prompt word is a negative prompt word or whether the arbitrary text prompt word is not a negative prompt word, and the classification model is trained based on a negative prompt word dataset and a non-negative prompt word dataset in the monitoring scenario; If the target category is a negative prompt word, then the text similarity between the first text feature and the second text feature of each standard text prompt word in the standard text prompt word library is obtained, and according to the ranking result of each text similarity, multiple candidate standard text prompt words with small text similarity are selected from each standard text prompt word; the second image-text similarity between the second text feature of each candidate standard text prompt word and the image feature corresponding to each candidate small image in the base library is calculated, and multiple second candidate images corresponding to each candidate standard text prompt word are retrieved and determined in the base library according to each second image-text similarity; Based on the second image-text similarity of each second candidate image, multiple second candidate images with higher second image-text similarity are selected from each second candidate image and output.

5. The image retrieval method according to claim 4, characterized in that, The method further includes: If the target category is any text prompt word that is not a negative prompt word, then return to the step of performing image retrieval in a preset base database based on the first text features to determine multiple first candidate images corresponding to the arbitrary text prompt word.

6. The image retrieval method according to any one of claims 1 to 5, characterized in that, The methods for establishing the base database include: Acquire multiple large monitoring images from a monitoring scene; each large monitoring image includes at least one target. The target detection model is used to detect the targets in each of the monitoring images, determine the detection boxes of the targets in each of the monitoring images, and crop the corresponding monitoring images according to the detection boxes of each target to obtain the candidate small images corresponding to each target. The candidate small images and their corresponding monitoring large images are bound together to obtain the base database.

7. The image retrieval method according to any one of claims 1 to 3, characterized in that, The extraction of contour features from the target standard thumbnail and the extraction of contour features from each of the first candidate images include: The contour features of the target standard small image and the contour features of each first candidate image are extracted using a preset contour feature extraction model, so as to obtain the contour features corresponding to the target standard small image and the contour features corresponding to each first candidate image. The contour feature extraction model is trained based on multiple sample images and the sample contour features of each sample image. Each sample contour feature is obtained by performing image feature extraction, clustering and contour extraction on each sample image, and then performing contour feature extraction on each sample contour image based on the GEI gait recognition model.

8. An image retrieval device, characterized in that, include: The text feature extraction module is used to obtain any text prompt words input by the user, and to extract the text features of the arbitrary text prompt words using a preset image-text matching model to obtain the first text feature; The first candidate image retrieval module is used to perform image retrieval in a preset base database based on the first text features to determine multiple first candidate images corresponding to the arbitrary text prompt word; the base database includes multiple candidate small images to be retrieved and / or a monitoring large image corresponding to each candidate small image; The standard corpus matching module is used to match the first text feature with a preset standard image corpus to determine the target standard small image corresponding to the first text feature; the standard image corpus includes a standard text prompt word library and multiple standard small images, and each standard text prompt word in the standard text prompt word library has a corresponding complete standard small image; The contour matching module is used to extract the contour features of the target standard small image and the contour features of each of the first candidate images, and adjust the order of the multiple first candidate images according to the contour features of the target standard small image and the contour features of each of the first candidate images to determine the multiple first candidate images sorted according to the target order.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image retrieval method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image retrieval method as described in any one of claims 1 to 7.

Images