Picture retrieval method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a picture retrieval method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring a picture input by a user to obtain a to-be-retrieved picture; inputting the to-be-retrieved picture into a pre-trained image description model to obtain a text description corresponding to the to-be-retrieved picture; performing preliminary retrieval on a preset picture library based on the text description to obtain a plurality of candidate similar pictures; performing correlation detection on each candidate similar picture in the plurality of candidate similar pictures based on the picture to be retrieved to obtain a correlation detection result of each candidate similar picture; and determining a correlation graph of the to-be-retrieved picture from the plurality of candidate similar graphs based on a correlation detection result. In the technical scheme provided by the embodiment of the invention, the image is retrieved and analyzed from multiple dimensions through combination of text description and correlation detection, so that the retrieval accuracy is improved, the application scene is expanded, and the use experience of a user is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an image retrieval method, device, electronic device, and storage medium. Background Art

[0002] In image search engines or e-commerce platform image comparison, it is necessary to retrieve similar images of user images. Existing image retrieval methods usually rely on direct pixel comparison or feature point matching methods to determine the similarity between different images. For example, the hash values ​​of two images are calculated and the similarity between the images is determined based on the hash values; or, feature extraction algorithms are used to extract specific features of the images and the similarity between the images is determined based on the extracted features. The images with higher similarity are then fed back to the user.

[0003] However, when the user's image undergoes complex transformations (such as rotation, scaling, and translation), the above image retrieval method will not be able to accurately retrieve similar images of the user's image, resulting in low retrieval accuracy and reduced user experience. Summary of the invention

[0004] The embodiments of the present application provide an image retrieval method, device, electronic device and storage medium, which realize the image retrieval function to solve the problem of low retrieval accuracy of images after complex conversion in the prior art.

[0005] In a first aspect, an embodiment of the present application provides an image retrieval method, the method comprising:

[0006] Get the picture input by the user and get the picture to be retrieved;

[0007] Input the image to be retrieved into the pre-trained image description model to obtain the text description corresponding to the image to be retrieved;

[0008] Perform a preliminary search of the preset image library based on the text description to obtain multiple candidate similar images;

[0009] Based on the image to be retrieved, a correlation test is performed on each of the multiple candidate similar images to obtain a correlation test result for each candidate similar image;

[0010] Based on the correlation detection result, a correlation graph of the image to be retrieved is determined from multiple candidate similarity graphs.

[0011] In a second aspect, an embodiment of the present application provides an image retrieval device, the device comprising:

[0012] The acquisition module is used to obtain the picture input by the user and obtain the picture to be retrieved;

[0013] A description generation module is used to input the image to be retrieved into a pre-trained image description model to obtain the text description corresponding to the image to be retrieved;

[0014] A retrieval module is used to perform a preliminary search on a preset image library based on text descriptions to obtain multiple candidate similar images;

[0015] A detection module, used for performing a correlation detection on each of the multiple candidate similar images based on the image to be retrieved, and obtaining a correlation detection result of each candidate similar image;

[0016] The determination module is used to determine the relevant graph of the image to be retrieved from multiple candidate similar graphs based on the correlation detection result.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0018] at least one processor; and a memory communicatively coupled to the at least one processor;

[0019] Among them, the memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the image retrieval method of any embodiment of the present application.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, an image retrieval method as in any embodiment of the present application is implemented.

[0021] In an embodiment of the present application, a picture input by a user can be obtained to obtain a picture to be retrieved, and the picture to be retrieved can be input into a pre-trained image description model to obtain a text description corresponding to the picture to be retrieved, and then a preliminary search is performed on a preset picture library based on the text description to obtain multiple candidate similar images, and a correlation test is performed on each of the multiple candidate similar images based on the picture to be retrieved to obtain a correlation test result for each candidate similar image, and then a correlation image of the picture to be retrieved is determined from the multiple candidate similar images based on the correlation test result, thereby realizing the image retrieval function. Traditional direct pixel comparison or feature point matching methods often find it difficult to accurately retrieve similar images when faced with complex transformations such as rotation, scaling, and translation of images. In the above technical solution, the image to be retrieved is first converted into a text description, and then a preliminary search is performed based on the text description. Although the image to be retrieved has undergone complex transformations, the essential content of the image to be retrieved can still be captured and compared in text form, thereby ignoring the impact of the complex geometric transformation of the image on the retrieval to a certain extent. Then, the preliminary retrieved candidate similar images are further subjected to correlation detection, which can ensure that the final related image is highly matched with the image to be retrieved in terms of content and features, thereby improving the retrieval accuracy and solving the problem of low retrieval accuracy of images after complex transformations in the prior art. Through the combination of text description and correlation detection, images can be retrieved and analyzed from multiple dimensions, thereby improving the accuracy of the retrieval. At the same time, it can be applied to image retrieval in various complex scenarios, expanding the application scenarios, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a flowchart of the image retrieval method provided in the embodiment of the present application;

[0024] Figure 2 is another flowchart of the image retrieval method provided in an embodiment of the present application;

[0025] Figure 3 is a structural schematic diagram of an image retrieval device provided in an embodiment of the present application;

[0026] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0028] It should be noted that the terms "first", "second", "target", and "original" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having", and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0029] Figure 1 This is a flowchart of the image retrieval method provided by an embodiment of the present application. This embodiment can be applied to scenarios where it is necessary to retrieve images that are similar or related to the content of the user's images. The image retrieval method provided by this embodiment can be executed by the image retrieval device provided by an embodiment of the present application, and the device can be implemented by software and / or hardware. In a specific embodiment, the image retrieval device can be integrated into an electronic device, and the electronic device can be a computer or a smart phone. The execution subject of this method can be an electronic device, see Figure 1 The image retrieval method of this embodiment includes but is not limited to the following steps:

[0030] S110: Acquire the picture input by the user to obtain the picture to be retrieved.

[0031] The image to be retrieved is a specific image input by the user and needs to search for similar images in a specific image library.

[0032] Specifically, the electronic device can pre-create a graphical user interface or web page interface to obtain a picture upload interface so that users can upload pictures. At this time, the picture upload interface can include a file selection button, a drag and drop area, or a camera access option to meet different user input methods.

[0033] When a user wants to upload a picture, the user can upload a picture in the picture upload interface of the electronic device by uploading a file, dragging or taking a photo. The picture at this time may be a picture after complex transformation (such as rotation, scaling and translation, etc.), or it may be a picture that has not been processed in any way.

[0034] Afterwards, the electronic device can detect the user's operation of uploading a picture, and determine the picture uploaded by the user as the picture to be retrieved.

[0035] S120: Input the image to be retrieved into a pre-trained image description model to obtain a text description corresponding to the image to be retrieved.

[0036] The image description model is a pre-trained neural network model that is used to understand the content of an image and convert it into a natural language description, similar to the ability of humans to describe the content of a painting or a photo in words. For example, the image description model may be a CLIP (Contrastive Language-Image Pre-training) model.

[0037] The text description is the output text of the image description model, that is, the image description model converts the content of the image to be retrieved into human-readable natural language text and outputs the natural language text.

[0038] Specifically, after obtaining the picture to be retrieved, a pre-trained image description model can be obtained, and the picture to be retrieved can be input into the pre-trained image description model. At this time, the image description model can extract the visual features of the picture to be retrieved through a multi-layer neural network, and based on the visual features, generate a piece of text corresponding to the content of the picture to be retrieved. After that, the electronic device can determine the output text of the image description as the text description corresponding to the picture to be retrieved.

[0039] S130: Perform a preliminary search on a preset image library based on the text description to obtain multiple candidate similar images.

[0040] The preset picture library is a pre-prepared database containing a large number of pictures. Exemplarily, the preset picture library can be a collection of pictures in a specific field or a general picture resource library.

[0041] Optionally, the preset image library may include multiple candidate images and an image description corresponding to each candidate image.

[0042] The candidate similar images are a group of images initially retrieved from a preset image library based on the text description of the image to be retrieved.

[0043] Specifically, after obtaining the text description corresponding to the image to be retrieved, a preliminary search can be performed on the preset image library based on the text description to obtain multiple candidate similar images. For example, the text description can be parsed, and key words, concepts, and themes can be extracted from the text description to obtain keywords. The keywords in the text description are then matched with the image descriptions in the preset image library. That is, for the current candidate image among multiple candidate images, if the image description of the current candidate image includes the keywords of the text description, the current candidate image can be determined as a candidate similar image, thereby obtaining multiple candidate similar images.

[0044] S140 , performing a correlation test on each of the multiple candidate similar graphs based on the image to be retrieved, and obtaining a correlation test result for each candidate similar graph.

[0045] Among them, the correlation detection result is the result obtained after further detection of each candidate similar image in the candidate similar images. Optionally, the correlation detection result may include the existence of geometric transformation and the existence of time correlation; the existence of geometric transformation indicates that there are geometric transformations such as rotation, scaling and translation between the image to be retrieved and the candidate similar image, and the candidate similar image at this time is the transformation image or original image of the image to be retrieved; the existence of time correlation indicates that there is a temporal correlation between the image to be retrieved and the candidate similar image, and the candidate similar image at this time is the preceding image or the succeeding image of the image to be retrieved in the time series.

[0046] Specifically, after obtaining multiple candidate similar images, a correlation test can be performed on each of the multiple candidate similar images based on the image to be retrieved to obtain a correlation test result for each candidate similar image. For example, since the image to be retrieved input by the user may be an image that has undergone complex transformations, a geometric transformation test can be performed on each candidate similar image based on the image to be retrieved, and it is determined whether there is a geometric transformation between the image to be retrieved and each candidate similar image, so as to perform a correlation test, and then the correlation test result of each candidate similar image is determined based on the geometric transformation test result, such as whether there is a geometric transformation or not; or, a time correlation test can be performed on each candidate similar image based on the image to be retrieved, and it is determined whether there is a time correlation between the image to be retrieved and each candidate similar image, so as to perform a correlation test, and then the correlation test result of each candidate similar image is determined based on the time correlation test result, such as whether there is a time correlation or not.

[0047] S150: Determine a correlation graph of the image to be retrieved from multiple candidate similarity graphs based on the correlation detection result.

[0048] Among them, the related graph is a set of images that are most similar or most relevant to the image to be retrieved among multiple candidate similar graphs; the related graph has a high similarity with the image to be retrieved in terms of content, theme and style; exemplarily, the related graph is the original image or transformed image of the image to be retrieved, or the related graph is the preceding or succeeding graph of the image to be retrieved in the time series.

[0049] Specifically, after obtaining the correlation detection result of each candidate similarity graph, the correlation graph of the image to be retrieved can be determined from multiple candidate similarity graphs based on the correlation detection result. For example, if the correlation detection result shows that there is a geometric transformation, the candidate similarity graph corresponding to the correlation detection result can be determined as the correlation graph, and the correlation graph at this time is the original image or the transformed graph of the image to be retrieved; if the correlation detection result shows that there is a time correlation, the candidate similarity graph corresponding to the correlation detection result can be determined as the correlation graph, and the correlation graph at this time is the preceding graph or the succeeding graph of the image to be retrieved in the time series.

[0050] The technical solution of the embodiment of the present application can obtain the picture input by the user, obtain the picture to be retrieved, and input the picture to be retrieved into a pre-trained image description model to obtain the text description corresponding to the picture to be retrieved, and then perform a preliminary search on the preset picture library based on the text description to obtain multiple candidate similar images, and perform a correlation test on each of the multiple candidate similar images based on the picture to be retrieved to obtain the correlation test result of each candidate similar image, and then determine the correlation image of the picture to be retrieved from the multiple candidate similar images based on the correlation test result, thereby realizing the picture retrieval function. Traditional direct pixel comparison or feature point matching methods often find it difficult to accurately retrieve similar images when faced with complex transformations such as rotation, scaling, and translation of images. In the above technical solution, the image to be retrieved is first converted into a text description, and then a preliminary search is performed based on the text description. Although the image to be retrieved has undergone complex transformations, the essential content of the image to be retrieved can still be captured and compared in text form, thereby ignoring the impact of the complex geometric transformation of the image on the retrieval to a certain extent. Then, the preliminary retrieved candidate similar images are further subjected to correlation detection, which can ensure that the final related image is highly matched with the image to be retrieved in terms of content and features, thereby improving the retrieval accuracy and solving the problem of low retrieval accuracy of images after complex transformations in the prior art. Through the combination of text description and correlation detection, images can be retrieved and analyzed from multiple dimensions, thereby improving the accuracy of the retrieval. At the same time, it can be applied to image retrieval in various complex scenarios, expanding the application scenarios, thereby improving the user experience.

[0051] The following further describes an image retrieval method provided by an embodiment of the present application. Figure 2is another flow chart of the image retrieval method provided by the embodiment of the present application. The embodiment of the present application is optimized based on the above embodiments. Figure 2 The method of this embodiment includes but is not limited to the following steps:

[0052] S201. Obtain the picture input by the user to obtain the picture to be searched.

[0053] S202: Input the image to be retrieved into a pre-trained image description model to obtain a text description corresponding to the image to be retrieved.

[0054] Optionally, the training process of the image description model is as follows: obtain multiple sample images and a sample text description corresponding to each sample image; input the sample image into the image encoder of the initial image description model to obtain an image feature vector, and input the corresponding sample text description into the text encoder of the initial image description model to obtain a text feature vector; calculate the similarity between the image feature vector and the text feature vector, and train and optimize the initial image description model with the goal of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs to obtain an image description model.

[0055] The sample images are image data used to train the image description model. The sample text description is the text content corresponding to the sample image, which is a textual description of the content, subject, object, action and other features contained in the sample image. It should be noted that the sample images and the sample text descriptions corresponding to the sample images are obtained from the Internet, and there is no need to manually annotate the text descriptions.

[0056] The initial image description model is an image description model that has been built before the training process begins but has not yet been optimized by any training data; the initial image description model may include an image encoder and a text encoder; the image encoder is used to convert the input image into an image feature vector and extract various visual features in the image; the text encoder is used to convert the input text into a text feature vector and capture the semantic information of the text.

[0057] A positive sample pair is a sample image and a sample text description that match each other during the training process, that is, the sample text description can accurately reflect the content of the sample image. A negative sample pair is a combination of sample images and sample text descriptions that do not match each other during the training process.

[0058] Specifically, positive sample pairs and negative sample pairs can be constructed, that is, sample images and sample text descriptions that match the sample images form positive sample pairs, and sample images and sample text descriptions that do not match the sample images form negative sample pairs. Secondly, the positive sample pairs and negative sample pairs are formed into training data sets, and the initial image description model is trained and optimized based on the training data sets. That is, the sample images can be input into an image encoder to obtain image feature vectors, and the corresponding sample text descriptions can be input into a text encoder to obtain text feature vectors. The image feature vectors and text feature vectors are then mapped into a unified feature space for comparative learning. The similarity between the image feature vectors and the text feature vectors is then calculated, and the loss of the initial image description model is calculated based on the contrast loss function. The purpose of the contrast loss function is to maximize the similarity of the positive sample pairs and minimize the similarity of the negative sample pairs. After that, the parameters of the initial image description model are updated through the back propagation algorithm to minimize the loss value to obtain the optimal initial image description model, that is, the image description model.

[0059] In an embodiment of the present application, by simultaneously processing data in both image and text modalities, the image description model can learn how to align image content with text descriptions, which helps the image description model to better understand objects and scenes in the image and the relationships between them, thereby improving the generation accuracy of the image description model and providing data support for subsequent preliminary retrieval.

[0060] S203: Perform a preliminary search on a preset image library based on the text description to obtain multiple candidate similar images.

[0061] Specifically, for the current candidate picture among the multiple candidate pictures, the similarity between the text description and the picture description corresponding to the current candidate picture can be calculated to obtain the text similarity, and when the text similarity is greater than the second preset threshold, the current candidate picture is determined to be a candidate similar picture. The text similarity is used to measure the similarity between the text description and the picture description corresponding to the current candidate picture in terms of semantic content; the higher the text similarity, the more similar the picture description is to the text description; the second preset threshold is a pre-set value, which is used to measure whether the similarity between the text description and the picture description corresponding to the candidate picture is sufficient to be determined as similar.

[0062] Specifically, any candidate image can be selected as the current candidate image, and then the similarity between the text description and the image description corresponding to the current candidate image can be calculated, that is, the text description can be vectorized to obtain a text description vector, and the image description corresponding to the current candidate image can be vectorized to obtain a picture description vector, and the vectorization method can be a bag-of-words model or a TF-IDF (Term Frequency-Inverse Document Frequency) method, etc.; then, the cosine similarity between the text description vector and the picture description vector is calculated to obtain the text similarity, and when the text similarity is greater than a second preset threshold, the current candidate image is determined as a candidate similar image; then, other candidate images are selected as the current candidate images, and the above process is repeated to determine whether each candidate image in the preset image library is a candidate similar image one by one, thereby screening out a set of images that are relatively similar to the image to be retrieved in terms of text description.

[0063] Afterwards, when it is necessary to retrieve the original image or the transformed image of the image to be retrieved, S204 may be executed; when it is necessary to retrieve the preceding image or the succeeding image of the image to be retrieved in the time sequence, S210 may be executed.

[0064] Optionally, a transformation-related graph retrieval control and a time-related graph retrieval control may be set, and when it is detected that the user triggers the transformation-related graph retrieval control, S204 is executed; when it is detected that the user triggers the time-related graph retrieval control, S210 is executed.

[0065] S204: for a current candidate similar graph among the multiple candidate similar graphs, determine an affine transformation matrix between the image to be retrieved and the current candidate similar graph.

[0066] Among them, the affine transformation matrix is ​​used to describe the affine transformation relationship between images, and can represent geometric transformation operations such as rotation, scaling, translation, and shearing.

[0067] Specifically, determine the affine transformation matrix between the image to be retrieved and the current candidate similarity image, including Sa1-Sa3:

[0068] Sa1. Perform feature point detection on the image to be retrieved to obtain the first feature point of the image to be retrieved, and perform feature point detection on the current candidate similar image to obtain the second feature point of the current candidate similar image.

[0069] Among them, the first feature point is a key point extracted from the image to be retrieved through a feature point detection algorithm, such as a corner point (i.e., the intersection of two edges) and a local extreme point on the edge; and the first feature point is invariant to complex transformations such as rotation, scaling, and translation of the image.

[0070] The second feature point is a key point extracted from the current candidate similarity graph by a feature point detection algorithm, and the second feature point is invariant to complex transformations such as rotation, scaling and translation of the image.

[0071] Specifically, a feature point detection algorithm can be used to perform feature point detection on the image to be retrieved and the current candidate similar image respectively. For example, a scale-invariant feature transform (SIFT) algorithm can be used to perform feature point detection on the image to be retrieved and the current candidate similar image respectively to obtain the first feature point of the image to be retrieved and the second feature point of the current candidate similar image.

[0072] Specifically, the image to be retrieved can be Gaussian blurred at different scales to obtain a series of images of different scales, and the difference between adjacent scale images can be calculated to construct a Gaussian difference pyramid. Then, in the Gaussian difference pyramid, local extreme points are detected, and then the detected extreme points are accurately located by fitting a three-dimensional quadratic function, and low-contrast extreme points and unstable edge response points are removed to obtain the first feature points to ensure that the first feature points have strong representativeness and invariance. At this time, there are multiple first feature points. After that, the above processing flow can be repeated to perform feature point detection on the current candidate similarity graph to obtain the second feature points. At this time, there are multiple second feature points.

[0073] After obtaining multiple first feature points and multiple second feature points, the direction of each first feature point in the multiple first feature points can be determined, that is, for the current first feature point in the multiple first feature points, a circular area with a specific radius is taken with the current first feature point as the center, and the gradient amplitude and direction of each pixel point in the circular area are calculated, and then the 360-degree direction range is divided into several direction intervals, and the pixel points are stored in the corresponding direction interval based on the direction of the pixel points, and then the sum of the gradient amplitudes of all pixels in each direction interval is counted, and the center direction corresponding to the direction interval of the maximum gradient amplitude sum is determined as the main direction of the current first feature point, so that the current first feature point has rotation invariance. After that, the above processing flow can be repeated to determine the direction of each second feature point in the multiple second feature points.

[0074] Sa2. Match the first feature point and the second feature point to obtain multiple matching point pairs.

[0075] The matching point pair is a pair of points consisting of a first feature point and a second feature point corresponding thereto.

[0076] Specifically, after obtaining the first feature point and the second feature point, a feature descriptor can be generated for each of the multiple first feature points. For example, for the current first feature point among the multiple first feature points, a neighborhood window of a fixed size (such as 16×16 pixels) is taken as the center with the current first feature point, and the neighborhood is divided into smaller sub-regions (such as 4×4 sub-regions). Then, a gradient direction histogram (such as a histogram in 8 directions) is calculated in each sub-region. At this time, each sub-region can be represented by a vector (such as an 8-dimensional vector), and then the vectors of all sub-regions are combined to obtain a feature descriptor (such as 128 dimensions), that is, the feature descriptor of the current first feature point. At this time, the feature descriptor is a vector, which can describe local features such as texture and shape around the current first feature point.

[0077] Afterwards, the above processing flow may be repeated to generate a feature descriptor for each of the plurality of second feature points.

[0078] Afterwards, the first feature point and the second feature point can be matched, that is, for the current first feature point among the multiple first feature points, the distance (such as Euclidean distance) between the feature descriptor of the current first feature point and the feature descriptors of all the second feature points can be calculated respectively, and when the distance is less than a preset distance threshold, the current first feature point and the corresponding second feature point are determined as a matching point pair. The preset distance threshold is a preset value used to determine whether the first feature point matches the second feature point.

[0079] Sa3. Determine the affine transformation matrix based on multiple matching point pairs.

[0080] Specifically, after obtaining multiple matching point pairs, the affine transformation matrix can be determined based on the multiple matching point pairs, that is, any matching point pair can be selected as the current matching point pair, and the coordinates of the two matching points in the current matching point pair are substituted into the affine transformation formula to obtain two affine transformation equations, and then other matching point pairs are selected as the current matching point pair, and the substitution process is repeated to obtain multiple affine transformation equations. At this time, the multiple affine transformation equations can be constructed as a linear equation group, and then the least squares method is used to solve the linear equation group to obtain the coefficients in the affine transformation equation, and these coefficients are the elements of the affine transformation matrix, thereby determining the elements in the affine transformation matrix to obtain the affine transformation matrix.

[0081] In an embodiment of the present application, by extracting invariant feature points, the influence of noise can be reduced, thereby improving noise resistance. Then, the invariant feature points are matched, which improves the matching accuracy, thereby improving the accuracy and efficiency of determining the affine transformation matrix, and providing data support for the subsequent determination of candidate transformation maps.

[0082] S205 . Transform the current candidate similarity graph based on the affine transformation matrix to obtain a candidate transformation graph.

[0083] Among them, the candidate transformation image is an image obtained by transforming the current candidate similarity image through an affine transformation matrix, so that the current candidate similarity image better meets the requirements for similarity comparison with the image to be retrieved in terms of shape and position.

[0084] Specifically, after obtaining the affine transformation matrix, the coordinates of each pixel in the current candidate similarity image can be transformed by the affine transformation matrix, that is, the product between the affine transformation matrix and the transposed matrix of the coordinates of each pixel can be calculated to obtain the transformed coordinates of each pixel (referred to as transformed coordinates).

[0085] Since the affine transformation matrix may transform the pixel coordinates of the image into non-integers, the transformed coordinates of each pixel point can be interpolated (such as bilinear interpolation) to obtain the transformed pixel value of each pixel point.

[0086] After transforming and interpolating all pixels of the current candidate similarity image, a new image, namely the candidate transformed image, can be obtained. At this time, compared with the current candidate similarity image, the candidate transformed image is closer to the geometric features of the image to be retrieved in terms of geometric shape and position.

[0087] S206: Calculate the similarity between the candidate transformation graph and the image to be retrieved to obtain the image similarity.

[0088] Among them, the image similarity is used to measure the similarity between the candidate transformation image and the image to be retrieved in terms of content, structure, color, etc.; the higher the image similarity, the more similar the candidate transformation image is to the image to be retrieved.

[0089] Specifically, after obtaining the candidate transformation graph corresponding to the current candidate similarity graph, the similarity between the candidate transformation graph and the image to be retrieved can be calculated. For example, the structural similarity (SSIM) between the candidate transformation graph and the image to be retrieved can be calculated to obtain the image similarity.

[0090] S207: Determine whether the image similarity is greater than a first preset threshold.

[0091] The first preset threshold is a pre-set value used to determine whether the similarity between the candidate transformation graph and the image to be retrieved reaches a level sufficient to determine that a geometric transformation exists.

[0092] Specifically, after the picture similarity is obtained, the picture similarity may be compared with a first preset threshold. If the picture similarity is greater than the first preset threshold, S208 may be executed; otherwise, S209 may be executed.

[0093] S208: When the image similarity is greater than a first preset threshold, determine that a correlation detection result of the current candidate similar image is that a geometric transformation exists.

[0094] Specifically, when the image similarity is greater than the first preset threshold, it indicates that the candidate transformation graph and the image to be retrieved have high similarity in geometric features and content, that is, there is an obvious geometric transformation relationship between the current candidate similarity graph and the image to be retrieved, that is, the current candidate similarity graph is the original image or the transformation graph of the image to be retrieved. At this time, it can be determined that the correlation detection result of the current candidate similarity graph is that there is a geometric transformation, and then S214 can be executed. By comparing the candidate transformation graph and the image to be retrieved, their content similarity after geometric adjustment can be more accurately evaluated, avoiding the similarity calculation deviation caused by the original geometric difference.

[0095] Optionally, by comparing the coordinates and directions of the matching points in each matching point pair, the type of geometric transformation between the image to be retrieved and the current candidate similarity image can be determined. For example, if the matching points have a systematic angular offset in direction, the type of geometric transformation can be determined to be rotation; if the distance between the matching points is proportionally reduced or enlarged, the type of geometric transformation can be determined to be scaling; if the direction of the matching points is completely flipped (for example, left to right or up to down), the type of geometric transformation can be determined to be flipping.

[0096] S209: When the image similarity is not greater than a first preset threshold, determine that the correlation detection result of the current candidate similar image is that there is no geometric transformation.

[0097] Specifically, when the image similarity is not greater than the first preset threshold, it indicates that the candidate transformation graph and the image to be retrieved have weak similarity in geometric features and content, that is, there is no obvious geometric transformation relationship between the current candidate similarity graph and the image to be retrieved, that is, the current candidate similarity graph is not the original image or transformation graph of the image to be retrieved. At this time, it can be determined that the correlation detection result of the current candidate similarity graph is that there is no geometric transformation, and then S214 can be executed.

[0098] It should be noted that after determining that the correlation detection result of the current candidate similarity graph is that there is a geometric transformation or there is no geometric transformation, other candidate similarity graphs can be selected as the current candidate similarity graphs, and S204 to S209 can be repeatedly executed to determine the correlation detection results of each candidate similarity graph in multiple candidate similarity graphs.

[0099] S210 , determining an optical flow field between the image to be retrieved and the current candidate similarity graph among the multiple candidate similarity graphs.

[0100] Among them, the optical flow field is a two-dimensional vector field that describes the movement of pixels in two pictures, that is, it represents the movement direction and amplitude of each pixel from the first picture to the second picture. Between video sequences or continuous image frames, due to factors such as the movement of objects and the movement of the camera, the pixels in the picture will change position. At this time, the optical flow field can be used to represent the movement information of these pixels, such as the displacement, speed and direction of movement.

[0101] Specifically, after obtaining multiple candidate similarity images, any candidate similarity image can be selected as the current candidate similarity image, and the optical flow field between the image to be retrieved and the current candidate similarity image can be determined, that is, a suitable optical flow calculation algorithm can be selected, such as a local optical flow calculation algorithm or a global optical flow calculation algorithm, and then the optical flow field matrix is ​​initialized, whose size is the same as the dimension of the image to be retrieved and the dimension of the current candidate similarity image, and the elements in the optical flow field matrix are initialized to zero, and then according to the selected optical flow calculation algorithm, the optical flow vector of each pixel point is calculated pixel by pixel (that is, the motion vector of the pixel point from the image to be retrieved to the current candidate similarity image), and the calculated optical flow vector is stored in the corresponding position of the optical flow field matrix to obtain the optical flow field between the image to be retrieved and the current candidate similarity image.

[0102] S211 , determining whether the motion information included in the optical flow field is continuous and consistent.

[0103] Specifically, after obtaining the optical flow field, for each pixel in the optical flow field, any pixel can be selected as the current pixel, and the difference between the optical flow vector of the current pixel and the optical flow vector of the adjacent pixel is calculated. At this time, there are multiple adjacent pixels. Then, when all the aforementioned differences are less than the preset continuous threshold, it can be determined that the optical flow of the current pixel is continuous, and then other pixels are selected as the current pixel, and the above process is repeated to determine whether the optical flow of each pixel in the optical flow field is continuous, and the number of pixels with continuous optical flow is counted, and then the ratio between the number of pixels with continuous optical flow and the total number of pixels is determined, and when the ratio is greater than the first preset ratio threshold, it can be determined that the motion information included in the optical flow field is continuous. Among them, the preset continuous threshold is a pre-set value used to determine whether the optical flow of the pixel is continuous. The first preset ratio threshold is a pre-set value used to determine whether the motion information included in the optical flow field is continuous.

[0104] Afterwards, for each pixel in the optical flow field, the direction and size of the optical flow vector of each pixel are calculated. If the ratio between the number of pixels with the same direction and the total number of pixels is greater than the second preset ratio threshold, it can be determined that the direction of movement of the optical flow field is consistent; and if the size of the optical flow vector of each pixel is within a preset interval, it can be determined that the size of movement of the optical flow field is consistent. The preset interval at this time is a value pre-set according to the average value of the sizes of multiple optical flow vectors in the optical flow field, which is used to determine whether the size of movement (i.e., the amplitude of movement) of the optical flow field is consistent. Among them, the second preset ratio threshold is a pre-set value, which is used to determine whether the direction of movement of the optical flow field is consistent.

[0105] If the motion information included in the optical flow field is continuous, the motion direction of the optical flow field is consistent, and the motion magnitude of the optical flow field is consistent, it can be determined that the motion information included in the optical flow field is continuous and consistent, and S212 can be executed at this time; otherwise, it can be determined that the motion information included in the optical flow field is not continuous and consistent, and S213 can be executed at this time.

[0106] S212: When the motion information included in the optical flow field is continuous and consistent, determine that the correlation detection result of the current candidate similarity graph is that there is a time correlation.

[0107] Specifically, when the motion information included in the optical flow field is continuous and consistent, it can be indicated that the current candidate similarity graph and the image to be retrieved are continuous in time series, that is, there is a time correlation between the current candidate similarity graph and the image to be retrieved, that is, the current candidate similarity graph is a preceding graph or a succeeding graph of the image to be retrieved in time series, and at this time, it can be determined that the correlation detection result of the current candidate similarity graph is that there is a time correlation. Afterwards, S214 can be executed.

[0108] S213: When the motion information included in the optical flow field is not continuous and consistent, determine that the correlation detection result of the current candidate similarity graph is that there is no time correlation.

[0109] Specifically, when the motion information included in the optical flow field is not continuous and consistent, it can be indicated that the current candidate similarity graph and the image to be retrieved are not continuous in time sequence, that is, there is no time correlation between the current candidate similarity graph and the image to be retrieved, that is, the current candidate similarity graph is not a preceding graph or a succeeding graph of the image to be retrieved in time sequence, and at this time, it can be determined that the correlation detection result of the current candidate similarity graph is that there is no time correlation. Afterwards, S214 can be executed.

[0110] It should be noted that after determining that the correlation detection result of the current candidate similarity graph is that there is time correlation or there is no time correlation, other candidate similarity graphs can be selected as the current candidate similarity graphs, and S210 to S213 can be repeatedly executed to determine the correlation detection results of each candidate similarity graph in multiple candidate similarity graphs.

[0111] It should be noted that when the original image or the transformed image of the image to be retrieved needs to be retrieved, the branch from S204 to S209 is executed; when the preceding image or the following image of the image to be retrieved in the time sequence needs to be retrieved, the branch from S210 to S213 is executed. Among them, there is no obvious order of execution between S204 to S209 and S210 to S213, and the execution order can be determined according to the actual situation, and this embodiment does not make specific restrictions on this.

[0112] S214: Determine a correlation graph of the image to be retrieved from multiple candidate similarity graphs based on the correlation detection result.

[0113] Specifically, when the correlation detection result shows that there is a geometric transformation or a time correlation, the corresponding candidate similar graph is determined as a related graph, which can more accurately find pictures similar to the picture to be retrieved, thereby improving the completeness and accuracy of the retrieval results, thereby improving the user experience.

[0114] For example, on an e-commerce platform, users may want to find display pictures of the same product from different angles (i.e., geometric transformation), or pictures of the same product taken at different times (i.e., time correlation). Correlation graphs can better meet user needs, provide results that are more in line with user expectations, and improve user satisfaction.

[0115] S215. Display relevant graphs.

[0116] Specifically, after the relevant graph is obtained, the retrieved relevant graph can be displayed on a display screen of the electronic device to facilitate user viewing, thereby improving the user's usage experience.

[0117] The technical solution of the embodiment of the present application can obtain the picture input by the user to obtain the picture to be retrieved, and input the picture to be retrieved into a pre-trained image description model to obtain the text description corresponding to the picture to be retrieved, and then perform a preliminary search on the preset picture library based on the text description to obtain multiple candidate similar images, and then determine the affine transformation matrix between the picture to be retrieved and the current candidate similar image for the current candidate similar image among the multiple candidate similar images, and transform the current candidate similar image based on the affine transformation matrix to obtain the candidate transformation image, and then calculate the similarity between the candidate transformation image and the picture to be retrieved to obtain the picture similarity, and when the picture similarity is greater than a first preset threshold, determine that the correlation detection result of the current candidate similar image is that there is a geometric transformation, by determining the affine transformation matrix and performing the corresponding transformation, the influence of the geometric transformation can be eliminated, and then the similarity between the pictures can be more accurately measured, and the determination accuracy and efficiency of the correlation detection results are improved, and data support is provided for the subsequent determination of the related image, thereby improving the accuracy of the retrieval and meeting the needs of users who want to retrieve the original image or the transformed image.

[0118] In addition, for the current candidate similar image among multiple candidate similar images, the optical flow field between the image to be retrieved and the current candidate similar image can be determined, and when the motion information included in the optical flow field is continuous and consistent, the correlation detection result of the current candidate similar image is determined to be that there is time correlation. By determining the optical flow field and analyzing the continuity and consistency of its motion information, the time correlation between images can be accurately captured, providing additional basis for similarity judgment, thereby improving the accuracy and efficiency of the correlation detection results, and providing data support for the subsequent determination of related images, thereby improving the accuracy of retrieval and meeting the needs of users who want to retrieve the preceding or succeeding images of the image.

[0119] Afterwards, based on the correlation detection results, the relevant graph of the image to be retrieved is determined from multiple candidate similar graphs and displayed, which improves the efficiency and accuracy of determining the relevant graph, thereby improving the retrieval accuracy in complex scenarios and meeting the various needs of users. At the same time, it can be applied to image retrieval in various complex scenarios, expanding the application scenarios and thus improving the user experience.

[0120] Figure 3 is a schematic diagram of the structure of the image retrieval device provided in the embodiment of the present application, referring to Figure 3 , the image retrieval device may include:

[0121] The acquisition module 310 is used to acquire the picture input by the user to obtain the picture to be retrieved;

[0122] Description generation module 320, used to input the image to be retrieved into a pre-trained image description model to obtain a text description corresponding to the image to be retrieved;

[0123] A retrieval module 330 is used to perform a preliminary search on a preset image library based on the text description to obtain a plurality of candidate similar images;

[0124] A detection module 340 is used to perform a correlation detection on each of the multiple candidate similar images based on the image to be retrieved, and obtain a correlation detection result of each candidate similar image;

[0125] The determination module 350 is used to determine the correlation graph of the image to be retrieved from multiple candidate similarity graphs based on the correlation detection result.

[0126] In one embodiment, the detection module 340 is specifically used to: determine the affine transformation matrix between the image to be retrieved and the current candidate similarity image for a current candidate similarity image among multiple candidate similarity images; transform the current candidate similarity image based on the affine transformation matrix to obtain a candidate transformation image; calculate the similarity between the candidate transformation image and the image to be retrieved to obtain the image similarity; when the image similarity is greater than a first preset threshold, determine that the correlation detection result of the current candidate similarity image is that there is a geometric transformation.

[0127] In one embodiment, the detection module 340 determines the affine transformation matrix between the image to be retrieved and the current candidate similarity image, including: performing feature point detection on the image to be retrieved to obtain the first feature point of the image to be retrieved, and performing feature point detection on the current candidate similarity image to obtain the second feature point of the current candidate similarity image; matching the first feature point and the second feature point to obtain multiple matching point pairs; determining the affine transformation matrix based on the multiple matching point pairs.

[0128] In one embodiment, the detection module 340 is specifically used to: determine the optical flow field between the image to be retrieved and the current candidate similarity image among multiple candidate similarity images; when the motion information included in the optical flow field is continuous and consistent, determine that the correlation detection result of the current candidate similarity image is that there is time correlation.

[0129] In one embodiment, the preset image library includes multiple candidate images and an image description corresponding to each candidate image. The retrieval module 330 is specifically used to: for a current candidate image among the multiple candidate images, calculate the similarity between the text description and the image description corresponding to the current candidate image to obtain the text similarity; when the text similarity is greater than a second preset threshold, determine that the current candidate image is a candidate similar image.

[0130] In one embodiment, the training process of the image description model in the description generation module 320 is as follows: obtain multiple sample images and a sample text description corresponding to each sample image; input the sample image into the image encoder of the initial image description model to obtain an image feature vector, and input the corresponding sample text description into the text encoder of the initial image description model to obtain a text feature vector; calculate the similarity between the image feature vector and the text feature vector, and train and optimize the initial image description model with the goal of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs to obtain an image description model.

[0131] In one embodiment, the determination module 350 is specifically used to: when the correlation detection result is that there is a geometric transformation or there is a time correlation, determine the corresponding candidate similarity graph as a correlation graph;

[0132] The image retrieval device also includes a display module, which is specifically used to: after determining a related graph of the image to be retrieved from multiple candidate similar graphs based on the correlation detection result, display the related graph.

[0133] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0134] The image retrieval device provided in this embodiment can be applied to the image retrieval method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0135] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. Figure 4 A block diagram of an exemplary electronic device 11 suitable for implementing the embodiments of the present application is shown. Figure 4 The electronic device 11 shown is only an example and should not bring any limitation to the function and scope of use of the present embodiment.

[0136] like Figure 4 As shown, the electronic device 11 is in the form of a general purpose computing electronic device. The components of the electronic device 11 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0137] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0138] The electronic device 11 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 11, including volatile and non-volatile media, removable and non-removable media.

[0139] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0140] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0141] The electronic device 11 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 11, and / or may communicate with any device that enables the electronic device 11 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the electronic device 11 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20.

[0142] like Figure 4 As shown, the network adapter 20 communicates with other modules of the electronic device 11 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 11, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0143] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, such as implementing an image retrieval method provided in any embodiment of the present application.

[0144] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements, for example, an image retrieval method provided by any embodiment of the present application.

[0145] The computer storage medium of this embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by instruction execution systems, devices or devices or used in combination with them.

[0146] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0147] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0148] Computer program code for performing the operation of the present application can be written in one or more programming languages ​​or combinations thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0149] It should be understood by those skilled in the art that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0150] In addition, the acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0151] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the inventive concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A picture retrieval method, characterized in that: The method comprises: Get the picture input by the user and get the picture to be retrieved; Input the image to be retrieved into a pre-trained image description model to obtain a text description corresponding to the image to be retrieved; Based on the text description, a preset image library is preliminarily searched to obtain multiple candidate similar images; Based on the image to be retrieved, a correlation test is performed on each of the multiple candidate similar images to obtain a correlation test result for each candidate similar image; Based on the correlation detection result, a correlation graph of the image to be retrieved is determined from the multiple candidate similarity graphs.

2. The image retrieval method according to claim 1, characterized in that: The step of performing a correlation test on each of the plurality of candidate similar images based on the image to be retrieved to obtain a correlation test result for each of the candidate similar images includes: For a current candidate similarity graph among the multiple candidate similarity graphs, determining an affine transformation matrix between the to-be-retrieved image and the current candidate similarity graph; Transforming the current candidate similarity graph based on the affine transformation matrix to obtain a candidate transformation graph; Calculating the similarity between the candidate transformation graph and the image to be retrieved to obtain the image similarity; When the picture similarity is greater than a first preset threshold, it is determined that a correlation detection result of the current candidate similar picture is that a geometric transformation exists.

3. The image retrieval method according to claim 2, characterized in that: The step of determining an affine transformation matrix between the image to be retrieved and the current candidate similar image comprises: Performing feature point detection on the image to be retrieved to obtain a first feature point of the image to be retrieved, and performing feature point detection on the current candidate similar image to obtain a second feature point of the current candidate similar image; Matching the first feature point and the second feature point to obtain a plurality of matching point pairs; The affine transformation matrix is ​​determined according to the plurality of matching point pairs.

4. The image retrieval method according to claim 1, characterized in that: The step of performing a correlation test on each of the plurality of candidate similar images based on the image to be retrieved to obtain a correlation test result for each of the candidate similar images includes: For a current candidate similarity graph among the multiple candidate similarity graphs, determining an optical flow field between the to-be-retrieved image and the current candidate similarity graph; When the motion information included in the optical flow field is continuous and consistent, it is determined that the correlation detection result of the current candidate similarity graph is that there is a time correlation.

5. The image retrieval method according to claim 1, characterized in that: The preset picture library includes a plurality of candidate pictures and a picture description corresponding to each candidate picture, and the preliminarily searching the preset picture library based on the text description to obtain a plurality of candidate similar pictures includes: For a current candidate picture among the multiple candidate pictures, calculating the similarity between the text description and the picture description corresponding to the current candidate picture to obtain text similarity; When the text similarity is greater than a second preset threshold, the current candidate image is determined to be the candidate similar image.

6. The image retrieval method according to claim 1, characterized in that: The training process of the image description model is as follows: Get multiple sample images and the sample text description corresponding to each sample image; Input the sample image into an image encoder of an initial image description model to obtain an image feature vector, and input the corresponding sample text description into a text encoder of the initial image description model to obtain a text feature vector; The similarity between the image feature vector and the text feature vector is calculated, and the initial image description model is trained and optimized with the goal of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs to obtain the image description model.

7. The image retrieval method according to claim 1, characterized in that: The determining the correlation graph of the to-be-retrieved image from the plurality of candidate similar graphs based on the correlation detection result comprises: When the correlation detection result indicates that there is a geometric transformation or a time correlation, determining the corresponding candidate similarity graph as the correlation graph; After determining the correlation graph of the to-be-retrieved image from the plurality of candidate similar graphs based on the correlation detection result, the method further includes: The correlation graph is displayed.

8. A picture retrieval device, characterized in that: The device comprises: The acquisition module is used to obtain the picture input by the user and obtain the picture to be retrieved; A description generation module, used to input the image to be retrieved into a pre-trained image description model to obtain a text description corresponding to the image to be retrieved; A retrieval module, used to perform a preliminary search on a preset image library based on the text description to obtain a plurality of candidate similar images; A detection module, used for performing a correlation detection on each of the plurality of candidate similar images based on the image to be retrieved, and obtaining a correlation detection result of each candidate similar image; A determination module is used to determine the correlation graph of the image to be retrieved from the multiple candidate similar graphs based on the correlation detection result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image retrieval method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image retrieval method as described in any one of claims 1 to 7 is implemented.

Images