Image retrieval method, device, electronic device, medium and program product

Abstract

The present disclosure relates to the field of artificial intelligence technology, and in particular to an image retrieval method, device, electronic device, medium and program product. The method comprises: obtaining a first text feature matrix, obtaining a first image feature matrix of each image to be detected in at least one image to be detected, so as to obtain at least one first image feature matrix, calculating at least one second image feature matrix according to at least one first image feature matrix and the first text feature matrix, determining a target image feature matrix whose similarity parameter with the first text feature matrix is ​​greater than or equal to a preset parameter from at least one second image feature matrix, and determining a target image corresponding to the target image feature matrix as an image matching the natural language information input by the user, so as to effectively improve the efficiency of image retrieval.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an image retrieval method, device, electronic device, medium and program product. Background Art

[0002] With the development of artificial intelligence technology, image retrieval has been widely used in the field of image processing. At present, image retrieval needs to generate natural language expressions from multiple images to be detected, compare the generated natural languages ​​with the natural languages ​​input by the user, and use the images to be detected corresponding to the natural languages ​​with high similarity as the target images, so as to obtain the target images that match the natural languages ​​input by the user among the multiple images to be detected.

[0003] However, when there are a large number of images to be detected, it is necessary to first obtain a large number of natural languages ​​corresponding to the images to be detected, thereby reducing the efficiency of image retrieval. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide an image retrieval method, device, electronic device, medium and program product.

[0005] In a first aspect, the present disclosure provides an image retrieval method, comprising:

[0006] Acquire a first text feature matrix, where the first text feature matrix is ​​determined according to natural language information input by a user;

[0007] Acquire a first image feature matrix of each image to be detected in at least one image to be detected to obtain at least one first image feature matrix;

[0008] Calculate at least one second image feature matrix according to the at least one first image feature matrix and the first text feature matrix;

[0009] Determine, from the at least one second image feature matrix, a target image feature matrix having a similarity parameter with the first text feature matrix that is greater than or equal to a preset parameter;

[0010] The target image corresponding to the target image feature matrix is ​​determined as an image matching the natural language information input by the user, and the target image is one or more of the at least one image to be detected.

[0011] Optionally, the calculating at least one second image feature matrix according to the at least one first image feature matrix and the first text feature matrix includes:

[0012] Mapping the at least one first image feature matrix to the first text feature matrix respectively to obtain at least one second text feature matrix;

[0013] At least one second image feature matrix is ​​obtained according to the at least one second text feature matrix and the at least one first image feature matrix.

[0014] Optionally, mapping the at least one first image feature matrix to the first text feature matrix to obtain at least one second text feature matrix includes:

[0015] Determine a first basis matrix and a second basis matrix of the first text feature matrix;

[0016] Mapping the at least one first image feature matrix to the first basis matrix respectively to obtain at least one third basis matrix corresponding to the first basis matrix; mapping the at least one first image feature matrix to the second basis matrix respectively to obtain at least one fourth basis matrix corresponding to the second basis matrix, so as to obtain the at least one second text feature matrix composed of the at least one third basis matrix and the at least one fourth basis matrix.

[0017] Optionally, obtaining at least one second image feature matrix according to the at least one second text feature matrix and at least one first image feature matrix includes:

[0018] According to at least one of the second text feature matrices, obtaining a weight value corresponding to each vector in at least one of the second text feature matrices;

[0019] At least one second image feature matrix is ​​obtained according to the weight values ​​and at least one of the first image feature matrices.

[0020] Optionally, the obtaining of the first text feature matrix, wherein the first text feature matrix is ​​determined according to natural language information input by the user, further includes:

[0021] Determining the dimension of the first text feature matrix;

[0022] According to the dimension, at least one first image feature matrix is ​​acquired, wherein the dimension size of the first text feature matrix is ​​the same as the dimension size of each first image feature matrix of the at least one first image feature matrix.

[0023] Optionally, the acquiring of a first text feature matrix, where the first text feature matrix is ​​determined according to natural language information input by a user, includes:

[0024] Inputting the natural language information input by the user into a natural language analysis model to obtain the first text feature matrix;

[0025] Acquiring a first image feature matrix of each image to be detected in at least one image to be detected to obtain at least one first image feature matrix includes:

[0026] The at least one image to be detected is input into an image target detection model to obtain at least one of the first image feature matrices.

[0027] In a second aspect, the present disclosure provides an image retrieval device, comprising:

[0028] A first text feature matrix acquisition module, used to acquire a first text feature matrix, where the first text feature matrix is ​​determined according to natural language information input by a user;

[0029] A first image feature matrix acquisition module, used to acquire a first image feature matrix of each image to be detected in at least one image to be detected, so as to obtain at least one first image feature matrix;

[0030] A second image feature matrix calculation module, configured to calculate at least one second image feature matrix according to the at least one first image feature matrix and the first text feature matrix;

[0031] A target image feature matrix determination module, used to determine, from the at least one second image feature matrix, a target image feature matrix having a similarity parameter with the first text feature matrix greater than or equal to a preset parameter;

[0032] The target image determination module is used to determine the target image corresponding to the target image feature matrix as an image matching the natural language information input by the user, and the target image is one or more of the at least one image to be detected.

[0033] In a third aspect, the present disclosure provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0034] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0035] In a fifth aspect, the present disclosure provides a computer program product, which, when executed on a computer, enables the computer to implement the steps of any one of the methods described in the first aspect.

[0036] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0037] The disclosed embodiment provides an image retrieval method, which obtains a first text feature matrix, the first text feature matrix is ​​determined according to the natural language information input by the user, obtains the first image feature matrix of each image to be detected in at least one image to be detected, so as to obtain at least one first image feature matrix, calculates at least one second image feature matrix according to the at least one first image feature matrix and the first text feature matrix, determines a target image feature matrix whose similarity parameter with the first text feature matrix is ​​greater than or equal to a preset parameter from the at least one second image feature matrix, and determines the target image corresponding to the target image feature matrix as an image matching the natural language information input by the user. In this way, by directly obtaining the natural language features corresponding to the natural language input by the user and the image features corresponding to the multiple images to be detected respectively, the natural language features are compared with the image features to determine the target image corresponding to the natural language information input by the user, compared with generating the corresponding natural language based on the multiple images to be detected and then comparing it with the natural language information input by the user, the efficiency of image retrieval is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of an image retrieval method provided by an embodiment of the present disclosure;

[0041] Figure 2 A flowchart of another image retrieval method provided by an embodiment of the present disclosure;

[0042] Figure 3 A flowchart of another image retrieval method provided by an embodiment of the present disclosure;

[0043] Figure 4 A flowchart of another image retrieval method provided by an embodiment of the present disclosure;

[0044] Figure 5A schematic diagram of the structure of an image retrieval device provided by an embodiment of the present disclosure;

[0045] Figure 6 This is a diagram of the internal structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0048] At present, image retrieval requires generating natural language expressions from multiple images to be detected, comparing the generated natural languages ​​with the natural languages ​​input by the user, and taking the images to be detected corresponding to the natural languages ​​with high similarity as the target images, so as to obtain the target images that match the natural languages ​​input by the user from multiple images to be detected. However, when there are a large number of images to be detected, it is necessary to first obtain the natural languages ​​corresponding to a large number of images to be detected, which reduces the efficiency of image retrieval.

[0049] Therefore, the present disclosure provides an image retrieval method, which obtains a first text feature matrix, obtains a first image feature matrix of each image to be detected in at least one image to be detected, so as to obtain at least one first image feature matrix, calculates at least one second image feature matrix according to at least one first image feature matrix and the first text feature matrix, and determines a target image feature matrix whose similarity parameter with the first text feature matrix is ​​greater than or equal to a preset parameter from at least one second image feature matrix, and determines the target image corresponding to the target image feature matrix as an image matching the natural language information input by the user. In this way, by directly obtaining the natural language features corresponding to the natural language input by the user, and the image features corresponding to the multiple images to be detected, respectively, the natural language features are compared with the image features to determine the target image corresponding to the natural language information input by the user, compared with generating the corresponding natural language based on the multiple images to be detected, and then comparing with the natural language information input by the user, so as to determine the target image, the efficiency of image retrieval is effectively improved.

[0050] The image retrieval method provided by the present disclosure can be applied to an image retrieval device, which can be various electronic devices such as personal computers, laptops, smart phones, tablet computers and portable wearable devices. Optionally, the device can also be a functional module or functional entity in these electronic devices that can implement the image retrieval method.

[0051] Figure 1 A flowchart of an image retrieval method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, specifically including:

[0052] S10, obtaining a first text feature matrix.

[0053] The first text feature matrix is ​​determined according to the natural language information input by the user. The natural language information input by the user can be text content input by the user to the terminal, or can be voice content input by the user to the terminal, but is not limited thereto and is not specifically limited in the present disclosure.

[0054] Exemplarily, the text content input by the user to the terminal may be, for example, "there is a computer on the desk", and the voice content input by the user to the terminal may be, for example, "there is a boat on the lake".

[0055] Specifically, a first text feature corresponding to natural language information input by a user into a terminal is obtained.

[0056] Based on the above embodiments, in some embodiments of the present application, one possible implementation method is:

[0057] The natural language information input by the user is input into the natural language analysis model to obtain a first text feature matrix.

[0058] Among them, the natural language analysis model refers to a text feature matrix used to extract the natural language information input by the user. Exemplarily, the natural language analysis model can be, for example, a bidirectional encoder representation from a transformer (Bidirectional Encoder Representation from Transformers, Bert). The Bert model includes a character vector feature extraction layer, a word vector feature extraction layer, and a position vector feature extraction layer for extracting text samples. The natural language analysis model is trained according to a training set, by obtaining a training sample set, the training sample set includes a text sample and a labeled text sample, the text sample is input into the initial natural language analysis model, and the output result corresponding to the text sample is obtained. According to the output result and the labeled text sample, the weight parameters of the initial natural language analysis model are adjusted until the initial natural language analysis model meets the preset convergence conditions, and the current initial natural language analysis model is determined to be a trained natural language analysis model, but is not limited to this, and the present disclosure is not specifically limited.

[0059] Exemplarily, when the above-mentioned natural language analysis model is a Bert model, the natural language information "There is a computer on the desk" input by the user is input into the natural language analysis model. The natural language analysis model such as the Bert model extracts the character vector, word vector, position vector and other features of "There is a computer on the desk", thereby obtaining the first text feature matrix corresponding to the natural language information input by the user.

[0060] S12, obtaining a first image feature matrix of each image to be detected in at least one image to be detected, so as to obtain at least one first image feature matrix.

[0061] The image to be detected refers to at least one image that needs to be retrieved, and the image to be detected includes one or more target images that match the natural language information input by the user.

[0062] Based on the above embodiments, in some embodiments of the present application, one possible implementation method is:

[0063] At least one image to be detected is input into an image object detection model to obtain at least one first image feature matrix.

[0064] The image target detection model refers to a model used to extract a first image feature matrix corresponding to at least one image to be detected.

[0065] Specifically, at least one image to be detected is input into a trained image target detection model, and a first image feature matrix corresponding to each image to be detected in at least one image to be detected is extracted according to the image target detection model to obtain at least one first image feature matrix.

[0066] On the basis of the above embodiments, in some embodiments of the present disclosure, the above-mentioned image target detection model can be, for example, a fast regional convolutional neural network, which is trained according to a training set, by obtaining a training sample set, the training sample set includes picture samples and labeled picture samples, the picture samples are input into the initial image target detection model, and the output results corresponding to the picture samples are obtained. According to the output results and the labeled picture samples, the weight parameters of the initial image target detection model are adjusted until the initial image target detection model meets the preset convergence conditions, and the current initial image target detection model is determined to be a trained image target detection model, but is not limited to this, and the present disclosure is not specifically limited.

[0067] It should be noted that in the above embodiment, the condition for judging whether the initial image target detection model has converged can be, for example, after completing a network model training, determining whether the output result of the current initial image target detection model meets the preset condition, for example, whether the error between the output result of the current fast regional convolutional neural network and the labeled image sample in the training sample set is less than the preset error. If it is not less than the preset error, that is, the output result of the current initial image target detection model does not meet the preset condition, return to continue training the current initial image target detection model until the error of the output result of the current initial image target detection model meets the preset condition, at which time, stop the network model training, and determine that the current image target detection model is a trained image target detection model.

[0068] S14, calculating at least one second image feature matrix according to the at least one first image feature matrix and the first text feature matrix.

[0069] Specifically, at least one second image feature matrix is ​​calculated based on a first text feature matrix extracted from natural language information input by a user terminal and at least one first image feature matrix extracted from at least one image to be detected.

[0070] S16, determining, from at least one second image feature matrix, a target image feature matrix whose similarity parameter with the first text feature matrix is ​​greater than or equal to a preset parameter.

[0071] The similarity parameter is used to determine the target image feature matrix in at least one second image feature matrix, and the preset parameter is used to determine the parameter set for the target image feature matrix in at least one second image feature matrix.

[0072] Exemplarily, the similarity parameter may be, for example, a distance, such as 11, or a cosine parameter of calculating the second image feature matrix and the first text feature matrix, but is not limited thereto. The present disclosure does not specifically limit this, and those skilled in the art may set it according to actual conditions.

[0073] Specifically, based on one or more second image feature matrices, a similarity parameter between each second image feature matrix and the first text feature matrix is ​​calculated. For multiple similarity parameters, when the similarity parameter is greater than or equal to a preset parameter, the second image feature matrix is ​​determined to be a target image feature matrix.

[0074] For example, the natural language information “There is a computer on the desk” input by the user is extracted through the natural language analysis model as the first text feature matrix C of “There is a computer on the desk”. For at least one image to be detected, for example, five images to be detected, the first image feature matrix V corresponding to the five images to be detected is extracted according to the image target detection model.11 、V 12 、V 13 、V 14 、V 15 , and based on the first text feature matrix C and the first image feature matrix V 11 、V 12 、V 13 、V 14 、V 15 Calculate the five second image feature matrices V 21 、V 22 、V 23 、V 24 、V 25 , respectively calculate the first text feature matrix C and the five second image feature matrices V 21 、V 22 、V 23 、V 24 、V 25 For example, if the distances are 10, 12, 5, 4, and 14, and the preset parameter is 11, then V in the five second image feature matrices is determined. 22 and V 25 The corresponding second image feature matrix, the corresponding first image feature matrix is ​​the target image feature matrix.

[0075] S18, determining the target image corresponding to the target image feature matrix as an image that matches the natural language information input by the user.

[0076] The target image is one or more of at least one image to be detected.

[0077] Specifically, the image to be detected corresponding to the target image feature matrix is ​​determined as the target image, that is, an image matching the natural language information input by the user is obtained.

[0078] In the technical solution provided by the embodiment of the present disclosure, by obtaining a first text feature matrix, obtaining the first image feature matrix of each image to be detected in at least one image to be detected, so as to obtain at least one first image feature matrix, according to the at least one first image feature matrix and the first text feature matrix, calculating at least one second image feature matrix, determining a target image feature matrix whose similarity parameter with the first text feature matrix is ​​greater than or equal to a preset parameter from the at least one second image feature matrix, and determining the target image corresponding to the target image feature matrix as an image matching the natural language information input by the user. In this way, by directly obtaining the natural language features corresponding to the natural language input by the user, and the image features corresponding to the multiple images to be detected respectively, the natural language features are compared with the image features to determine the target image corresponding to the natural language information input by the user, compared with generating the corresponding natural language based on the multiple images to be detected, and then comparing with the natural language information input by the user, so as to determine the target image, the efficiency of image retrieval is effectively improved.

[0079] In some embodiments of the present application, Figure 2 As shown, at least one second image feature matrix is ​​calculated based on at least one first image feature matrix and a first text feature matrix, including:

[0080] S141, mapping at least one first image feature matrix to a first text feature matrix to obtain at least one second text feature matrix.

[0081] Among them, mapping refers to associating the feature information contained in at least one first image feature matrix to the first text feature matrix respectively. The mapping method can be, for example, by matrix multiplying the first image feature matrix with the first text feature matrix, but is not limited to this and is not specifically limited in the present disclosure.

[0082] Specifically, each first image feature matrix in at least one first image feature matrix is ​​mapped to the first text feature matrix to obtain a second text feature matrix mapped by each first image feature matrix.

[0083] For example, the first text feature matrix C is the first image feature matrix V corresponding to the five images to be detected. 11 、V 12 、V 13 、V 14 、V 15 , the five first image feature matrices V 11 、V 12 、V 13 、V 14 、V 15Mapped to the first text feature matrix C respectively, we get five second text feature matrices C 21 , C 22 , C 23 , C 24 , C 25 .

[0084] S142: Obtain at least one second image feature matrix according to at least one second text feature matrix and at least one first image feature matrix.

[0085] Specifically, based on each second text feature matrix in the at least one second text feature matrix obtained and the corresponding first image feature matrix in the at least one first image feature matrix, the second image feature matrix corresponding to each current first image feature matrix is ​​determined to obtain at least one second image feature matrix.

[0086] In this way, this embodiment maps at least one first image feature matrix to the first text feature matrix to obtain at least one second text feature matrix, and obtains at least one second image feature matrix based on at least one second text feature matrix and at least one first image feature matrix, thereby combining the first image feature matrix corresponding to the image to be detected with the first text feature matrix corresponding to the natural language information, and further determining the target image based on the second image feature matrix and the first text feature matrix, thereby effectively improving the efficiency of image retrieval.

[0087] In some embodiments of the present application, Figure 3 As shown, at least one first image feature matrix is ​​mapped to a first text feature matrix to obtain at least one second text feature matrix, including:

[0088] S1411, determining a first basis matrix and a second basis matrix of the first text feature matrix.

[0089] Among them, the first basis matrix and the second basis matrix are the main feature matrix components of the first text feature matrix, that is, through the first text feature matrix, at least two basis matrices are selected, and the original first text feature matrix is ​​transformed into this group of basis matrices. In this way, the multidimensional data information in the first text feature matrix can be represented by the basis matrix, thereby realizing the dimensionality reduction of the multidimensional data information. Exemplarily, the first basis matrix and the second basis matrix corresponding to the first text feature matrix can be obtained by means of covariance, but is not limited to this, and the present disclosure is not limited to this.

[0090] Exemplarily, the above-mentioned first text feature matrix is ​​C. According to the first text feature matrix C, its corresponding first basis matrix is ​​determined to be A, and the second basis matrix is ​​determined to be B, that is, the first text feature matrix C can be expressed as: C=αA+βB, wherein α represents the coefficient of the first basis matrix A, and β represents the coefficient of the second basis matrix B.

[0091] S1412, respectively mapping at least one first image feature matrix to the first basis matrix to obtain at least one third basis matrix corresponding to the first basis matrix; respectively mapping at least one first image feature matrix to the second basis matrix to obtain at least one fourth basis matrix corresponding to the second basis matrix, so as to obtain at least one second text feature matrix composed of at least one third basis matrix and at least one fourth basis matrix.

[0092] For example, the first image feature matrix V corresponding to the five images to be detected is 11 、V 12 、V 13 、V 14 、V 15 , the five first image feature matrices V 11 、V 12 、V 13 、V 14 、V 15 The measurements are mapped to the first basis matrix A respectively, and the corresponding third basis matrix A is obtained. 11 , A 12 , A 13 , A 14 , A 15 , the five first image feature matrices V 11 、V 12 、V 13 、V 14 、V 15 The measurements are mapped to the second basis matrix B respectively, and the corresponding fourth basis matrix B is obtained. 11 , B 12 , B 13 , B 14 , B 15 , and according to the third basis matrix A 11 , A 12 , A 13 , A 14 , A 15 And the fourth basis matrix B 11 , B 12 , B 13 , B 14 , B 15 The corresponding second text feature matrix C 21 , C 22 , C 23 , C24 , C 25 .

[0093] In this way, this embodiment achieves dimensionality reduction of multidimensional data information by utilizing at least two basis matrices to represent the multidimensional data information in the first text feature matrix, and further maps at least one first image feature matrix to at least two basis matrices respectively, and further obtains the second text feature matrix. In this way, by reducing the dimension of the data information in the first text feature matrix, the complexity of mapping is reduced, and the efficiency of image retrieval is effectively improved.

[0094] In some embodiments of the present application, Figure 4 As shown, at least one second text feature matrix and at least one first image feature matrix are obtained, including:

[0095] S1421: Obtain a weight value corresponding to each vector in at least one second text feature matrix according to at least one second text feature matrix.

[0096] Specifically, each first image feature matrix in at least one first image feature matrix is ​​mapped to a first text feature matrix to obtain at least one second text feature matrix, and further, a weight value corresponding to each vector in the second text feature matrix is ​​obtained.

[0097] Exemplarily, following the above embodiment, for the second text feature matrix C 21 , C 22 , C 23 , C 24 , C 25 , using the normalized exponential function Softmax to the second text feature matrix C 21 , C 22 , C 23 , C 24 , C 25 Score them respectively to get the second text feature matrix C 21 , C 22 , C 23 , C 24 , C 25 The weight value corresponding to each vector in each second text feature matrix.

[0098] It should be noted that the normalized exponential function Softmax refers to "compressing" a multidimensional vector containing arbitrary real numbers into another multidimensional real vector so that the range of each element is between (0,1) and the sum of all elements is 1. This function is mostly used in multi-classification problems, that is, to present the results of multi-classification in the form of probability.

[0099] S1422: Obtain at least one second image feature matrix according to the weight values ​​and at least one first image feature matrix.

[0100] Exemplarily, according to the second text feature matrix C in the above embodiment 21 , C 22 , C 23 , C 24 , C 25 The weight value corresponding to each vector in each second text feature matrix in the text feature matrix is ​​determined to determine the weight matrix corresponding to each second text feature matrix. Further, according to each second text feature matrix C 21 , C 22 , C 23 , C 24 , C 25 The corresponding first image feature matrix V 11 、V 12 、V 13 、V 14 、V 15 , and the weight matrix corresponding to each second text feature matrix, determine the second image feature matrix V 21 、V 22 、V 23 、V 24 、V 25 .

[0101] In this way, this embodiment obtains the weight value of each vector in the second text feature matrix, and obtains the second image feature matrix based on the weight value and the first image feature matrix, thereby realizing the association between the feature information of the image to be detected and the feature information of the natural language, and effectively improving the efficiency of image retrieval.

[0102] On the basis of the above embodiments, in some embodiments of the present disclosure, after obtaining a first text feature matrix, where the first text feature matrix is ​​determined according to natural language information input by a user, the method further includes:

[0103] Determine the dimensions of the first text feature matrix.

[0104] At least one first image feature matrix is ​​obtained according to the dimension.

[0105] The dimension of the first text feature matrix is ​​the same as the dimension of each first image feature matrix of the at least one first image feature matrix.

[0106] Specifically, based on the natural language information input by the user into the terminal, a natural language analysis model is used to extract a first text feature matrix of the natural language information. The dimension of the first text feature matrix is ​​determined by the natural language analysis model. According to the dimension, the dimension of the image target detection model is set, so that the dimension of each first image feature matrix of at least one first image feature matrix obtained is the same as the dimension of the first text feature matrix.

[0107] The disclosed embodiment also provides an image retrieval device for executing any one of the image retrieval methods provided in the above embodiments, and having the corresponding beneficial effects of the image retrieval method.

[0108] Figure 5 A schematic diagram of the structure of an image retrieval device provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the image retrieval device includes: a first text feature matrix acquisition module 100, a first image feature matrix acquisition module 120, a second image feature matrix calculation module 140, a target image feature matrix determination module 160 and a target image determination module 180.

[0109] The first text feature matrix acquisition module 100 is used to acquire a first text feature matrix, and the first text feature matrix is ​​determined according to natural language information input by a user.

[0110] The first image feature matrix acquisition module 120 is used to acquire the first image feature matrix of each of the at least one image to be detected to obtain at least one first image feature matrix.

[0111] The second image feature matrix calculation module 140 is used to calculate at least one second image feature matrix according to at least one first image feature matrix and a first text feature matrix.

[0112] The target image feature matrix determination module 160 is used to determine, from at least one second image feature matrix, a target image feature matrix whose similarity parameter with the first text feature matrix is ​​greater than or equal to a preset parameter.

[0113] The target image determination module 180 is used to determine the target image corresponding to the target image feature matrix as an image matching the natural language information input by the user, and the target image is one or more of the at least one image to be detected.

[0114] In one implementation of an embodiment of the present invention, the second image feature matrix calculation module 140 is specifically used to map at least one first image feature matrix to a first text feature matrix to obtain at least one second text feature matrix; and obtain at least one second image feature matrix based on at least one second text feature matrix and at least one first image feature matrix.

[0115] In an implementation manner of an embodiment of the present invention, the second image feature matrix calculation module 140 is specifically further used to determine the first basis matrix and the second basis matrix of the first text feature matrix; map at least one of the first image feature matrices to the first basis matrix, respectively, to obtain at least one third basis matrix corresponding to the first basis matrix; map at least one of the first image feature matrices to the second basis matrix, respectively, to obtain at least one fourth basis matrix corresponding to the second basis matrix, so as to obtain at least one second text feature matrix composed of at least one third basis matrix and at least one fourth basis matrix.

[0116] In one implementation of an embodiment of the present invention, the second image feature matrix calculation module 140 is specifically used to obtain a weight value corresponding to each vector in at least one second text feature matrix based on at least one second text feature matrix; and obtain at least one second image feature matrix based on the weight value and at least one first image feature matrix.

[0117] In an implementation of an embodiment of the present invention, the first text feature matrix acquisition module 100 further includes a dimension determination module, which is used to determine the dimension of the first text feature matrix.

[0118] The first image feature matrix acquisition module 120 is used to acquire at least one first image feature matrix according to the dimension, wherein the dimension size of the first text feature matrix is ​​the same as the dimension size of each first image feature matrix of the at least one first image feature matrix.

[0119] In an implementation of an embodiment of the present invention, the first text feature matrix acquisition module 100 is specifically used to input the natural language information input by the user into the natural language analysis model to obtain the first text feature matrix.

[0120] In an implementation of an embodiment of the present invention, the first image feature matrix acquisition module 120 is specifically used to input at least one image to be detected into an image target detection model to obtain at least one first image feature matrix.

[0121] The disclosed embodiment can directly obtain the natural language features corresponding to the natural language input by the user and the image features corresponding to multiple images to be detected, and then compare the natural language features with the image features to determine the target image corresponding to the natural language information input by the user. Compared with generating the corresponding natural language based on multiple images to be detected and then comparing it with the natural language information input by the user to determine the target image, the efficiency of image retrieval is effectively improved.

[0122] The device of this embodiment can be used to perform the above Figures 1 to 4 The technical solution of any of the method embodiments shown has similar implementation principles and technical effects, which will not be repeated here.

[0123] The present disclosure provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the image retrieval method provided by the present disclosure can be implemented. For example, when the processor executes the computer program, the image retrieval method provided by the present disclosure can be implemented. Figures 1 to 4 The technical solution of any of the method embodiments shown has similar implementation principles and technical effects, which will not be repeated here.

[0124] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image retrieval method provided by the embodiment of the present disclosure can be implemented. For example, when the computer program is executed by a processor, the image retrieval method provided by the embodiment of the present disclosure can be implemented. Figures 1 to 4 The technical solution of any of the method embodiments shown has similar implementation principles and technical effects, which will not be repeated here.

[0125] The present disclosure provides a computer program product. When the computer program product is run on a computer, the computer can implement the image retrieval method provided by the embodiment of the present disclosure. For example, the computer can implement Figures 1 to 4 The technical solution of any of the method embodiments shown has similar implementation principles and technical effects, which will not be repeated here.

[0126] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0127] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image retrieval method, characterized in that: The method comprises: Acquire a first text feature matrix, where the first text feature matrix is ​​determined according to natural language information input by a user; Acquire a first image feature matrix of each image to be detected in at least one image to be detected to obtain at least one first image feature matrix; Calculate at least one second image feature matrix according to the at least one first image feature matrix and the first text feature matrix; Determine, from the at least one second image feature matrix, a target image feature matrix having a similarity parameter with the first text feature matrix that is greater than or equal to a preset parameter; Determine the target image corresponding to the target image feature matrix as an image matching the natural language information input by the user, wherein the target image is one or more of the at least one image to be detected; The step of calculating at least one second image feature matrix based on the at least one first image feature matrix and the first text feature matrix includes: Mapping the at least one first image feature matrix to the first text feature matrix respectively to obtain at least one second text feature matrix; Obtain at least one second image feature matrix according to the at least one second text feature matrix and the at least one first image feature matrix; Wherein, obtaining at least one second image feature matrix according to the at least one second text feature matrix and at least one first image feature matrix comprises: According to at least one of the second text feature matrices, obtaining a weight value corresponding to each vector in at least one of the second text feature matrices; Obtaining at least one second image feature matrix according to the weight values ​​and at least one of the first image feature matrices; The mapping of the at least one first image feature matrix to the first text feature matrix to obtain at least one second text feature matrix includes: Determine a first basis matrix and a second basis matrix of the first text feature matrix; Mapping the at least one first image feature matrix to the first basis matrix respectively to obtain at least one third basis matrix corresponding to the first basis matrix; mapping the at least one first image feature matrix to the second basis matrix respectively to obtain at least one fourth basis matrix corresponding to the second basis matrix, so as to obtain the at least one second text feature matrix composed of the at least one third basis matrix and the at least one fourth basis matrix.

2. The method according to claim 1, characterized in that The method further comprises: obtaining a first text feature matrix, wherein the first text feature matrix is ​​determined according to natural language information input by a user; Determining the dimension of the first text feature matrix; According to the dimension, at least one first image feature matrix is ​​acquired, wherein the dimension size of the first text feature matrix is ​​the same as the dimension size of each first image feature matrix of the at least one first image feature matrix.

3. The method according to claim 1, characterized in that The obtaining of a first text feature matrix, where the first text feature matrix is ​​determined according to natural language information input by a user, includes: Inputting the natural language information input by the user into a natural language analysis model to obtain the first text feature matrix; Acquiring a first image feature matrix of each image to be detected in at least one image to be detected to obtain at least one first image feature matrix includes: The at least one image to be detected is input into an image target detection model to obtain at least one of the first image feature matrices.

4. An image retrieval device, characterized in that: include: A first text feature matrix acquisition module, used to acquire a first text feature matrix, where the first text feature matrix is ​​determined according to natural language information input by a user; A first image feature matrix acquisition module, used to acquire a first image feature matrix of each image to be detected in at least one image to be detected, so as to obtain at least one first image feature matrix; A second image feature matrix calculation module, configured to calculate at least one second image feature matrix according to the at least one first image feature matrix and the first text feature matrix; A target image feature matrix determination module, used to determine, from the at least one second image feature matrix, a target image feature matrix having a similarity parameter with the first text feature matrix greater than or equal to a preset parameter; A target image determination module, used to determine the target image corresponding to the target image feature matrix as an image matching the natural language information input by the user, wherein the target image is one or more of the at least one image to be detected; The second image feature matrix calculation module is specifically used to: map the at least one first image feature matrix to the first text feature matrix to obtain at least one second text feature matrix; obtain at least one second image feature matrix according to the at least one second text feature matrix and the at least one first image feature matrix; Wherein, when obtaining at least one second image feature matrix according to the at least one second text feature matrix and at least one first image feature matrix, the second image feature matrix calculation module is specifically used to: obtain a weight value corresponding to each vector in at least one second text feature matrix according to at least one second text feature matrix; obtain at least one second image feature matrix according to the weight value and at least one first image feature matrix; The second image feature matrix calculation module, when obtaining at least one second image feature matrix according to the at least one second text feature matrix and at least one first image feature matrix, is specifically used to: determine a first basis matrix and a second basis matrix of the first text feature matrix; Mapping the at least one first image feature matrix to the first basis matrix respectively to obtain at least one third basis matrix corresponding to the first basis matrix; mapping the at least one first image feature matrix to the second basis matrix respectively to obtain at least one fourth basis matrix corresponding to the second basis matrix, so as to obtain the at least one second text feature matrix composed of the at least one third basis matrix and the at least one fourth basis matrix.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

7. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is enabled to execute the steps of the method according to any one of claims 1 to 3.

Images