Image retrieval sorting method, device and equipment and readable storage medium

Abstract

The invention provides an image retrieval sorting method and device, equipment and a readable storage medium. The method comprises the steps that a query image and matched media resources are acquired; determining a first interaction feature according to the image feature of the query image and the feature information of the matched media resource; screening the image feature, the feature information and the first interaction feature to obtain a target feature; and sorting the matched media resources according to a second interaction feature between the target features. According to the method, after the query image and the matched media resource are obtained, interaction is carried out between the image features and the feature information of the media resource to obtain the interactive features with richer features, and then the features are screened to remove irrelevant features in the interactive features and retain the target features with higher importance; therefore, the matching media resources are sequenced by further utilizing the interaction features between the target features subsequently, and the effectiveness of the final sequencing result can be ensured while the model calculation amount can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image retrieval and sorting method, device, equipment and readable storage medium. Background Art

[0002] Image search is a search method based on image recognition technology. When users upload pictures, the system will automatically identify the content in the pictures and then return relevant search results.

[0003] Image search services usually require sorting of large amounts of media data in order to present the most relevant content in search results. Currently, related technologies have difficulty achieving a balance between search efficiency and search result accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide an image retrieval and ranking method, device, equipment and readable storage medium to address the above technical problems, so as to improve the efficiency and accuracy of image retrieval and ranking.

[0005] In a first aspect, the present application provides an image retrieval and ranking method, comprising:

[0006] Acquire a query image to be retrieved and matching media resources corresponding to the query image;

[0007] determining a first interaction feature between the query image and the matching media resource according to an image feature of the query image and feature information of the matching media resource;

[0008] Screening the image feature, the feature information, and the first interactive feature to obtain a target feature;

[0009] The matching media resources are sorted according to the second interaction feature between the target features to obtain sorted matching media resources.

[0010] As a feasible embodiment of the present application, the first interaction feature is a low-order interaction feature;

[0011] The determining, according to the image feature of the query image and the feature information of the matching media resource, a first interaction feature between the query image and the matching media resource comprises:

[0012] The image features of the query image and the feature information of the matching media resources are input into a preset factorization machine model to obtain a first interaction feature between the query image and the matching media resources.

[0013] As a feasible embodiment of the present application, the step of inputting the image feature of the query image and the feature information of the matching media resource into a preset factor decomposition machine model to obtain a first interaction feature between the query image and the matching media resource includes:

[0014] Acquire user features corresponding to the query image;

[0015] The image features of the query image, the user features, and the feature information of the matching media resources are input into a preset factorization machine model to obtain a first interaction feature between the query image and the matching media resources.

[0016] As a feasible embodiment of the present application, the screening of the image feature, the feature information and the first interactive feature to obtain the target feature includes:

[0017] The image feature, the feature information and the first interactive feature are input into a pre-trained feature screening model to obtain a target feature.

[0018] As a feasible embodiment of the present application, the second interaction feature is a high-order interaction feature;

[0019] The step of sorting the matching media resources according to the second interaction feature between the target features to obtain sorted matching media resources includes:

[0020] Inputting the target features into a deep convolutional layer in a pre-trained deep network model to obtain a second interaction feature between the target features;

[0021] Inputting the second interactive feature into a linear layer in the deep network model to obtain a matching degree between the query image and each of the matching media resources;

[0022] The matching media resources are sorted according to the matching degree to obtain sorted matching media resources.

[0023] As a feasible embodiment of the present application, before the step of determining the first interaction feature between the query image and the matching media resource according to the image feature of the query image and the feature information of the matching media resource, the method further includes:

[0024] Processing the query image to obtain initial sparse features and initial dense features of the query image;

[0025] A first normalization process is performed on the initial sparse features, and a second normalization process is performed on the initial dense features to obtain image features of the query image.

[0026] As a feasible embodiment of the present application, before the step of determining the first interaction feature between the query image and the matching media resource according to the image feature of the query image and the feature information of the matching media resource, the method further includes:

[0027] In response to a trigger instruction for a query image, obtaining a reference text corresponding to the query image;

[0028] The reference text and the query image are processed to obtain image features of the query image.

[0029] As a feasible embodiment of the present application, the step of obtaining a query image to be retrieved and a matching media resource corresponding to the query image includes:

[0030] Acquire a query image to be retrieved and a query feature corresponding to the query image;

[0031] According to the similarity between the query feature and feature information of each initial media resource in a preset database, a matching media resource corresponding to the query image is determined from the initial media resources.

[0032] In a second aspect, the present application provides an image retrieval and ranking device, comprising:

[0033] An acquisition module, used to acquire a query image to be retrieved and matching media resources corresponding to the query image;

[0034] an interaction module, configured to determine a first interaction feature between the query image and the matching media resource according to an image feature of the query image and feature information of the matching media resource;

[0035] A screening module, used for screening the image feature, the feature information and the first interactive feature to obtain a target feature;

[0036] The sorting module is used to sort the matching media resources according to the second interaction feature between the target features to obtain sorted matching media resources.

[0037] In a third aspect, the present application further provides a computer device, the computer device comprising:

[0038] one or more processors;

[0039] Memory; and

[0040] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the image retrieval and ranking method provided above.

[0041] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the image retrieval and ranking method provided above.

[0042] The image retrieval and ranking method provided in the embodiment of the present application, after obtaining the query image and matching media resources, first interacts between the image features and the feature information of the media resources to obtain interactive features with richer features, and then screens the features to omit irrelevant features and retain target features with higher importance, so as to further use the interactive features between the target features to sort the matching media resources, which can effectively reduce the amount of model calculation while ensuring the validity of the final ranking result. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic diagram of the steps of an image retrieval and sorting method provided in an embodiment of the present application;

[0045] Figure 2a A schematic diagram of a process flow for implementing image retrieval and sorting based on a neural network model provided in an embodiment of the present application;

[0046] Figure 2b A schematic diagram of the structure of a lightweight feature screening model provided in an embodiment of the present application;

[0047] Figure 2c A schematic diagram of a model structure for implementing image retrieval and ranking provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a flow chart of steps for fusing user features to obtain interaction features provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a process flow for standardizing dense features and sparse features based on different standardization methods provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a process flow for determining image features provided in an embodiment of the present application;

[0051] Figure 6A schematic diagram of a process flow for determining matching media resources provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of an image retrieval and sorting device provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0055] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0056] In the description of the present application, the word "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0057] To facilitate understanding of the image retrieval and ranking method provided in the embodiment of the present application, the application scenario of the image retrieval and ranking method is first described. Specifically, image retrieval and ranking are mainly used in the case where a user inputs an image, and the program system returns some media resources related to the image from the database, such as note text, images, videos and other resources. In this scenario, two stages are usually included, namely the recall stage and the fine sorting stage. In the recall stage, the program system is often required to quickly and preliminarily screen out a certain number of media resources that may be related to the image input by the user from the database. For example, this number is usually in the hundreds or thousands. Then, in the fine sorting stage, the preliminary screened media resources and images are further sorted for relevance, so as to display several media resources with higher relevance to the user through sorting. In this process, the sorting effect of the fine sorting stage will significantly affect the user's search experience.

[0058] For the refined ranking stage, most of the related technologies use deep learning to achieve the final ranking of matching results. For example, the more common ones are DeepFM (Deep Factorization Machine) model, DCN (Deep Cross Network) model, WDL (Wide & Deep Learning) model, etc. Specifically, for ease of understanding, taking the DeepFM model as an example, in the usual ranking process, the features of the retrieved image and the matching media resources are first extracted, where these features can be divided into sparse features (features that appear discontinuously in the data set, and most of the values ​​are zero) and dense features (features that appear frequently or regularly in the data set, and most of the values ​​are non-zero). Then, the sparse features are used as the input of the FM (Factorization Machine) structure in the DeepFM model to achieve low-order interaction of features, and the dense features are used as the input of the DNN (Deep Nueral Network) structure in the DeepFM model to achieve high-order interaction of features, and finally the correlation between each matching media resource and the retrieved image is determined using the obtained feature interaction results to complete the final retrieval result ranking.

[0059] It can be seen that in the process of sorting matching results based on deep learning, it is often necessary to first interact the extracted image features and the feature information of the media resources to obtain interactive features with richer feature information, so as to better identify the similarity between the image and the text, thereby improving the accuracy of the final sorting results. However, the more features are used for feature interaction, the more computational effort the model will have, thus affecting the efficiency of the model sorting. In specific application scenarios, this will result in a long waiting process, thus affecting the user experience to a certain extent.

[0060] It is precisely to solve the above problems that the present application provides an image retrieval sorting method, device, equipment and readable storage medium, which completes the sorting of matching media resources by screening features during feature interaction, omitting irrelevant features, and using the retained features with higher importance to interact again, thereby effectively improving the efficiency of model calculation while ensuring the final sorting effect, reducing waiting during the sorting process, and thus improving user experience. Specifically, the image retrieval sorting method is usually installed on the image retrieval sorting device in the form of a computer program, and the image retrieval sorting device is usually set in the form of a processor in a computer device (such as a terminal device installed with a program that can be used to implement image retrieval), and the image retrieval sorting device in the computer device executes the computer program corresponding to the image retrieval sorting method to execute the image retrieval sorting method provided in the embodiment of the present application.

[0061] For details, please refer to Figure 1 , Figure 1 A schematic flow chart of the steps of an image retrieval and ranking method provided in an embodiment of the present application, specifically, comprising steps S110 to S140:

[0062] S110: Acquire a query image to be retrieved and matching media resources corresponding to the query image.

[0063] In an embodiment of the present application, a query image refers to an image that a user determines to be retrieved by inputting instructions. Specifically, there are many ways to determine a query image. For example, a user may drag a specified image to a search bar, or a user may upload a specified image to a program system, or the query image may be determined by selecting a search instruction from a displayed selection menu in response to an instruction triggered by a user on an image. The method of determining a query image will not be described in detail in the embodiments of the present application. For ease of description, the specific application scenario will be described later, taking the example of a user left-clicking an image in a note text and selecting a search instruction from the displayed selection menu. At this time, the image is the query image to be retrieved.

[0064] On this basis, the matching media resources corresponding to the query image refer to a number of media resources that may be relevant to the query image, such as note text, images, videos, etc., which are initially screened out from the preset database after the query image is determined through the platform's recall algorithm, such as common collaborative filtering, vectorized recall, etc. Under normal circumstances, the number of matching media resources is generally in the hundreds or thousands. After the matching media resources are determined by the recall algorithm upstream, these matching media resources will be recorded in the platform's recall system database for further use in the subsequent sorting process. The content that is more likely to be related to the query image will be displayed first through sorting, thereby improving the user's search experience. For details, please refer to the subsequent steps and their explanations.

[0065] S120: Determine a first interaction feature between the query image and the matching media resource according to an image feature of the query image and feature information of the matching media resource.

[0066] In the embodiment of the present application, after determining the query image and the matching media resources to be sorted, it is usually necessary to use the feature information of the query image and the feature information of the matching media resources to determine the similarity between the query item and each matching media resource. Specifically, under normal circumstances, the feature information of the query image generally refers to the image features obtained by performing image processing on the query image, such as image recognition, image encoding, etc.

[0067] The feature information of matching media resources usually needs to consider the type of media resources. For example, for text media resources, their feature information generally refers to text features obtained through text processing, such as text segmentation, word encoding, etc. For image media resources, the extraction of their feature information can be similar to the query image, obtained through image recognition, image encoding, etc. For video media resources, the extraction of their feature information can be obtained by fusing the image features obtained by image processing of each image frame in the video, or by using video introduction or video attribute information, such as video producer, video time, etc., through text segmentation, word encoding, etc. Of course, in some platforms, media resources may also include multiple of the aforementioned text, image, video or audio, such as text and image resources, text and video resources, etc., but when extracting features, it can be considered to extract each type of feature information separately, and finally splice and fuse them to obtain the feature information of the complete media resource.

[0068] Specifically, since there are many ways to extract feature information, for example, the encoding layers in common deep neural networks can all realize feature encoding of videos, images, and texts to obtain feature information, the embodiments of the present application do not elaborate on the specific methods of determining the image features of the query image and matching the feature information of the media resources.

[0069] Furthermore, considering that in a specific application scenario, the user may determine the query image by clicking on an image in a note text, that is, the image features of the query image at this time, in addition to the features contained in the image itself, can also fuse the text features of the note text to which it belongs with the features contained in the image itself to obtain the image features of the query image, thereby enriching the image features of the query image and improving the effect of subsequent sorting.

[0070] In addition, it should be noted that both image features and text features can be roughly divided into sparse features and dense features. For example, dense features generally include the structural content of the image itself, while sparse features generally include the frequency of occurrence of each word in the text. In order to improve the final sorting effect, it is often necessary to standardize these features. However, due to the different characteristics of dense features and sparse features, in order to further improve the sorting effect, this application proposes a method of standardizing dense features and sparse features based on different standardization methods to obtain the final image features and text features. For details, please refer to the relevant description of the subsequent embodiments.

[0071] After the image features of the query image and the feature information of the matching media resources are determined, in order to enrich the feature information, in the embodiment of the present application, the image features and the feature information of the media resources are often interacted to obtain the first interactive feature. Specifically, the interactive method here can be selected based on actual needs, but considering that the features will be interacted again later, the first interactive feature here can be a low-order interactive feature that is relatively simple to process and has a relatively small amount of calculation, that is, linear processing of the feature on the plane.

[0072] Specifically, low-order interaction features can be obtained by linearly processing the feature information of image features and media resources on a plane through factor decomposition. The factor decomposition machine, i.e., the FM model, is a feature cross-talk method based on the factor decomposition machine, which can capture the second-order interaction relationship between features. Each feature will be mapped to a latent vector space, and then the interaction weight between the features is obtained by calculating the inner product of the two feature latent vectors, so as to further obtain the interaction features between the features based on the interaction weight. Through the FM model, the interaction relationship between the features can be better captured, thereby improving the accuracy of the ranking prediction of the subsequent model.

[0073] S130, screening the image feature, the feature information and the first interactive feature to obtain a target feature.

[0074] After completing the interaction of the feature information of the image features and the media resources and obtaining a larger number of interactive features, if all the interactive features are again subjected to nonlinear interaction in the high-dimensional space, the amount of calculation will increase exponentially, which will seriously affect the calculation efficiency of the subsequent model. Therefore, in the embodiment of the present application, the image features, the feature information of the media resources and the obtained first interactive features will be further screened to filter out irrelevant features and retain as much as possible the relevant features with higher importance as the target features. In other words, in the process of screening the image features, the feature information of the media resources and the first interactive features, it is necessary to fully explore the importance of each feature as much as possible. Therefore, as a feasible embodiment of the present application, a lightweight feature screening model is proposed to adaptively mine the importance of the original features and the diversified interactive features. Of course, it should be noted that, under normal circumstances, in order to enable the feature screening model to adaptively mine the importance of the original features and the diversified interactive features, so as to screen out the target features with higher importance from the image features, the feature information of the media resources and the first interactive features, under normal circumstances, the model needs to be trained. Therefore, for ease of understanding, the scheme will be specifically described in conjunction with the model training process.

[0075] S140: Sort the matching media resources according to the second interaction feature between the target features to obtain sorted matching media resources.

[0076] After the above-mentioned screening of the original features and low-order interactive features and retaining a small number of target features with higher importance, in an embodiment of the present application, the second interactive features between these target features will be used again to sort the matching media resources to obtain specific sorted matching media resources. Specifically, the above process is usually also implemented in the form of a deep network model, that is, the high-dimensional feature interaction between the target features is first realized through the deep convolution layer of the deep network model, and then the final numerical value is further output using the linear layer. The numerical value is usually between 0 and 1, which describes the matching degree (similarity) between the query image and the matching media resources, wherein the higher the numerical value, that is, the closer to 1, the higher the matching degree between the query image and the matching media resources. Subsequently, different matching media resources can be sorted based on the matching degree between the query image and each matching media resource, thereby outputting the final sorted matching media resources and displaying them to the user.

[0077] It can be seen that the above provides a specific implementation scheme for realizing image retrieval ranking, which can be based on different models (model structures) to realize the processing of different steps in image retrieval ranking. Therefore, in order to facilitate understanding of the technical solution of this application, please refer to Figure 2a ,The following provides a schematic flow diagram of the steps for implementing image retrieval and ranking based on a neural network model, which is described in detail below.

[0078] Image retrieval sorting usually includes three processes of low-order feature interaction, feature screening and high-order feature interaction sorting, among which different processes are usually implemented based on different neural network model processing. Considering that there are many network model structures that can be used to implement low-order feature interaction, feature screening and high-order feature interaction in the related technology, therefore, the embodiment of the present application mainly provides a specific feasible implementation scheme. In actual application scenarios, users can appropriately replace any model used in the processing process based on actual needs, such as the factor decomposition machine model, feature screening model and deep network model provided below, which should be regarded as within the scope of protection claimed in the present application. Specifically, it includes steps S210 to S250:

[0079] S210: Inputting the image features of the query image and the feature information of the matching media resources into a preset factorization machine model to obtain low-order interaction features between the query image and the matching media resources.

[0080] In the embodiment of the present application, by using a pre-set factor decomposition machine model to process the image features of the query image and the feature information of the matching media resources, it is possible to achieve linear interaction between the image features and the feature information of the media resources in a low-dimensional plane space, thereby obtaining low-order interaction features between the query image and the matching media resources. Of course, in addition to the factor decomposition machine model to achieve low-order interaction of features, it is also feasible to achieve low-order interaction of features based on a vector machine model, etc. Of course, other network model structures that can achieve linear combination of features on a spatial plane to obtain low-order interaction features are all within the scope of protection required by this application.

[0081] In addition, it should be noted that, considering that different users have different preferences for different notes in actual application scenarios, the user factor can also be appropriately considered in the process of sorting matching media resources. Therefore, as a feasible embodiment of the present application, in the process of realizing the feature interaction of image features and feature information of media resources to obtain low-order interaction features, user features can also be integrated. For details, please refer to Figure 3 , Figure 3 A schematic flow chart of a step of fusing user features to obtain interaction features provided in an embodiment of the present application, specifically, comprising steps S310 to S320:

[0082] S310: Obtain user features corresponding to the query image.

[0083] In the embodiment of the present application, the user characteristics corresponding to the query image refer to the user characteristics of the user who triggers the image query instruction, wherein the user characteristics may include characteristics such as the media resources that the user has viewed historically and the time when each media resource was viewed. Of course, after obtaining user permissions, the user characteristics may also include user portraits such as gender, age, etc. The embodiment of the present application does not elaborate on the user characteristics here.

[0084] S320: Input the image features of the query image, the user features, and the feature information of the matching media resources into a preset factorization machine model to obtain a first interaction feature between the query image and the matching media resources.

[0085] In an embodiment of the present application, after determining the feature information of the image features, user features, and media resources as mentioned above, similar to step S210, by inputting the feature information of the image features, user features, and media resources into a preset factor decomposition machine model, the factor decomposition machine model can complete the linear combination of these features in the plane space, thereby obtaining a number of first interaction features, that is, low-order interaction features.

[0086] S220, inputting the image features, the information features, and the low-order interaction features into a pre-trained feature screening model to obtain target features.

[0087] After a large number of low-order interaction features are obtained through feature interaction as mentioned above, in order to avoid the excessive number of features seriously affecting the efficiency of high-order feature interaction, in an embodiment of the present application, image features, feature information of media resources and low-order interaction features are further input into a pre-trained feature screening model that is adapted to mining the importance of original features and diversified interaction features, thereby screening and retaining target features with higher importance, thereby reducing the amount of calculation for subsequent high-order feature interactions without losing important feature information.

[0088] For more information, please refer to Figure 2b As shown, Figure 2bA structural schematic diagram of a lightweight feature screening model provided for an embodiment of the present application, specifically, the feature screening model is usually composed of Pooling (pooling layer), LinearLayer (linear layer), ReLU (linear rectification function, i.e., modified linear unit), Dropout (random deactivation, used to reduce the overfitting phenomenon of the model) and Softmax (normalized exponential function, used to output the final result) structure, wherein, after the original feature Feature is processed by each structure in the model in turn, a vector describing the importance of each original feature Feature can be obtained, and then after cross-multiplication with the original feature Feature, the refined feature Refined Feature with higher importance can be retained, i.e., the target feature. Specifically, after the original feature Feature is processed by the pooling layer of the feature screening model, the feature can be effectively compressed to remove redundant information, reduce the amount of data of the parameters input to the linear layer, and thus improve the model processing efficiency. When the compressed features are processed by the linear layer, the linear layer can use the trained model parameters to further extract the linear feature information of the compressed features and describe them in the form of latent vectors. The linear rectification function ReLU can improve the nonlinear fitting ability of the neural network and enhance the expression ability of the model. Therefore, the latent vector of the linear feature information describing the features obtained by the linear layer processing can further extract the nonlinear feature information of the features after being processed by the linear rectification function and express it in the form of a latent vector. The latent vector after the above processing will continue to be input into the random inactivation layer for processing, in which the model parameters in the random inactivation layer are randomly invalidated with a certain probability during the training process, thereby reducing the overfitting phenomenon of the model during the training process. The final output result will be mapped to a value between 0 and 1 after being processed by the normalized exponential function, which is used to describe the importance of the feature. Specifically, the closer the value is to 1, the more important the feature is after model analysis and processing, and it should be retained. Conversely, the closer the value is to 0, the less important the feature is after model analysis and processing, and it can be considered for elimination. Therefore, after the above feature screening model completes the processing of each original feature Feature, the more important refined features can be retained based on the value output by the normalized exponential function. Of course, the above Figure 2b It only shows a structural diagram of a feasible feature screening model. In fact, based on this model structure and based on actual needs, the user makes routine deletions, replacements and additions to the model structure itself. The resulting other model structures that can also achieve feature screening should be deemed to be within the scope of protection required by this application.

[0089] Of course, in actual application, the feature screening model contains several model parameters, which often need to be determined in advance through training. Subsequently, the specific implementation steps for model training will be provided in combination with the deep network model.

[0090] S230, inputting the target features into a deep convolutional layer in a pre-trained deep network model to obtain high-order interaction features between the target features.

[0091] After adaptively mining the importance of original features and diversified interactive features through the feature screening model and retaining more important target features while discarding a large number of irrelevant features, we can subsequently use the high-order interactive features between these target features to better achieve the sorting of matching media resources while effectively reducing the amount of calculation.

[0092] Specifically, high-order interactions of features can often be achieved through deep network models. The deep convolutional layers in the deep network models can not only realize linear combinations of features, but also nonlinear combinations of features, thereby obtaining high-order interaction features of target features in higher-dimensional space.

[0093] S240: Input the high-order interaction features into a linear layer in the deep network model to obtain a matching degree between the query image and each of the matching media resources.

[0094] Furthermore, after determining the high-order interaction features in the higher-dimensional space as mentioned above, these high-order interaction features are parsed by using a linear layer, and using the final activation function, the degree of match between the query image and the matching media resources can be described by a numerical value between 0 and 1, wherein the higher the numerical value, the higher the degree of match between the image and the matching media resources.

[0095] S250: Sort the matching media resources according to the matching degree to obtain sorted matching media resources.

[0096] In the embodiment of the present application, the matching media resources are sorted in descending order in combination with the above-mentioned calculated matching degrees, so as to obtain the sorted and displayed matching media resources.

[0097] Of course, it should be noted that in the aforementioned process, in order to train a feature screening network that can mine the importance of original features and diversified interactive features, and a deep network model that realizes matching calculation based on high-order feature interactions, it is usually necessary to train with a certain amount of samples. Specifically, the training samples usually include training sample images, training sample media resources, and matching labels between them. Specifically, the training process is as follows:

[0098] (1) Constructing an initial feature screening model and an initial deep network model for initializing model parameters;

[0099] (2) Inputting the training sample images and training sample media resources into the FM model to obtain low-order interaction features, and then inputting the training sample images, training sample media resources and low-order interaction features into the initial feature screening model to obtain training features;

[0100] (3) inputting the training features into the initial deep network model to predict the matching degree between the training sample images and the training sample media resources by utilizing the high-order interaction features between the training features, thereby obtaining the predicted matching degree;

[0101] (4) According to the difference between the predicted matching degree and the matching degree label, combined with the idea of ​​back propagation, the parameters in the initial feature screening model and the initial deep network model are optimized and adjusted to obtain the updated feature screening model and deep network model;

[0102] (5) Based on the contents of the aforementioned steps (2) to (3), the updated feature screening model and the deep network model are used for further processing to obtain an updated predicted matching degree, and the difference between the predicted matching degree and the matching degree label is used again, combined with the idea of ​​back propagation, to optimize and adjust the parameters of the updated feature screening model and the deep network model again until certain conditions are met. For example, after the parameters of the feature screening model and the deep network model are optimized for a certain number of times, or when the difference between the predicted matching degree and the matching degree label obtained for several consecutive times is small, the current feature screening model and the deep network model can be determined as the network model finally used for image retrieval and ranking.

[0103] Specifically, to understand the above process, please refer to Figure 2c , Figure 2c A schematic diagram of a model structure for implementing image retrieval and ranking is provided in an embodiment of the present application, as detailed below.

[0104] When the image retrieval sorting is triggered, the user features (user), the image features of the query image (query), and the feature information of the matching media resources (doc) are extracted to form the final feature vector. After being processed based on different feature standardization processes, the interaction of low-order features is realized through the factorization machine (FM) model. After the interactive features are obtained, they are input into the lightweight feature screening model LightModule together with the original features, including user features, image features, and feature information of media resources, to obtain the screened features, and are processed by multiple convolutional layers in the deep network model to obtain the final matching result. After performing the above operations on each matching media resource, the matching media resources can be sorted using the matching degree between the query image and each matching media resource.

[0105] The image retrieval and ranking method provided in the embodiment of the present application, after obtaining the query image and matching media resources, first interacts between the image features and the feature information of the media resources to obtain interactive features with richer features, and then screens the features to omit irrelevant features and retain target features with higher importance, so as to further use the interactive features between the target features to sort the matching media resources, which can effectively reduce the amount of model calculation while ensuring the validity of the final ranking result.

[0106] In addition, in the process of retrieving and sorting images, in order to improve the final sorting effect, it is often necessary to standardize the features. Considering the differences between sparse features and dense features contained in the features, please refer to Figure 4 Specifically, the image features of the query image are used as an example to describe the schematic flow chart of the steps of standardizing the dense features and the sparse features using different standardization methods, specifically including steps S410 to S420:

[0107] S410: Process the query image to obtain initial sparse features and initial dense features of the query image.

[0108] In an embodiment of the present application, the initial sparse features of the query image may include the encoding features of the entity types in the image obtained by image recognition processing, and the initial dense features generally refer to the image structured features obtained by grayscale processing of the image. The definitions of sparse features and dense features are not repeated in this embodiment of the present application. Those skilled in the art can determine whether the features of different dimensions obtained by processing the query image belong to sparse features or density features based on the definitions of sparse features and dense features. Similarly, the sparse features and dense features obtained by matching media resources are not repeated in this embodiment of the present application.

[0109] S420, performing a first normalization process on the initial sparse features, and performing a second normalization process on the initial dense features, so as to obtain image features of the query image.

[0110] After determining the initial sparse features and the initial dense features of the image as mentioned above, taking into account the characteristics of the sparse features and the dense features, in an embodiment of the present application, the initial sparse features are standardized by means of SyncBatchNorm, wherein SyncBatchNorm is a batch normalization method that can achieve synchronous standardization, thereby improving the training efficiency and accuracy of the model, while for dense features, they are standardized by means of LayerNor, wherein LayerNor can better adapt to the differences between different samples by standardizing the feature dimensions of each sample, thereby improving the generalization ability of the model.

[0111] Of course, in addition to the standardized processing methods provided above, standardized processing of sparse features and dense features can also be achieved based on other conventional standardized methods, but obviously different standardized processing will ultimately affect the results of image retrieval and ranking. Users can adopt corresponding standardized processing methods based on actual needs to complete the processing of sparse features and dense features.

[0112] Furthermore, considering that in a specific application scenario, the user may determine the query image by clicking on an image in a note text, that is, the image features may also include text features in the note text associated with the query image. For details, please refer to Figure 5 ,like Figure 5 As shown, Figure 5 A schematic diagram of a process flow for determining image features provided in an embodiment of the present application, specifically, comprising steps S510 to S520:

[0113] S510: In response to a trigger instruction for a query image, obtain a reference text corresponding to the query image.

[0114] In an embodiment of the present application, in response to a user clicking on an image in a note text to trigger a query instruction for the image, the note text at this time is the reference text corresponding to the query image.

[0115] S520: Process the reference text and the query image to obtain image features of the query image.

[0116] Compared with the aforementioned implementation scheme of obtaining the image features of the query image by performing image processing on the query image, in the embodiment of the present application, a number of reference text features are additionally obtained by performing text encoding, text segmentation and other processing methods on the text content in the reference text. The reference text features are added and fused with the features obtained by performing image processing on the query image to obtain the image features of the query image.

[0117] In addition, considering that the embodiment of the present application is mainly used in the fine sorting stage of image retrieval sorting, that is, it is necessary to determine in advance through a recall algorithm a number of matching media resources that are relevant to the query image, therefore, as a feasible embodiment of the present application, please refer to Figure 6 , Figure 6 A schematic diagram of a process flow for determining matching media resources provided in an embodiment of the present application specifically includes steps S610 to S620:

[0118] S610: Acquire a query image to be retrieved and a query feature corresponding to the query image.

[0119] In an embodiment of the present application, the query features corresponding to the query image may be the same as the aforementioned image features, and of course may also be query features with less information obtained by using other simplified image processing methods, so as to facilitate the subsequent screening of suitable content from massive database information as matching media resources.

[0120] S620: Determine, from the initial media resources, matching media resources corresponding to the query image according to the similarity between the query feature and feature information of each initial media resource in a preset database.

[0121] In an embodiment of the present application, in the recall stage, by calculating the similarity between the query features and the feature information of each initial media resource in a preset database, and performing preliminary screening and filtering, a certain number of matching media resources corresponding to the query image can be determined from all the media resources contained in the database, and the matching media resources will subsequently be further sorted and displayed through the image retrieval sorting method mentioned above.

[0122] In order to better implement the image retrieval and ranking method provided in the embodiment of the present application, on the basis of the image retrieval and ranking method provided in the embodiment of the present application, an image retrieval and ranking device is also provided in the embodiment of the present application, such as Figure 7 As shown, the image retrieval and sorting device 700 includes:

[0123] An acquisition module 710 is used to acquire a query image to be retrieved and matching media resources corresponding to the query image;

[0124] An interaction module 720, configured to determine a first interaction feature between the query image and the matching media resource according to an image feature of the query image and feature information of the matching media resource;

[0125] A screening module 730, configured to screen the image feature, the feature information and the first interactive feature to obtain a target feature;

[0126] The sorting module 740 is configured to sort the matching media resources according to the second interaction feature between the target features to obtain sorted matching media resources.

[0127] In some embodiments of the present application, the interaction module 720 is further used to input the image features of the query image and the feature information of the matching media resources into a preset factor decomposition machine model to obtain a first interaction feature between the query image and the matching media resources.

[0128] In some embodiments of the present application, the interaction module 720 is also used to obtain user features corresponding to the query image; input the image features of the query image, the user features, and the feature information of the matching media resources into a preset factor decomposition machine model to obtain a first interaction feature between the query image and the matching media resources.

[0129] In some embodiments of the present application, the screening module 730 is further used to input the image feature, the feature information and the first interactive feature into a pre-trained feature screening model to obtain a target feature.

[0130] In some embodiments of the present application, the sorting module 740 is also used to input the target features into the deep convolutional layer in the pre-trained deep network model to obtain a second interaction feature between the target features; input the second interaction feature into the linear layer in the deep network model to obtain the matching degree between the query image and each of the matching media resources; and sort each of the matching media resources according to the matching degree to obtain sorted matching media resources.

[0131] In some embodiments of the present application, before the step of determining the first interaction feature between the query image and the matching media resource based on the image features of the query image and the feature information of the matching media resource, the interaction module 720 is also used to process the query image to obtain the initial sparse features and initial dense features of the query image; perform a first normalization process on the initial sparse features, and perform a second normalization process on the initial dense features to obtain the image features of the query image.

[0132] In some embodiments of the present application, before the step of determining the first interaction feature between the query image and the matching media resource based on the image features of the query image and the feature information of the matching media resource, the interaction module 720 is also used to obtain a reference text corresponding to the query image in response to a trigger instruction for the query image; and process the reference text and the query image to obtain the image features of the query image.

[0133] In some embodiments of the present application, the acquisition module 710 is also used to obtain a query image to be retrieved and a query feature corresponding to the query image; and determine a matching media resource corresponding to the query image from the initial media resources based on the similarity between the query feature and feature information of each initial media resource in a preset database.

[0134] The specific definition of the image retrieval and ranking device can be found in the definition of the image retrieval and ranking method above, which will not be repeated here. Each module in the above-mentioned image retrieval and ranking device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0135] In some embodiments of the present application, the image retrieval and ranking device 700 can be implemented in the form of a computer program. The computer program can be implemented in Figure 8 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the image retrieval and sorting device 700, for example, Figure 7 The acquisition module 710, interaction module 720, screening module 730 and sorting module 740 are shown. The computer program composed of various program modules enables the processor to execute the steps of the image retrieval and sorting method of each embodiment of the present application described in this specification.

[0136] For example, Figure 8 The computer device shown can be Figure 7 The acquisition module 710 in the image retrieval and sorting device 700 shown in the figure performs step S110. The computer device can perform step S120 through the interaction module 720. The computer device can perform step S130 through the screening module 730. The computer device can perform step S140 through the sorting module 740. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external computer device through a network connection. When the computer program is executed by the processor, an image retrieval and sorting method is implemented.

[0137] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] In some embodiments of the present application, a computer device is provided, comprising one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the following steps:

[0139] Acquire a query image to be retrieved and matching media resources corresponding to the query image;

[0140] determining a first interaction feature between the query image and the matching media resource according to an image feature of the query image and feature information of the matching media resource;

[0141] Screening the image feature, the feature information, and the first interactive feature to obtain a target feature;

[0142] The matching media resources are sorted according to the second interaction feature between the target features to obtain sorted matching media resources.

[0143] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: inputting the image features of the query image and the feature information of the matching media resources into a preset factor decomposition machine model to obtain a first interaction feature between the query image and the matching media resources.

[0144] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: obtaining user features corresponding to the query image; inputting the image features of the query image, the user features, and the feature information of the matching media resources into a preset factor decomposition machine model to obtain a first interaction feature between the query image and the matching media resources.

[0145] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: inputting the image feature, the feature information and the first interactive feature into a pre-trained feature screening model to obtain a target feature.

[0146] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: inputting the target features into the deep convolution layer in the pre-trained deep network model to obtain a second interaction feature between the target features; inputting the second interaction feature into the linear layer in the deep network model to obtain the degree of match between the query image and each of the matching media resources; and sorting each of the matching media resources according to the degree of match to obtain sorted matching media resources.

[0147] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: processing the query image to obtain initial sparse features and initial dense features of the query image; performing a first normalization process on the initial sparse features, and performing a second normalization process on the initial dense features to obtain image features of the query image.

[0148] In some embodiments of the present application, the processor further implements the following steps when executing the computer program: in response to a trigger instruction for a query image, obtaining a reference text corresponding to the query image; and processing the reference text and the query image to obtain image features of the query image.

[0149] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: obtaining a query image to be retrieved, and a query feature corresponding to the query image; and determining a matching media resource corresponding to the query image from the initial media resources based on the similarity between the query feature and feature information of each initial media resource in a preset database.

[0150] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Any reference to memory, storage, information library or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0151] In some embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program. The computer program is loaded by a processor, so that the processor performs the following steps:

[0152] Acquire a query image to be retrieved and matching media resources corresponding to the query image;

[0153] determining a first interaction feature between the query image and the matching media resource according to an image feature of the query image and feature information of the matching media resource;

[0154] Screening the image feature, the feature information, and the first interactive feature to obtain a target feature;

[0155] The matching media resources are sorted according to the second interaction feature between the target features to obtain sorted matching media resources.

[0156] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: inputting the image features of the query image and the feature information of the matching media resources into a preset factor decomposition machine model to obtain a first interaction feature between the query image and the matching media resources.

[0157] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: obtaining user features corresponding to the query image; inputting the image features of the query image, the user features, and the feature information of the matching media resources into a preset factor decomposition machine model to obtain a first interaction feature between the query image and the matching media resources.

[0158] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: inputting the image feature, the feature information and the first interactive feature into a pre-trained feature screening model to obtain a target feature.

[0159] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: inputting the target features into the deep convolution layer in the pre-trained deep network model to obtain a second interaction feature between the target features; inputting the second interaction feature into the linear layer in the deep network model to obtain the matching degree between the query image and each of the matching media resources; and sorting each of the matching media resources according to the matching degree to obtain sorted matching media resources.

[0160] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: processing the query image to obtain initial sparse features and initial dense features of the query image; performing a first normalization process on the initial sparse features, and performing a second normalization process on the initial dense features to obtain image features of the query image.

[0161] In some embodiments of the present application, the processor further implements the following steps when executing the computer program: in response to a trigger instruction for a query image, obtaining a reference text corresponding to the query image; and processing the reference text and the query image to obtain image features of the query image.

[0162] In some embodiments of the present application, when the processor executes the computer program, it also implements the following steps: obtaining a query image to be retrieved, and a query feature corresponding to the query image; and determining a matching media resource corresponding to the query image from the query media resources based on the similarity between the query feature and feature information of each initial media resource in a preset database.

[0163] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The above is a detailed introduction to an image retrieval and sorting method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image retrieval and ranking method, characterized in that: include: Acquire a query image to be retrieved and matching media resources corresponding to the query image; determining a first interaction feature between the query image and the matching media resource according to an image feature of the query image and feature information of the matching media resource; Screening the image feature, the feature information, and the first interactive feature to obtain a target feature; The matching media resources are sorted according to the second interaction feature between the target features to obtain sorted matching media resources.

2. The image retrieval and ranking method according to claim 1, characterized in that: The first interaction feature is a low-order interaction feature; The determining, according to the image feature of the query image and the feature information of the matching media resource, a first interaction feature between the query image and the matching media resource comprises: The image features of the query image and the feature information of the matching media resources are input into a preset factorization machine model to obtain a first interaction feature between the query image and the matching media resources.

3. The image retrieval and ranking method according to claim 2, characterized in that: The step of inputting the image feature of the query image and the feature information of the matching media resource into a preset factorization machine model to obtain a first interaction feature between the query image and the matching media resource includes: Acquire user features corresponding to the query image; The image features of the query image, the user features, and the feature information of the matching media resources are input into a preset factorization machine model to obtain a first interaction feature between the query image and the matching media resources.

4. The image retrieval and ranking method according to claim 1, characterized in that: The screening of the image feature, the feature information and the first interactive feature to obtain a target feature includes: The image feature, the feature information and the first interactive feature are input into a pre-trained feature screening model to obtain a target feature.

5. The image retrieval and ranking method according to claim 1, characterized in that: The second interaction feature is a high-order interaction feature; The step of sorting the matching media resources according to the second interaction feature between the target features to obtain sorted matching media resources includes: Inputting the target features into a deep convolutional layer in a pre-trained deep network model to obtain a second interaction feature between the target features; Inputting the second interactive feature into a linear layer in the deep network model to obtain a matching degree between the query image and each of the matching media resources; The matching media resources are sorted according to the matching degree to obtain sorted matching media resources.

6. The image retrieval and ranking method according to claim 1, characterized in that: Before the step of determining the first interaction feature between the query image and the matching media resource according to the image feature of the query image and the feature information of the matching media resource, the method further includes: Processing the query image to obtain initial sparse features and initial dense features of the query image; A first normalization process is performed on the initial sparse features, and a second normalization process is performed on the initial dense features to obtain image features of the query image.

7. The image retrieval and ranking method according to claim 1, characterized in that: Before the step of determining the first interaction feature between the query image and the matching media resource according to the image feature of the query image and the feature information of the matching media resource, the method further includes: In response to a trigger instruction for a query image, obtaining a reference text corresponding to the query image; The reference text and the query image are processed to obtain image features of the query image.

8. The image retrieval and ranking method according to any one of claims 1 to 7, characterized in that: The step of obtaining a query image to be retrieved and a matching media resource corresponding to the query image includes: Acquire a query image to be retrieved and a query feature corresponding to the query image; According to the similarity between the query feature and feature information of each initial media resource in a preset database, a matching media resource corresponding to the query image is determined from the initial media resources.

9. An image retrieval and ranking device, characterized in that: include: An acquisition module, used to acquire a query image to be retrieved and matching media resources corresponding to the query image; an interaction module, configured to determine a first interaction feature between the query image and the matching media resource according to an image feature of the query image and feature information of the matching media resource; A screening module, used for screening the image feature, the feature information and the first interactive feature to obtain a target feature; The sorting module is used to sort the matching media resources according to the second interaction feature between the target features to obtain sorted matching media resources.

10. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the image retrieval and ranking method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the image retrieval and ranking method according to any one of claims 1 to 8.

Images