Interactive image retrieval method based on multi-modal fusion

Abstract

The invention belongs to the field of image retrieval, and particularly relates to an interactive image retrieval method based on multi-modal fusion. Comprising the following steps: encoding a text, a sketch and an image by adopting a deep neural network model to obtain a text embedding vector, a sketch embedding vector and an image embedding vector; fusing the text embedding vector and the sketch embedding vector to obtain a fused embedding vector; constructing a reward function according to the image embedding vector and the fusion embedding vector; constructing and training a reinforcement learning model containing a reward function, and finishing model training when a preset number of training times is reached; the sketch of the user and the text description of the retrieval object by the user are obtained in real time, the sketch and the text are input into the trained reinforcement learning model for real-time retrieval, and the model feeds back the retrieved image to the user; the user further optimizes the sketch according to the feedback image until a target image satisfying the user is retrieved; according to the method, the stable performance of retrieval feedback can be ensured, and better retrieval experience is realized.

Description

Technical Field

[0001] The present invention belongs to the field of image retrieval, and in particular relates to an interactive image retrieval method based on multimodal fusion. Background Art

[0002] With the widespread use of touch screen devices, the field of computer vision has made significant progress in sketch-related issues. Among them, sketch-based image retrieval has received special attention due to its wide commercial application potential. Although traditional sketch-based image retrieval technology has made great progress, its promotion in practical applications still faces some obstacles, such as the long time required to draw a complete sketch and the limitation of user drawing ability. Traditional methods require the use of complete sketches for target recognition, but in commercial application scenarios, users often do not have enough time to draw high-quality sketches. Therefore, in practice, people hope to be able to retrieve the target image quickly and accurately in the shortest time, even if the user has limited drawing skills.

[0003] The prior art proposes a real-time retrieval method for hand-drawn images based on multimodal data fusion. The method performs real-time retrieval based on the number of sketches drawn by the artist each time and returns the retrieval results. The artist continues to add drawings based on the retrieval results until the target person appears. This method provides a hybrid mode of human-computer interaction, thereby improving retrieval efficiency.

[0004] However, the existing technology ignores the problems caused by differences in user drawing skills. Since different users draw different sketches in different styles, especially in the early stages of drawing, the retrieval results may be affected by the randomness of the sketch strokes, resulting in the feedback image results not accurately reflecting the user's intentions. This instability of the retrieval results may seriously affect the user experience in practical applications, causing operational inconveniences and other problems. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes an interactive image retrieval method based on multimodal fusion, which includes:

[0006] S1: Obtain a multimodal dataset, including text, sketches, and images; use a deep neural network model to encode the multimodal data to obtain text embedding vectors, sketch embedding vectors, and image embedding vectors;

[0007] S2: Fusion of text embedding vector and sketch embedding vector to obtain fused embedding vector;

[0008] S3: Construct a reward function based on the image embedding vector and the fusion embedding vector;

[0009] S4: Build a reinforcement learning model including a reward function and train it. When the preset number of training times is reached, the model training is completed.

[0010] S5: Obtain the user's sketch and the user's text description of the retrieval object in real time, input the sketch and text into the trained reinforcement learning model for real-time retrieval, and the model feeds back the retrieved image to the user; the user further optimizes the sketch based on the feedback image until the target image that satisfies the user is retrieved.

[0011] Preferably, the deep neural network model includes an image branch and a text branch;

[0012] The image branch includes an image encoder and a fully connected layer for processing images and sketches;

[0013] The text branch includes a text encoder and a fully connected layer for processing text.

[0014] Preferably, the process of constructing the reward function includes:

[0015] The inverse of the ranking of the target image corresponding to each sketch is used as the image similarity reward;

[0016] Calculate the multimodal feature fusion reward based on the fusion embedding vector and the target image embedding vector;

[0017] Calculate the step penalty based on the number of sketch steps, design error adjustment and stable period reward; use the image similarity reward as the basic reward; calculate the global reward based on the basic reward, step penalty, error adjustment and stable period reward;

[0018] The reward function is constructed by taking the sum of image similarity reward, multimodal feature fusion reward and global reward as the total reward.

[0019] Furthermore, the formula for calculating the multimodal feature fusion reward is:

[0020]

[0021]

[0022] in, represents the multimodal feature fusion reward, Represents the fused embedding vector v * and the target image embedding vector The similarity.

[0023] Furthermore, the formula for calculating the step penalty is:

[0024]

[0025] in, represents the step penalty of the i-th step, step represents the current step number, and total q Indicates the maximum number of steps.

[0026] Furthermore, the error adjustment is expressed as:

[0027]

[0028] Among them, R Err Represents the error adjustment value, α represents the adjustment coefficient, R(i) represents the base reward at the i-th step, R(i-SErrA) represents the base reward at SErrA steps before the current step i, and SErrA represents the number of interval steps.

[0029] Furthermore, the stationary period reward is expressed as:

[0030]

[0031] in, Represents the stable reward value of the i-th step, Rank i represents the ranking of the target image at the i-th step, and PlatB represents the plateau threshold.

[0032] Furthermore, the formula for calculating the global reward is:

[0033]

[0034] in, represents the global reward of the i-th step, R(i) represents the basic reward of the i-th step, represents the step penalty of the i-th step, R Err Indicates the error adjustment value, represents the stable reward value of the i-th step, SErrA represents the reward error threshold, and n represents the total number of sketch steps.

[0035] The beneficial effects of the present invention are as follows: the present invention adopts a reinforcement learning framework, and in view of the problem that the sparse strokes in the initial stage of sketch drawing lead to an unstable retrieval process, the present invention proposes to design a reward optimization mechanism for the retrieval feedback process, and optimizes the matching strategy between sketches and positive and negative samples. This method can not only improve the retrieval performance, but also constrain the results of the current sketch according to the feedback results of the previous sketch in the sketch drawing process, ensure the stability of the retrieval feedback, and achieve a better retrieval experience. The present invention adopts a human-computer hybrid interaction method to perform portrait retrieval. After obtaining the retrieval results of the model each time, the user does not rely on a high-level painter, and the user can further optimize the sketch according to the feedback results until the target image is retrieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the training process of the reinforcement learning model of the interactive image retrieval method in the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The present invention proposes an interactive image retrieval method based on multimodal fusion, such as Figure 1 As shown, the method includes:

[0039] S1: Obtain a multimodal dataset, including text, sketches, and images; use a deep neural network model to encode the multimodal data to obtain text embedding vectors, sketch embedding vectors, and image embedding vectors.

[0040] Obtain a multimodal data set, including sketches, images, and text descriptions corresponding to the sketches; if the present invention is applied to face image retrieval, construct a small batch of training data based on the face image-sketch data set; the small batch training data includes a sketch sequence S = {s1, s2, ... s n}, the text T of the sketch, and the face image; the face image includes the target face image corresponding to the sketch.

[0041] A deep neural network model is used to encode multimodal data to obtain text embedding vectors, sketch embedding vectors, and image embedding vectors; specifically:

[0042] The deep neural network model includes an image branch and a text branch; the image branch includes an image encoder E I (·) and fully connected layers Used to process images and sketches to obtain sketch embedding vectors and image embedding vectors.

[0043] The text branch includes the text encoder E T (·) and fully connected layers Used to process text and obtain text embedding vectors.

[0044] S2: Fuse the text embedding vector and the sketch embedding vector to obtain the fused embedding vector.

[0045] In the preliminary sketching stage, due to the lack of details, it is difficult to capture the complete semantics with visual information alone, so it is necessary to introduce text description as auxiliary information to enhance the semantic expression of the sketch. The text embedding vector and the sketch embedding vector are fused (the fusion strategy is: the text embedding vector and the sketch embedding vector of the same dimension are fused into one using vector multiplication (i.e. Hadamard product) to obtain the fused embedding vector.

[0046] S3: Construct a reward function based on the image embedding vector and the fused embedding vector.

[0047] The reward function designed by the present invention includes three parts: image similarity reward, multimodal feature fusion reward and global reward.

[0048] Image Similarity Reward:

[0049] The retrieval goal of the present invention is similar to the application of reinforcement learning in optimizing non-differentiable task indicators in computer vision. The present invention hopes to retrieve the target image using the least strokes. When retrieving sketches, the higher the matching degree between the sketch and the target image, the higher the position of the target image retrieved by the sketch in the ranking and the higher the reward. Therefore, the optimization goal is to maximize the reverse ranking of the sketch sequence, and the reward is designed according to this goal:

[0050]

[0051] in, represents the image similarity reward of the i-th step sketch, Rank si represents the target image ranking of the sketch in step i; s i The higher the ranking (i.e. the smaller the ranking number), the greater the corresponding reward.

[0052] Multimodal feature fusion reward:

[0053] In order to evaluate the consistency between the fused multimodal features and the target image at the visual and semantic levels, this paper proposes a similarity-based reward mechanism. This mechanism guides the model optimization generation process by calculating the cosine similarity between the sketch-text pair and the target image embedding vector:

[0054]

[0055]

[0056] in, represents the multimodal feature fusion reward, Represents the fused embedding vector v * and the target image embedding vector Here, cosine similarity is used to ensure that the generated sketch is consistent with the target image at the descriptive text level.

[0057] Global Rewards:

[0058] (1) Basic reward: For a sketch sequence S, the basic reward value is determined by the inverse of the retrieval ranking of each sketch, that is, the image similarity reward is used as the basic reward.

[0059] (2) Step penalty: Step penalty is a mechanism designed to regulate the dynamic behavior of the model throughout the retrieval process. The penalty function decreases as the number of steps increases, that is, in the early steps of the retrieval process, the penalty is lower and the reward is higher, thereby encouraging the model to make accurate predictions early on:

[0060]

[0061] in, represents the step penalty of the i-th step, step represents the current step number, and total q Indicates the maximum number of steps.

[0062] (3) Error adjustment: Error adjustment aims to impose a penalty on the error at a specific step. As the number of sketch strokes increases, the retrieval results do not increase monotonically, but fluctuate. Therefore, a reward error threshold (SErrA) is designed. At a specific SErrA*m step, if the retrieval result increases compared to the previous one, the impact on the reward is small. Once the error exceeds the tolerance, the reward will be greatly affected:

[0063]

[0064] Among them, R Err Represents the error adjustment value, α represents the adjustment coefficient, R(i) represents the base reward at the i-th step, and R(i-SErrA) represents the base reward at SErrA steps before the current step i.

[0065] (4) Steady-state rewards: As the completeness of the sketch increases, the retrieval results generally show an upward trend. When a specific stable-state threshold PlatB is reached, the reward no longer increases and remains stable:

[0066]

[0067] in, represents the stable period reward value of the i-th step, and PlatB represents the stable period threshold.

[0068] The global reward function guides the model to learn to retrieve images quickly and accurately in the early steps and maintain stable performance in the later steps through the mechanism of rewards and penalties. i The global rewards are:

[0069]

[0070] in, represents the global reward of the i-th step, R(i) represents the basic reward of the i-th step, represents the step penalty of the i-th step, R Err Indicates the error adjustment value, represents the stable reward value of the i-th step, SErrA represents the reward error threshold, and n represents the total number of sketch steps.

[0071] The sum of image similarity reward, multimodal feature fusion reward and global reward is used as the total reward to construct the reward function R si .

[0072] S4: Build a reinforcement learning model including a reward function and train it. When the preset number of training times is reached, the model training is completed.

[0073] Building a reinforcement learning model:

[0074] In the model design of the present invention, the retrieval process is modeled as a Markov decision process in a reinforcement learning environment, and the state space represented by the search results and input conditions (the entire sequence of the sketch drawing process) and the action space consisting of various possible search behaviors are defined respectively. The sketch feature vector can be converted into a decision action by training the policy network parameters including the pre-trained image encoder and the fully connected layer. In the retrieval process of each step of the sketch, the distribution parameters of the action are predicted by this policy network, and then the action to be taken in the current step is sampled from this distribution. The training goal of the model is to maximize the cumulative reward. Specifically, the reward and the discounted return of each time step are calculated, and then the loss function is attempted to be minimized by gradient updating, which is the product of the negative return and the probability of action selection. The introduction of the gradient update strategy enables the model to make the optimal behavior selection according to the given state in a reinforcement learning environment. Therefore, this model fully combines the advantages of deep learning and reinforcement learning, and realizes dynamic and effective behavior decision-making in complex image search tasks.

[0075] The image encoder acts as a key branch of the agent in the reinforcement learning framework. The feature vector v of the sketch si Used to generate actions. The generation of actions is based on a random continuous Gaussian strategy defined by a multivariate normal distribution. Define the strategy as π θ (α i |t i ), where θ represents the image encoder E containing the pre-training I (·) and fully connected layers The policy network parameters are gradually updated during the iteration process. For each step of the sketch s i , according to the strategy π θ (α i |t i ) is constructed by the distribution parameters predicted by the network. Sampling is performed according to this distribution to determine the action α of the current step i , where action α i In status iThe probability of θ (α i |t i ) is determined and sampled from it. This approach combines the power of deep learning with the decision-making process of reinforcement learning, making the strategy both dynamic and efficient. Among them:

[0076] α i ~γ(v si ,∑)

[0077] Among them, action α i is from the mean v si , sampled from a multivariate normal distribution with a covariance matrix of ∑.

[0078] The training goal is to maximize the expected cumulative reward. This paper adopts a variant of the nearest neighbor strategy optimization, in which the objective function is usually defined as the expected log probability of the reward. Since maximizing the log probability of the reward is equivalent to minimizing its negative value, this objective function is often expressed in the form of a loss function and is minimized in practical applications. For each time step i, i∈n, the following calculation is performed to obtain the discounted reward

[0079]

[0080] Among them, n represents the length of the sketch sequence, R si is the reward at the current time step, and δ is the discount factor. The loss function l(θ) is the negative of the sum of the products of the log probability and the discounted reward, and can be written as:

[0081]

[0082] Therefore, the objective function ζ(θ), which the present invention hopes to maximize, is the inverse of this loss function.

[0083]

[0084] Where n is the total number of time steps, π θ (α i |t i ) is the policy network, α i is in i The action taken in the state. Updating the policy parameters by gradient ascent (or minimizing the loss by gradient descent in practice) is actually trying to maximize the expected total reward, thereby training the policy network to learn to choose the best action in a given state.

[0085] When the preset number of training times is reached, the model training is completed.

[0086] S5: Obtain the user's sketch and the user's text description of the retrieval object in real time, input the sketch and text into the trained reinforcement learning model for real-time retrieval, and the model feeds back the retrieved image to the user; the user further optimizes the sketch based on the feedback image until the target image that satisfies the user is retrieved.

[0087] When users cannot accurately describe the target person's features through existing photos or text, they can use a touch screen device to hand-draw the person's portrait for retrieval. Specifically, the portrait sketch drawn by the user will be input into a trained deep neural network model, which will perform retrieval based on the input sketch and return the k images that are most similar to the sketch. In this way, users no longer rely on language or existing images, but only need to hand-draw the target person from memory to find the corresponding matching image.

[0088] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An interactive image retrieval method based on multimodal fusion, characterized in that: include: S1: Obtain a multimodal dataset, including text, sketches, and images; use a deep neural network model to encode the multimodal data to obtain text embedding vectors, sketch embedding vectors, and image embedding vectors; S2: Fusion of text embedding vector and sketch embedding vector to obtain fused embedding vector; S3: Construct a reward function based on the image embedding vector and the fusion embedding vector; S4: Build a reinforcement learning model including a reward function and train it. When the preset number of training times is reached, the model training is completed. S5: Obtain the user's sketch and the user's text description of the retrieval object in real time, input the sketch and text into the trained reinforcement learning model for real-time retrieval, and the model feeds back the retrieved image to the user; the user further optimizes the sketch based on the feedback image until the target image that satisfies the user is retrieved.

2. The interactive image retrieval method based on multimodal fusion according to claim 1, characterized in that: The deep neural network model includes an image branch and a text branch; The image branch includes an image encoder and a fully connected layer for processing images and sketches; The text branch includes a text encoder and a fully connected layer for processing text.

3. The interactive image retrieval method based on multimodal fusion according to claim 1, characterized in that: The process of building a reward function includes: The inverse of the ranking of the target image corresponding to each sketch is used as the image similarity reward; Calculate the multimodal feature fusion reward based on the fusion embedding vector and the target image embedding vector; Calculate the step penalty based on the number of sketch steps, design error adjustment and stable period reward; use the image similarity reward as the basic reward; calculate the global reward based on the basic reward, step penalty, error adjustment and stable period reward; The reward function is constructed by taking the sum of image similarity reward, multimodal feature fusion reward and global reward as the total reward.

4. The interactive image retrieval method based on multimodal fusion according to claim 3, characterized in that: The formula for calculating the multimodal feature fusion reward is: in, represents the multimodal feature fusion reward, Represents the fused embedding vector v * and the target image embedding vector The similarity.

5. The interactive image retrieval method based on multimodal fusion according to claim 3, characterized in that: The formula for calculating the step penalty is: in, represents the step penalty of the i-th step, step represents the current step number, and total q Indicates the maximum number of steps.

6. The interactive image retrieval method based on multimodal fusion according to claim 3, characterized in that: The error adjustment is expressed as: Among them, R Err Represents the error adjustment value, α represents the adjustment coefficient, R(i) represents the base reward at the i-th step, R(i-SErrA) represents the base reward at SErrA steps before the current step i, and SErrA represents the number of interval steps.

7. The interactive image retrieval method based on multimodal fusion according to claim 3, characterized in that: The stationary period reward is expressed as: in, Represents the stable reward value of the i-th step, Rank i represents the ranking of the target image at the i-th step, and PlatB represents the plateau threshold.

8. The interactive image retrieval method based on multimodal fusion according to claim 3, characterized in that: The formula for calculating the global reward is: in, represents the global reward of the i-th step, R(i) represents the basic reward of the i-th step, represents the step penalty of the i-th step, R Err Indicates the error adjustment value, represents the stable reward value of the i-th step, SErrA represents the reward error threshold, and n represents the total number of sketch steps.

Images