An Interactive Image Retrieval Method Based on Multimodal Fusion

Abstract

This invention belongs to the field of image retrieval, specifically relating to an interactive image retrieval method based on multimodal fusion. It includes: encoding text, sketches, and images using a deep neural network model to obtain text embedding vectors, sketch embedding vectors, and image embedding vectors; fusing the text embedding vectors and sketch embedding vectors to obtain a fused embedding vector; constructing a reward function based on the image embedding vector and the fused embedding vector; constructing and training a reinforcement learning model containing the reward function, completing model training after reaching a preset number of training iterations; acquiring the user's sketch and the user's text description of the search object in real time, inputting the sketch and text into the trained reinforcement learning model for real-time retrieval, and the model providing feedback to the user with the retrieved image; the user further optimizes the sketch based on the feedback image until a satisfactory target image is retrieved; this invention ensures stable retrieval feedback performance and achieves a superior retrieval experience.

Description

Technical Field

[0001] This invention belongs to the field of image retrieval, specifically relating to an interactive image retrieval method based on multimodal fusion. Background Technology

[0002] With the widespread adoption of touchscreen devices, the field of computer vision has made significant progress in sketch-related problems. Among these, sketch-based image retrieval has received particular attention due to its broad commercial application potential. Although traditional sketch-based image retrieval techniques have made considerable progress, their widespread adoption in practical applications still faces some obstacles, such as the time-consuming process of drawing a complete sketch and limitations in users' drawing skills. Traditional methods require the use of a complete sketch for target recognition, but in commercial applications, users often do not have enough time to draw high-quality sketches. Therefore, in practice, there is a desire to retrieve target images quickly and accurately in the shortest possible time, even if the user's drawing skills are limited.

[0003] Existing technology proposes a real-time retrieval method for hand-drawn images based on multimodal data fusion. This method performs real-time retrieval based on the few strokes of the artist's sketch each time and returns the retrieval results. The artist continues to supplement the drawing based on the retrieval results until the target character appears. This approach provides a hybrid mode of human-computer interaction, thereby improving retrieval efficiency.

[0004] However, existing technologies overlook the problems arising from differences in users' drawing skills. Because different users create sketches with varying styles, especially in the early stages of drawing, search results may be affected by the randomness of brushstrokes, leading to images that fail to accurately reflect the user's intent. This instability in search results can severely impact the user experience in practical applications, causing inconvenience and other problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an interactive image retrieval method based on multimodal fusion, which includes:

[0006] S1: Obtain a multimodal dataset, including text, sketches, and images; use a deep neural network model to encode the multimodal data to obtain text embedding vectors, sketch embedding vectors, and image embedding vectors;

[0007] S2: Merge the text embedding vector and the sketch embedding vector to obtain the merged embedding vector;

[0008] S3: Construct a reward function based on the image embedding vector and the fused embedding vector;

[0009] S4: Construct and train a reinforcement learning model that includes a reward function. When the preset number of training iterations is reached, the model training is complete.

[0010] S5: Real-time acquisition of user sketches and text descriptions of the search target, inputting the sketches and text into a trained reinforcement learning model for real-time retrieval, and the model provides the retrieved images back to the user; the user further optimizes the sketch based on the feedback images until a target image satisfactory to the user is retrieved.

[0011] Preferably, the deep neural network model includes an image branch and a text branch;

[0012] The image branch includes an image encoder and fully connected layers for processing images and sketches;

[0013] The text branch includes a text encoder and a fully connected layer for processing the text.

[0014] Preferably, the process of constructing the reward function includes:

[0015] The image similarity reward is calculated based on the reciprocal of the ranking of the target image corresponding to each sketch.

[0016] The multimodal feature fusion reward is calculated based on the fused embedding vector and the target image embedding vector;

[0017] Calculate step penalty based on sketching steps, design error adjustment and plateau period reward; use image similarity reward as base reward; calculate global reward based on base reward, step penalty, error adjustment and plateau period reward;

[0018] The reward function is constructed by summing the image similarity reward, the multimodal feature fusion reward, and the global reward.

[0019] Furthermore, the formula for calculating the multimodal feature fusion reward is as follows:

[0020]

[0021]

[0022] in, This represents the reward for multimodal feature fusion. Represents the fused embedding vector v * Embedded vector of target image The similarity.

[0023] Furthermore, the formula for calculating the step penalty is as follows:

[0024]

[0025] in, This represents the step penalty for step i, where step represents the current step number, and total represents the total step penalty. q This indicates the maximum number of steps.

[0026] Furthermore, the error adjustment is expressed as:

[0027]

[0028] Among them, R Err Let R(i) represent the error adjustment value, α represent the adjustment coefficient, R(i) represent the base reward at step i, R(i-SErrA) represent the base reward at steps SErrA before the current step i, and SErrA represent the interval step number.

[0029] Furthermore, the reward during the stable period is expressed as follows:

[0030]

[0031] in, Rank represents the reward value during the stationary period at step i. i This represents the ranking of the target image at step i, and PlatB represents the stationary period threshold.

[0032] Furthermore, the formula for calculating the global reward is:

[0033]

[0034] in, Let R(i) represent the global reward at step i, and let R(i) represent the base reward at step i. R represents the step penalty for the i-th step. Err Indicates the error adjustment value. Let SErrA represent the reward value during the steady-state period at step i, SErrA represent the reward error threshold, and n represent the total number of sketching steps.

[0035] The beneficial effects of this invention are as follows: This invention employs a reinforcement learning framework to address the instability in the retrieval process caused by sparse strokes in the initial stages of sketching. This invention proposes a reward optimization mechanism for the retrieval feedback process, optimizing the matching strategy between the sketch and positive / negative samples. This method not only improves retrieval performance but also constrains the current sketch result based on the feedback from the previous sketching step, ensuring stable retrieval feedback performance and achieving a superior retrieval experience. This invention uses a hybrid human-computer interaction approach for portrait retrieval. After each retrieval result from the model, users do not rely on highly skilled artists; they can further optimize their sketches based on the feedback until the target image is retrieved. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the reinforcement learning model training process of the interactive image retrieval method in this invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] This invention proposes an interactive image retrieval method based on multimodal fusion, such as... Figure 1 As shown, the method includes:

[0039] S1: Obtain a multimodal dataset, including text, sketches, and images; use a deep neural network model to encode the multimodal data to obtain text embedding vectors, sketch embedding vectors, and image embedding vectors.

[0040] Obtain a multimodal dataset, including sketches, images, and corresponding text descriptions of the sketches; if this invention is applied to face image retrieval, construct mini-batch training data based on the face image-sketch dataset; the mini-batch training data includes a sketch sequence S = {s1, s2, ... s}. n The text T of the sketch, and the face image; the face image includes the target face image corresponding to the sketch.

[0041] A deep neural network model is used to encode multimodal data, resulting in text embedding vectors, sketch embedding vectors, and image embedding vectors; specifically:

[0042] Deep neural network models include image branches and text branches; the image branch includes an image encoder E. I (·) and fully connected layers Used to process images and sketches, resulting in sketch embedding vectors and image embedding vectors.

[0043] The text branch includes the text encoder E. T (·) and fully connected layers Used to process text, resulting in text embedding vectors.

[0044] S2: Merge the text embedding vector and the sketch embedding vector to obtain the merged embedding vector.

[0045] In the initial sketching stage, due to the lack of detail, it is difficult to capture the complete semantics based solely on visual information. Therefore, it is necessary to introduce textual descriptions as auxiliary information to enhance the semantic expression of the sketch. The text embedding vector and the sketch embedding vector are fused (the fusion strategy is: text embedding vectors and sketch embedding vectors of the same dimension are fused into one using vector multiplication (i.e., Hadamard product) to obtain the fused embedding vector.

[0046] S3: Construct a reward function based on the image embedding vector and the fused embedding vector.

[0047] The reward function designed in this invention consists of three parts: image similarity reward, multimodal feature fusion reward, and global reward.

[0048] Image similarity reward:

[0049] The retrieval objective of this invention is similar to the application of reinforcement learning in optimizing non-differentiable task metrics in computer vision. This invention aims to retrieve the target image using the fewest possible strokes. In sketch retrieval, the higher the match between the sketch and the target image, the higher the ranking of the retrieved target image, and the higher the reward. Therefore, the optimization objective is to maximize the reverse ranking of the sketch sequence, and the reward is designed accordingly:

[0050]

[0051] in, Rank represents the image similarity reward for the i-th sketch. si Indicates the ranking of the target image in the i-th sketch; s i The higher the ranking (i.e., the smaller the ranking number), the greater the corresponding reward.

[0052] Multimodal feature fusion reward:

[0053] To evaluate the visual and semantic consistency between the fused multimodal features and the target image, this invention proposes a similarity-based reward mechanism. This mechanism guides the model to optimize the generation process by calculating the cosine similarity between the sketch-text pair and the target image embedding vector.

[0054]

[0055]

[0056] in, This represents the reward for multimodal feature fusion. Represents the fused embedding vector v * Embedded vector of target image The similarity is used here to ensure that the generated sketches are consistent with the target image at the descriptive text level.

[0057] Global Rewards:

[0058] (1) Basic reward: For a sketch sequence S, the basic reward value is determined by the reciprocal of the retrieval ranking of each sketch, that is, the image similarity reward is used as the basic reward.

[0059] (2) Step Penalty: The step penalty is a mechanism designed to regulate the dynamic behavior of the model throughout the retrieval process. This penalty function decreases as the number of steps increases; that is, in the early steps of the retrieval process, the penalty is lower and the reward is higher, thereby encouraging the model to make accurate predictions early on.

[0060]

[0061] in, This represents the step penalty for step i, where step represents the current step number, and total represents the total step penalty. q This indicates the maximum number of steps.

[0062] (3) Error Adjustment: Error adjustment aims to penalize errors at specific steps. As the number of sketch strokes increases, the search results do not increase monotonically but fluctuate. Therefore, a reward error threshold (SErrA) is designed. At a specific SErrA*m step, if the search results increase compared to before, the impact on the reward is small. Once the error exceeds the tolerance, the reward will be significantly affected.

[0063]

[0064] Among them, R Err Let R(i) represent the error adjustment value, α represent the adjustment coefficient, R(i) represent the base reward at step i, and R(i-SErrA) represent the base reward at steps SErrA before the current step i.

[0065] (4) Plateau Period Reward: As the completeness of the sketch increases, the search results generally show an upward trend. When a specific plateau period threshold PlatB is reached, the reward no longer increases and remains stable.

[0066]

[0067] in, Let represent the reward value during the steady-state period at step i, and PlatB represent the threshold value during the steady-state period.

[0068] The global reward function guides the model to learn to retrieve images quickly and accurately in early steps through a reward and penalty mechanism, and maintains stable performance in later steps. The current step s... i The global reward is:

[0069]

[0070] in, Let R(i) represent the global reward at step i, and let R(i) represent the base reward at step i. R represents the step penalty for the i-th step. Err Indicates the error adjustment value. Let SErrA represent the reward value during the steady-state period at step i, SErrA represent the reward error threshold, and n represent the total number of sketching steps.

[0071] The reward function R is constructed by summing the image similarity reward, the multimodal feature fusion reward, and the global reward. si .

[0072] S4: Construct and train a reinforcement learning model that includes a reward function. When the preset number of training iterations is reached, the model training is complete.

[0073] Building reinforcement learning models:

[0074] In the model design of this invention, the retrieval process is modeled as a Markov decision process in a reinforcement learning environment. A state space, represented by the search results and input conditions (the entire sequence of the sketching process), and an action space, consisting of various possible search behaviors, are defined. By training the parameters of a policy network containing a pre-trained image encoder and fully connected layers, the sketch feature vector can be converted into decision actions. During each sketch retrieval step, this policy network predicts the distribution parameters of the actions, and then samples from this distribution to obtain the action to be taken in the current step. The training objective of the model is to maximize the cumulative reward. Specifically, the reward and discounted return at each time step are calculated, and then an attempt is made to minimize the loss function through gradient updates. The loss function is the product of the negative reward and the probability of action selection. The introduction of the gradient update strategy enables the model to make optimal action selection based on a given state in a reinforcement learning environment. Therefore, this model fully combines the advantages of deep learning and reinforcement learning, achieving dynamic and effective action decision-making in complex image search tasks.

[0075] In reinforcement learning frameworks, the image encoder acts as a key branch of the agent. The feature vector v of the sketch... si This is used to generate actions, which are based on a random continuous Gaussian policy defined by a multivariate normal distribution. The policy is defined as π. θ (α i |t i ), where θ represents the image encoder E containing the pre-trained image. I (·) and fully connected layers The policy network parameters are updated gradually during the iteration process. For each step of the sketch... i According to strategy π θ (α i |t i The network predicts distribution parameters to construct the algorithm. Sampling is then performed based on this distribution to determine the action α for the current step. i action α i In state s iThe probability of π is given by θ (α i |t i This method determines and samples from these sources. It combines the power of deep learning with the decision-making process of reinforcement learning, resulting in a policy that is both dynamic and efficient. Specifically:

[0076] α i ~γ(v si ,∑)

[0077] Among them, action α i From the mean v si The sampled data is obtained from a multivariate normal distribution with a covariance matrix of ∑.

[0078] The training objective is to maximize the expected cumulative reward. This invention employs a variant of nearest neighbor policy optimization, where the objective function is typically defined as the expected log probability of the reward. Since maximizing the log probability of the reward is equivalent to minimizing its negative value, this objective function is often expressed as a loss function and is minimized in practical applications. For each time step i, i∈n, the following calculation is performed to obtain the discounted reward.

[0079]

[0080] Where n represents the length of the sketch sequence, R si Here, θ is the reward at the current time step, and δ is the discount factor. The loss function l(θ) is the negative of the sum of the products of the logarithmic probability and the discounted reward, and can be written as:

[0081]

[0082] Therefore, the objective function ζ(θ), which is the function that this invention aims to maximize, is the negative of this loss function.

[0083]

[0084] Where n is the total number of time steps, π θ (α i |t i ) is the policy network, α i It is in s i The action taken in a given state. Updating policy parameters through gradient ascent (or, in practice, minimizing the loss through gradient descent) is essentially an attempt to maximize the expected total reward, thereby training the policy network to learn to choose the optimal action for a given state.

[0085] The model training is complete when the preset number of training iterations is reached.

[0086] S5: Real-time acquisition of user sketches and text descriptions of the search target, inputting the sketches and text into a trained reinforcement learning model for real-time retrieval, and the model provides the retrieved images back to the user; the user further optimizes the sketch based on the feedback images until a target image satisfactory to the user is retrieved.

[0087] When users cannot accurately describe the features of a target person using existing photos or text, they can search for them by drawing a portrait of the person using a touchscreen device. Specifically, the user's portrait sketch is input into a trained deep neural network model, which performs a search based on the input sketch, returning the k images most similar to the sketch. In this way, users no longer rely on language or existing images; they can find corresponding matching images simply by drawing the target person from memory.

[0088] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An interactive image retrieval method based on multimodal fusion, characterized in that, include: S1: Obtain a multimodal dataset, including text, sketches, and images; use a deep neural network model to encode the multimodal data to obtain text embedding vectors, sketch embedding vectors, and image embedding vectors; Deep neural network models include image branches and text branches; the image branch includes an image encoder and fully connected layers for processing images and sketches; the text branch includes a text encoder and fully connected layers for processing text. S2: Merge the text embedding vector and the sketch embedding vector to obtain the merged embedding vector; S3: Construct a reward function based on the image embedding vector and the fused embedding vector; the process of constructing the reward function includes: The image similarity reward is calculated based on the reciprocal of the ranking of the target image corresponding to each sketch. The multimodal feature fusion reward is calculated based on the fused embedding vector and the target image embedding vector; the formula for calculating the multimodal feature fusion reward is: in, This represents the reward for multimodal feature fusion. Represents the fused embedding vector v * Embedded vector of target image Similarity; The step penalty is calculated based on the number of sketching steps, and error adjustment and plateau period rewards are designed; image similarity reward is used as the base reward; the global reward is calculated based on the base reward, step penalty, error adjustment, and plateau period reward; the formula for calculating the step penalty is: in, This represents the step penalty for step i, where step represents the current step number, and total represents the total step penalty. q Indicates the maximum number of steps; Error adjustment is expressed as: Among them, R Err Let R(i) represent the error adjustment value, α represent the adjustment coefficient, R(i) represent the base reward at step i, R(i-SErrA) represent the base reward at steps SErrA before the current step i, and SErrA represent the interval step number. The reward during the stable period is expressed as follows: in, Rank represents the reward value during the stationary period at step i. i This represents the ranking of the target image at step i, and PlatB represents the stationary period threshold. The reward function is constructed by summing the image similarity reward, the multimodal feature fusion reward, and the global reward. The formula for calculating the global reward is: in, Let SErrA represent the global reward at step i, SErrA represent the reward error threshold, and n represent the total number of sketching steps. S4: Construct and train a reinforcement learning model that includes a reward function. When the preset number of training iterations is reached, the model training is complete. S5: Real-time acquisition of user sketches and text descriptions of the search target, inputting the sketches and text into a trained reinforcement learning model for real-time retrieval, and the model provides the retrieved images back to the user; the user further optimizes the sketch based on the feedback images until a target image satisfactory to the user is retrieved.

Images