A media library image retrieval method based on multimodal model

Abstract

A media resource library image retrieval method based on a multimodal model includes the following steps: S1, collecting images and corresponding text data, and preprocessing the images; S2, uniformly encoding the images and the corresponding text data to generate an overall scene description of the image and a multi-label description of the image; S3, converting the overall scene description of the image and the multi-label description of the image into a high-dimensional embedding vector through a multimodal embedding fusion network, and fusing them; S4, converting the user's search text into a vector, and using a similarity calculation method with the fused image vector to determine the corresponding image. The media resource library image retrieval method based on a multimodal model provided by the present invention can more accurately describe the image content by combining image and text information, thereby achieving more efficient image search.

Description

Technical Field

[0001] The present invention belongs to the field of computer application technology, and in particular relates to a media resource library image retrieval method based on a multimodal model. Background Art

[0002] With the development of the Internet and smart devices, the scale of image libraries continues to expand. The Chinese content community has a massive amount of graphic and text content, and the amount of image data has reached hundreds of millions. Its graphic and text content is rich in variety and has complex semantics. The retrieval and matching of semantically similar content has strong business value in search, recommendation, commercial advertising and other scenarios. Usually, people use neural network labeling and content representation methods to aggregate graphic and text data with similar semantic content. Due to the extremely large scale of images, how to efficiently and accurately retrieve a group of semantically similar images from massive image data for a single image has become a complex and important problem. Traditional image search methods mostly rely on basic visual feature matching, such as image feature extraction and comparison, which has problems such as high computational complexity and low search efficiency. The emergence of multimodal models has provided new ideas for image search. Therefore, in order to make retrieval more accurate and efficient, a picture retrieval method combined with a multimodal model is needed. The present invention solves this technical problem. Summary of the Invention

[0003] The present invention provides a media resource library image retrieval method based on a multimodal model, which can more accurately describe the image content by combining image and text information, thereby achieving more efficient image search.

[0004] A method for retrieving images from a media library based on a multimodal model includes the following steps:

[0005] S1, collect pictures and their corresponding text data, and preprocess the pictures;

[0006] S2. Encode the image and the corresponding text data uniformly to generate the overall scene description and multi-label description of the image;

[0007] S3. Convert the overall scene description and multi-label description of the image into high-dimensional embedding vectors through a multimodal embedding fusion network and fuse them.

[0008] S4. Convert the user's search text into a vector and use a similarity calculation method with the fused image vector to determine the corresponding image.

[0009] Furthermore, the step S1 specifically includes the following steps:

[0010] S11, collecting pictures and corresponding news text data as well as corresponding text descriptions and tag data;

[0011] S12. Perform size standardization: Use OpenCV to read the image and resize it to the specified width and length using the resize function;

[0012] S13. Perform color space conversion: Use opencv to read the image and convert it into a grayscale image using the COLOR_BGR2GRAY function.

[0013] S14. Divide the data into training set, validation set and test set.

[0014] Furthermore, the step S2 specifically includes the following steps:

[0015] S21. Use the GLM-4V model to uniformly encode the image and the corresponding news text data to generate the overall scene description of the image. and multi-label description of images ;

[0016] S22. Assume that the input image sequence is , the news text data prompt word corresponding to the picture is ,in, Represents the nth input picture, Indicates the nth news data prompt word corresponding to the input image;

[0017] S23, input sequence I and sequence X into the model to obtain and Text sequence.

[0018] Furthermore, the step S3 specifically includes the following steps:

[0019] S31. Use the Embedding model to and Encode it and convert it into a fixed-length vector. The converted embedding vector is and ;

[0020] S32. Define two initialization weight matrices and ;

[0021] S33. Using weight matrix and right and Perform linear transformation and then add the results to get the final fusion vector ;

[0022] S34, establishing a loss function;

[0023] S35, fusion vector Stored in the database.

[0024] Furthermore, the calculation formula in step S33 is as follows:

[0025] ,

[0026] Among them, b represents the bias vector commonly used in neural networks, and Relu represents the activation function.

[0027] Furthermore, the specific contents of step S34 are as follows:

[0028] In order for the model to find the correct description from an input sequence of length n when given an image i and related news x, it is necessary to minimize the loss of correct pairings and maximize the loss of incorrect pairings. The calculation formula is as follows:

[0029] ,

[0030] in, represents the similarity score of the positive sample, represents the average similarity score of all negative samples, and m is a boundary parameter used to control the distance between positive and negative samples.

[0031] Furthermore, the specific contents of step S4 are as follows:

[0032] S41. When a user inputs text for image query, the text data input by the user is converted into a vector Q;

[0033] S42, all Q and database Perform weighted fusion similarity calculation to obtain a similarity score;

[0034] S43. Sort the related images according to the similarity scores and use them as the user's search results.

[0035] Furthermore, the calculation formula in step S42 is as follows:

[0036] ,

[0037] Where M represents the number of all images in the database.

[0038] The technical effects of the present invention are as follows:

[0039] (1) This solution combines image and text information to encode the image and its corresponding text data and convert them into a fusion vector. When the user searches, the text data entered by the user can be converted into a vector, and then the similarity between this vector and the fusion vector is calculated to determine the most similar fusion vector. The image corresponding to the fusion vector is the search result, thus realizing the image retrieval function in a new way.

[0040] (2) Since this solution combines image and text information to achieve accurate image search, the traditional method is to extract and compare features of images based on computer vision features, which is computationally complex and inefficient. Compared with the traditional method, this solution uses a multimodal model, which allows this solution to search for corresponding images more accurately, is computationally simple, and has higher search efficiency, thereby improving the practicality of this solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the present invention.

[0042] Figure 2 This is a flowchart of the offline phase. DETAILED DESCRIPTION

[0043] The technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and drawings.

[0044] See also Figure 1 、 Figure 2 , a media library image retrieval method based on a multimodal model, comprising the following steps:

[0045] S1, collect pictures and their corresponding text data, and preprocess the pictures;

[0046] S2. Encode the image and the corresponding text data uniformly to generate the overall scene description and multi-label description of the image;

[0047] S3. Convert the overall scene description and multi-label description of the image into high-dimensional embedding vectors through a multimodal embedding fusion network and fuse them.

[0048] S4. Convert the user's search text into a vector and use a similarity calculation method with the fused image vector to determine the corresponding image.

[0049] In this embodiment, S1 to S3 are performed in an offline stage, and S4 is performed in an online stage. The collection method in S1 may be to query an internal database.

[0050] Furthermore, the step S1 specifically includes the following steps:

[0051] S11, collecting pictures and corresponding news text data as well as corresponding text descriptions and tag data;

[0052] S12. Perform size standardization: Use OpenCV to read the image and resize it to the specified width and length using the resize function;

[0053] S13. Perform color space conversion: Use opencv to read the image and convert it into a grayscale image using the COLOR_BGR2GRAY function.

[0054] S14. Divide the data into training set, validation set and test set.

[0055] Furthermore, the step S2 specifically includes the following steps:

[0056] S21. Use the GLM-4V model to uniformly encode the image and the corresponding news text data to generate the overall scene description of the image. and multi-label description of images ;

[0057] S22. Assume that the input image sequence is ,

[0058] The news text data prompt word corresponding to the picture is ,

[0059] in, Represents the nth input picture, Indicates the nth news data prompt word corresponding to the input image;

[0060] S23, input sequence I and sequence X into the model to obtain and Text sequence.

[0061] Furthermore, the step S3 specifically includes the following steps:

[0062] S31. Use the Embedding model to and Encode it and convert it into a vector of fixed length 512. The converted embedding vector is and ;

[0063] S32. Define two initialization weight matrices and ;

[0064] S33. Using weight matrix and right and Perform linear transformation and then add the results to get the final fusion vector ;

[0065] S34, establishing a loss function;

[0066] S35, fusion vector Specifically, after the model training is completed, S32 and S33 are executed for all the data in the database to generate the corresponding fusion vector , and store it in the database

[0067] Furthermore, the calculation formula in step S33 is as follows:

[0068] ,

[0069] Among them, b represents the bias vector commonly used in neural networks, and Relu represents the activation function.

[0070] Furthermore, the specific contents of step S34 are as follows:

[0071] The goal of the training phase is to enable the model to learn to find the correct description from an input sequence of length n when given an image i and related news x. This is achieved by minimizing the loss of correct pairings and maximizing the loss of incorrect pairings. The calculation formula is as follows: ,

[0072] in, Represents the similarity score of the positive sample. The similarity calculation score here uses cosine similarity. It represents the average similarity score of all negative samples. m is a boundary parameter used to control the distance between positive and negative samples. In this embodiment, the value of m is 0.2.

[0073] Furthermore, the specific contents of step S4 are as follows:

[0074] S41. When a user inputs text for image query, the text data input by the user is converted into a vector Q;

[0075] S42, all Q and database Perform weighted fusion similarity calculation to obtain a similarity score;

[0076] S43. Sort the related images according to the similarity scores and use them as the user's search results.

[0077] Furthermore, the calculation formula in step S42 is as follows:

[0078] ,

[0079] Where M represents the number of all images in the database.

[0080] The working principle of the present invention is as follows:

[0081] First, collect the pictures and the corresponding news text data, corresponding text descriptions and label data, then preprocess the pictures, and then use the GLM-4V model to uniformly encode the pictures and the corresponding news text data, and then use the model to obtain the overall scene description of the pictures. and multi-label description of images , and then the two are encoded and converted into vectors through the Multimodal Embedding Fusion Network (MEFN) and , and then the corresponding fusion vector is obtained by fusion , and establish a loss function. Finally, when the user searches, the text data entered by the user is converted into a vector Q, and Q is compared with all the vectors in the database. A weighted fusion similarity calculation is performed to determine the most similar fusion vector. The image corresponding to the fusion vector is the search result, and the multiple retrieved images can be sorted according to the similarity score, thereby realizing the image retrieval function in a new way.

[0082] The above embodiments are only preferred embodiments of the present invention. Those skilled in the art can derive other embodiments from the above embodiments without creative work. Therefore, this application protects not only the above embodiments, but also the scope consistent with the principles and features of this application.

Claims

1. A media resource image retrieval method based on a multimodal model, characterized in that: The following steps are involved: S1, collecting pictures and corresponding text data, and preprocessing the pictures; S2, uniformly encode the image and the corresponding text data to generate the overall scene description of the image and the multi-label description of the image; S3, convert the overall scene description of the image and the multi-label description of the image into a high-dimensional embedding vector through a multimodal embedding fusion network and fuse them; S4, converting the user's search text into a vector, and using a similarity calculation method with the fused image vector to determine the corresponding image; The step S1 specifically includes the following steps: S11, collecting pictures and corresponding news text data as well as corresponding text descriptions and tag data; S12. Standardize the size: use opencv to read the image and adjust the image to the specified width and length through the resize function; S13, perform color space conversion: Color space conversion: use opencv to read the image, and convert the image into a grayscale image through the COLOR_BGR2GRAY function; S14, divide the data into training set, validation set and test set; The step S2 specifically includes the following steps: S21. Use the GLM-4V model to uniformly encode the image and the corresponding news text data to generate the overall scene description of the image. and multi-label description of images ; S22. Assume that the input image sequence is , the news text data prompt word corresponding to the picture is ,in, Represents the nth input picture, Indicates the nth news data prompt word corresponding to the input image; S23, input sequence I and sequence X into the model to obtain and Text sequence; The step S3 specifically comprises the following steps: S31. Use the Embedding model to and Encode it and convert it into a vector of fixed length. The converted embedding vector is and ; S32. Define two initialization weight matrices and ; S33. Using weight matrix and right and Perform linear transformation and then add the results to get the final fusion vector ; S34, establishing a loss function; S35, fusion vector Stored in the database.

2. According to the multimodal model-based media resource library image retrieval method according to claim 1, it is characterized in that: The calculation formula in step S33 is as follows: ， Among them, b represents the bias vector commonly used in neural networks, and Relu represents the activation function.

3. The media resource library image retrieval method based on a multimodal model according to claim 1, characterized in that: The specific contents of step S34 are as follows: In order for the model to find the correct description from an input sequence of length n when given an image i and related news x, it is necessary to minimize the loss of correct pairings and maximize the loss of incorrect pairings. The calculation formula is as follows: ， in, represents the similarity score of the positive sample, represents the average similarity score of all negative samples, and m is a boundary parameter used to control the distance between positive and negative samples.

4. The media resource library image retrieval method based on a multimodal model according to claim 1, characterized in that: The specific contents of step S4 are as follows: S41, when the user inputs text to perform image query, convert the text data input by the user into a vector Q; S42, all Q and database Perform weighted fusion similarity calculation to obtain a similarity score; S43. Sort the related images according to the similarity scores as search results for the user.

5. The media resource library image retrieval method based on a multimodal model according to claim 4 is characterized in that: The calculation formula in step S42 is as follows: ， Where M represents the number of all images in the database.

Images