Image and text retrieval method in civil aviation field based on multimodal pre-training model

Abstract

The present invention provides a method for image-text retrieval in the civil aviation field based on a multimodal pre-trained model. The method comprises collecting two types of multimodal information, namely, image-text pairs and image-label pairs, to construct a multimodal dataset in the civil aviation field; cleaning the dataset using a self-supervised mean-shift clustering method to construct a high-quality multimodal dataset in the civil aviation field; evaluating the quality of images in the dataset using a reference-free evaluation method and deleting images with low evaluation; and performing a question-based image search task based on a BriVL model adjusted using a frozen training method. The present invention constructs a high-quality multimodal dataset in the civil aviation field by utilizing a mean-shift clustering method and a reference-free image quality assessment method. BriVL is adjusted using a frozen training method using the dataset, thereby shortening the model training time. In the text-based image search task on the civil aviation field multimodal dataset used for testing, the adjusted BriVL achieves an excellent recall rate.

Description

Technical Field

[0001] The present invention relates to retrieval technology in the field of civil aviation, and in particular to a civil aviation field image and text retrieval method based on a multimodal pre-training model. Background Art

[0002] Civil aviation is rapidly expanding. Statistics show that civil aviation passenger turnover now accounts for nearly one-third of the total transportation system. However, search engines like Baidu offer overly broad image and text retrieval methods, making it difficult to accurately locate the information passengers are looking for. In this context, building multimodal datasets in the civil aviation sector to address image and text matching is essential.

[0003] There are two main image-text matching algorithms used in current pre-training models: cross-modal similarity measurement and public space feature learning. The former has a significantly slower response speed than the latter. The Air Travel APP requires a high response speed for search results, so this article needs to select a model based on public space feature learning. BriVL is a model based on public space feature information and is currently the largest pre-training model in the Chinese multimodal field. There are currently three problems with applying pre-training models to the civil aviation field: there are no existing multimodal datasets for civil aviation and they need to be obtained from the internet; BriVL's encoder has too many parameters and takes too long to train; and current optimizations for multi-label tasks are mostly based on a single modality, making it difficult to directly migrate to multi-modal multi-label tasks. Summary of the Invention

[0004] The purpose of the present invention is to provide a civil aviation field image and text retrieval method based on a multimodal pre-trained model, which uses a mean-shift clustering method and a reference-free evaluation method to clean and screen the data obtained from the network, delete low-quality data in the data set, thereby constructing a multimodal data set in the civil aviation field and improving the data set quality, and adjusting the BriVL pre-trained on a general data set to the multimodal data in the civil aviation field to adapt to the civil aviation field.

[0005] To achieve the above objectives, the present invention provides a method for image and text retrieval in the civil aviation field based on a multimodal pre-training model, which specifically includes the following steps:

[0006] S1. Collect multimodal information of the first and second types of training data to construct a multimodal dataset in the civil aviation field.

[0007] S11. Collect the first type of training data. Collect image-text pairs from the carnoc civil aviation resource network and the open source Wukong dataset to form the first type of training data for adjusting the fully connected layer of the BriVL pre-trained model.

[0008] S12, collecting the second type of training data;

[0009] S121. Divide the tag hierarchy structure; based on actual application scenarios and the inherent logical relationships of tags, divide the tags into a three-level tag structure and divide the tags into five types of tags;

[0010] S122. Acquire multiple images of each type of label from the Internet, and acquire multiple images for compound labels, to form a second type of training data set, where the second type of training data set includes the image-label pair data;

[0011] S2, data cleaning;

[0012] S21. Clean up missing value images, remove images with empty content, image-text pairs corresponding to texts with lengths less than a first threshold, and redundant texts with a total length greater than a second threshold;

[0013] S22, using the self-supervised mean-shift clustering method to determine whether the image-label pair data in the second type of training data set matches;

[0014] S221. Randomly select a data point from the unlabeled data points as the center;

[0015] S222. Mark all points whose distance to the center is less than r and record them as set M. Assuming that all x points in set M belong to cluster c, the count of cluster c is increased by x.

[0016] S223. Calculate the vector from the center center to each element in M, add all the vectors together to obtain the shift vector, and add the center center and the shift vector to obtain the new center center.

[0017] S224: Repeat steps S222 and S223 until the modulus of the shift vector shift converges below a preset threshold. Record the value of the center at this point. Classify all elements encountered in this process into cluster c and increase the count of cluster c.

[0018] S225, if the distance between the center center at this time and the center center of an existing cluster c' is less than a preset threshold, then merge cluster c with the existing cluster c', and add all elements in M ​​to the existing cluster c';

[0019] S226, repeat steps S221 to S225 until all data points have been marked at least once;

[0020] S3, evaluate the image quality and delete low-quality images;

[0021] S4, based on the BriVL model, conduct a question-based image search task;

[0022] S41. Using the joint dataset, the BriVL model is adjusted through the frozen training method. The MLP layers in the two encoders are retrained using the acquired data to make them fit the characteristics of image-text pairs in the civil aviation field.

[0023] S42, calculating the image-text similarity value using the adjusted BriVL model;

[0024] S421, encode the text using Word2Vec to calculate word features, keep the Word2Vec model parameters unchanged during the training process, encode the image using CNN, and also keep the CNN network parameters unchanged during the training process;

[0025] S422, using an MLP layer with learnable parameters to map the text vectors obtained by Word2Vec and the image vectors obtained by CNN into the same semantic space;

[0026] S423, calculating the cosine similarity of each image-text pair as the similarity value of the image-text pair;

[0027] S43. Sort all images in descending order according to the similarity values ​​of their image-text pairs, so that the higher the ranking, the higher the recommendation degree of the image search result.

[0028] Furthermore, step S3 specifically includes the following steps:

[0029] S31. Based on all images in the same label category, calculate the corresponding three indicators: standard deviation, average gradient and entropy;

[0030] S32. For all images, sort the three indicators in descending order;

[0031] S33. Delete from the data set images whose standard deviation, average gradient, or entropy is in the bottom 5%.

[0032] Furthermore, in the three-level label structure described in step S121, each leaf node corresponds to an atomic label, and each label existing in a non-leaf node is an intermediate label used to assist multi-classification.

[0033] Furthermore, in step S22, the image-label pair includes a first source obtained by using a search engine and a second source selected from the Wukong open source dataset.

[0034] Preferably, in step S21, the first threshold is 2 and the second threshold is 10.

[0035] Preferably, in step S31, the standard deviation identifies the degree of discreteness of the image grayscale. A small standard deviation means that the image grayscale distribution is concentrated, and the image is a low-quality image; the average gradient identifies the clarity and contrast of the image texture. A large average gradient means that the image is a high-quality image; the entropy is a measure of the amount of information. A large entropy value of a picture means that the picture contains a large amount of information.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The multimodal data sampling method for the civil aviation field proposed in this invention obtains the initial dataset through crawler acquisition and open source dataset retrieval. The initial dataset is cleaned and optimized using the mean-shift clustering method and a reference-free image quality assessment method, thereby filtering out low-quality data and constructing a high-quality multimodal dataset for the civil aviation field.

[0038] 2. This paper uses a self-constructed multimodal dataset in the civil aviation field to adjust the BriVL pre-trained model to adapt the model to the data characteristics of the civil aviation field, thereby enhancing its image and text feature extraction capabilities in the civil aviation field;

[0039] 3. The present invention uses a frozen training method to adjust BriVL, freezing several convolutional layers at the front of the model and only training the parameters of the last two layers of BriVL, thereby shortening the model training time; in the text-to-image search task on the multimodal dataset in the civil aviation field used for testing, the adjusted BriVL achieved a maximum recall rate of 38.4, significantly exceeding other multimodal methods used for comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of the civil aviation field image and text retrieval method based on the multimodal pre-training model of the present invention;

[0041] Figure 2 This is a label hierarchy diagram of the present invention;

[0042] Figure 3 It is an illustration of a low-quality image of the present invention;

[0043] Figure 4 This is a flow chart of BriVL calculation of image-text similarity values ​​of the present invention. DETAILED DESCRIPTION

[0044] The exemplary embodiments, features, and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0045] The present invention provides a Figure 1 The civil aviation field image and text retrieval method based on the multimodal pre-training model shown includes:

[0046] S1. Collect two types of multimodal information: image-text pairs and image-label pairs, and construct a multimodal dataset in the civil aviation field.

[0047] S11. Collect the first type of training data, namely image-text pairs. We collected 10,000 image-text pairs from the carnoc Civil Aviation Resource Network and retrieved 50,000 civil aviation-related image-text pairs from the open-source Wukong dataset. Before cleaning, these 60,000 image-text pairs were used to adjust the fully connected layers of the BriVL pre-trained model, making the pre-trained model's image and text encoders more responsive to civil aviation data.

[0048] S12. Collect the second type of training data, namely image-label pair data.

[0049] S121, divide the label hierarchy structure; according to the actual use scenario and the internal logical relationship of the label, divide the label into Figure 2 In the three-level label structure shown, each leaf node corresponds to an atomic label. Each label that exists in the leaf node is not a real label used for classification, but only an intermediate label used to assist multi-classification.

[0050] The labels are divided into five categories: text, food, environment, people and behavior; the text category is further divided into boarding passes and other text such as tickets, books, certificates, slogans, etc.; food is divided into in-flight meals, non-in-flight meals and drinks; the environment is divided into outdoor environment and indoor environment, and the outdoor environment is further divided into airplanes, scenery, buildings and airport exteriors. If the image contains the outline of the main body of the aircraft, then the image will be added with an airplane label. Scenery includes sky, aerial photography, mountains and water. Indoor environment is further divided into hotel, home, airport lobby, shopping mall, restaurant and cabin; people are divided into men, women and crowds; behavior is divided into security check and waiting.

[0051] The main subject of architectural images should be the building. If the image contains a large number of landscape elements that make it difficult to highlight the main body of the building, it should be labeled as architecture + landscape + related elements; screenshot labels include flight screenshots, weather screenshots, mobile WeChat, tickets, electronic maps, etc.

[0052] S122. For each type of label, obtain 5K images from the Internet. For compound labels, obtain 125K images. A total of 150K images constitute the second type of training data set.

[0053] S2. Data cleaning.

[0054] S21. Clean up images with missing values. Using Python's built-in function cv2.imread(), we can identify whether the downloaded image is empty, thereby determining link validity and removing images with empty content. For text, some text is too short, so all image-text pairs with text lengths less than 2 are deleted. If the same text appears more than 10 times, it is considered meaningless. Such text is deleted and output to the corresponding record file for manual verification of whether it is indeed meaningless.

[0055] S22. Use the self-supervised mean-shift clustering method to process inconsistent data in image-label pairs; there are two sources of image-label pairs, one is to use search engines, and the other is to filter from the Wukong open source data set; for images under the same label, the correlation between the images from the former and the label shows an obvious downward trend, and for the last 10K or so data, the correlation between the images and the labels is low. This is because the return order of the search engine is affected by the image-label correlation; use the self-supervised mean-shift clustering method to process inconsistent data. In the data obtained under the same label, the images with high correlation account for the majority, and the images with low correlation account for only a small part. Therefore, the present invention uses BriVL's image-text encoder to embed the image-text pairs, and then deletes the discrete points generated after clustering under each tag. Assuming that there are several data points that need to be clustered in the multidimensional space, when the window size r is given, the mean-shift process is as follows.

[0056] S221. Randomly select a data point from the unlabeled data points as the center.

[0057] S222. Mark all points whose distance to the center is less than r and record them as a set M. Assuming that all x points in the set M belong to cluster c, the count of cluster c is increased by x.

[0058] S223. Calculate the vector from the center center to each element in M, add all the vectors together to obtain the movement vector shift, and add the center center and the movement vector shift together to obtain the new center center.

[0059] S224. Repeat steps S222 and S223 until the modulus of the shift vector shift converges below a preset threshold. Record the value of the center at this time. Classify all elements encountered in this process into cluster c and increase the count of cluster c.

[0060] S225 , if the distance between the center center at this time and the center center of an existing cluster c′ is less than a preset threshold, then merge cluster c with the existing cluster c′, and add all elements in M ​​to the existing cluster c′.

[0061] S226. Repeat steps S221 to S225 until all data points have been marked at least once.

[0062] S3, use the no-reference evaluation method to evaluate the image quality and delete the images with low evaluation; Figure 3 The pictures shown have low resolution and concentrated grayscale and are not suitable for training models, so it is necessary to evaluate the pictures and delete the ones with low evaluation. Due to the wide variety of picture labels, it is relatively complicated to find a suitable ideal picture for each category, so the present invention adopts a reference-free evaluation method. Three indicators, standard deviation, average gradient and entropy, are used for reference-free evaluation; the standard deviation identifies the degree of discreteness of the grayscale of the image. The smaller the standard deviation, the more concentrated the grayscale distribution of the image, that is, the contrast of the entire image is too low and the image quality is poor. The average gradient identifies the clarity and contrast of the image texture. It is generally believed that it can represent the clarity of the image. The larger the average gradient, the higher the image quality. Entropy is a measure of the amount of information, which is a concept based on information theory. The larger the entropy value of a picture, the greater the amount of information it contains.

[0063] S31. Based on all images in the same label category, calculate the corresponding three no-reference indicators: standard deviation, average gradient, and entropy.

[0064] S32. For all images, sort the three indicators in descending order.

[0065] S33. Delete from the dataset any of the images whose standard deviation, average gradient, and entropy are ranked in the bottom 5%.

[0066] S4. Based on the BriVL model, perform question-based image search tasks; BriVL's cross-modal pre-training model is based on image-text retrieval tasks, so its main goal is to train the text encoder and image encoder separately so that the text encoder and image encoder map the image and text to the same subspace.

[0067] S41. Using the joint dataset, the BriVL model is adjusted through the frozen training method, and the MLP layers in the two encoders are retrained using the acquired data to make them fit the characteristics of image-text pairs in the civil aviation field.

[0068] S42, Figure 4 The figure shows the process of calculating the image-text similarity value using the adjusted BriVL model, which specifically includes the following steps:

[0069] S421. Calculate word features of the text using Word2Vec encoding, keep the Word2Vec model parameters unchanged during the training process, encode the image using CNN, and also keep the CNN network parameters unchanged during the training process.

[0070] S422. Use the parameter-learnable MLP layer to map the text vector obtained by Word2Vec and the image vector obtained by CNN into the same semantic space.

[0071] S423 , calculating the cosine similarity of each image-text pair as the similarity value of the image-text pair.

[0072] S43. Sort all images in descending order according to the similarity values ​​of their image-text pairs, so that the higher the ranking, the higher the recommendation degree of the image search result.

[0073] The civil aviation field currently lacks multimodal datasets for model training, and multimodal datasets directly obtained from the Internet using crawlers also have the problem of poor image and text quality. The multimodal data sampling method designed in the present invention for the civil aviation field obtains the initial dataset based on crawler acquisition and open source dataset retrieval. Subsequently, the mean-shift clustering method and the reference-free image quality assessment method are used to clean and optimize the initial dataset, screen out low-quality data, and thus construct a high-quality multimodal dataset for the civil aviation field.

[0074] Pre-training models such as BriVL all use general datasets to complete pre-training, but there are differences in the distribution of civil aviation field datasets and general datasets, so directly using pre-training parameters is not effective. In addition, BriVL has the problem of taking a long time to train. The present invention uses the constructed civil aviation field multimodal dataset to adjust the BriVL pre-training model, so that the model adapts to the data characteristics of the civil aviation field, thereby enhancing its image and text feature extraction capabilities in the civil aviation field. In terms of time consumption, the present invention uses the frozen training method to adjust BriVL, freezes several convolutional layers in front of the model, and only trains the parameters of the last two layers of BriVL, thereby shortening the model training time. In the text-to-image search task on the civil aviation field multimodal dataset used for testing, the adjusted BriVL achieved a highest recall rate of 38.4, significantly exceeding other multimodal methods used for comparison.

[0075] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A method for image and text retrieval in the civil aviation field based on a multimodal pre-trained model, characterized by: It includes the following steps: S1. Collect multimodal information of the first and second types of training data to construct a multimodal dataset in the civil aviation field. S11. Collect the first type of training data. Collect image-text pairs from the carnoc civil aviation resource network and the open source Wukong dataset to form the first type of training data for adjusting the fully connected layer of the BriVL pre-trained model. S12, collecting the second type of training data; S121. Divide the tag hierarchy structure; based on actual application scenarios and the inherent logical relationships of tags, divide the tags into a three-level tag structure and divide the tags into five types of tags; S122. Acquire multiple images of each type of label from the Internet, and acquire multiple images for compound labels, to form a second type of training data set, where the second type of training data set includes the image-label pair data; S2, data cleaning; S21. Clean up missing value images, remove images with empty content, image-text pairs corresponding to texts with lengths less than a first threshold, and redundant texts with a total length greater than a second threshold; S22, using the self-supervised mean-shift clustering method to determine whether the image-label pair data in the second type of training data set matches; S221. Randomly select a data point from the unlabeled data points as the center; S222. Mark all points whose distance to the center is less than r and record them as set M. Assuming that all x points in set M belong to cluster c, the count of cluster c is increased by x. S223. Calculate the vector from the center center to each element in M, add all the vectors together to obtain the shift vector, and add the center center and the shift vector to obtain the new center center. S224: Repeat steps S222 and S223 until the modulus of the shift vector shift converges below a preset threshold. Record the value of the center at this point. Classify all elements encountered in this process into cluster c and increase the count of cluster c. S225, if the distance between the center center at this time and the center center of an existing cluster c' is less than a preset threshold, then merge cluster c with the existing cluster c', and add all elements in M ​​to the existing cluster c'; S226, repeat steps S221 to S225 until all data points have been marked at least once; S3, evaluate the image quality and delete low-quality images; S4, based on the BriVL model, conduct a question-based image search task; S41. Using the joint dataset, the BriVL model is adjusted through the frozen training method. The MLP layers in the two encoders are retrained using the acquired data to make them fit the characteristics of image-text pairs in the civil aviation field. S42, calculating the image-text similarity value using the adjusted BriVL model; S421, encode the text using Word2Vec to calculate word features, keep the Word2Vec model parameters unchanged during the training process, encode the image using CNN, and also keep the CNN network parameters unchanged during the training process; S422, using an MLP layer with learnable parameters to map the text vectors obtained by Word2Vec and the image vectors obtained by CNN into the same semantic space; S423, calculating the cosine similarity of each image-text pair as the similarity value of the image-text pair; S43. Sort all images in descending order according to the similarity values ​​of their image-text pairs, so that the higher the ranking, the higher the recommendation degree of the image search result.

2. The method for image-text retrieval in the civil aviation field based on a multimodal pre-training model according to claim 1 is characterized in that: Step S3 specifically includes the following steps: S31. Based on all images in the same label category, calculate the corresponding three indicators: standard deviation, average gradient and entropy; S32. For all images, sort the three indicators in descending order; S33. Delete from the data set images whose standard deviation, average gradient, or entropy is in the bottom 5%.

3. The method for image-text retrieval in the civil aviation field based on a multimodal pre-training model according to claim 1, characterized in that: In the three-level label structure described in step S121, each leaf node corresponds to an atomic label, and each label existing in a non-leaf node is an intermediate label used to assist multi-classification.

4. The method for image-text retrieval in the civil aviation field based on a multimodal pre-training model according to claim 1, characterized in that: In step S22 , the image-label pair includes a first source obtained by using a search engine and a second source selected from the Wukong open source dataset.

5. The method for image-text retrieval in the civil aviation field based on a multimodal pre-training model according to claim 1, characterized in that: In step S21 , the first threshold is 2 and the second threshold is 10.

6. The method for image and text retrieval in the civil aviation field based on a multimodal pre-training model according to claim 2, characterized in that: In step S31, the standard deviation identifies the degree of discreteness of the image grayscale. A small standard deviation means that the image grayscale distribution is concentrated, and the image is a low-quality image; the average gradient identifies the clarity and contrast of the image texture. A large average gradient means that the image is a high-quality image; the entropy is a measure of the amount of information. A large entropy value of the image means that the image contains a large amount of information.

Images