A method for determining keyword similarity of aviation literature by fusing multi-modal semantic correlation graph atlas

Abstract

The application relates to an aviation document keyword similarity judgment method based on a fusion multimodal semantic correlation graph, which comprises the following steps: acquiring aviation field keywords, constructing an aviation document keyword correlation graph, constructing an aviation document keyword multimodal correlation graph, and constructing an image-text cross-modal retrieval model based on the acquired data set; a multidimensional similar word recall sorting model is constructed, aviation keyword semantic similarity fusion of text content and image content is calculated, and a similar keyword dictionary is constructed; the aviation document and keyword correlation graph constructed by the application improves aviation field keyword quality; the internal connection between the keywords constructed in the form of the multimodal correlation graph optimizes the problem that semantic expression between the keywords can only be directed to one mode of text information; through the constructed semantic similarity calculation model of the fusion of the text and the image, a near-synonym dictionary which can be used for an aviation standard field search engine is formed.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of aviation information processing, and particularly relates to an aviation document keyword similarity determination method fusing a multi-modal semantic association graph. BACKGROUND

[0002] In the application process of the aviation standard knowledge service system, a key requirement of the user is similarity query of the existing standards, which is related to the establishment of new standards and the use of existing standards. The search engine of the standard knowledge service system mainly takes keywords as the basis for recall and sorting. In the use process, the user often needs to input multiple semantically similar keywords to obtain perfect, accurate and non-missing results, which makes the standard query inefficient and often causes the omission of query results. The search engine in the aviation standard knowledge service system can improve the recall rate of semantic search by introducing a similar keyword dictionary, but there are many problems in constructing the aviation field semantic similar word dictionary, mainly including the following aspects:

[0003] Currently, there are few authoritative keyword libraries in the aviation field, and there is a lack of high-quality, sufficient and highly usable synonym and near-synonym word tables. In the process of judging semantic similarity, the similarity is mainly determined by natural language processing, and for such short texts as words, the effect is not good due to the lack of sufficient context information. In the process of judging the semantic similarity of aviation field words, the method of learning the semantic similarity of aviation field words through machine learning needs to manually annotate a large amount of corpus and train to obtain a similarity determination model.

[0004] The present application mainly solves the following three problems:

[0005] 1. By constructing an association relationship graph of aviation documents and keywords in the aviation field, the problem of lack of high-quality keywords in the aviation field is solved.

[0006] 2. By constructing a multi-modal association graph in the aviation field, the text semantic relationship and image content relationship between keywords are represented, and the internal relationship between keywords is constructed in the form of a multi-modal association graph, solving the problem that the semantic expression between keywords only has one modal information of text.

[0007] 3. By constructing a set of unsupervised semantic similarity calculation models fusing text content and image content, the similarity relationship between keywords is calculated, a near-synonym dictionary for the aviation standard field search engine is formed, and the problem of lack of keyword semantic similarity determination method is solved. SUMMARY

[0008] The purpose of the present application is to construct the correlation graph of aviation documents and keywords in the aviation field, obtain high-quality available keywords in the aviation field, construct a multi-modal correlation graph in the aviation field, represent the text semantic connection and image content connection between keywords, further construct an aviation document keyword similarity determination method based on the multi-modal semantic correlation graph, form a near-synonym dictionary that can be used for an aviation standard field search engine, and solve the problem of inaccurate keyword semantic similarity determination method.

[0009] To achieve the above-mentioned goal, the present application provides an aviation document keyword similarity determination method based on a multi-modal semantic correlation graph, which specifically comprises the following steps:

[0010] S1, constructing a correlation relationship graph of aviation document keywords;

[0011] S11, obtaining aviation field keywords and constructing an aviation document keyword dataset; using the standard name in the aviation standard library as input, using the literature title to search for aviation documents and keyword information related to aviation standards on the Internet as the aviation document keyword dataset;

[0012] S12, for each keyword , performing word frequency statistics, taking all keywords as a set , keyword , the word frequency of keyword is the number of occurrences of keyword in the aviation document keyword dataset;

[0013] S13, if a keyword and the search object keyword appear in the keyword list of the same aviation document, then the keyword and the search object keyword are defined as having a direct co-occurrence relationship, and the co-occurrence probability calculation method of keyword and search object keyword is: taking all keywords as , keyword , search object keyword , in this case, the probability of the occurrence of keyword when the search object keyword occurs is :

[0014] ;

[0015] wherein represents the probability of the occurrence of search object when keyword occurs, express and The number of times it appears at the same time, Indicates keywords Number of occurrences;

[0016] S14: Obtain the co-occurrence matrix of all keywords with the help of the calculation results of step S13 The co-occurrence matrix vector express ;

[0017] S15. Construct a sentence vector encoder for aviation document titles, extract features from aviation document titles, and construct similarity relationships between aviation document titles.

[0018] S16, the front-end result of the cosine similarity between each aviation document title obtained in step S15 is represented in the form of a graph, and a correlation graph of aviation document keywords is constructed; the nodes in the graph include document titles, document keywords, and document titles containing keywords, and the relationships include ,in, represents the title of the i-th document, Indicates the i-th keyword, which refers to The literature contains Keywords; ,in, represents the title of the i-th document, represents the jth document title, express and The similarity between the two refers to the title of the i-th document and the jth document title Between Similarity, ,in represents the i-th document keyword, Indicates the jth document keyword, which refers to the keyword Keywords when appearing Probability of simultaneous occurrence;

[0019] S2, construct a multimodal association map of aviation literature keywords;

[0020] S21, using the keywords obtained in step S11, using the first image search and the second image search to obtain images related to the keywords;

[0021] S22, perform image cleaning, remove damaged images and gif format images, and convert png and jpeg format images into jpg format images to form a first image data set with keyword labels. For each image All have labels ;

[0022] S23, calculating a multidimensional feature descriptor of the image; for the first image data set obtained in step S21 and step S22, respectively calculating a deep neural network descriptor, a grayscale histogram descriptor, a color layout descriptor, a color moment descriptor, and a histogram of oriented gradients descriptor;

[0023] S24, image-text cross-modal retrieval model training: for the first image dataset obtained in step S22, construct an image-text cross-modal retrieval model;

[0024] S3. Build a multi-dimensional similar word recall ranking model to calculate the semantic similarity of aviation keywords that integrate text content and image content;

[0025] S31. Construct a recall ranking model based on word form similarity;

[0026] S32. Construct a recall ranking model based on word meaning similarity;

[0027] S33, obtain similar candidate word sequence; give any keyword , get and keywords Similar candidate word list ;

[0028] S34, using the multimodal knowledge graph constructed in step S1 and step S2 to calculate the indirect co-occurrence similarity value of each candidate word ;

[0029] S35. Calculate the image similarity value of each candidate word using the multimodal knowledge graph constructed in step S1 and step S2 ;

[0030] S36, obtain by calculating the comprehensive similarity Similarity value , obtain keywords with similar meanings to the target word and build a similar keyword dictionary:

[0031] ;

[0032] in, Indicates keywords and candidate keywords The comprehensive similarity judgment score of .

[0033] Furthermore, step S15 specifically includes the following steps:

[0034] S151. Use aviation literature titles as training data and use the SimCSE model to unsupervisedly train the text sentence vector encoder.

[0035] S152, vector encoding is performed on all aeronautical literature titles by using the trained encoder, and the cosine similarity between each aeronautical literature title is calculated, and the similarity front-end result is analyzed:

[0036] ;

[0037] wherein, respectively represent the feature vectors of two literature titles, respectively represent the numerical values of each dimension of the feature vector.

[0038] Further, step S23 specifically comprises the following steps:

[0039] S231, calculating a deep neural network descriptor; feature extraction is performed on the image by using a deep neural network, and a ResNet neural network model and a Vision Transformer (ViT) neural network model are used for image vectorization to generate a deep neural network feature vector ;

[0040] S232, calculating a gray scale histogram descriptor, and performing statistics on the gray scale distribution of the image by using a gray scale histogram; the gray scale histogram represents the frequency of the gray scale size of all pixels in the image, and the gray scale feature vector is calculated by using the following formula :

[0041] ;

[0042] wherein, is the gray scale level of the pixel of the image, is the number of pixels with a gray scale , is the total number of pixels in the image;

[0043] S233, calculating a color layout descriptor, and the extraction process includes image segmentation, representative color selection, DCT transformation and zigzag scanning, and finally obtaining a color layout feature vector ;

[0044] S234, calculating a histogram of oriented gradients descriptor, obtaining the statistical quantity of the local image gradient direction information, and further obtaining a direction gradient feature vector .

[0045] Further, step S24 specifically comprises the following steps:

[0046] S241, obtaining the image and the label from the first image data set, and expanding the label content description , content description with image composing text-image pairs ;

[0047] S242, constructing a text encoder, using a Transformer model to generate a text modal vector , with a dimension of N;

[0048] S243, constructing an image encoder, using a ResNet and Vision Transformer (ViT) model to generate an image modal vector with a dimension of N;

[0049] S244, linearly projecting the text modal vector and the image modal vector to the same vector space, calculating the cosine similarity between the two modalities, so that the similarity of N matched text-image pairs is maximized, and the similarity of unmatched text-image pairs is minimized, using symmetric cross-entropy loss for training, and selecting 60% of the pictures for random cropping enhancement during training;

[0050] S245, training the model to obtain a text-image cross-modal retrieval model .

[0051] Further, step S31 specifically includes the following steps:

[0052] S311, constructing a segmentation dictionary , the segmentation dictionary contains terms and keywords in the target field;

[0053] S312, using a segmentation tool and the segmentation dictionary to perform fine-grained segmentation on all keywords ;

[0054] S313, for keywords and the corresponding feature words after fine-grained segmentation , using the BM25 algorithm to define the keyword similarity value between the keyword and the keyword , given a keyword , which contains feature words , the relevance score of the keyword and the keyword is:

[0055] ;

[0056] wherein Indicates the inverse document frequency of the query term, which measures how much information this feature word provides; Characteristic words In keywords The frequency of occurrence in , Represents the adjustment factor, usually k1=2,b=0.75, where Indicates keywords length, Expressed as the average length of all keywords.

[0057] Further, step S32 specifically includes the following steps:

[0058] S321, use the Chinese sentence-BERT model to vectorize each keyword, and encode the variable-length keywords Convert to a fixed-length 768-dimensional vector ;

[0059] S322, using cosine similarity to obtain keywords With keywords The vector similarity value between :

[0060] ;

[0061] in, Indicates keywords The eigenvector of With keywords The eigenvector of The similarity value between represent the eigenvectors, and Indicates the length of a vector.

[0062] Preferably, step S33 specifically includes the following steps:

[0063] S331, through the word form similarity calculation formula in step S313, from all keywords Get and A list of top five keyword candidates ranked by word form similarity , arranged from high to low by word form similarity value, including , then normalize and get candidate keywords With keywords The word form similarity value of ;

[0064] S332, using the word meaning similarity calculation formula in step S322, from all keywords Get A keyword list with the top five keywords ranked by semantic similarity : , the word meaning similarity values ​​are arranged from high to low and normalized to obtain candidate keywords With keywords The word meaning similarity value ;

[0065] S333: Merge the candidate words obtained in step S331 and step S332 to obtain a comprehensive candidate word sequence , is a list of candidate keywords with similar word forms. is a list of all candidate keywords with similar meanings, where the similarity value of each candidate keyword is for:

[0066] .

[0067] Preferably, step S34 specifically includes the following steps:

[0068] S341. Define common co-occurring words ,in and target keywords appear in the keyword list of the same aviation document; and and candidate keywords Also appears in the keywords of the same aviation document; when this relationship exists, it is called the target keyword and candidate keywords Have indirect co-occurrence relationship;

[0069] S342, calculating the indirect co-occurrence similarity value using the association relationship map of aviation literature keywords obtained in step S1;

[0070] = ;

[0071] in, Indicates candidate keywords With keywords The co-occurrence relationship score of yes Keywords when appearing The probability of occurrence, yes Candidate keywords when they appear Probability of occurrence.

[0072] Preferably, step S35 specifically includes the following steps:

[0073] S351, target keywords Input step S24 constructs a text-image cross-modal retrieval model , and obtains the image result ranked first ;

[0074] S352, input each candidate keyword to the text-image cross-modal retrieval model constructed in step S24 , and obtain the image result ranked first ;

[0075] S353, for and , calculate the cosine distance similarity of , , , respectively by querying the descriptor of the image in the multi-modal knowledge graph constructed in step S23 , , and take the average value, finally obtain the image similarity value as:

[0076] ;

[0077] Wherein, represents the similarity value calculated by the image semantics of the keyword and the candidate keyword , represents the cosine similarity of the image descriptor of the keyword (in any one of the description ways of image description , , , ) and the image descriptor of the candidate keyword .

[0078] Preferably, in step S244, the judgment expression for making the similarity of N matched image-text pairs maximum and the similarity of unmatched image-text pairs minimum is:

[0079] ;

[0080] Wherein, represents the i-th image-text pair, 1 for matching and 0 for mismatching, is the probability that the model predicts the image-text pair as a positive example, and N is the number of image-text pairs.

[0081] Compared with the prior art, the present application has the following beneficial effects:

[0082] 1. By constructing a correlation graph of aviation literature and keywords in the aviation field, we fully explored the relationship and implicit similarity logic between keywords in aviation literature within a field, expanded the keyword vocabulary, and solved the problem of lack of high-quality keywords in the aviation field;

[0083] 2. By constructing a multimodal association graph in the aviation field, we can show the textual semantic connections and image content connections between keywords. This multimodal association graph can be used to build the intrinsic connections between keywords, solving the problem where the semantic expression between keywords only has one modal information: text.

[0084] 3. By constructing an unsupervised semantic similarity calculation model that integrates text content and image content, the similarity features of text modality in terms of word form and word meaning, as well as the visual features of image modality, are integrated to mine the semantic relationship between keywords and calculate the similarity between keywords. This forms a synonym dictionary that can be used in search engines in the aviation standards field, solving the problem of the lack of a method for determining keyword semantic similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 This is a flowchart of the aviation literature keyword similarity determination method integrating multimodal semantic association graphs according to the present invention;

[0086] Figure 2 A schematic diagram of the association relationship map of aviation literature keywords constructed for the present invention;

[0087] Figure 3 This is a flow chart of a method for calculating color layout descriptors in the aviation literature keyword similarity determination method integrating a multimodal semantic association graph according to the present invention;

[0088] Figure 4 This is a flow chart of a method for calculating a directional gradient histogram descriptor in the aviation literature keyword similarity determination method integrating a multimodal semantic association graph according to the present invention;

[0089] Figure 5 The text-image cross-modal retrieval model is obtained by training the aviation literature keyword similarity determination method based on the multimodal semantic association graph of the present invention. Schematic diagram. DETAILED DESCRIPTION

[0090] The exemplary embodiments, features, and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0091] like Figure 1As shown, the present application provides a kind of fusion multimodal semantic association graph of aviation document keyword similarity determination method, it includes:

[0092] S1, the association relationship graph of aviation document keyword is constructed;

[0093] S11, the key word of aviation field is acquired, and the key word data set of aviation document is constructed;Using standard name in aviation standard library as input, from Internet, such as the paper abstract query interface of open paper of ZhiNet, the scientific research paper and key word information related to aviation standard are retrieved, as the key word library of aviation field.

[0094] S12, for each key word , carry out word frequency statistics, all key words are as set , key word , the word frequency of key word is the number of times of key word in aviation document key word data set;

[0095] S13, if a key word With the search object key word In the key word list of the same aviation document, then define the key word With search object key word There is direct co-occurrence relationship, and The co-occurrence probability calculation method of , key word , search object key word In this case, the probability of search object key word When key word Appear is :

[0096] ;

[0097] Wherein Indicate the probability of search object When key word Appear, Indicate the number of times of With Simultaneous appearance, Indicate the number of times of key word Appear;

[0098] S14, by the calculation result of step S13, the co-occurrence matrix of all key words is obtained Wherein the co-occurrence matrix vector Indicate ;

[0099] S15, construct a sentence vector encoder for aviation literature titles, extract features from aviation literature titles, and construct similarity relationships between aviation literature titles;

[0100] S151, use the SimCSE model to unsupervisedly train the text sentence vector encoder with the aviation literature titles as training data.

[0101] S152, vector encode all aviation literature titles using the trained encoder, and calculate the cosine similarity between each aviation literature title. Take the top 10 results in terms of similarity, i.e., the top 10 similar results:

[0102] ;

[0103] where, respectively represent the feature vectors of two literature titles, respectively represent the numerical values of each dimension of the feature vector.

[0104] S16, represent the top results of the cosine similarity between each aviation literature title obtained in step S15 in the form of a graph, and construct an association graph of aviation literature keywords. The nodes in the graph include literature titles, literature keywords, and literature titles containing keywords, and the relationships include where, represents the i-th literature title, represents the i-th keyword, which means the keyword appears in the literature; where, represents the i-th literature title, represents the j-th literature title, represents the similarity between , which means that the i-th literature title and the j-th literature title have similarity, where represents the i-th literature keyword, represents the j-th literature keyword, which means that the keyword appears at the same time;

[0105] S2, construct a multi-modal association graph of aviation field keywords.

[0106] ​​S21, the keywords obtained by means of step S11 are used to obtain images related to the keywords by using first picture search and second picture search, wherein the first picture search is a Google picture search, and the second picture search is a Bing picture search.

[0107] S22, image cleaning is performed to remove damaged images and gif format images, and png and jpeg format images are converted into jpg format images to form a first image data set with keyword labels, and each image has a label .

[0108] S23, multi-dimensional feature descriptor calculation of images; the first image data set obtained in steps S21 and S22 is used to calculate deep neural network descriptors, gray scale histogram descriptors, color layout descriptors, color moment descriptors and direction gradient histogram descriptors.

[0109] S231, calculation of deep neural network descriptors; feature extraction of images is performed by using a deep neural network, and ResNet neural network model and Vision Transformer (ViT) neural network model are used for image vectorization to generate deep neural network feature vectors .

[0110] S232, calculation of gray scale histogram descriptors; the gray scale histogram is used to count the gray scale distribution of images; the gray scale histogram represents the frequency of the gray scale size of all pixels in the image, and the gray scale feature vector is calculated by using the following formula :

[0111] ;

[0112] wherein, is the gray scale level of the pixel of the image, is the number of pixels with the gray scale , and is the total number of pixels in the image.

[0113] S233, calculation of color layout descriptors; as shown in Figure 3 , the color layout descriptor is a kind of efficient local color feature description in the mpeg-7 multimedia content standard description, and the extraction process includes image segmentation, representative color selection, DCT transformation and zigzag scanning, and finally the color layout feature vector is obtained.

[0114] S234, the direction gradient histogram (HOG) is a kind of feature descriptor used for target detection in the field of computer vision and image processing, as shown in Figure 4As shown, the direction gradient histogram descriptor is calculated to obtain the statistical quantity of the local image gradient direction information, and then the direction gradient feature vector is obtained .

[0115] S24, image-text cross-modal retrieval model training; for the first image data set obtained in step S22, an image-text cross-modal retrieval model is constructed, mainly including a training data set, a text encoder, an image encoder, and an aviation field keyword image-text cross-modal retrieval model is obtained by training.

[0116] S241, obtaining an image and a label from the first image data set. extended content description , content description and image to form a text-image pair .

[0117] S242, constructing a text encoder, using a Transformer model to generate a text modal vector , the vector dimension is N.

[0118] S243, constructing an image encoder, using ResNet and Vision Transformer (ViT) model to generate an image modal vector dimension N.

[0119] S244, linearly projecting the text modal vector and the image modal vector to the same vector space, calculating the cosine similarity between the two modalities, making the similarity of N matched image-text pairs maximum and the similarity of unmatched image-text pairs minimum, and the judgment expression is:

[0120] ;

[0121] wherein, represents the i-th image-text pair, 1 for matching and 0 for mismatching, is the probability that the model predicts the image-text pair as a positive example, and N is the number of image-text pairs.

[0122] Symmetric cross-entropy loss is used for training, and 60% of the pictures are randomly selected for random cropping enhancement during training.

[0123] S245, training the model to obtain a text-image cross-modal retrieval model as shown in Figure 5 .

[0124] S3, construct a multi-dimensional similar word recall ranking model to calculate the semantic similarity of aviation keywords fused with text content and image content; the multi-dimensional similar word recall ranking model mainly calculates the similarity between keywords through constructing the morphological similarity and semantic similarity between keywords .

[0125] S31, construct a recall ranking model based on morphological similarity.

[0126] S311, construct a word segmentation dictionary , which contains terms and keywords in the target field.

[0127] S312, use word segmentation tools and dictionaries to perform fine-grained word segmentation on all keywords , and the feature words of each keyword , such as "Laval nozzle", can be divided into "Laval nozzle", "Laval", and "nozzle".

[0128] S313, for keywords and the corresponding feature words after fine-grained word segmentation, use the BM25 algorithm to define the keyword similarity value between keywords , given a keyword , which contains feature words , the relevance score between keywords and keywords is:

[0129] ;

[0130] where represents the inverse document frequency of the query term, which measures how much information this feature word provides; represents the frequency of the feature word in the keyword , , represents the adjustment factor, usually k1=2,b=0.75, where represents the length of the keyword , represents the average length of all keywords.

[0131] S32, construct a recall ranking model based on semantic similarity. ​​​​​​

[0132] S321, use the Chinese sentence-BERT model to vectorize each keyword, and encode the variable-length keywords Convert to a fixed-length 768-dimensional vector .

[0133] S322, using cosine similarity to obtain keywords With keywords The vector similarity value between :

[0134] ;

[0135] in, Indicates keywords The eigenvector of With keywords The eigenvector of The similarity value between represent the eigenvectors, and Indicates the length of a vector.

[0136] S33, obtain similar candidate word sequence; give any keyword , get the Similar candidate word list .

[0137] S331, through the word form similarity calculation formula in step S313, from all keywords Get and A list of top five keyword candidates ranked by word form similarity , arranged from high to low by word form similarity value, including , then normalize and get candidate keywords With keywords The word form similarity value of .

[0138] S332, using the word meaning similarity calculation formula in step S322, from all keywords Get A keyword list with the top five keywords ranked by semantic similarity : , the word meaning similarity values ​​are arranged from high to low and normalized to obtain candidate keywords With keywords The word meaning similarity value .

[0139] S333, merge the candidate words obtained in steps S331 and S332 to obtain a comprehensive candidate word sequence , is a list of all word-form similar candidate keywords, is a list of all word meaning similar candidate keywords, where the similarity value of each candidate keyword is:

[0140] .

[0141] S34, calculate the indirect co-occurrence similarity value of each candidate word using the multi-modal knowledge graph constructed in steps S1 and S2 .

[0142] S341, define common co-occurrence words where and the target keyword appear in the keyword list of the same aviation document; at the same time and the candidate keyword also appear in the keywords of the same aviation document; when this relationship exists, the target keyword has an indirect co-occurrence relationship with the candidate keyword .

[0143] S342, calculate the indirect co-occurrence similarity value using the association relationship graph of the aviation document keywords obtained in step S1.

[0144] = ;

[0145] where, represents the co-occurrence relationship score of the candidate keyword and the keyword , is the probability of the keyword appearing when it appears, is the probability of the selected keyword appearing when it appears.

[0146] S35, calculate the image similarity value of each candidate word using the multi-modal knowledge graph constructed in steps S1 and S2 .

[0147] S351, input the target keyword into the text-image cross-modal retrieval model constructed in step S24 to obtain the image result ranked first .

[0148] S352, each candidate keyword Input to the text-image cross-modal retrieval model constructed in step S24 , get the top ranked image result .

[0149] S353, for and By querying the descriptor of the image in the multimodal knowledge graph constructed in step S23 、 、 、 , respectively calculated and The cosine distance similarity of the image is calculated and the average value is taken. The final image similarity value is:

[0150] ;

[0151] in, Indicates keywords and candidate keywords The similarity value calculated by image semantics, Indicates keywords Image descriptor (In the image description 、 、 、 Choose one of the descriptions) and candidate keywords Image descriptor The cosine similarity of .

[0152] S36, obtain by calculating the comprehensive similarity Similarity value , obtain keywords with similar meanings to the target word and build a similar keyword dictionary:

[0153] ;

[0154] in, Indicates keywords and candidate keywords The comprehensive similarity judgment score of .

[0155] The application improves the association ability of the field search / recommendation engine for the user input keyword by means of expanding the synonym library of the field keyword, constructs a multi-modal graph capable of expressing the correlation of the keyword by using the keyword in the aviation document and the related image on the Internet, and mines more synonymous keywords by using the machine learning and graph analysis technology. By constructing a set of unsupervised semantic similarity calculation model fusing the text content and the image content, the semantic relationship between the keywords is mined by fusing the similar features of the text mode in the word form and the word meaning and the visual features of the image mode, the similarity between the keywords is calculated, a synonymous keyword dictionary capable of being used in the aviation standard field search engine is formed, and the problem of the missing keyword semantic similarity judgment method is solved.

[0156] The above-described embodiments are only used to describe the preferred embodiments of the present application, and are not used to limit the scope of the present application, and various modifications and improvements made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application shall fall within the protection scope of the present application defined by the claims.

Claims

1. A method for determining aviation literature keyword similarity by integrating a multimodal semantic association graph, characterized by: It includes the following steps: S1. Construct a correlation graph of aviation literature keywords; S11. Obtain keywords in the aviation field and construct a keyword dataset of aviation literature; Using the standard names in the aviation standards database as input, we searched the Internet for aviation literature and keyword information related to aviation standards based on the literature titles to create the aviation literature keyword dataset. S12, for each keyword w i , perform word frequency statistics, take all keywords as set W, keyword wi∈W, and its word frequency c(w i ) is the keyword w i The number of times it appears in the aviation literature keyword dataset S13. If a keyword w i and the search target keyword w j appears in the keyword list of the same aviation document, then define w i With w j There is a direct co-occurrence relationship, w i With w j The co-occurrence probability calculation method is: keyword w i ∈W, w j ∈W, in this case w i When it appears j The probability of occurrence is p(w j |w i ): ； Where p(w j |w i ) indicates the keyword w i Retrieve object w when it appears j The probability of occurrence, c(w i ,w j ) represents w i With w j Number of simultaneous occurrences; S14, using the calculation results of step S13, obtain the co-occurrence matrix M of all keywords, where the co-occurrence matrix vector M ij represents p(w j |w i ); S15. Construct a sentence vector encoder for aviation document titles, extract features from aviation document titles, and construct similarity relationships between aviation document titles. S16, the front-end result of the cosine similarity between each aviation document title obtained in step S15 is represented in the form of a graph to construct the association relationship map; the nodes in the association relationship map include document titles and document keywords; the edges in the association relationship map include 、 、 ;right , T s represents the sth document title, W i Indicates the i-th document keyword, which refers to T t The literature contains W i ;right ,T s represents the title of the sth document, T t Indicates the title of the t-th document, Simcos st Indicates T s With T t The similarity between them refers to T s and T t Simcos st Similarity, Where W i represents the i-th document keyword, W j Indicates the jth document keyword, which refers to W i When W appears j Probability of simultaneous occurrence; S2, construct a multimodal association map of aviation literature keywords; S21, using the aviation field keywords obtained in step S11, using the first image search and the second image search to obtain images related to the aviation field keywords; S22, perform image cleaning, remove damaged images and gif format images, and convert png and jpeg format images into jpg format images to form a first image data set with keyword labels. For each image i k , all have label i k-label ∈W; S23, calculating a multidimensional feature descriptor of the image; for the acquired first image data set, respectively calculating a deep neural network descriptor, a grayscale histogram descriptor, a color layout descriptor, a color moment descriptor, and a directional gradient histogram descriptor; S24. Constructing an image-text cross-modal retrieval model for the acquired first image dataset; S3. Constructing a multi-dimensional similar word recall ranking model to calculate the semantic similarity of aviation keywords that integrate text content and image content; S31. Construct a recall ranking model based on word form similarity; S32. Construct a recall ranking model based on word meaning similarity; S33, obtain the word form similarity in W with w i The candidate word list Wk with the top q word form similarity values can ; Obtain the words in W that are similar to w through word semantic similarity i The candidate word list Wv with the top q semantic similarity values can ; get and w i Similar candidate word list Wcan, Wcan=Wk can ∪Wv can ; Determine the candidate word w in the candidate word list by word form similarity and word meaning similarity canq and w i The text similarity between ; S34, using the association relationship map to calculate With w i Co-occurrence relationship score ; S35, using the multimodal association map to calculate With w i Image similarity value ; S36. Calculate comprehensive similarity , Get keywords with similar meanings to the target word and build a dictionary of similar keywords: ； in, Indicates keyword w i and candidate keywords w canq The comprehensive similarity judgment score of .

2. The aviation document keyword similarity determination method based on the fusion of multimodal semantic association graphs according to claim 1 is characterized in that: Step S15 specifically includes the following steps: S151. Use aviation literature titles as training data and use the SimCSE model to unsupervisedly train the text sentence vector encoder. S152. Use the trained encoder to vector encode all aviation document titles, calculate the cosine similarity between each aviation document title, and analyze the similarity front-end results: ； Among them, A and B represent the feature vectors of two literature titles respectively. i ,B i Represents the value of each dimension of the feature vector.

3. The aviation document keyword similarity determination method based on the fusion of multimodal semantic association graphs according to claim 2 is characterized in that: Step S23 specifically includes the following steps: S231. Calculate the deep neural network descriptor; extract features from the image using the deep neural network, vectorize the image using the ResNet neural network model and the Vision Transformer neural network model, and generate a deep neural network feature vector ; S232, calculate the grayscale histogram descriptor, and use the grayscale histogram to perform statistics on the grayscale distribution of the image; the grayscale histogram represents the frequency of occurrence of the grayscale size of all pixels in the image, and the grayscale feature vector is calculated using the following formula ; ； Among them, p r (r k ) is the gray level of the pixel in the image, n k is a grayscale r k The number of pixels, n' is the total number of pixels in the image; S233, calculate the color layout descriptor, the extraction process includes image segmentation, representative color selection, DCT transformation and Z-shaped scanning, and finally obtain the color layout feature vector ; S234, calculate the directional gradient histogram descriptor, obtain the statistics of the local image gradient direction information, and then obtain the directional gradient feature vector .

4. The aviation document keyword similarity determination method based on the fusion of multimodal semantic association graphs according to claim 3 is characterized in that: Step S24 specifically includes the following steps: S241, obtain image i according to the first image data set. k With label i k-label , label i k-label Extended content description k-sentence , content description k-sentence With image i k Composing a text-image pair P; S242. Build a text encoder and use the Transformer model to generate a text modality vector v T , the vector dimension is N; S243. Build an image encoder and use the ResNet and Vision Transformer models to generate the image modality vector v I The dimension is N; S244, the text modal vector v T and the image modality vector v I Linearly project the images into the same vector space and calculate the cosine similarity between the two modalities, so that the similarity of N' matching image-text pairs is maximized and the similarity of non-matching image-text pairs is minimized. The model is trained using a symmetric cross-entropy loss, and 60% of the images are randomly cropped and augmented during training. S245. Train the model to obtain the text-image cross-modal retrieval model crosssearch txt-img .

5. The aviation document keyword similarity determination method based on the fusion of multimodal semantic association graphs according to claim 1 is characterized in that: Step S31 specifically includes the following steps: S311, constructing a word segmentation dictionary dic, which contains terms and keywords in the target field; S312, using the word segmentation tool and the word segmentation dictionary dic to perform fine-grained word segmentation on all keywords W; S313, for the keyword W and the corresponding feature word FW after fine-grained word segmentation, use the BM25 algorithm to define the keyword w i With the keyword w j The keyword similarity value between them, given a keyword w i , which contains the feature word fw i1 ,fw i2 …fw im , keyword w j With the keyword w i The relevance score keyword (w j ,w i )for: ； Where IDF(fw ik ) represents the inverse document frequency of the query term, which measures how much information this feature word provides; f(fw ik ,w j ) represents the feature word fw ik In the keyword w j The frequency of occurrence, k1, b represents the adjustment factor, usually k1 = 2, b = 0.75, where |w j | indicates the keyword w j , avgdl is the average length of all keywords.

6. The aviation document keyword similarity determination method based on the fusion of multimodal semantic association graphs according to claim 3 is characterized in that: Step S32 specifically includes the following steps: S321, use the Chinese sentence-BERT model to vectorize each keyword, and encode the variable-length keyword w i Convert to a fixed-length 768-dimensional vector ; S322, using cosine similarity to obtain keyword w i With the keyword w j The vector similarity score between veccos (w i ,w j ): ； in, Indicates keyword w i The eigenvector of With the keyword w j The eigenvector of The similarity value between represent the eigenvectors, and Indicates the length of a vector.

7. The method for determining aviation document keyword similarity by integrating multimodal semantic association graphs according to claim 1, characterized in that: Step S33 specifically includes the following steps: S331, using the word form similarity calculation formula in step S313, obtain the top five keyword candidate list Wk with word form similarity to wi from all keywords W can , arranged from high to low by word form similarity value, including wk can1 ,wk can ,wk can ,wk can4 ,wk can5 , and then normalize it to get the candidate keyword wk canq With the keyword w i The word form similarity score keyword (wk canq ,w i ),q=1…5; S332, using the word meaning similarity calculation formula in step S322, obtain the word meaning similarity with w from all keywords W i Keyword list with the top five keywords ranked by semantic similarity Wv can , arranged from high to low by word meaning similarity value, including wv can1 ,wv can2 ,wv can3 ,wv can4 ,wv can5 , and perform normalization processing to obtain the candidate keyword wv canq With the keyword w i The word meaning similarity score veccos (w i ,wv canq ), q = 1…5; S333: Merge the candidate words obtained in step S331 and step S332 to obtain a comprehensive candidate word sequence W can =Wk can ∪Wv can , Wk can is a list of candidate keywords with similar word forms, Wv can is a list of all candidate keywords with similar meanings, where the similarity value of each candidate keyword is for: .

8. The aviation document keyword similarity determination method based on the fusion of multimodal semantic association graphs according to claim 1 is characterized in that: Step S34 specifically includes the following steps: S341. Define the co-occurring term co , where term co The target keyword w appears in the keyword list of the same aviation document; at the same time, term co With the candidate keyword w can Also appear in the keywords of the same aviation document; when they have this relationship, the target keyword w and the candidate keyword w can Have indirect co-occurrence relationship; S342, calculating the indirect co-occurrence similarity value using the association relationship map of aviation literature keywords obtained in step S1; ； in, Indicates candidate keywords With the keyword w i The co-occurrence relationship score, p(w i |term co ) is the term co When the keyword w appears i The probability of occurrence, It is term co Candidate keywords when they appear Probability of occurrence.

9. The aviation document keyword similarity determination method based on the fusion of multimodal semantic association graphs according to claim 1 is characterized in that: Step S35 specifically includes the following steps: S351, input the target keyword w into the text-image cross-modal retrieval model crosssearch constructed in step S24 txt-img , get the image result I ranked first w ; S352, each candidate keyword Input to the text-image cross-modal retrieval model crosssearch constructed in step S24 txt-img , get the top ranked image result ; S353, for I w and By querying the descriptor of the image in the multimodal knowledge graph constructed in step S23 Calculate I separately w and The cosine distance similarity of the image is calculated and the average value is taken. The final image similarity value is: ； in, Indicates keyword w i and candidate keywords The similarity value calculated by image semantics, Indicates keyword w i Image descriptor and candidate keywords Image descriptor The cosine similarity of For the image descriptor Choose one of the descriptions.

10. The aviation document keyword similarity determination method based on the fusion of multimodal semantic association graphs according to claim 4 is characterized in that: In step S244, the judgment expression for maximizing the similarity of N' matching image-text pairs and minimizing the similarity of unmatched image-text pairs is: ； Among them, y ii Indicates the ii-th image-text pair, matching is 1, mismatch is 0, p ii The model will be the picture and text ii The probability of predicting a positive example, N' is the number of image-text pairs.

Images