Scientific and technological text picture and text matching algorithm

Abstract

The invention discloses a scientific and technological text picture and text matching algorithm, and relates to the technical field of artificial intelligence. The problems that with rapid development of the science and technology field, new text and picture types emerge continuously, it cannot be ensured that a database can cover and accurately recognize the new types, texts and pictures in different fields have different characteristics and rules, a recommendation system cannot adapt to requirements of different fields, and targeted recommendation services cannot be achieved are solved. Through big data crawling and topic screening, the diversity and correlation of a data set are ensured, so that the algorithm can better adapt to the rapid development requirement of the science and technology field, new types of texts and pictures are covered and accurately recognized, and the establishment and entity alignment of a science and technology text mapping knowledge domain are improved. According to the method, the deep connection between the text and the image can be captured in the matching degree calculation, the matching precision is improved, and the reliability and the accuracy of a matching result are improved by introducing weight distribution and an attention mechanism.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a scientific and technological text image and text matching algorithm. Background Art

[0002] With the rapid development of big data and artificial intelligence technology, the matching of text and images has become an important research topic. Regarding text image and text matching, the Chinese patent with publication number CN111339338B discloses a text image matching recommendation method based on deep learning. The recommendation method adopted is based on the constantly enriched user operation record database, using image recognition and keyword extraction algorithm based on feature vector distribution, constructing keyword feature vectors, and combining user operation record database, vector similarity algorithm and set similarity algorithm to achieve the best matching recommendation between pictures and text. After the inputter enters the text and picture data, it can automatically recommend the homepage picture entered this time and the display position of the entered picture in the text. So as to eliminate the work of the inputter to understand the text, manually select pictures, and adjust the picture position.

[0003] Although the above patent reduces the workload of data entry personnel through automation, the following problems still exist:

[0004] With the rapid development of science and technology, new types of text and images continue to emerge. It is impossible to ensure that the database can cover and accurately identify these new types. Texts and images in different fields have different characteristics and rules. The recommendation system cannot adapt to the needs of different fields and cannot provide targeted recommendation services. Summary of the invention

[0005] The purpose of the present invention is to provide a scientific text image and text matching algorithm, which can better identify and understand new types of text and images through big data crawling and topic screening, combined with entity alignment of scientific text knowledge graph. The feature extraction and fusion process takes into account the characteristics of different fields, so that the algorithm can provide personalized matching services according to the needs of different fields to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A scientific text image and text matching algorithm includes the following steps:

[0008] Step 1: Data preprocessing: Classify and identify the collected scientific and technological text images and texts, and preprocess the texts and images based on the recognition results;

[0009] Step 2: Feature extraction: Encode the preprocessed text data and convert it into vector representation, extract the converted text features, perform feature extraction on the image data, capture the visual features in the image, fuse the text features with the visual features, and build an association model between text and image;

[0010] Step 3: Matching degree calculation: Calculate the similarity between the fused text features and visual features based on the association model;

[0011] Step 4: Model output: Set a matching threshold based on the matching calculation result. When the matching degree exceeds the set matching threshold, the image and text are considered to match, and the matching result is output;

[0012] Step 5: Evaluation and optimization: Evaluate the constructed association model, adjust the corresponding parameters according to the evaluation results, and continuously optimize the model through multiple iterative training until satisfactory performance indicators are achieved.

[0013] Furthermore, in step 1, the text and the picture are preprocessed, specifically:

[0014] Text preprocessing: including removing stop words, punctuation marks and non-semantic information of numbers, performing stem extraction, word form restoration and text segmentation operations;

[0015] Image preprocessing: including adjusting the image size and format, normalizing the image, and also including image denoising and grayscale operations.

[0016] Furthermore, the data preprocessing in step 1 further includes:

[0017] Crawl text and image data based on big data, determine the crawled text theme, and filter the corresponding text image based on the text theme, establish a mapping relationship between text and image, and draw a scientific and technological text knowledge graph;

[0018] The source data platforms corresponding to the text and image data are obtained, and multimodal text knowledge graphs of the source data platforms are formed respectively, and the texts and images in the multimodal text knowledge graphs are entity aligned.

[0019] Furthermore, in the multimodal text knowledge graph, when any text data is retrieved, image data related to the text is retrieved for the multimodal text knowledge graph. At the same time, text data from different source data platforms but representing the same or similar content are presented separately in the form of a text knowledge graph.

[0020] Furthermore, the feature extraction in step 2 is specifically as follows:

[0021] Text feature extraction: Convert the segmented text into a vector representation of a fixed dimension, summarize the word vectors, and select the keyword corresponding to the word vector with the smallest distance to the standard word vector as the target keyword and the text feature of the text;

[0022] Image feature extraction: Capture the edge information of the image data, extract the texture and shape features in the image based on the edge information, extract high-level semantic features based on the texture and shape features, and perform feature encoding to obtain a fixed-dimensional feature vector as the visual feature of the image;

[0023] Feature fusion: Align the feature dimensions of the text feature vector and the image feature vector, and concatenate them in the dimensions to form a new joint feature vector, and build an association model based on the joint feature vector.

[0024] Furthermore, the feature extraction in step 2 also includes: feature extraction based on text features and visual features, comparing the difference between text and image as a loss function, extracting corresponding writing style features and emotional features in the text, and obtaining corresponding changes in visual features of the image through changes in writing style features and emotional features.

[0025] Furthermore, the matching degree calculation in step 3 is specifically as follows:

[0026] Assign corresponding weights based on different feature categories in the feature extraction results;

[0027] The similarity between text and image in this type of feature is calculated based on the cosine similarity algorithm, and the similarity of each type of feature is weighted and summed according to its weight to obtain a comprehensive similarity score;

[0028] Based on the attention mechanism, the attention scores of each word or sentence in the text and each region in the image are calculated to construct a matching matrix between the text and the image.

[0029] The comprehensive similarity score and the matching matrix between the text and the image are input into the trained association model to calculate the final matching score between the text and the image.

[0030] Furthermore, the matching degree calculation in step 3 further includes:

[0031] Obtain the relationships and attributes of text and image data in the scientific and technological text knowledge graph;

[0032] Identify entities in text data and link the identified entities to corresponding nodes in the scientific text knowledge graph;

[0033] Extracting attribute information related to entities from the scientific and technological text knowledge graph and mapping the attribute information into a feature vector;

[0034] Analyze the relationships in the knowledge graph of scientific and technological texts, extract relationship features based on the analysis results, and determine the strength of association between entities;

[0035] The joint feature vector is updated by combining the attribute information mapped into the feature vector and the association strength between entities.

[0036] Furthermore, the model output in step 4 is specifically:

[0037] Determine the matching threshold: Draw a histogram based on the matching scores, determine the distribution of the matching scores, and extract the low-density area of ​​the score distribution of matching and non-matching samples as the matching threshold;

[0038] Matching result judgment: Compare the matching score of each text and image pair with the set matching threshold;

[0039] If it is higher than the set matching threshold, it is considered a match;

[0040] Otherwise, it is judged as mismatch;

[0041] Record each judgment result, including the status of whether it matches or not and the corresponding matching score, and output the matching result in a formatted manner.

[0042] Furthermore, the matching result includes successfully matched pictures, text pairs and matching scores.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] Through big data crawling and topic screening, the diversity and relevance of the data set are ensured, so that the algorithm can better adapt to the rapid development of the science and technology field, cover and accurately identify new types of text and pictures, and the establishment of scientific and technological text knowledge graphs and entity alignment helps to capture the deep connection between text and images in the matching calculation, thereby improving the matching accuracy. The feature extraction and fusion process takes into account the characteristics of text and pictures in different fields, so that the algorithm can adapt to the needs of different fields and provide targeted recommendation services. The introduction of weight allocation and attention mechanism makes the key features get more attention in the matching process, thereby improving the reliability and accuracy of the matching results. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a step diagram of the scientific text image and text matching algorithm of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In order to solve the technical problems that with the rapid development of science and technology, new types of text and images are constantly emerging, and it is impossible to ensure that the database can cover and accurately identify these new types, texts and images in different fields have different characteristics and rules, and the recommendation system cannot adapt to the needs of different fields and cannot provide targeted recommendation services, please refer to Figure 1 , this embodiment provides the following technical solutions:

[0048] A scientific text image and text matching algorithm includes the following steps:

[0049] Step 1: Data preprocessing: Classify and identify the collected scientific and technological text images and texts, and preprocess the texts and images based on the recognition results, specifically:

[0050] Text preprocessing: including removing non-semantic information (such as stop words, punctuation marks and numbers), performing stemming, word form restoration and text segmentation operations, which helps to extract the keywords and core content of the text and improve the accuracy and efficiency of subsequent feature extraction;

[0051] Image preprocessing: including adjusting the image size and format, normalizing the image, and performing image denoising and grayscale operations to reduce redundant information in the image data and highlight key features to improve the effect of feature extraction;

[0052] Step 2: Feature extraction: Encode the preprocessed text data and convert the text data into vector representation, extract the converted text features, such as grammatical sentence features, semantic features, context features, etc., extract features from the image data, capture the visual features in the image, such as edges, textures, shapes, etc., and encode these features into vector form; fuse the text features with the visual features to build a correlation model between text and image;

[0053] Step 3: Matching degree calculation: Calculate the similarity between the fused text features and visual features based on the association model, specifically:

[0054] Assign corresponding weights based on different feature categories in the feature extraction results (such as text features, text sentiment features, image visual features, etc.), where higher weights are assigned to style and sentiment features;

[0055] The similarity between text and image in this type of feature is calculated based on the cosine similarity algorithm, and the similarity of each type of feature is weighted and summed according to its weight to obtain a comprehensive similarity score;

[0056] Based on the attention mechanism, the attention scores of each word or sentence in the text and each region in the image are calculated to construct a matching matrix between the text and the image. Each element in the matrix represents the matching strength between an element (word or sentence) in the text and a region in the image.

[0057] Inputting the comprehensive similarity score and the matching matrix between the text and the image into the trained association model to calculate the final matching score between the text and the image, which reflects the similarity between the text and the image;

[0058] In this embodiment, style and sentiment features can often reflect the deep meaning of text and the emotional expression of images. Giving them higher weights can ensure that these key information are fully considered in the matching process. Through weight distribution, the algorithm can pay more attention to the features that have a greater impact on the matching results, thereby improving the relevance and accuracy of the matching, so that the algorithm can adjust the calculation method of similarity according to different application scenarios and needs, increase the flexibility of the algorithm, and provide more personalized matching services. By comprehensively considering multiple feature dimensions and fine-grained matching relationships, the algorithm can more accurately identify and match related texts and images;

[0059] Step 4: Model output: Set the matching threshold according to the matching calculation result. When the matching degree exceeds the set matching threshold, the image and text are considered to match, and the matching result is output, which is as follows:

[0060] Determine the matching threshold: Draw a histogram based on the matching scores, determine the distribution of the matching scores, and extract the low-density area of ​​the score distribution of matching and non-matching samples as the matching threshold;

[0061] Matching result judgment: Compare the matching score of each text and image pair with the set matching threshold;

[0062] If it is higher than the set matching threshold, it is considered a match;

[0063] Otherwise, it is judged as mismatch;

[0064] Record each judgment result, including the status of match or not and the corresponding match score, and output the matching result in a formatted manner;

[0065] The matching result includes the successfully matched pictures, text pairs and matching scores;

[0066] Step 5: Evaluation and optimization: Evaluate the constructed association model, adjust the corresponding parameters in the data preprocessing, feature extraction, matching calculation and other steps according to the evaluation results, and continuously optimize the model through multiple iterative training until satisfactory performance indicators are achieved. Performance indicators include accuracy, recall rate, F1 score and other indicators. These indicators are used to evaluate the performance of the association model.

[0067] In this embodiment, the intrinsic information of the text is fully captured by encoding and converting it into a vector representation, and the captured visual features provide a rich description of the image content. The diversity of extraction ensures that the deep information of the text and image is fully mined. The feature fusion enhances the correlation between the text and the image, and provides a richer information dimension for the matching algorithm. The weight allocation and attention mechanism improve the flexibility of the matching algorithm and its sensitivity to key features. The combination of the comprehensive similarity score and the matching matrix makes the matching results more comprehensive and reliable. The continuous evaluation and optimization process improves the accuracy and robustness of the algorithm to adapt to different data sets and application scenarios, thereby improving the applicability of the algorithm.

[0068] In this embodiment, the data preprocessing in step 1 further includes:

[0069] Crawl text and image data based on big data, determine the crawled text theme, and filter the corresponding text image based on the text theme, establish a mapping relationship between text and image, and draw a scientific and technological text knowledge graph;

[0070] Acquire each source data platform corresponding to the text and image data, respectively form a multimodal text knowledge graph of each source data platform, and perform entity alignment on the text and image in the multimodal text knowledge graph;

[0071] In the multimodal text knowledge graph, when any text data is retrieved, image data related to the text is retrieved for the multimodal text knowledge graph. At the same time, text data from different source data platforms but representing the same or similar content are presented separately in the form of a text knowledge graph.

[0072] In this embodiment, through big data crawling and topic screening, the diversity and relevance of the data set are ensured, providing a high-quality data foundation for subsequent processing, establishing a mapping relationship between text and image, and drawing a scientific and technological text knowledge graph, which helps to intuitively display the relationship between data and improve the interpretability of data. The construction of entity alignment and multimodal text knowledge graphs enhances the ability to integrate information between different source data platforms, integrates knowledge from different fields and sources, promotes knowledge innovation and discovery, improves data utilization efficiency, and can retrieve relevant image data based on text data, which facilitates multimodal content analysis and retrieval, helps to discover differences and connections between data, and improves the accuracy and practicality of scientific and technological text image and text matching algorithms.

[0073] In this embodiment, the feature extraction in step 2 is specifically as follows:

[0074] Text feature extraction: Convert the segmented text into a vector representation of a fixed dimension, summarize the word vectors, and select the keyword corresponding to the word vector with the smallest distance to the standard word vector as the target keyword and the text feature of the text;

[0075] Image feature extraction: Capture the edge information of the image data, extract the texture and shape features in the image based on the edge information, extract high-level semantic features based on the texture and shape features, and perform feature encoding to obtain a fixed-dimensional feature vector as the visual feature of the image;

[0076] Feature fusion: Align the feature dimensions of the text feature vector and the image feature vector to ensure that the dimensions of the text feature vector and the image feature vector are consistent. This can be achieved through an embedding layer or a fully connected layer, and they are spliced ​​in dimension to form a new joint feature vector. The joint feature vector fuses the feature information of the text and the image, that is, the text feature vector and the image feature vector interact with each other to enhance the correlation between the two. An association model is constructed based on the joint feature vector for subsequent matching degree calculation.

[0077] In this embodiment, the feature extraction in the step 2 also includes: feature extraction based on text features and visual features, comparing the difference between the text and the image as a loss function, extracting the corresponding writing style features and emotional features in the text, and obtaining the corresponding changes in the visual features of the image through the changes in the writing style features and emotional features.

[0078] In this embodiment, by extracting keywords and high-level semantic features, the accuracy and relevance of text and image feature representation are improved, feature fusion enhances the correlation between text and image, and helps to calculate the matching degree more accurately. The extraction of writing style and emotional features provides a richer information dimension for the matching algorithm, which can capture the deep meaning of the text and the emotional expression of the image. Comparing the difference between text and image as a loss function helps the model to better focus on and reduce these differences during the learning process, thereby improving the matching quality and improving the performance of the scientific text image and text matching algorithm.

[0079] In this embodiment, the matching degree calculation in step 3 further includes:

[0080] Obtain the relationships and attributes of text and image data in the scientific and technological text knowledge graph;

[0081] In the knowledge graph of scientific texts, relationships define the connections between different entities, such as "belongs to", "used for", "related to", etc. In the context of scientific texts, relationships may include "technology applied to", "field belongs to", "concept related to", etc. Attributes describe the characteristics of entities, such as "publication year", "author", "technical category", etc. These attributes provide additional information for entities and help to more accurately describe and distinguish entities;

[0082] Identify entities in text data (such as technical terms, product names, company names, etc.) and link the identified entities to the corresponding nodes in the scientific and technological text knowledge graph;

[0083] Extracting attribute information related to entities from the scientific and technological text knowledge graph and mapping the attribute information into a feature vector;

[0084] Analyze the relationships in the knowledge graph of scientific and technological texts, extract relationship features based on the analysis results, and determine the strength of association between entities;

[0085] The joint feature vector is updated by combining the attribute information mapped to the feature vector and the association strength between entities, ensuring that the updated feature vector can simultaneously reflect the semantic information of the text, the visual features of the image, and the structured information in the knowledge graph of the scientific text.

[0086] In this embodiment, by linking entities in the text to nodes in the scientific and technological text knowledge graph, it is possible to better understand the semantic content of the text, more accurately understand the context and intent of the text, make more accurate judgments when dealing with polysemous words or ambiguous expressions, and provide an additional dimension for the matching of text and images. For example, if the image shows a certain technical equipment and the text mentions the application field of the technology, the scientific and technological text knowledge graph can reveal the connection between the two, help capture the deep connection between text and image, and map the attribute information and relationship features in the scientific and technological text knowledge graph into feature vectors, which helps to improve the matching accuracy.

[0087] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A scientific text image and text matching algorithm, characterized in that: The following steps are involved: Step 1: Data preprocessing: Classify and identify the collected scientific and technological text images and texts, and preprocess the texts and images based on the recognition results; Step 2: Feature extraction: Encode the preprocessed text data and convert it into vector representation, extract the converted text features, perform feature extraction on the image data, capture the visual features in the image, fuse the text features with the visual features, and build an association model between text and image; Step 3: Matching degree calculation: Calculate the similarity between the fused text features and visual features based on the association model; Step 4: Model output: Set a matching threshold based on the matching calculation result. When the matching degree exceeds the set matching threshold, the image and text are considered to match, and the matching result is output; Step 5: Evaluation and optimization: Evaluate the constructed association model, adjust the corresponding parameters according to the evaluation results, and continuously optimize the model through multiple iterative training until satisfactory performance indicators are achieved.

2. The scientific text image and text matching algorithm as claimed in claim 1, characterized in that: In the step 1, the text and the picture are preprocessed, specifically: Text preprocessing: including removing stop words, punctuation marks and non-semantic information of numbers, performing stem extraction, word form restoration and text segmentation operations; Image preprocessing: including adjusting the image size and format, normalizing the image, and also including image denoising and grayscale operations.

3. The scientific text image and text matching algorithm as claimed in claim 2, characterized in that: The data preprocessing in step 1 further includes: Crawl text and image data based on big data, determine the crawled text theme, and filter the corresponding text image based on the text theme, establish a mapping relationship between text and image, and draw a scientific and technological text knowledge graph; The source data platforms corresponding to the text and image data are obtained, and multimodal text knowledge graphs of the source data platforms are formed respectively, and the texts and images in the multimodal text knowledge graphs are entity aligned.

4. The scientific text image and text matching algorithm as claimed in claim 3, characterized in that: In the multimodal text knowledge graph, when any text data is retrieved, image data related to the text is retrieved for the multimodal text knowledge graph. At the same time, text data from different source data platforms but representing the same or similar content are presented separately in the form of a scientific and technological text knowledge graph.

5. The scientific text image and text matching algorithm as claimed in claim 4, characterized in that: The feature extraction in step 2 is specifically as follows: Text feature extraction: Convert the segmented text into a vector representation of a fixed dimension, summarize the word vectors, and select the keyword corresponding to the word vector with the smallest distance to the standard word vector as the target keyword and the text feature of the text; Image feature extraction: Capture the edge information of the image data, extract the texture and shape features in the image based on the edge information, extract high-level semantic features based on the texture and shape features, and perform feature encoding to obtain a fixed-dimensional feature vector as the visual feature of the image; feature Fusion: Align the feature dimensions of the text feature vector and the image feature vector, and concatenate them in the dimensions to form a new joint feature vector, and build an association model based on the joint feature vector.

6. The scientific text image and text matching algorithm as claimed in claim 5, characterized in that: The feature extraction in the step 2 also includes: feature extraction based on text features and visual features, comparing the difference between the text and the image as a loss function, extracting the corresponding writing style features and emotional features in the text, and obtaining the corresponding changes in the visual features of the image through the changes in the writing style features and emotional features.

7. The scientific text image and text matching algorithm as claimed in claim 6, characterized in that: The matching degree calculation in step 3 is specifically as follows: Assign corresponding weights based on different feature categories in the feature extraction results; The similarity between text and image in this type of feature is calculated based on the cosine similarity algorithm, and the similarity of each type of feature is weighted and summed according to its weight to obtain a comprehensive similarity score; Based on the attention mechanism, the attention scores of each word or sentence in the text and each region in the image are calculated to construct a matching matrix between the text and the image. The comprehensive similarity score and the matching matrix between the text and the image are input into the trained association model to calculate the final matching score between the text and the image.

8. The scientific text image and text matching algorithm as claimed in claim 7, characterized in that: The matching degree calculation in step 3 further includes: Obtain the relationships and attributes of text and image data in the scientific and technological text knowledge graph; Identify entities in text data and link the identified entities to corresponding nodes in the scientific text knowledge graph; Extracting attribute information related to entities from the scientific and technological text knowledge graph and mapping the attribute information into a feature vector; Analyze the relationships in the knowledge graph of scientific and technological texts, extract relationship features based on the analysis results, and determine the strength of association between entities; The joint feature vector is updated by combining the attribute information mapped into the feature vector and the association strength between entities.

9. A scientific text image and text matching algorithm as claimed in claim 8, characterized in that: The model output in step 4 is specifically: Determine the matching threshold: Draw a histogram based on the matching scores, determine the distribution of the matching scores, and extract the low-density area of ​​the score distribution of matching and non-matching samples as the matching threshold; Matching result judgment: Compare the matching score of each text and image pair with the set matching threshold; If it is higher than the set matching threshold, it is considered a match; Otherwise, it is judged as mismatch; Record each judgment result, including the status of whether it matches or not and the corresponding matching score, and output the matching result in a formatted manner.

10. A scientific text image and text matching algorithm as claimed in claim 9, characterized in that: The matching result includes successfully matched pictures, text pairs and matching scores.

Images