A method for matching pictures and texts of scientific texts

Abstract

The application discloses a kind of science and technology text picture and text matching method, it is related to artificial intelligence technical field.In order to solve with the rapid development of science and technology field, new text and picture type constantly emerge, cannot ensure that database can cover and accurately identify these new types, text and picture of different fields have different characteristics and regularity, recommendation system cannot adapt to the needs of different fields, cannot targeted recommendation service problem;Through big data crawling and theme screening, the diversity and relevance of the data set are ensured, so that the algorithm can better adapt to the needs of the rapid development of the science and technology field, cover and accurately identify new types of text and pictures, the establishment and entity alignment of science and technology text knowledge graph help to capture the deep connection between text and image in matching degree calculation, improve the matching precision, the introduction of weight distribution and attention mechanism improves the reliability and accuracy of matching result.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for matching scientific and technological text images and texts. Background Art

[0002] With the rapid development of big data and artificial intelligence technology, the matching of text and images has become an important research topic. Regarding text-image and text matching, the Chinese patent with publication number CN111339338B discloses a text-image matching recommendation method based on deep learning. The recommendation method adopted is based on a continuously enriched user operation record database, using image recognition and a keyword extraction algorithm based on feature vector distribution to construct a keyword feature vector, and combining the user operation record database, vector similarity algorithm and set similarity algorithm to achieve the best matching recommendation between images and text. In order to achieve the automatic recommendation of the homepage picture entered and the display position of the entered picture in the text after the inputter enters the text and picture data. In order to eliminate the work of the inputter in understanding the text, manually selecting pictures, and adjusting the picture position.

[0003] Although the above patent reduces the workload of data entry personnel through automation, the following problems still exist:

[0004] With the rapid development of science and technology, new types of text and images are constantly emerging. It is impossible to ensure that the database can cover and accurately identify these new types. Texts and images in different fields have different characteristics and rules. The recommendation system cannot adapt to the needs of different fields and cannot provide targeted recommendation services. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for matching scientific text images and texts. Through big data crawling and topic screening, combined with entity alignment of scientific text knowledge graphs, it can better identify and understand new types of texts and images. The feature extraction and fusion process takes into account the characteristics of different fields, so that the algorithm can provide personalized matching services according to the needs of different fields, so as to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for matching scientific text images and texts, comprising the following steps:

[0008] Step 1: Data preprocessing: Classify and identify the collected scientific and technological text images and texts, and preprocess the texts and images based on the recognition results;

[0009] Step 2: Feature extraction: Encode the preprocessed text data and convert it into a vector representation. Extract the converted text features. Perform feature extraction on the image data to capture the visual features in the image. Fuse the text features with the visual features to build a correlation model between the text and the image.

[0010] Step 3: Matching degree calculation: Calculate the similarity between the fused text features and visual features based on the association model;

[0011] Step 4: Model output: Set a matching threshold based on the matching calculation results. When the matching degree exceeds the set matching threshold, the image and text are considered to match, and the matching result is output;

[0012] Step 5: Evaluation and optimization: Evaluate the constructed association model, adjust the corresponding parameters according to the evaluation results, and continuously optimize the model through multiple iterative training until satisfactory performance indicators are achieved.

[0013] Furthermore, in step 1, the text and image are pre-processed, specifically:

[0014] Text preprocessing: including removing stop words, punctuation marks, and non-semantic information of numbers, and performing stemming, lemmatization, and text segmentation operations;

[0015] Image preprocessing: including adjusting the image size and format, normalizing the image, and also including image denoising and grayscale operations.

[0016] Furthermore, the data preprocessing in step 1 further includes:

[0017] Crawl text and image data based on big data, determine the crawled text topics, and filter corresponding text images based on the text topics, establish a mapping relationship between text and images, and draw a scientific and technological text knowledge graph;

[0018] Obtain each source data platform corresponding to the text and image data, form a multimodal text knowledge graph for each source data platform, and perform entity alignment on the text and image in the multimodal text knowledge graph.

[0019] Furthermore, in the multimodal text knowledge graph, when any text data is retrieved, image data related to the text is retrieved for the multimodal text knowledge graph. At the same time, text data from different source data platforms but representing the same or similar content are presented separately in the form of a text knowledge graph.

[0020] Furthermore, the feature extraction in step 2 is specifically as follows:

[0021] Text feature extraction: Convert the segmented text into a fixed-dimensional vector representation, aggregate the word vectors, and select the keyword corresponding to the word vector with the smallest distance to the standard word vector as the target keyword and the text feature of the text;

[0022] Image feature extraction: Capture the edge information of the image data, extract the texture and shape features in the image based on the edge information, extract high-level semantic features based on the texture and shape features, and perform feature encoding to obtain a fixed-dimensional feature vector as the visual feature of the image;

[0023] Feature fusion: Align the feature dimensions of the text feature vector and the image feature vector, and concatenate them in the dimensions to form a new joint feature vector. Build an association model based on the joint feature vector.

[0024] Furthermore, the feature extraction in step 2 also includes: feature extraction based on text features and visual features, comparing the difference between text and image as a loss function, extracting the corresponding writing style features and emotional features in the text, and obtaining the corresponding visual feature changes of the image through the changes in the writing style features and emotional features.

[0025] Furthermore, the matching degree calculation in step 3 is specifically as follows:

[0026] Assign corresponding weights based on different feature categories in the feature extraction results;

[0027] The cosine similarity algorithm is used to calculate the similarity between text and image in this type of feature. The similarity of each type of feature is weighted and summed according to its weight to obtain a comprehensive similarity score.

[0028] Based on the attention mechanism, the attention scores of each word or sentence in the text and each region in the image are calculated to construct a matching matrix between the text and the image.

[0029] The comprehensive similarity score and the matching matrix between the text and the image are input into the trained association model to calculate the final matching score between the text and the image.

[0030] Furthermore, the matching degree calculation in step 3 further includes:

[0031] Obtain the relationships and attributes of text and image data in the scientific text knowledge graph;

[0032] Identify entities in text data and link the identified entities to corresponding nodes in the scientific text knowledge graph;

[0033] Extracting attribute information related to entities from the scientific text knowledge graph and mapping the attribute information into feature vectors;

[0034] Analyze the relationships in the knowledge graph of scientific and technological texts, extract relationship features based on the analysis results, and determine the strength of association between entities;

[0035] The joint feature vector is updated by combining the attribute information mapped into the feature vector and the association strength between entities.

[0036] Furthermore, the model output in step 4 is specifically as follows:

[0037] Determine the matching threshold: Draw a histogram based on the matching scores to determine the distribution of the matching scores, and extract the low-density area of ​​the score distribution of matching and non-matching samples as the matching threshold;

[0038] Matching result judgment: Compare the matching score of each text and image pair with the set matching threshold;

[0039] If it is higher than the set matching threshold, it is considered a match;

[0040] Otherwise, it is judged as mismatch;

[0041] Record each judgment result, including the status of whether it matches or not and the corresponding matching score, and output the matching result in a formatted manner.

[0042] Furthermore, the matching result includes successfully matched pictures and text pairs and matching scores.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] Through big data crawling and topic screening, the diversity and relevance of the data set are ensured, enabling the algorithm to better adapt to the rapid development needs of the science and technology field, covering and accurately identifying new types of text and images. The establishment of the knowledge graph of scientific and technological texts and entity alignment help capture the deep connection between text and images in the matching calculation, thereby improving the matching accuracy. The feature extraction and fusion process takes into account the characteristics of text and images in different fields, enabling the algorithm to adapt to the needs of different fields and provide targeted recommendation services. The introduction of weight distribution and attention mechanism allows key features to receive more attention in the matching process, thereby improving the reliability and accuracy of the matching results. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a step diagram of the scientific text image and text matching algorithm of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] To address the technical issues that with the rapid development of science and technology, new text and image types are constantly emerging, and the database cannot be guaranteed to cover and accurately identify these new types. Texts and images in different fields have different characteristics and patterns, and the recommendation system cannot adapt to the needs of different fields and cannot provide targeted recommendation services. Figure 1 , this embodiment provides the following technical solutions,

[0048] A method for matching scientific text images and texts, comprising the following steps:

[0049] Step 1: Data preprocessing: Classify and identify the collected scientific and technological text images and texts, and preprocess the texts and images based on the recognition results. Specifically:

[0050] Text preprocessing: This includes removing non-semantic information (such as stop words, punctuation marks, and numbers), performing stemming, lemmatization, and text segmentation operations, which helps extract keywords and core content from the text and improve the accuracy and efficiency of subsequent feature extraction;

[0051] Image preprocessing: This includes adjusting the image size and format, normalizing the image, and performing image denoising and grayscale operations to reduce redundant information in the image data and highlight key features to improve feature extraction.

[0052] Step 2: Feature Extraction: Encode the preprocessed text data and convert it into a vector representation. Extract the converted text features, such as grammatical sentence features, semantic features, and contextual features. Perform feature extraction on the image data to capture visual features in the image, such as edges, textures, and shapes, and encode these features into vector form. Fuse the text features with the visual features to construct a correlation model between text and image.

[0053] Step 3: Matching degree calculation: Calculate the similarity between the fused text features and visual features based on the association model, specifically:

[0054] Assign corresponding weights based on different feature categories in the feature extraction results (such as text features, text sentiment features, image visual features, etc.), where style and sentiment features are assigned higher weights;

[0055] The cosine similarity algorithm is used to calculate the similarity between text and image in this type of feature. The similarity of each type of feature is weighted and summed according to its weight to obtain a comprehensive similarity score.

[0056] Based on the attention mechanism, the attention score of each word or sentence in the text and each region in the image is calculated, and a matching matrix between the text and the image is constructed. Each element in the matrix represents the matching strength between an element (word or sentence) in the text and a region in the image.

[0057] Inputting the comprehensive similarity score and the matching matrix between the text and the image into the trained association model to calculate the final matching score between the text and the image, which reflects the similarity between the text and the image;

[0058] In this embodiment, style and sentiment features often reflect the deeper meaning of text and the emotional expression of images. Giving them higher weights ensures that this key information is fully considered during the matching process. By assigning weights, the algorithm can pay more attention to features that have a greater impact on the matching results, thereby improving the relevance and accuracy of the matching. This allows the algorithm to adjust the similarity calculation method according to different application scenarios and needs, increasing the algorithm's flexibility and providing more personalized matching services. By comprehensively considering multiple feature dimensions and fine-grained matching relationships, the algorithm can more accurately identify and match relevant text and images.

[0059] Step 4: Model output: Set a matching threshold based on the matching calculation results. When the matching degree exceeds the set matching threshold, the image and text are considered to match, and the matching result is output, specifically:

[0060] Determine the matching threshold: Draw a histogram based on the matching scores to determine the distribution of the matching scores, and extract the low-density area of ​​the score distribution of matching and non-matching samples as the matching threshold;

[0061] Matching result judgment: Compare the matching score of each text and image pair with the set matching threshold;

[0062] If it is higher than the set matching threshold, it is considered a match;

[0063] Otherwise, it is judged as mismatch;

[0064] Record each judgment result, including the status of whether it matches or not and the corresponding matching score, and output the matching results in a formatted manner;

[0065] The matching results include successfully matched images, text pairs, and matching scores;

[0066] Step 5: Evaluation and Optimization: Evaluate the constructed association model and adjust the corresponding parameters in steps such as data preprocessing, feature extraction, and matching calculation based on the evaluation results. Through multiple iterative training, continuously optimize the model until satisfactory performance indicators are achieved. Performance indicators include accuracy, recall rate, F1 score, etc. These indicators are used to evaluate the performance of the association model.

[0067] In this embodiment, by encoding and converting to vector representation, the intrinsic information of the text is fully captured, and the captured visual features provide a rich description of the image content. The diversity of extraction ensures that the deep information of the text and image is fully mined. Feature fusion enhances the correlation between text and image, providing a richer information dimension for the matching algorithm. The weight distribution and attention mechanism improve the flexibility of the matching algorithm and its sensitivity to key features. The combination of comprehensive similarity score and matching matrix makes the matching results more comprehensive and reliable. The continuous evaluation and optimization process improves the accuracy and robustness of the algorithm to adapt to different data sets and application scenarios, thereby improving the applicability of the algorithm.

[0068] In this embodiment, the data preprocessing in step 1 further includes:

[0069] Crawl text and image data based on big data, determine the crawled text topics, and filter corresponding text images based on the text topics, establish a mapping relationship between text and images, and draw a scientific and technological text knowledge graph;

[0070] Obtaining source data platforms corresponding to text and image data, forming multimodal text knowledge graphs for each source data platform, and performing entity alignment on the text and images in the multimodal text knowledge graphs;

[0071] In the multimodal text knowledge graph, when any text data is retrieved, image data related to the text is retrieved from the multimodal text knowledge graph. At the same time, text data from different source data platforms but representing the same or similar content are presented separately in the form of a text knowledge graph.

[0072] In this embodiment, through big data crawling and topic screening, the diversity and relevance of the data set are ensured, providing a high-quality data foundation for subsequent processing, establishing a mapping relationship between text and images, and drawing a scientific and technological text knowledge graph, which helps to intuitively display the relationship between data and improve the interpretability of data. The construction of entity alignment and multimodal text knowledge graphs enhances the ability to integrate information between different source data platforms, integrates knowledge from different fields and sources, promotes knowledge innovation and discovery, improves data utilization efficiency, and can retrieve relevant image data based on text data, which facilitates multimodal content analysis and retrieval, helps to discover differences and connections between data, and improves the accuracy and practicality of scientific and technological text image and text matching algorithms.

[0073] In this embodiment, the feature extraction in step 2 is specifically as follows:

[0074] Text feature extraction: Convert the segmented text into a fixed-dimensional vector representation, aggregate the word vectors, and select the keyword corresponding to the word vector with the smallest distance to the standard word vector as the target keyword and the text feature of the text;

[0075] Image feature extraction: Capture the edge information of the image data, extract the texture and shape features in the image based on the edge information, extract high-level semantic features based on the texture and shape features, and perform feature encoding to obtain a fixed-dimensional feature vector as the visual feature of the image;

[0076] Feature fusion: Align the feature dimensions of the text feature vector and the image feature vector to ensure that the dimensions of the text feature vector and the image feature vector are consistent. This can be achieved through an embedding layer or a fully connected layer, and spliced ​​in dimension to form a new joint feature vector. The joint feature vector fuses the feature information of the text and image, that is, the text feature vector and the image feature vector interact with each other, so that the correlation between the two is enhanced. An association model is constructed based on the joint feature vector for subsequent matching calculations.

[0077] In this embodiment, the feature extraction in step 2 also includes: feature extraction based on text features and visual features, comparing the difference between text and image as a loss function, extracting the corresponding writing style features and emotional features in the text, and obtaining the corresponding visual feature changes of the image through the changes in the writing style features and emotional features.

[0078] In this embodiment, by extracting keywords and high-level semantic features, the accuracy and relevance of text and image feature representation are improved. Feature fusion enhances the correlation between text and image, which helps to calculate the matching degree more accurately. The extraction of writing style and emotional features provides a richer information dimension for the matching algorithm, which can capture the deep meaning of the text and the emotional expression of the image. Comparing the difference between text and image as a loss function helps the model to better focus on and reduce these differences during the learning process, thereby improving the matching quality and improving the performance of the scientific text image and text matching algorithm.

[0079] In this embodiment, the matching degree calculation in step 3 further includes:

[0080] Obtain the relationships and attributes of text and image data in the scientific text knowledge graph;

[0081] In the knowledge graph of scientific texts, relationships define the connections between different entities, such as "belongs to," "used for," and "related to." In the context of scientific texts, relationships may include "technology applied to," "field belonging to," and "concept related to." Attributes describe the characteristics of entities, such as "publication year," "author," and "technology category." These attributes provide additional information for entities, helping to more accurately describe and distinguish them.

[0082] Identify entities in text data (such as technical terms, product names, company names, etc.) and link the identified entities to the corresponding nodes in the scientific text knowledge graph;

[0083] Extracting attribute information related to entities from the scientific text knowledge graph and mapping the attribute information into feature vectors;

[0084] Analyze the relationships in the knowledge graph of scientific and technological texts, extract relationship features based on the analysis results, and determine the strength of association between entities;

[0085] The joint feature vector is updated by combining the attribute information mapped to the feature vector and the association strength between entities to ensure that the updated feature vector can simultaneously reflect the semantic information of the text, the visual features of the image, and the structured information in the knowledge graph of scientific text.

[0086] In this embodiment, by linking entities in the text to nodes in the scientific and technological text knowledge graph, it is possible to better understand the semantic content of the text, more accurately understand the context and intention of the text, make more accurate judgments when dealing with polysemous words or ambiguous expressions, and provide an additional dimension for the matching of text and images. For example, if the image shows a certain technical equipment and the text mentions the application field of the technology, the scientific and technological text knowledge graph can reveal the connection between the two, which helps to capture the deep connection between the text and the image. By mapping the attribute information and relationship features in the scientific and technological text knowledge graph into the feature vector, it helps to improve the matching accuracy.

[0087] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for matching scientific text images and texts, characterized in that: The following steps are involved: Step 1: Data preprocessing: Classify and identify the collected scientific and technological text images and texts, and preprocess the texts and images based on the recognition results; Step 2: Feature extraction: Encode the preprocessed text data and convert it into a vector representation. Extract the converted text features. Perform feature extraction on the image data to capture the visual features in the image. Fuse the text features with the visual features to build a correlation model between the text and the image. Step 3: Matching degree calculation: Calculate the similarity between the fused text features and visual features based on the association model, which also includes: Obtain the relationships and attributes of text and image data in the scientific text knowledge graph; Identify entities in text data and link the identified entities to corresponding nodes in the scientific text knowledge graph; Extracting attribute information related to entities from the scientific text knowledge graph and mapping the attribute information into feature vectors; Analyze the relationships in the knowledge graph of scientific and technological texts, extract relationship features based on the analysis results, and determine the strength of association between entities; Update the joint feature vector by combining the attribute information mapped to the feature vector and the association strength between entities; Step 4: Model output: Set a matching threshold based on the matching calculation results. When the matching degree exceeds the set matching threshold, the image and text are considered to match, and the matching result is output; Step 5: Evaluation and optimization: Evaluate the constructed association model, adjust the corresponding parameters according to the evaluation results, and continuously optimize the model through multiple iterative training until satisfactory performance indicators are achieved.

2. The method for matching scientific text images and texts according to claim 1, characterized in that: In step 1, the text and image are preprocessed, specifically: Text preprocessing: including removing stop words, punctuation marks, and non-semantic information of numbers, and performing stemming, lemmatization, and text segmentation operations; Image preprocessing: including adjusting the image size and format, normalizing the image, and also including image denoising and grayscale operations.

3. The method for matching scientific text images and texts according to claim 2, wherein: The data preprocessing in step 1 further includes: Crawl text and image data based on big data, determine the crawled text topics, and filter corresponding text images based on the text topics, establish a mapping relationship between text and images, and draw a scientific and technological text knowledge graph; Obtain each source data platform corresponding to the text and image data, form a multimodal text knowledge graph for each source data platform, and perform entity alignment on the text and image in the multimodal text knowledge graph.

4. A method for matching scientific text images and texts according to claim 3, characterized in that: In the multimodal text knowledge graph, when any text data is retrieved, image data related to the text is retrieved from the multimodal text knowledge graph. At the same time, text data from different source data platforms but representing the same or similar content are presented separately in the form of a scientific and technological text knowledge graph.

5. A method for matching scientific and technological text images and texts according to claim 4, characterized in that: The feature extraction in step 2 is specifically as follows: Text feature extraction: Convert the segmented text into a fixed-dimensional vector representation, aggregate the word vectors, and select the keyword corresponding to the word vector with the smallest distance to the standard word vector as the target keyword and the text feature of the text; Image feature extraction: Capture the edge information of the image data, extract the texture and shape features in the image based on the edge information, extract high-level semantic features based on the texture and shape features, and perform feature encoding to obtain a fixed-dimensional feature vector as the visual feature of the image; feature Fusion: Align the feature dimensions of the text feature vector and the image feature vector, and concatenate them in the dimensions to form a new joint feature vector. Build an association model based on the joint feature vector.

6. A method for matching scientific text images and texts according to claim 5, characterized in that: The feature extraction in step 2 also includes: feature extraction based on text features and visual features, comparing the difference between the text and the image as a loss function, extracting the corresponding writing style features and emotional features in the text, and obtaining the corresponding visual feature changes of the image through the changes in the writing style features and emotional features.

7. A method for matching scientific text images and texts according to claim 6, characterized in that: The matching degree calculation in step 3 is specifically as follows: Assign corresponding weights based on different feature categories in the feature extraction results; The cosine similarity algorithm is used to calculate the similarity between text and image in this type of feature. The similarity of each type of feature is weighted and summed according to its weight to obtain a comprehensive similarity score. Based on the attention mechanism, the attention scores of each word or sentence in the text and each region in the image are calculated to construct a matching matrix between the text and the image. The comprehensive similarity score and the matching matrix between the text and the image are input into the trained association model to calculate the final matching score between the text and the image.

8. The method for matching scientific text images and texts according to claim 1, wherein: The model output in step 4 is specifically: Determine the matching threshold: Draw a histogram based on the matching scores to determine the distribution of the matching scores, and extract the low-density area of ​​the score distribution of matching and non-matching samples as the matching threshold; Matching result judgment: Compare the matching score of each text and image pair with the set matching threshold; If it is higher than the set matching threshold, it is considered a match; Otherwise, it is judged as mismatch; Record each judgment result, including the status of whether it matches or not and the corresponding matching score, and output the matching result in a formatted manner.

9. A method for matching scientific text images and texts according to claim 8, characterized in that: The matching result includes successfully matched pictures, text pairs and matching scores.

Images