A text-image matching method and system based on semantic segmentation and feature association

Abstract

This invention proposes a text-image matching method and system based on semantic segmentation and feature association, relating to the field of semantic processing technology. The method includes: performing semantic analysis on target text and target images in response document samples to obtain response text features for different paragraphs and response image features for each target image; mapping the response text features and response image features to the same dimensional space to obtain text-image combinations, using these combinations as input to an initial association model, and training the initial association model using the cosine similarity between the response text features and response image features as output; optimizing the initial association model using a loss function and a stochastic gradient descent optimization algorithm to obtain a target association model; preprocessing the question text to generate corresponding question text features, inputting these features into the target association model to obtain the corresponding text-image combinations, and displaying the text paragraphs and target images corresponding to the text-image combinations.

Description

Technical Field

[0001] This invention relates to the field of semantic processing technology, and in particular to a method and system for image-text matching based on semantic segmentation and feature association. Background Technology

[0002] Traditional large-scale models, when processing user knowledge base documents, focus primarily on extracting and summarizing textual information, paying little attention to non-textual elements such as images and charts. Images are typically treated as mere attachments, without exploring their intrinsic connections to the text. In terms of information retrieval and matching, they mainly rely on textual keywords for searching, failing to accurately match text with corresponding image content. When answering user questions, they struggle to provide comprehensive, illustrated solutions. For complex document structures, such as corporate policy texts containing multiple chapters and charts, current technologies lack effective semantic segmentation capabilities, limiting the accuracy of knowledge understanding and application. Summary of the Invention

[0003] In view of this, the present invention proposes a text-image matching method and system based on semantic segmentation and feature association.

[0004] The technical solution of this invention is implemented as follows: The first aspect of this invention provides a text-image matching method based on semantic segmentation and feature association, comprising:

[0005] Semantic analysis is performed on the target text and target image in the response document sample to obtain the response text features of different paragraphs and the response image features of each target image; the response text features include text vocabulary information, text grammar information and text semantic information; the response image features include image color information, image texture information, image shape information and image semantic information.

[0006] The response text features and response image features are mapped to the same dimensional space to obtain a text-image combination. The text-image combination is used as the input of the initial association model. The cosine similarity between the response text features and the response image features is used as the output to train the initial association model. The initial association model is then optimized by combining a loss function and a stochastic gradient descent optimization algorithm to obtain the target association model.

[0007] The question text is preprocessed to generate corresponding question text features. The question text features are then input into the target association model to obtain the corresponding text image combination. The text paragraph and target image corresponding to the text image combination are then displayed. The question text features are of the same type as the response text features.

[0008] Based on the above technical solutions, preferably, the step of performing semantic analysis on the target text and target image in the response document sample to obtain the response text features of different paragraphs and the response image features of each target image includes:

[0009] The target text in the response document sample is split to obtain text vocabulary information; the text vocabulary information includes words or phrases.

[0010] The vocabulary information is tagged with part-of-speech tags to determine the text grammar information of each vocabulary item in the target text;

[0011] A syntax tree is constructed based on the text syntax information in the response document sample to determine the text semantic information between the text paragraphs.

[0012] Based on the above technical solutions, preferably, the step of splitting the target text in the response document sample to obtain text vocabulary information includes:

[0013] The initial weight of each word or phrase in the response document sample is obtained based on the TF-IDF algorithm; the initial weight is positively correlated with the word frequency.

[0014] The TextRank algorithm is used to obtain the correlation between different words or phrases, and the initial weights are adjusted based on the correlation to obtain optimized weights;

[0015] The words or phrases are sorted based on the optimized weights to determine the keywords in the text vocabulary information.

[0016] Based on the above technical solutions, preferably, the step of performing semantic analysis on the target text and target image in the response document sample to obtain the response text features of different paragraphs and the response image features of each target image includes:

[0017] Obtain the color histogram of the target image, and statistically analyze the distribution of different colors in the target image to obtain image color information;

[0018] The gray-level co-occurrence matrix method is used to calculate the texture features of the target image in multiple different directions, and the local binary mode algorithm is introduced to supplement the texture features with local texture details to obtain the image texture information;

[0019] The contours of objects in the target image are extracted using an edge detection algorithm to obtain image shape information;

[0020] The target image is processed using a pre-trained convolutional neural network, and the semantic information of the image is obtained by adaptively weighting and fusing multi-scale features.

[0021] Based on the above technical solutions, preferably, the initial association model includes a text tower and an image tower. The step of using the combination of text and images as input to the initial association model, and using the cosine similarity between the response text features and the response image features as output to train the initial association model includes:

[0022] The response text features are input into the text tower, and the response image features are input into the image tower. The text tower and the image tower are trained by combining the cosine similarity output by the output layer.

[0023] Based on the above technical solutions, preferably, the optimization of the initial association model by combining the loss function and the stochastic gradient descent optimization algorithm to obtain the target association model includes:

[0024] The difference between the predicted association score and the true association score of the initial association model is determined by using cross-entropy loss, triplet loss and contrastive loss. The parameters of the initial association model are continuously adjusted by the stochastic gradient descent optimization algorithm until the difference reaches a preset threshold, thus obtaining the target association model.

[0025] Based on the above technical solutions, preferably, the step of preprocessing the problem text to generate corresponding problem text features, and inputting the problem text features into the target association model to obtain the corresponding text-image combination, includes:

[0026] The question text is preprocessed to generate corresponding question vocabulary information, question grammar information, and question semantic information;

[0027] The question vocabulary information, the question grammar information, and the question semantic information are input into the target association model to obtain the response text features and the associated response image features.

[0028] Furthermore, a second aspect of the present invention provides a text-image matching system based on semantic segmentation and feature association, comprising: a feature acquisition module, a model optimization module, and a result display module; wherein,

[0029] The feature acquisition module is configured to perform semantic analysis on the target text and target image in the response document sample, and acquire the response text features of different paragraphs and the response image features of each target image; the response text features include text vocabulary information, text grammar information and text semantic information; the response image features include image color information, image texture information, image shape information and image semantic information.

[0030] The model optimization module is configured to map the response text features and the response image features to the same dimensional space to obtain a text-image combination, and use the text-image combination as the input of the initial association model, use the cosine similarity of the response text features and the response image features as the output to train the initial association model, and combine the loss function and stochastic gradient descent optimization algorithm to optimize the initial association model to obtain the target association model;

[0031] The result display module is configured to preprocess the question text to generate corresponding question text features, input the question text features into the target association model to obtain the corresponding text image combination, and display the text paragraph and target image corresponding to the text image combination; the question text features are of the same type as the response text features.

[0032] More preferably, a third aspect of the present invention provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the image-text matching method based on semantic segmentation and feature association described in the first aspect.

[0033] More preferably, a fourth aspect of the present invention provides a computer storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the image-text matching method based on semantic segmentation and feature association described in the first aspect.

[0034] The image-text matching method and system based on semantic segmentation and feature association of the present invention have the following advantages over the prior art:

[0035] 1. By performing semantic analysis on the target text and target images in the response document samples, the response text features of different paragraphs and the response image features of each target image are obtained. The response text features and response image features are then associated through an association model. Thus, when the question text is input, the corresponding text-image combination is determined based on the question text features, and the corresponding text paragraph and target image are displayed. This improves the matching accuracy and efficiency between text paragraphs and corresponding image content, and ensures the reliability of the response.

[0036] 2. By performing semantic analysis on the target text and target images in the response document samples, we obtain the text vocabulary information, text grammar information, and text semantic information of different paragraphs, as well as the image color information, image texture information, image shape information, and image semantic information of each target image. Based on this, we realize the association between text paragraphs and target images, improve the matching accuracy of text and images, and the comprehensiveness of the response questions.

[0037] 3. By dynamically weighting different words or phrases based on the TF-IDF and TextRank algorithms, interference from high-frequency but inefficient words is reduced, and the recognition rate of low-frequency keywords is improved. At the same time, by combining the logical relationships between words, semantic fragmentation is avoided, and the accurate extraction of keywords is ultimately achieved, ensuring the accuracy of subsequent information retrieval and response. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating a text-image matching method based on semantic segmentation and feature association, provided in an embodiment of the present invention.

[0040] Figure 2 A schematic diagram of the structure of a text-image matching system based on semantic segmentation and feature association provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] In some embodiments, such as Figure 1 As shown, Figure 1 This is a flowchart illustrating a text-image matching method based on semantic segmentation and feature association, provided by an embodiment of the present invention. The text-image matching method based on semantic segmentation and feature association provided by the present invention includes:

[0044] S110, perform semantic analysis on the target text and target image in the response document sample, and obtain the response text features of different paragraphs and the response image features of each target image; the response text features include text vocabulary information, text grammar information and text semantic information; the response image features include image color information, image texture information, image shape information and image semantic information.

[0045] In this embodiment, natural language processing (NLP) techniques are used to perform semantic analysis on the target text in the response document samples. Combining document structure features and domain knowledge, the target text can be divided into different paragraphs according to semantic logic, and key response text features can be extracted. Here, text lexical information can include words, phrases, and some keywords; text grammatical information can include sentence structure and modifiers, such as subject-verb-object structures, attributives, and adverbs; and text semantic information can include text meaning and contextual relationships. For target images in the response document samples, image recognition techniques can be used to analyze the target images and extract response image features. Commonly used image recognition models include YOLO and Faster R-CNN.

[0046] In some embodiments, semantic analysis is performed on the target text and target image in the response document sample to obtain the response text features of different paragraphs and the response image features of each target image, including:

[0047] The target text in the response document sample is split to obtain text vocabulary information; the text vocabulary information includes words or phrases.

[0048] Vocabulary information is tagged with parts of speech to determine the textual grammatical information of each text vocabulary in the target text;

[0049] A syntax tree is constructed based on the text syntax information in the response document sample to determine the text semantic information between text paragraphs.

[0050] In this embodiment, the BERT-BiLSTM-CRF model can be used for named entity recognition, trained on the MSRA-NER dataset. Meanwhile, a domain entity extension module is added, and entity recognition is optimized through industry dictionaries, such as "insulator" and "load power consumption" in the power grid domain, and "convolutional layer" and "confidence interval" in the academic domain. Stanford Parser is used for dependency syntactic analysis to extract the subject-predicate-object structure, construct a syntactic tree to determine the relationship between sentence components. Meanwhile, semantic role labeling (SRL) is introduced to identify roles such as "agent" and "patient" to clarify the association between actions and objects in the text. Chinese word segmentation uses the jieba segmentation tool, combined with the HMM model for out-of-vocabulary word recognition, and the dictionary is updated monthly to cover the industry professional vocabulary library; English word segmentation uses NLTK, combined with WordNet synonym forest for word form reduction, and a domain synonym table is supplemented, such as "deep learning" and "deep neural network". For paragraph division, the target text in the response document sample is divided into paragraphs according to the obtained text vocabulary information, text grammar information, and text semantic information, resulting in multiple text paragraphs of the response document sample. When dividing based on rules, when encountering a period, question mark, or exclamation mark and the first letter of the subsequent text is capitalized, it is divided into a new paragraph. When the paragraph length exceeds 200 words, it is forced to be segmented according to formats such as chapter numbers and sub-chapter numbers. Meanwhile, a document structure adaptation rule is added. For text blocks containing reference marks such as "Figure X" and "Table X", they are preferentially merged with the corresponding chart-related paragraphs. When dividing based on semantics, the cosine similarity is used to calculate the semantic relevance of adjacent text blocks. The window size can be set to 50 words. When the similarity is lower than 0.3, it is divided into different paragraphs. A dynamic threshold adjustment mechanism is introduced to optimize the division accuracy according to the document domain type. For example, the threshold for power grid documents is set to 0.25, and the threshold for academic documents is set to 0.35. For keyword extraction in the response document sample, keywords in the response document sample can be extracted through a preset algorithm, removing stop words that are not crucial to the text theme, such as "of", "is", "in", etc. Meanwhile, domain stop word filtering is added, such as general terms like "technical specifications" and "execution standards" in power grid documents.

[0051] In some embodiments, the target text in the response document sample is split to obtain text vocabulary information, including:

[0052] Based on the TF-IDF algorithm, the initial weight of each word or phrase in the response document sample is obtained; the initial weight is positively correlated with the word frequency;

[0053] The TextRank algorithm is used to obtain the relevance between different words or phrases, and the initial weight is adjusted based on the relevance to obtain the optimized weight;

[0054] Based on optimized weights, words or phrases are sorted to determine keywords in the text's vocabulary information.

[0055] In this embodiment, the TF-IDF algorithm is used to calculate the weight of each word or phrase in the user document. The higher the weight, the more important the word or phrase. Simultaneously, the TextRank algorithm is integrated, taking into account core components in the syntactic tree, such as nouns in a subject-verb-object structure, to adjust the weights. Words are sorted by weight, and the words or phrases with the highest weights are selected as keywords. The semantic type of the keywords is also marked, such as "device name," "operation steps," or "parameter value."

[0056] In some embodiments, semantic analysis is performed on the target text and target image in the response document sample to obtain the response text features of different paragraphs and the response image features of each target image, including:

[0057] Obtain the color histogram of the target image, and statistically analyze the distribution of different colors in the target image to obtain image color information;

[0058] The gray-level co-occurrence matrix method is used to calculate the texture features of the target image in multiple different directions, and the local binary mode algorithm is introduced to supplement the texture features with local texture details to obtain the image texture information;

[0059] Edge detection algorithms are used to extract the contours of objects in a target image to obtain image shape information;

[0060] The target image is processed using a pre-trained convolutional neural network, and the semantic information of the image is obtained by adaptively weighting and fusing multi-scale features.

[0061] In this embodiment, for image color information, the distribution of different colors in the target image can be statistically analyzed by calculating the color histogram of the target image. The RGB color space is converted to HSV space, with the H channel divided into 32 bins, the S channel into 16 bins, and the V channel into 16 bins. A normalized color histogram is calculated, and color saliency analysis is added. The Itti-Koch model is used to identify color-prominent regions in the target image, such as red warning signs, and assigns a 1.5 times weight to the color features of these regions. For image texture information extraction, the Gray-Level Co-occurrence Matrix (GLCM) method is used to calculate texture features in four directions: 0°, 45°, 90°, and 135°. The window size is set to 16×16 pixels, and four feature values—contrast, correlation, energy, and entropy—are extracted. A Local Binary Mode (LMM) algorithm is also introduced to supplement local texture details in the target image. For image shape information extraction, an edge detection algorithm can be used to extract the contours of objects in the target image to obtain shape features. For example, using the Canny edge detection operator with a high-to-low threshold ratio of 2:1 and a Gaussian kernel size of 5×5, shape complexity and roundness are calculated after extracting the object contour. Shape complexity can be expressed as perimeter² / area, and roundness as 4π×area / perimeter². A new object detection module, such as YOLOv8, is added to identify core objects in the target image, such as equipment or chart axes, calculating shape features only for the core object region to exclude background interference. For image semantic information extraction, a pre-trained convolutional neural network can be used to process the image, obtaining feature vectors from the intermediate layers of the network. For example, ResNet-50 can be used as the backbone network, pre-trained on the ImageNet dataset, extracting the feature vector of the 4th convolutional block with a dimension of 2048; features from Conv2, Conv3, and Conv4 layers are also extracted, and multi-scale features are fused through an adaptive weighted attention mechanism. A new image semantic supplementation module is also added, using OCR to recognize text in the image, such as the equipment model "CK6140," and im2txt to generate image description text, such as "CNC machine tool spindle structure diagram," thus obtaining the image semantic information.

[0062] S120: Map the response text features and response image features to the same dimensional space to obtain a text-image combination. Use the text-image combination as the input of the initial association model and use the cosine similarity between the response text features and response image features as the output to train the initial association model. Combine the loss function and stochastic gradient descent optimization algorithm to optimize the initial association model and obtain the target association model.

[0063] In this embodiment, the fully connected layers and non-linear activation functions, such as ReLU, of the initial association model can be used to map the response text features and response image features from the original dimension to the target dimension, ensuring that the mapped response text features and response image features have the same dimension. The mapped text and image vectors are concatenated along the feature dimension to form a 2D combined vector, and the response text features and response image features are fused through learnable weight parameters.

[0064] In some embodiments, the initial association model includes a text tower and an image tower. The text-image combination is used as input to the initial association model, and the cosine similarity between the response text features and the response image features is used as output to train the initial association model, including:

[0065] The text features of the response are input into the text tower, and the image features of the response are input into the image tower. The text tower and the image tower are trained by combining the cosine similarity output by the output layer.

[0066] In this embodiment, the text tower can consist of an LSTM and an attention layer. The weights of the attention layer are combined with the semantic type of the keywords, such as "device name" with a weight of 1.2 and "parameter value" with a weight of 0.8. The image tower includes ResNet-50 and global average pooling, and a new domain adaptation layer is added to dynamically adjust the network parameters based on domain labels, such as "power grid" and "academic". The output layer calculates the association score using cosine similarity and introduces a dynamic weight factor for optimization. The dynamic weight factor is jointly determined by the document domain type and the user's historical feedback. For example, the image feature weight for the power grid domain is 1.3, the text feature weight for the academic domain is 1.2, and the weight of the text-image combination manually adjusted by the user is increased by 0.2.

[0067] In some embodiments, the initial association model is optimized by combining a loss function and a stochastic gradient descent optimization algorithm to obtain a target association model, including:

[0068] The difference between the predicted association score and the true association score of the initial association model is determined by using cross-entropy loss, triplet loss and contrastive loss. The parameters of the initial association model are continuously adjusted by the stochastic gradient descent optimization algorithm until the difference reaches a preset threshold, thus obtaining the target association model.

[0069] Cross-entropy loss is common in binary classification, so it will not be discussed further here.

[0070] The triplet loss can be expressed as:

[0071] ;

[0072] The contrast loss can be expressed as:

[0073] ;

[0074] in, Represents cosine similarity. As a positive sample, For negative samples, when y=1, it means that the two samples belong to the same category or are similar; when y=0, it means that the two samples belong to different categories or are not similar.

[0075] The newly added domain loss term can be represented as:

[0076] ;

[0077] in, Similarity between positive sample pairs in the same domain This loss term, representing the similarity of positive sample pairs across domains, can enhance the model's domain adaptability and improve the model's training convergence speed.

[0078] In this embodiment, response document samples are used to train the initial association model. The model parameters are continuously adjusted through an optimization algorithm to minimize the loss function. A stochastic gradient descent optimization algorithm is adopted. For example, the initial learning rate can be set to 0.001, and the learning rate decays by a factor of 0.1 after every 10 training rounds. At the same time, a learning rate warm-up mechanism is introduced, in which the learning rate is linearly increased from 0.0001 to 0.001 in the first 3 rounds to avoid initial parameter oscillation. During the training process, a domain validation set is added. When the association accuracy of a certain domain validation set does not improve for 3 consecutive rounds, domain fine-tuning is triggered, and additional samples from the domain are used to update the local parameters until the loss function converges.

[0079] S130: Preprocess the question text to generate corresponding question text features, input the question text features into the target association model to obtain the corresponding text image combination, and display the text paragraph and target image corresponding to the text image combination; the question text features and the response text features are of the same type.

[0080] Preprocessing of problem text can include text cleaning and stop word removal. For example, for text containing HTML tags, regular expressions can be used to match and remove the tag parts. A stop word list can be constructed, and the segmented text can be matched against the stop word list to remove words appearing in the list.

[0081] In some embodiments, the question text is preprocessed to generate corresponding question text features, and the question text features are input into a target association model to obtain the corresponding text-image combination, including:

[0082] The question text is preprocessed to generate corresponding question vocabulary information, question grammar information, and question semantic information;

[0083] The vocabulary, grammar, and semantic information of the question are input into the target association model to obtain the text features of the response and the associated image features of the response.

[0084] In this embodiment, preprocessing also includes word segmentation and text vectorization. For example, for Chinese text, the Jieba word segmentation tool is used; for English text, tools such as NLTK or SpaCy can be used for word segmentation. The BERT model is used to encode the segmented question text, obtain the context-aware vector representation of each word, and aggregate these vectors to obtain the feature vector of the entire question text, ultimately generating the corresponding question vocabulary information, question grammar information, and question semantic information.

[0085] In one example, the TextCNN model is used for question intent recognition. Trained on a self-built question intent dataset, it achieves a 90% accuracy rate in recognizing intents such as "equipment model query" and "process step inquiry." A new intent segmentation module is added, further subdividing "process step inquiry" into "operation steps," "maintenance steps," and "troubleshooting steps," improving intent recognition accuracy to 94%. TF-IDF or TextRank algorithms are used to extract question keywords, which are then converted into vector representations. These keywords are combined with historical dialogue context to obtain the semantic feature vector of the question. Domain keywords in the question are also labeled, such as "tower construction" and "line erection," corresponding to the power grid domain, to trigger the domain adaptation parameters of the association model. For question search and matching, the obtained question text feature vector is input into the trained association model for searching, yielding target text and target images from response document samples related to the question. A hierarchical retrieval mechanism is employed. First, the BM25 algorithm is used for coarse screening of text paragraphs, prioritizing paragraphs within the same domain based on keywords relevant to the question. Next, an association model is used to calculate the image-text correlation for fine screening, dynamically adjusting weighting factors. For example, when the user intent is "equipment structure query," the image feature weight is increased to 1.4. The similarity between the question text features and the response text features of each paragraph in the response document sample is calculated. Text paragraphs are ranked based on similarity, and those with higher similarity are selected. The association model then searches for target images associated with these paragraphs. A new image priority ranking is added, adjusting the image ranking based on the matching degree between the core objects in the target images and the question keywords to improve matching accuracy. After returning the matching results to the user, the user's actions are recorded, such as whether they clicked on an image or manually adjusted the image-text correspondence. If the user adjusts the results, the adjusted image-text pairs are used as new training samples, marked as "high confidence samples," and added to the model's incremental training set. Incremental training is performed once a week to update the model parameters. At the same time, the matching accuracy under different domains and intentions is statistically analyzed. When the accuracy of a certain scenario is lower than 85%, the weight factor and segmentation threshold of that scenario are automatically optimized to achieve dynamic model iteration.

[0086] In some embodiments, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the structure of a text-image matching system based on semantic segmentation and feature association, provided in an embodiment of the present invention. The present invention provides a text-image matching system 200 based on semantic segmentation and feature association, comprising: a feature acquisition module 210, a model optimization module 220, and a result display module 230; wherein,

[0087] The feature acquisition module 210 is configured to perform semantic analysis on the target text and target image in the response document sample, and to acquire the response text features of different paragraphs and the response image features of each target image; the response text features include text vocabulary information, text grammar information and text semantic information; the response image features include image color information, image texture information, image shape information and image semantic information.

[0088] The model optimization module 220 is configured to map the response text features and response image features to the same dimensional space to obtain a text-image combination, and use the text-image combination as the input of the initial association model, and use the cosine similarity between the response text features and response image features as the output to train the initial association model, and combine the loss function and stochastic gradient descent optimization algorithm to optimize the initial association model to obtain the target association model;

[0089] The result display module 230 is configured to preprocess the question text to generate corresponding question text features, input the question text features into the target association model to obtain the corresponding text image combination, and display the text paragraph and target image corresponding to the text image combination; the question text features are of the same type as the response text features.

[0090] In some embodiments, the feature acquisition module 210 is specifically configured as follows:

[0091] The target text in the response document sample is split to obtain text vocabulary information; the text vocabulary information includes words or phrases.

[0092] Vocabulary information is tagged with parts of speech to determine the textual grammatical information of each text vocabulary in the target text;

[0093] A syntax tree is constructed based on the text syntax information in the response document sample to determine the text semantic information between text paragraphs.

[0094] In some embodiments, the feature acquisition module 210 is specifically configured as follows:

[0095] The initial weight of each word or phrase in the response document sample is obtained based on the TF-IDF algorithm; the initial weight is positively correlated with the word frequency.

[0096] The TextRank algorithm is used to obtain the correlation between different words or phrases, and the initial weights are adjusted based on the correlation to obtain optimized weights;

[0097] Based on optimized weights, words or phrases are sorted to determine keywords in the text's vocabulary information.

[0098] In some embodiments, the feature acquisition module 210 is specifically configured as follows:

[0099] Obtain the color histogram of the target image, and statistically analyze the distribution of different colors in the target image to obtain image color information;

[0100] The gray-level co-occurrence matrix method is used to calculate the texture features of the target image in multiple different directions, and the local binary mode algorithm is introduced to supplement the texture features with local texture details to obtain the image texture information;

[0101] Edge detection algorithms are used to extract the contours of objects in a target image to obtain image shape information;

[0102] The target image is processed using a pre-trained convolutional neural network, and the semantic information of the image is obtained by adaptively weighting and fusing multi-scale features.

[0103] In some embodiments, the initial association model includes a text tower and an image tower, and the model optimization module 220 is specifically configured as follows:

[0104] The text features of the response are input into the text tower, and the image features of the response are input into the image tower. The text tower and the image tower are trained by combining the cosine similarity output by the output layer.

[0105] In some embodiments, the model optimization module 220 is specifically configured as follows:

[0106] The difference between the predicted association score and the true association score of the initial association model is determined by using cross-entropy loss, triplet loss and contrastive loss. The parameters of the initial association model are continuously adjusted by the stochastic gradient descent optimization algorithm until the difference reaches a preset threshold, thus obtaining the target association model.

[0107] In some embodiments, the result display module 230 is specifically configured as follows:

[0108] The question text is preprocessed to generate corresponding question vocabulary information, question grammar information, and question semantic information;

[0109] The vocabulary, grammar, and semantic information of the question are input into the target association model to obtain the text features of the response and the associated image features of the response.

[0110] It should be noted that the image-text matching system based on semantic segmentation and feature association provided in this application embodiment and the image-text matching method based on semantic segmentation and feature association provided in this application embodiment are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned image-text matching method based on semantic segmentation and feature association, and the repeated parts will not be described again.

[0111] In some embodiments, please refer to Figure 3 , Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 300 provided in this application includes a processor 310 and a memory 320; the memory 320 stores a computer program, wherein the computer program, when executed by the processor, implements the aforementioned image-text matching method based on semantic segmentation and feature association.

[0112] Specifically, processor 310 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 310 may also include onboard memory for caching purposes. Processor 310 may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0113] The memory 320 may be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, the memory 320 may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, apparatuses, or propagation media. Specific examples of the memory 320 include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and may also be random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0114] This application also provides a computer-readable medium storing a computer program that, when executed by a processor, implements the above-described image-text matching method based on semantic segmentation and feature association. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0115] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0116] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A text-image matching method based on semantic segmentation and feature association, characterized in that, include: Semantic analysis is performed on the target text and target image in the response document sample to obtain the response text features of different paragraphs and the response image features of each target image; the response text features include text vocabulary information, text grammar information and text semantic information; the response image features include image color information, image texture information, image shape information and image semantic information. The response text features and response image features are mapped to the same dimensional space to obtain a text-image combination. The text-image combination is used as the input of the initial association model. The cosine similarity between the response text features and the response image features is used as the output to train the initial association model. The initial association model is then optimized by combining a loss function and a stochastic gradient descent optimization algorithm to obtain the target association model. The question text is preprocessed to generate corresponding question text features. The question text features are then input into the target association model to obtain the corresponding text image combination. The text paragraph and target image corresponding to the text image combination are then displayed. The question text features are of the same type as the response text features.

2. The image-text matching method based on semantic segmentation and feature association as described in claim 1, characterized in that, The semantic analysis of the target text and target image in the response document sample, to obtain the response text features of different paragraphs and the response image features of each target image, includes: The target text in the response document sample is split to obtain text vocabulary information; the text vocabulary information includes words or phrases. The vocabulary information is tagged with part-of-speech tags to determine the text grammar information of each vocabulary item in the target text; A syntax tree is constructed based on the text syntax information in the response document sample to determine the text semantic information between the text paragraphs.

3. The image-text matching method based on semantic segmentation and feature association as described in claim 2, characterized in that, The step of splitting the target text in the response document sample to obtain text vocabulary information includes: The initial weight of each word or phrase in the response document sample is obtained based on the TF-IDF algorithm; the initial weight is positively correlated with the word frequency. The TextRank algorithm is used to obtain the correlation between different words or phrases, and the initial weights are adjusted based on the correlation to obtain optimized weights; The words or phrases are sorted based on the optimized weights to determine the keywords in the text vocabulary information.

4. The image-text matching method based on semantic segmentation and feature association as described in claim 1, characterized in that, The semantic analysis of the target text and target image in the response document sample, to obtain the response text features of different paragraphs and the response image features of each target image, includes: Obtain the color histogram of the target image, and statistically analyze the distribution of different colors in the target image to obtain image color information; The gray-level co-occurrence matrix method is used to calculate the texture features of the target image in multiple different directions, and the local binary mode algorithm is introduced to supplement the texture features with local texture details to obtain the image texture information; The contours of objects in the target image are extracted using an edge detection algorithm to obtain image shape information; The target image is processed using a pre-trained convolutional neural network, and the semantic information of the image is obtained by adaptively weighting and fusing multi-scale features.

5. The image-text matching method based on semantic segmentation and feature association as described in claim 1, characterized in that, The initial association model includes a text tower and an image tower. The process of using the combination of text and images as input to the initial association model, and using the cosine similarity between the response text features and the response image features as output to train the initial association model includes: The response text features are input into the text tower, and the response image features are input into the image tower. The text tower and the image tower are trained by combining the cosine similarity output by the output layer.

6. The image-text matching method based on semantic segmentation and feature association as described in claim 5, characterized in that, The initial association model is optimized by combining a loss function and a stochastic gradient descent optimization algorithm to obtain the target association model, including: The difference between the predicted association score and the true association score of the initial association model is determined by using cross-entropy loss, triplet loss and contrastive loss. The parameters of the initial association model are continuously adjusted by the stochastic gradient descent optimization algorithm until the difference reaches a preset threshold, thus obtaining the target association model.

7. The image-text matching method based on semantic segmentation and feature association as described in claim 1, characterized in that, The process of preprocessing the question text to generate corresponding question text features, and then inputting these features into the target association model to obtain the corresponding text-image combination includes: The question text is preprocessed to generate corresponding question vocabulary information, question grammar information, and question semantic information; The question vocabulary information, question grammar information, and question semantic information are input into the target association model to obtain response text features and associated response image features.

8. A text-image matching system based on semantic segmentation and feature association, characterized in that, include: The module comprises a feature acquisition module, a model optimization module, and a results display module; among which, The feature acquisition module is configured to perform semantic analysis on the target text and target image in the response document sample, and acquire the response text features of different paragraphs and the response image features of each target image; the response text features include text vocabulary information, text grammar information and text semantic information; the response image features include image color information, image texture information, image shape information and image semantic information. The model optimization module is configured to map the response text features and the response image features to the same dimensional space to obtain a text-image combination, and use the text-image combination as the input of the initial association model, use the cosine similarity of the response text features and the response image features as the output to train the initial association model, and combine the loss function and stochastic gradient descent optimization algorithm to optimize the initial association model to obtain the target association model; The result display module is configured to preprocess the question text to generate corresponding question text features, input the question text features into the target association model to obtain the corresponding text image combination, and display the text paragraph and target image corresponding to the text image combination; the question text features are of the same type as the response text features.

9. An electronic device comprising a processor and a memory; said memory having a storage for a computer program, wherein, When the computer program is executed by the processor, it implements the image-text matching method based on semantic segmentation and feature association as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the image-text matching method based on semantic segmentation and feature association as described in any one of claims 1 to 7.

Images