A topic text matching method, device and electronic equipment

Abstract

The present invention provides a topic text matching method, device and electronic device, and relates to the field of computer technology. The method comprises: when the correlation between text information and image information in a content text is greater than a preset threshold, obtaining the position feature of the image information, the position feature being used to characterize the distance relationship between the image information and each paragraph information in the text information; inputting the position feature, the text information and the image information into a multimodal pre-training model, and outputting a first intermediate vector corresponding to the content text; and determining the correlation between the content text and the topic text according to the first intermediate vector and the second intermediate vector corresponding to the topic text. The scheme of the present invention can accurately determine the correlation between the topic text and the content text using the image information in the content text.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a topic text matching method, device and electronic equipment. Background Art

[0002] Among the many research directions of Natural Language Processing (NLP) and vector retrieval (Elasticsearch, ES), text similarity has always been a hot research topic. Text similarity refers to the similarity between two texts. It has become an indispensable step in many natural language processing and vector database applications, such as search engines, vector retrieval and other fields. Currently, the more common text relevance calculation is the calculation between two plain texts. The semantic relevance between texts is strong and the semantic expression is relatively complete.

[0003] In the application scenario of matching thematic text with content text, thematic text is expressed at the word granularity, while content text is expressed at the sentence granularity. There is inconsistency in the semantic completeness of these two texts. In addition, there is often image information in the content text. For example, a fire picture is consistent with the "fire" information in the thematic text. Therefore, how to use the image information in the content text to measure the relevance between thematic text and the content text is a technical issue that needs to be considered.

[0004] Currently, a commonly used method is to directly convert image information into vector information and merge it with the content text vector to enhance the expression of the content text vector. However, this solution does not fully utilize the image information, and the matching accuracy between the topic text and the content text is low. Summary of the invention

[0005] The purpose of the present invention is to provide a topic text matching method, device and electronic device, which solves the problem of how to use the image information in the content text to accurately determine the relevance between the topic text and the content text.

[0006] To achieve the above object, an embodiment of the present invention provides a topic text matching method, comprising:

[0007] When the correlation between the text information and the picture information in the content text is greater than a preset threshold, obtaining a position feature of the picture information, wherein the position feature is used to characterize a distance relationship between the picture information and each paragraph information in the text information;

[0008] Inputting the position feature, the text information and the image information into a multimodal pre-training model, and outputting a first intermediate vector corresponding to the content text;

[0009] The relevance between the content text and the topic text is determined according to the first intermediate vector and a second intermediate vector corresponding to the topic text.

[0010] Optionally, the location feature includes one or more of the following:

[0011] The distance between the upper left corner coordinates of the image information and each paragraph information;

[0012] The distance between the lower right corner coordinates of the image information and each paragraph information;

[0013] The spatial similarity distance between the image information and each paragraph information.

[0014] Optionally, the above topic text matching method further includes:

[0015] Constructing a topic forest according to the logical information and keywords in the topic text; wherein the topic forest includes at least one topic tree, and the topic tree has distributed feature word vectors as leaf nodes;

[0016] According to the distributed feature word vector, the second intermediate vector corresponding to the topic text is obtained.

[0017] Optionally, determining the relevance between the content text and the topic text according to the first intermediate vector and a second intermediate vector corresponding to the topic text includes:

[0018] Acquire an event theme vector according to the content text and the topic text;

[0019] Using the attention mechanism, the event topic vector and the first intermediate vector are fused with different weights to obtain a content text vector;

[0020] Using the attention mechanism, the event topic vector and the second intermediate vector are fused with different weights to obtain a topic text vector;

[0021] The correlation between the content text and the topic text is determined according to the topic text vector and the content text vector.

[0022] Optionally, acquiring the event theme vector according to the content text and the topic text includes:

[0023] Obtaining a first topic vector corresponding to the content text;

[0024] Obtaining a second theme vector corresponding to the theme text;

[0025] The first topic vector and the second topic vector are fused with different weights by using the attention mechanism to obtain the event topic vector.

[0026] Optionally, the above topic text matching method further includes:

[0027] Obtaining a first topic probability distribution of the text information and a second topic probability distribution of the image information;

[0028] Input the first topic probability distribution and the second topic probability distribution into a linear transformation layer respectively, and output a first high-dimensional vector and a second high-dimensional vector;

[0029] fusing the first high-dimensional vector and the second high-dimensional vector through a gating unit to obtain a multimodal topic feature vector;

[0030] The multimodal topic feature vector is input into a multi-layer perceptron (MLP) classifier, and a multimodal topic distribution is output, where the multimodal topic distribution is used to characterize the relevance between the text information and the image information.

[0031] To achieve the above object, an embodiment of the present invention provides a topic text matching device, comprising:

[0032] A first acquisition module, configured to acquire, when the correlation between the text information and the image information in the content text is greater than a preset threshold, a position feature of the image information, wherein the position feature is used to characterize a distance relationship between the image information and each paragraph information in the text information;

[0033] A second acquisition module, used for inputting the position feature, the text information and the image information into a multimodal pre-training model, and outputting a first intermediate vector corresponding to the content text;

[0034] The first determination module is used to determine the relevance between the content text and the topic text according to the first intermediate vector and a second intermediate vector corresponding to the topic text.

[0035] To achieve the above object, an embodiment of the present invention provides an electronic device, including a processor and a transceiver, wherein the processor is used to:

[0036] When the correlation between the text information and the picture information in the content text is greater than a preset threshold, obtaining a position feature of the picture information, wherein the position feature is used to characterize a distance relationship between the picture information and each paragraph information in the text information;

[0037] Inputting the position feature, the text information and the image information into a multimodal pre-training model, and outputting a first intermediate vector corresponding to the content text;

[0038] The relevance between the content text and the topic text is determined according to the first intermediate vector and a second intermediate vector corresponding to the topic text.

[0039] To achieve the above objectives, an embodiment of the present invention provides an electronic device, including a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the thematic text matching method as described above is implemented.

[0040] To achieve the above objective, an embodiment of the present invention provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the above-mentioned thematic text matching method are implemented.

[0041] To achieve the above objective, an embodiment of the present invention provides a computer program product, including computer instructions, which implement the steps in the above-mentioned thematic text matching method when executed by a processor.

[0042] The beneficial effects of the above technical solution of the present invention are as follows:

[0043] In the above scheme, when it is determined that the correlation between the text information and the image information in the content text is greater than a preset threshold, the position feature of the image information is obtained, and the position feature is used to characterize the distance relationship between the image information and each paragraph information in the text information; the position feature, text information and image information are input into the multimodal pre-training model, and the first intermediate vector corresponding to the content text is output; according to the first intermediate vector and the second intermediate vector corresponding to the topic text, the correlation between the content text and the topic text is determined. This scheme can avoid the interference of image information irrelevant to the content text on the matching results, and at the same time, the position feature of the image information is integrated to realize multimodal input to increase the richness of the intermediate semantic information, thereby improving the matching accuracy of the topic text and the content text. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a topic text matching method according to an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of location feature information according to an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of constructing a topic tree according to an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the architecture of a topic matching model according to an embodiment of the present invention;

[0048] Figure 5Schematic diagram of steps for obtaining a first intermediate vector according to an embodiment of the present invention;

[0049] Figure 6 It is a structural diagram of a topic text matching device according to an embodiment of the present invention;

[0050] Figure 7 is a structural diagram of an electronic device according to an embodiment of the present invention;

[0051] Figure 8 FIG. 4 is a structural diagram of an electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0053] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0054] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0055] like Figure 1 As shown, a topic text matching method according to an embodiment of the present invention comprises the following steps:

[0056] Step 101: when the correlation between text information and image information in the content text is greater than a preset threshold, obtain the position feature of the image information, wherein the position feature is used to characterize the distance relationship between the image information and each paragraph information in the text information.

[0057] The distance relationship includes but is not limited to at least one of the following: the number of text lines between the two texts, the number of text columns between the two texts, the semantic distance, etc.

[0058] Step 102: Input the position feature, the text information and the image information into a multimodal pre-training model, and output a first intermediate vector corresponding to the content text.

[0059] That is, Emb s1 =MultiTransformer(word+position+images);

[0060] Among them, MultiTransformer is a multimodal pre-training model, word is text information, position is position feature, images is image information, Emb s1 is the first intermediate vector, which is the semantic vector of the content text.

[0061] It should be pointed out that the multimodal pre-trained model is a deep learning model obtained by pre-training on a large-scale multimodal dataset. It can simultaneously process and understand multiple types of data, such as text, images, audio, etc.

[0062] Step 103: Determine the relevance between the content text and the topic text according to the first intermediate vector and the second intermediate vector corresponding to the topic text.

[0063] Among them, the relevance can reflect the overall semantic connection between the content text and the topic text.

[0064] In the above embodiment, when the correlation between the text information and the image information in the content text is greater than a preset threshold, the position feature of the image information is obtained, so that the image information that is not related to the text information can be filtered out. Moreover, since the position feature of the image information can characterize the positional relationship and semantic relationship between the image information and each paragraph information, the position feature, text information and image information are input into the multimodal pre-training model. The auxiliary information provided by the position feature can enrich the intermediate semantic information, help the multimodal pre-training model to better understand the text content, and improve the matching accuracy of the content text and thematic text.

[0065] In some embodiments, the location characteristics include one or more of the following:

[0066] The distance between the upper left corner coordinates of the image information and each paragraph information;

[0067] The distance between the lower right corner coordinates of the image information and each paragraph information;

[0068] The spatial similarity distance between the image information and each paragraph information.

[0069] For example, it is assumed that the content text includes 10 text paragraphs and one picture information; wherein the coordinate of the upper left corner of the picture information is (x 0 ,y 0 ), the coordinate of the lower right corner of the image information is (x 1 ,y 1 ), the spatial similarity distance includes z 0 and z 1 , where z 0 Represents the cosine distance between each paragraph text and image information, z0 Indicates the Euclidean distance between each paragraph's text and image information.

[0070] Then, treat the area where each text paragraph is located as a rectangular image, and obtain the coordinates of the upper left corner of the rectangular image (x A ,y A ), the coordinates of the lower right corner of the rectangular image (x B ,y B ), based on the position coordinates, Figure 2 The feature vector shown may include:

[0071] 10 text paragraphs x A Respectively with x 0 The vectors corresponding to the number of rows apart are: E X1 、E X2 、E X3 ,……,E X9 、E X10 ;

[0072] y of 10 text paragraphs A Respectively with y 0 The vectors corresponding to the number of columns apart are: E Y1 、E Y2 、E Y3 ,……,E Y9 、E Y10 ;

[0073] 10 text paragraphs x B Respectively with x 1 The vectors corresponding to the number of rows apart are: E X11 、E X12 、E X13 ,……,E X110 ;

[0074] y of 10 text paragraphs B Respectively with y 1 The vectors corresponding to the number of columns apart are: E Y11 、E Y12 、E Y13 ,……,E Y110 ;

[0075] The cosine distance z between 10 text paragraphs and image information 0 The corresponding vectors are: E Z1 、E Z2 、E Z3 ,……,E Z10 ;

[0076] The Euclidean distance z between 10 text paragraphs and image information 1 The corresponding vectors are: EZ11 、E Z12 、E Z13 ,……,E Z110 ;

[0077] Furthermore, the final position feature is obtained by concatenating E1, E2, E3, E4, E5, E6, E7, E8, E9, and E10, where: E1 = E X1 +E Y1 +E X11 +E Y11 +E Z1 +E Z11 , E2=E X2 +E Y2 +E X12 +E Y12 +E Z2 +E Z12 , E3=E X3 +E Y3 +E X13 +E Y13 +E Z3 +E Z13 , ..., E10 = E X10 +E Y10 +E X20 +E Y110 +E Z10 +E Z20 .

[0078] It is understandable that the length of the vector depends on the number of text paragraphs.

[0079] Among them, the cosine distance z between each paragraph text and image information 0 The calculation method includes: converting the text paragraph and the picture information into feature vectors respectively, and taking the cosine distance between the two feature vectors as the cosine distance z between the paragraph text and the picture information 0 .

[0080] Among them, the Euclidean distance z between each paragraph text and image information 1 The calculation method includes: converting the text paragraph and the image information into feature vectors respectively, and taking the Euclidean distance between the two feature vectors as the co-Euclidean distance z between the paragraph text and the image information 1 .

[0081] In the above embodiment, the distance between the upper left corner coordinate of the image information and each paragraph information and the distance between the lower right corner coordinate of the image information and each paragraph information can represent the distance and position relationship between the image information and each paragraph information, and the spatial similarity distance between the image information and each paragraph information can reflect the association relationship or semantic similarity between the image information and each paragraph information. This information can help the multimodal pre-training model understand the text content more accurately, thereby improving the accuracy of the first intermediate vector.

[0082] In some embodiments, the above topic text matching method further includes:

[0083] Constructing a topic forest according to the logical information and keywords in the topic text; wherein the topic forest includes at least one topic tree, and the topic tree has distributed feature word vectors as leaf nodes;

[0084] According to the distributed feature word vector, the second intermediate vector corresponding to the topic text is obtained.

[0085] For example, Figure 3 As shown, assuming that the topic text is: "A+B|(B+C)+D", then according to the outermost "+" (logical relationship), the topic text is divided into three parts: "A", "B|(B+C)" and "D"; then, for the three parts "A", "B|(B+C)" and "D", the topic trees are constructed respectively, and each topic tree includes at least one leaf node composed of keywords, and each leaf node is represented by a feature word vector corresponding to the keyword, and the feature word vector changes dynamically with the training of the model. That is, the word string "A" corresponds to a topic tree, the word string "B|(B+C)" corresponds to a topic tree, and the word string "D" corresponds to a topic tree.

[0086] The steps of constructing the thematic tree include:

[0087] Step 1: After removing invalid words such as adjectives and adverbs from the word string "A", word string "B|(B+C)" and word string "D" that have been segmented once in the topic forest, perform keyword segmentation to obtain the feature word dictionary corresponding to each word string.

[0088] Step 2: Based on the BERT pre-trained model, obtain the feature word vector corresponding to each feature word; here, what is stored is the distributed representation of the word, that is, word embedding, and the feature word vector changes dynamically with the training of the model.

[0089] The calculation method of the feature word vector is as follows: w =BERT(word); where Emb wis the feature word vector, BERT is the BERT Encoder, and word is the feature word.

[0090] It should be pointed out that in the prior art, in the process of processing thematic texts, thematic texts are generally directly converted into texts without logical information. Since they cannot form complete sentences, the semantic integrity of thematic texts is weak, and the thematic matching effect is not good. If the logical keywords of thematic texts are directly converted into one keyword, the logical information in the thematic texts is ignored. For example, if the thematic keyword texts contain "and" and "or" logical words, if the logical relationship is ignored and the relevance is directly calculated based on the keywords, the logical information is lost. Putting keywords that may be in different logics together will cause confusion in the matching keyword information. For example, (E Province, F District, Fire) and (H Province, Highway Accident) are mixed together, and it is uncertain where the fire occurred and where the highway accident occurred.

[0091] In this embodiment, by constructing a thematic forest and a thematic tree corresponding to the thematic text, a distributed feature word vector can be obtained, the feature word vectors within the thematic tree are superimposed, and the vectors between the thematic trees are spliced ​​to obtain a second intermediate vector corresponding to the thematic text. This ensures the uniformity within the thematic tree and the difference between the thematic trees, retains the logical information in the thematic text, and can improve the accuracy of the correlation calculation between the thematic text and the content text.

[0092] In some embodiments, in the above step 103, determining the relevance between the content text and the topic text according to the first intermediate vector and the second intermediate vector corresponding to the topic text includes:

[0093] Acquire an event theme vector according to the content text and the topic text;

[0094] Using the attention mechanism, the event topic vector and the first intermediate vector are fused with different weights to obtain a content text vector;

[0095] Using the attention mechanism, the event topic vector and the second intermediate vector are fused with different weights to obtain a topic text vector;

[0096] The correlation between the content text and the topic text is determined according to the topic text vector and the content text vector.

[0097] It should be pointed out that since the keywords in the title text are descriptions of the event theme information, such as fire events, fire locations, fire time and other event keywords, when extracting vector information from the topic text and content text, it will be more accurate to calculate the relevance of the topic text and content text by combining the event theme information of the text.

[0098] In this embodiment, an attention mechanism is introduced to integrate the event theme vector into the first intermediate vector and the second intermediate vector respectively with different weights to obtain a thematic text vector and a content text vector. This enables the thematic text vector and the content text vector to have event theme information, thereby enhancing the semantic expression of the original vector and improving the effect of text granularity theme matching.

[0099] In some embodiments, obtaining an event theme vector according to the content text and the topic text includes:

[0100] Obtaining a first topic vector corresponding to the content text;

[0101] Obtaining a second theme vector corresponding to the theme text;

[0102] The first topic vector and the second topic vector are fused with different weights by using the attention mechanism to obtain the event topic vector.

[0103] In specific implementation, the event topic vector can be obtained based on the pre-trained topic model (Latent Dirichlet Allocation, LDA) combined with the attention mechanism. Among them, LDA is a commonly used topic model for extracting potential topics from text data. Specifically, firstly, the first topic vector corresponding to the content text and the second topic vector corresponding to the topic text are obtained based on LDA; further, the first topic vector and the second topic vector are input into the attention module, and the event topic vector is generated through the attention mechanism.

[0104] Specifically, the step of obtaining the second topic vector corresponding to the topic text based on LDA includes:

[0105] Step 1: Output the keyword topic vector corresponding to the topic keyword based on the LDA model.

[0106] That is, W key =TopicModel(word); where TopicModel is the LDA model, word is the topic keyword, and W key is the keyword topic vector.

[0107] Step 2: When the topic text includes multiple topic keywords, the average value of the keyword topic vectors corresponding to the multiple topic keywords is used as the second topic vector.

[0108] Specifically, the step of obtaining the first topic vector corresponding to the content text based on LDA includes:

[0109] Step 1: Output the topic vector corresponding to each word in the content text based on the LDA model.

[0110] That is, Dwi =TopicModel(word i ); TopicModel is the LDA model, word i is the i-th word, D wi is the topic vector corresponding to the i-th word.

[0111] Step 2: Take the average value of the topic vectors corresponding to each word in a sentence as the sentence topic vector.

[0112] Right now, Among them, W sen is the sentence topic vector, N is the number of words included in the sentence, i and N are positive integers, and i is from 1 to N.

[0113] Step 3: When the content text includes multiple sentences, the average value of the sentence topic vectors corresponding to the multiple sentences is used as the first topic vector.

[0114] It should be pointed out that an event topic model can be constructed based on the LDA model and the attention mechanism, and the thematic text and content text can be used as the input of the event topic model to output an event topic vector. In the process of training LDA, the event topic is dynamically expanded. After the training is completed, the event topic information within the corpus can basically be covered.

[0115] In the above embodiment, different from the traditional solution of combining thematic text and content text to obtain thematic vector, the present application will separately obtain various corresponding thematic vectors for thematic text and content text, and then use the attention mechanism to form the final event thematic vector, and then merge the event thematic vector into the original first intermediate vector and the second intermediate vector to generate thematic text vector and content text vector. In this way, the semantic expression of the original vector can be enhanced, and the effect of text granularity thematic matching can be improved.

[0116] In some embodiments, the above topic text matching method further includes:

[0117] Obtaining a first topic probability distribution of the text information and a second topic probability distribution of the image information;

[0118] Input the first topic probability distribution and the second topic probability distribution into a linear transformation layer respectively, and output a first high-dimensional vector and a second high-dimensional vector;

[0119] fusing the first high-dimensional vector and the second high-dimensional vector through a gating unit to obtain a multimodal topic feature vector;

[0120] The multimodal topic feature vector is input into a multi-layer perceptron (MLP) classifier, and a multimodal topic distribution is output, where the multimodal topic distribution is used to characterize the relevance between the text information and the image information.

[0121] In specific implementation, the correlation between text information and image information can be determined based on the generation model, wherein the generation model includes a decoder, a linear change layer, a gating unit, and an MLP classifier.

[0122] Exemplarily, the processing includes:

[0123] Step 1: Obtain the image topic probability distribution pi corresponding to the image information and the text topic probability distribution pt corresponding to the text information through the decoder;

[0124] That is, pi=Decoder(images), pt=Decoder(word);

[0125] Among them, images is image information, word is text information, and Decoder is decoder.

[0126] Step 2: Input pi and pt into a linear transformation layer respectively, and project them into a first high-dimensional vector and a second high-dimensional vector;

[0127] Step 3: The high-dimensional vectors of the image topic distribution pi and the text topic probability distribution pt are fused through the gating unit to obtain the multimodal topic feature vector pm.

[0128] That is, pm=GAN(pi,pt), where GAN is a gated unit.

[0129] Step 4: Input the multimodal topic feature vector pm into the MLP classifier and predict the output multimodal topic distribution ps, where ps represents the correlation between image information and text information.

[0130] That is, ps = softmax(mlp(pm)); where mlp is the MLP classifier and softmax is a mathematical function used to convert the output of the MLP classifier into a probability distribution.

[0131] In the above embodiment, the correlation between text information and image information in the text content can be accurately identified, so that image information irrelevant to the text information can be filtered out based on a preset correlation threshold, thereby avoiding image information irrelevant to the text information from interfering with the text content vector and affecting the matching accuracy.

[0132] The following is combined with Figure 4 This paper provides an illustrative introduction to the model architecture and training process for thematic text matching.

[0133] like Figure 4 The model as a whole includes:

[0134] Thematic text input module, intermediate processing module and content text input module. Thematic text input module and content text input module are used to generate the second intermediate vector of thematic text and the first intermediate vector of content text respectively. The intermediate processing module is used to obtain the event theme vector of thematic text and content text; further, using the attention module, the first intermediate vector and the second intermediate vector are respectively fused with the event theme vector to obtain thematic text vector and content text vector; finally, the loss function is calculated based on thematic text vector and content text vector; through iterative training, the trained model architecture for thematic text matching can be obtained.

[0135] Among them, the thematic text input module is built based on BERT Encoder, and thematic forests and thematic trees are introduced to process thematic keyword text to retain logical information; the intermediate processing module is built based on the LDA model and attention mechanism; the content text input module is built based on the multimodal pre-training model and BERT Encoder, and the location features of image information are introduced.

[0136] In addition, if Figure 5 As shown in the figure, after the text information and the image information are correlated, if the two are correlated, they are input into the multimodal pre-training model. For the output result of the multimodal pre-training model, the attention mechanism is used to distinguish the weights of the text content and the image content in the text relevance calculation, and the first intermediate vector is obtained. The calculation method is as follows:

[0137] Among them, a i and μ i is the weight coefficient, a i and μ i Add up to 1, w i is the text information in text content i, p i is the image information in text content i, N is the total number of training samples, Emb s is the first intermediate vector corresponding to the text content i.

[0138] It should be pointed out that in the training process of the entire framework, the loss function includes:

[0139] loss 1 =-log(σ(s·cos(w i ,s i )))+log(1-σ(s·cos(w i ,s ni )));

[0140] loss 2 =max(0,-[cos(w i ,s i )-cos(w i ,s ni )]+margin);

[0141] Among them, loss 1 is the first loss value calculated by the cross entropy loss function. The first loss is used to characterize the relevance between the topic text and the content text. 2 is the second loss value calculated based on the margin rank loss function. The second loss value is used to characterize the irrelevance between the topic text and the content text. Based on the difference between the first loss value and the second loss value, the relevance between the topic text and the content text can be obtained. σ is the Sigmod activation function, s is the preset adjustment parameter, (w i ,s i ) represents a positive sample pair, (w i ,s ni ) represents a negative sample pair, w i is the thematic text training sample, s i and ni Two content text training samples.

[0142] In this way, two different loss functions are used to constrain the distances of two different vectors, so that the topic text is closer to its related content text and farther from the irrelevant content text.

[0143] In the above scheme, a topic keyword text processing method with logical symbols is proposed in the topic text input module, and an event topic model that can be dynamically trained and expanded is introduced in the intermediate processing module, so that the extracted intermediate semantic vector carries event topic information, fully considering the event topic information of topic keywords and content text, and improving the topic matching effect. In the content text input processing module, after effectively identifying the relevant image information in the content text, multimodal input is realized using text information and image information, and combined with the attention mechanism, the weights given to image information and text information in the relevance calculation are weighed. In addition, the entire intermediate vector extraction task is optimized in combination with the location information of the text and the image, so that the extracted semantic vector has richer semantic information, which improves the accuracy of topic text matching.

[0144] like Figure 6 As shown, a topic text matching device 600 according to an embodiment of the present invention includes:

[0145] A first acquisition module 601 is used to acquire a position feature of the image information when the correlation between the text information and the image information in the content text is greater than a preset threshold, wherein the position feature is used to characterize the distance relationship between the image information and each paragraph information in the text information;

[0146] A second acquisition module 602 is used to input the position feature, the text information and the image information into a multimodal pre-training model, and output a first intermediate vector corresponding to the content text;

[0147] The first determination module 603 is used to determine the relevance between the content text and the topic text according to the first intermediate vector and the second intermediate vector corresponding to the topic text.

[0148] Optionally, the location feature includes one or more of the following:

[0149] The distance between the upper left corner coordinates of the image information and each paragraph information;

[0150] The distance between the lower right corner coordinates of the image information and each paragraph information;

[0151] The spatial similarity distance between the image information and each paragraph information.

[0152] Optionally, the topic text matching device 600 further includes:

[0153] A first processing module is used to construct a topic forest according to the logical information and keywords in the topic text; wherein the topic forest includes at least one topic tree, and the topic tree has distributed feature word vectors as leaf nodes;

[0154] The second processing module is used to obtain the second intermediate vector corresponding to the topic text according to the distributed feature word vector.

[0155] Optionally, the first determining module 603 includes:

[0156] A first determination submodule is used to obtain an event theme vector according to the content text and the topic text;

[0157] A second determination submodule is used to use an attention mechanism to fuse the event topic vector and the first intermediate vector with different weights to obtain a content text vector;

[0158] A third determination submodule is used to use an attention mechanism to fuse the event topic vector and the second intermediate vector with different weights to obtain a topic text vector;

[0159] The fourth determination submodule is used to determine the relevance between the content text and the topic text according to the topic text vector and the content text vector.

[0160] Optionally, the first determining submodule includes:

[0161] A first acquisition unit, configured to acquire a first topic vector corresponding to the content text;

[0162] A second acquisition unit, used to acquire a second theme vector corresponding to the theme text;

[0163] The third acquisition unit is used to utilize the attention mechanism and adopt different weights to fuse the first topic vector and the second topic vector to obtain the event topic vector.

[0164] Optionally, the topic text matching device 600 further includes:

[0165] A third acquisition module, used to acquire a first topic probability distribution of the text information and a second topic probability distribution of the picture information;

[0166] a fourth acquisition module, configured to input the first topic probability distribution and the second topic probability distribution into a linear transformation layer respectively, and output a first high-dimensional vector and a second high-dimensional vector;

[0167] a fifth acquisition module, configured to fuse the first high-dimensional vector and the second high-dimensional vector through a gating unit to obtain a multimodal topic feature vector;

[0168] The sixth acquisition module is used to input the multimodal topic feature vector into a multi-layer perceptron MLP classifier and output a multimodal topic distribution, where the multimodal topic distribution is used to characterize the correlation between the text information and the image information.

[0169] The topic text matching device 600 is a device corresponding to the method. All implementation means in the above method embodiment are applicable to the embodiment of the topic text matching device 600 and can achieve the same technical effect.

[0170] like Figure 7As shown, an electronic device 700 of an embodiment of the present invention includes a processor 710 and a transceiver 720, wherein the processor is used to obtain a position feature of the image information when the correlation between the text information and the image information in the content text is greater than a preset threshold, and the position feature is used to characterize the distance relationship between the image information and each paragraph information in the text information; input the position feature, the text information and the image information into a multimodal pre-training model, and output a first intermediate vector corresponding to the content text; and determine the correlation between the content text and the topic text according to the first intermediate vector and the second intermediate vector corresponding to the topic text.

[0171] Optionally, the location feature includes one or more of the following:

[0172] The distance between the upper left corner coordinates of the image information and each paragraph information;

[0173] The distance between the lower right corner coordinates of the image information and each paragraph information;

[0174] The spatial similarity distance between the image information and each paragraph information.

[0175] Optionally, the processor is further configured to:

[0176] Constructing a topic forest according to the logical information and keywords in the topic text; wherein the topic forest includes at least one topic tree, and the topic tree has distributed feature word vectors as leaf nodes;

[0177] According to the distributed feature word vector, the second intermediate vector corresponding to the topic text is obtained.

[0178] Optionally, the processor is specifically configured to:

[0179] Acquire an event theme vector according to the content text and the topic text;

[0180] Using the attention mechanism, the event topic vector and the first intermediate vector are fused with different weights to obtain a content text vector;

[0181] Using the attention mechanism, the event topic vector and the second intermediate vector are fused with different weights to obtain a topic text vector;

[0182] The correlation between the content text and the topic text is determined according to the topic text vector and the content text vector.

[0183] Optionally, the processor is specifically configured to:

[0184] Obtaining a first topic vector corresponding to the content text;

[0185] Obtaining a second theme vector corresponding to the theme text;

[0186] The first topic vector and the second topic vector are fused with different weights by using the attention mechanism to obtain the event topic vector.

[0187] Optionally, the processor is further configured to:

[0188] Obtaining a first topic probability distribution of the text information and a second topic probability distribution of the image information;

[0189] Input the first topic probability distribution and the second topic probability distribution into a linear transformation layer respectively, and output a first high-dimensional vector and a second high-dimensional vector;

[0190] fusing the first high-dimensional vector and the second high-dimensional vector through a gating unit to obtain a multimodal topic feature vector;

[0191] The multimodal topic feature vector is input into a multi-layer perceptron (MLP) classifier, and a multimodal topic distribution is output, where the multimodal topic distribution is used to characterize the relevance between the text information and the image information.

[0192] The electronic device of this embodiment, when the correlation between the text information and the picture information in the content text is greater than a preset threshold, obtains the position feature of the picture information, the position feature is used to characterize the distance relationship between the picture information and each paragraph information in the text information; inputs the position feature, the text information and the picture information into a multimodal pre-training model, outputs a first intermediate vector corresponding to the content text; and determines the correlation between the content text and the topic text according to the first intermediate vector and the second intermediate vector corresponding to the topic text. Since the position feature can characterize the positional relationship and semantic relationship between the picture information and each paragraph information, the position feature, the text information and the picture information are input into the multimodal pre-training model, and the auxiliary information provided by the position feature can enrich the intermediate semantic information, help the multimodal pre-training model to better understand the text content, and improve the matching accuracy of the content text and the topic text.

[0193] An electronic device according to another embodiment of the present invention, such as Figure 8 As shown, it includes a transceiver 810, a processor 800, a memory 820, and a program or instruction stored in the memory 820 and executable on the processor 800; when the processor 800 executes the program or instruction, the above-mentioned method for matching thematic texts is implemented.

[0194] The transceiver 810 is used to receive and send data under the control of the processor 800 .

[0195] Among them, Figure 8 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 800 and various circuits of memory represented by memory 820 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 810 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 830 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0196] The processor 800 is responsible for managing the bus architecture and general processing, and the memory 820 can store data used by the processor 800 when performing operations.

[0197] A computer-readable storage medium according to an embodiment of the present invention stores a program or instruction thereon, and when the program or instruction is executed by a processor, the steps in the above-mentioned topic text matching method are implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0198] To achieve the above objective, an embodiment of the present invention provides a computer program product, including computer instructions, which implement the steps of the above-mentioned topic text matching method when executed by a processor.

[0199] It should be further explained that the terminals described in this specification include but are not limited to smart phones, tablet computers, etc., and many of the functional components described are called modules in order to more particularly emphasize the independence of their implementation methods.

[0200] In the embodiment of the present invention, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

[0201] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

[0202] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

[0203] The above exemplary embodiments are described with reference to the accompanying drawings, and many different forms and embodiments are feasible without departing from the spirit and teachings of the present invention. Therefore, the present invention should not be constructed as a limitation of the exemplary embodiments proposed herein. More specifically, these exemplary embodiments are provided so that the present invention will be perfect and complete, and the scope of the present invention will be conveyed to those who are familiar with the technology. In these figures, the component sizes and relative sizes may be exaggerated for clarity. The terms used here are only based on the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used herein, unless the text clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to include these multiple forms. It will be further understood that the terms "including" and / or "comprising" when used in this specification indicate the presence of the features, integers, steps, operations, components and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, components, components and / or their groups. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of that range and any subranges therebetween.

[0204] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A topic text matching method, characterized in that: include: When the correlation between the text information and the picture information in the content text is greater than a preset threshold, obtaining a position feature of the picture information, wherein the position feature is used to characterize a distance relationship between the picture information and each paragraph information in the text information; Inputting the position feature, the text information and the image information into a multimodal pre-training model, and outputting a first intermediate vector corresponding to the content text; The relevance between the content text and the topic text is determined according to the first intermediate vector and a second intermediate vector corresponding to the topic text.

2. The topic text matching method according to claim 1, characterized in that: The location characteristics include one or more of the following: The distance between the upper left corner coordinates of the image information and each paragraph information; The distance between the lower right corner coordinates of the image information and each paragraph information; The spatial similarity distance between the image information and each paragraph information.

3. The topic text matching method according to claim 1, characterized in that: The method further comprises: Constructing a topic forest according to the logical information and keywords in the topic text; wherein the topic forest includes at least one topic tree, and the topic tree has distributed feature word vectors as leaf nodes; According to the distributed feature word vector, the second intermediate vector corresponding to the topic text is obtained.

4. The topic text matching method according to claim 1, characterized in that: The determining the relevance between the content text and the topic text according to the first intermediate vector and the second intermediate vector corresponding to the topic text includes: Acquire an event theme vector according to the content text and the topic text; Using the attention mechanism, the event topic vector and the first intermediate vector are fused with different weights to obtain a content text vector; Using the attention mechanism, the event topic vector and the second intermediate vector are fused with different weights to obtain a topic text vector; The correlation between the content text and the topic text is determined according to the topic text vector and the content text vector.

5. The topic text matching method according to claim 4, characterized in that: The step of obtaining an event theme vector according to the content text and the topic text includes: Obtaining a first topic vector corresponding to the content text; Obtaining a second theme vector corresponding to the theme text; The first topic vector and the second topic vector are fused with different weights by using the attention mechanism to obtain the event topic vector.

6. The topic text matching method according to claim 1, characterized in that: The method further comprises: Obtaining a first topic probability distribution of the text information and a second topic probability distribution of the image information; Input the first topic probability distribution and the second topic probability distribution into a linear transformation layer respectively, and output a first high-dimensional vector and a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector through a gating unit to obtain a multimodal topic feature vector; The multimodal topic feature vector is input into a multi-layer perceptron (MLP) classifier, and a multimodal topic distribution is output, where the multimodal topic distribution is used to characterize the relevance between the text information and the image information.

7. A topic text matching device, characterized in that: include: A first acquisition module, configured to acquire, when the correlation between the text information and the image information in the content text is greater than a preset threshold, a position feature of the image information, wherein the position feature is used to characterize a distance relationship between the image information and each paragraph information in the text information; A second acquisition module, used for inputting the position feature, the text information and the image information into a multimodal pre-training model, and outputting a first intermediate vector corresponding to the content text; The first determination module is used to determine the relevance between the content text and the topic text according to the first intermediate vector and a second intermediate vector corresponding to the topic text.

8. An electronic device, characterized in that: include: transceivers and processors; The processor is used to: When the correlation between the text information and the picture information in the content text is greater than a preset threshold, obtaining a position feature of the picture information, wherein the position feature is used to characterize a distance relationship between the picture information and each paragraph information in the text information; Inputting the position feature, the text information and the image information into a multimodal pre-training model, and outputting a first intermediate vector corresponding to the content text; The relevance between the content text and the topic text is determined according to the first intermediate vector and a second intermediate vector corresponding to the topic text.

9. An electronic device, comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; characterized in that when the processor executes the program or instruction, the thematic text matching method according to any one of claims 1 to 6 is implemented.

10. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps in the thematic text matching method according to any one of claims 1 to 6 are implemented.

11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the thematic text matching method according to any one of claims 1 to 6.

Images