A method, device and equipment for extracting key information from a picture-text and a storage medium

Abstract

The application provides a method and device for extracting key information from images and texts, equipment and a storage medium, relates to the technical field of image-text information extraction, and solves the problems of low accuracy and incompleteness of the current key information extraction method. First, the obtained document is preprocessed to obtain image-text information composed of pictures and words, then a pre-trained word vector extraction model and a graph vector extraction model are obtained, the word vector extraction model and the graph vector extraction model are used to receive the image-text information, the feature vectors of the words and the pictures are extracted, the image-text information is taken as a node, the relationship between the image-text information is taken as an edge, a graph network model is constructed, the weight scores of the nodes are calculated and sorted based on the feature vectors of the words and the pictures, and the image-text information corresponding to the nodes with top-n weight scores is extracted as key image-text information, so that the accuracy of the extracted key information is high, and the completeness of the extracted key information is ensured.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of image and text information extraction, specifically to a method, apparatus, device, and storage medium for extracting key image and text information. Background Technology

[0002] Key information in text and images, as representative information within an article or document, has wide applications in text classification, keyword matching, document and sentence similarity matching, intelligent search, recommendation, and intelligent dialogue. With the explosive growth of information, extracting key information from text and images enables rapid understanding of a document from massive amounts of data, significantly improving information retrieval speed. Therefore, researching methods for extracting key information from text and images is of great significance.

[0003] Key information in text and images includes the document's keywords and key images. Keywords represent the words containing the most information in the document, enabling readers to quickly grasp the document's central idea. Key images represent pictures closely related to the document's keywords, allowing readers to quickly understand the document's basic structure. Traditional methods for extracting key information in text and images mostly extract keywords or key images from the document. Keyword extraction relies on the TextRank algorithm, which uses a co-occurrence window reflecting the relationships between local words to rank subsequent keywords and directly extract keywords from the document itself. However, the TextRank algorithm focuses only on the co-occurrence window between words, ignoring the influence of grammar and word order on the meaning of the text. It is not sensitive to the semantic expression of sentences, resulting in low accuracy in keyword extraction. Existing patent literature discloses a keyword extraction method. This method first obtains the text to be processed and performs word segmentation on the text to obtain candidate keywords corresponding to the text. Then, it searches for the word vectors corresponding to the candidate keywords in a word vector model, which includes the word vectors of the candidate keywords. Next, it constructs a word similarity matrix of the candidate keywords based on the word vectors. Finally, it sorts the candidate keywords based on the word similarity matrix to extract the keywords of the text to be processed. However, existing image and text key information extraction methods only extract keywords or key images from the document, ignoring the relationship between images and text in the document. They cannot extract both keywords and key images from the document at the same time, resulting in incomplete extraction of key information from the document. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a method, apparatus, device and storage medium for extracting key information from images and text, to solve the problems of low accuracy and incomplete extraction of key information in the prior art.

[0005] The method includes: according to one aspect of the present invention, a method for extracting key information from images and text is provided, the method including:

[0006] Obtain the document, preprocess the document to obtain graphic and textual information composed of images and words;

[0007] The pre-trained word vector extraction model and graph vector extraction model are invoked to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively.

[0008] A graph network model is constructed using the text and image information in the document as nodes and the relationships between the text and image information as edges.

[0009] Calculate the weight scores of nodes in the graph network model based on the feature vectors of words and images;

[0010] The nodes are sorted by their weight scores, and the image and text information corresponding to the top-n nodes by weight score is taken as the image and text key information of the document, where n is a positive integer.

[0011] In one alternative approach, preprocessing the document to obtain graphic information consisting of images and words includes:

[0012] The document is divided to obtain the original text and original images within it;

[0013] The original text is divided into several sentences, each sentence is segmented into words, and stop words are removed from each sentence to obtain the words in the document.

[0014] The original image is resized to a preset size to obtain the image in the document.

[0015] In one alternative approach, the graph-text information relationships include word relationships and word-graph relationships. The step of constructing a graph network model using graph-text information in the document as nodes and graph-text information relationships as edges includes:

[0016] Use words and images as nodes in a graph network model;

[0017] Edges are set for nodes that have word relationships or word-graph relationships to construct a graph network model.

[0018] In one alternative approach, the calculation of weight scores for nodes in the graph network model based on word feature vectors and image feature vectors includes:

[0019] The inner product of the feature vectors of words between nodes in the graph network model, and the inner product of the feature vectors of words and images are used as the weights of the edges in the graph network model.

[0020] Set the initial values ​​for the weights of the edges between nodes and the weight scores of the nodes;

[0021] Using the weights of the edges between nodes and the initial weight scores of the nodes, the weight scores of the nodes are calculated iteratively.

[0022] Determine whether the weight score obtained in each iteration converges to the preset threshold. If so, stop the next iteration calculation of the node's weight score and output the final weight score of the node; otherwise, execute the next iteration calculation of the node's weight score until the weight score obtained in the iteration calculation converges to the preset threshold.

[0023] In one optional approach, the key image and text information includes key images and keywords; the step of using the image and text information corresponding to the top-n nodes by weight score as the key image and text information of the document includes:

[0024] The words corresponding to the top-n nodes in terms of weight score are used as the keywords, and the images corresponding to the top-n nodes in terms of weight score are used as the key images.

[0025] In one alternative approach, the word vector extraction model is an ELMO model, which consists of a bidirectional LSTM network structure.

[0026] In one alternative approach, the graph vector extraction model is a convolutional neural network model, which consists of sequentially connected convolutional layers, pooling layers, and fully connected layers.

[0027] According to another aspect of the present invention, a device for extracting key information from images and text is provided, comprising: a document processing module, a feature vector extraction module, a graph network model construction module, a weight calculation unit module, and a key information extraction module from images and text;

[0028] The document processing module is used to acquire documents, preprocess the documents, and obtain graphic information composed of images and words;

[0029] The feature vector extraction module is used to call the pre-trained word vector extraction model and graph vector extraction model to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively.

[0030] The graph network model construction module is used to construct a graph network model using the text and image information in the document as nodes and the relationships between the text and image information as edges.

[0031] The weight calculation unit module is used to calculate the weight scores of nodes in the graph network model based on the feature vectors of words and images.

[0032] The image and text key information extraction module is used to sort the weight scores of nodes and take the image and text information corresponding to the top-n nodes as the image and text key information of the document, where n is a positive integer.

[0033] According to another aspect of the present invention, a device for extracting key information from images and text is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0034] The memory is used to store at least one executable instruction, which causes the processor to perform the operation of the image and text key information extraction method.

[0035] The present invention also proposes a computer-readable storage medium storing at least one executable instruction, which, when executed on a key information extraction device, causes the key information extraction device to perform the operation of a key information extraction method.

[0036] This invention first preprocesses the document to obtain image and text information composed of images and words, achieving the goal of simultaneously obtaining images and words, avoiding the drawback of missing information due to extracting only words or images from the document. Then, it calls a trained word vector extraction model and a graph vector extraction model to receive the image and text information, facilitating the extraction of feature vectors of words and images from the image and text information. Next, it constructs a graph network model using the image and text information in the document as nodes and the relationships between image and text information as edges, enabling the graph network model to associate images and words in the document. Furthermore, based on the feature vectors of words and images, it calculates and ranks the weight scores of nodes in the graph network model, extracting the image and text information corresponding to the top-n nodes as key image and text information. This allows the extracted key image and text information to completely help readers quickly understand the central idea and basic ideas of the document, achieving high accuracy in extracting key image and text information and ensuring the completeness of the extracted key image and text information.

[0037] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0038] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, the entire drawing...

[0039] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0040] Figure 1 A flowchart illustrating a first embodiment of a method for extracting key information from images and text provided by the present invention is shown.

[0041] Figure 2 This diagram illustrates the principle block diagram of a first embodiment of a method for extracting key information from images and text provided by the present invention.

[0042] Figure 3 A schematic diagram of the graph network model provided in the first embodiment of the present invention is shown;

[0043] Figure 4 A schematic diagram of the structure of a first embodiment of a graphic key information extraction device provided by the present invention is shown.

[0044] Figure 5 The diagram shows a structural schematic of an embodiment of a key information extraction device for images and text provided by the present invention. Detailed Implementation

[0045] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0046] Figure 1 A flowchart of a first embodiment of a method for extracting key information from images and text provided by the present invention is shown. This method is executed by an image and text key information extraction device. Figure 1 As shown, the method includes the following steps:

[0047] Step 110: Obtain the document, preprocess the document to obtain graphic information composed of images and words;

[0048] Specifically, when acquiring documents, a preset time unit can be used as an extraction cycle to extract the text and image information of each document. The preset time unit can be set according to actual needs, such as one second, thirty seconds, one minute, half an hour, one hour, or one day. The text and image information is used to reflect the specific text and image content of the document. Through the text and image information, readers can grasp the overall layout and ideas of the document from a global perspective, achieving the goal of understanding a document in detail. As you can understand, each document has corresponding text and image information in each extraction cycle. The source of each document is not limited, such as obtaining documents through the websites of government agencies or publishing institutions, obtaining documents through search engines, obtaining documents through full-text databases, or obtaining documents through free electronic journal websites, etc.

[0049] The preprocessing of the document to obtain graphic information composed of images and words includes:

[0050] See Figure 1 and Figure 2 The document d is divided to obtain the original text and original images in the document;

[0051] The original text is divided into n sentences, resulting in d = [S1, S2, S3, ..., S...]. n ], where S n S represents the nth sentence; for each sentence S n Perform word segmentation and remove S from each sentence. n The stop words are obtained from the document, resulting in: S i =[w i1 ,w i2 ,w i3 ,……,w in ], where w in Words that represent each sentence;

[0052] The original image is resized to a preset size to obtain the image in the document.

[0053] Step 120: Call the pre-trained word vector extraction model and graph vector extraction model to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively;

[0054] Among them, see Figure 1 The word vector extraction model is one of the following models: ELMO model, BERT model, Word2Vec model, and LDA model. Step 120 only takes the ELMO model as an example, but it is not limited to the ELMO model. The ELMO model is a pre-trained model, similar to the BERT model. The ELMO model can be downloaded and used directly, or it can be trained using the acquired image and text information.

[0055] The ELMO model consists of a bidirectional LSTM network structure. This structure can fully learn the semantics of sentences, compensating for the shortcomings of the Bag-of-Words (BBO) model (TextRank), which ignores the influence of grammar and word order on the meaning of text and is not sensitive to the semantic representation of sentences, resulting in low keyword extraction accuracy. LSTM networks, also known as Long Short-Term Memory structures, are a variant of traditional RNNs. Compared with classic RNNs, they can effectively capture semantic relationships between long sequences and alleviate gradient vanishing or exploding phenomena. At the same time, the structure of LSTM networks is more complex; an LSTM network structure consists of multiple sequentially connected basic units A, each with input and output. Each basic unit A includes a forget gate, an input gate, and an output gate. The input gates to basic unit A are C. t-1 h t-1 and X t The basic unit A outputs C. t and h t The goalkeeper selectively forgets information from the previous step.

[0056] Forget gate: ft = σ*(Wf*[ht-1,xt]+bf), selectively forgets the information from the previous step;

[0057] Input gate: it = σ*(Wi*[ht-1,xt]+bi)*tanh(Wc*[ht-1,xt]+bc);

[0058] At this point, update the state of C: Ct = Ct-1*ft + it;

[0059] Output gate: ot=σ*(Wo*[ht-1,xt]+bo)ht=ot*tanh(Ct)

[0060] During the input process, information from the preceding part is passed to the following part, so that the output word vector will have semantic meaning.

[0061] The image vector extraction model described in step 120 is a convolutional neural network (CNN) model. This CNN model consists of sequentially connected convolutional layers, pooling layers, and fully connected layers. The images segmented from the text information undergo multiple layers of convolution, pooling, and fully connected layers to reduce image dimensionality, ultimately transforming them into a one-dimensional vector. This one-dimensional vector contains the image's features; however, these features are not those visible to the naked eye, but rather features specific to the CNN model. The CNN model primarily employs convolution and pooling operations. During convolution, the image's features are continuously extracted through convolution and sum operations; convolution and sum are equivalent to feature extractors. During pooling, because the model's parameters are independent of image size while the computational load is related to image size, pooling significantly reduces the computational load by reducing the image size.

[0062] The graph network model proposed in step 120 integrates the TextRank algorithm, the ELMO model, and the convolutional neural network model. The TextRank algorithm is a graph-based ranking algorithm model for keyword extraction and document summarization. It is an improvement on Google's PageRank algorithm for ranking the importance of web pages. It extracts keywords by utilizing the co-occurrence information or semantics between words within a document. It can extract the keywords and keyword groups of a given text and use an extractive automatic summarization method to extract the key sentences of the text. The basic idea of ​​TextRank is to regard a document as a network of words, in which the links represent the semantic relationships between words. The basic steps of the TextRank algorithm are as follows: First, the given text d is divided into n sentences. Then, each sentence is analyzed and stop words are removed to obtain the words contained in each sentence. Next, a word graph is constructed, where V is the set of nodes composed of generated words. Then, edges between any two nodes are constructed using co-occurrence relations: an edge exists between two nodes only if their corresponding words co-occur within a window of length K, where K represents the window size, i.e., at most K words co-occur, typically K is 2. Finally, the weights of the nodes are calculated iteratively until convergence. Each node has an initial weight, and the weights between the nodes are... The edges connecting the nodes also have different weights. The weight of each node is calculated iteratively through edge transmission until it stabilizes, thus obtaining the weight of each node. Finally, the weights of the nodes are sorted to obtain the most important t words, which are used as the top-t keywords. In the process of using the TextRank algorithm model, the co-occurrence of words in a window is independent of the order of words, which is a bag-of-words model. The order in which words appear in a sentence has no effect on the weight of the edges. Therefore, the TextRank algorithm model has drawbacks when constructing the graph network model and calculating the weights between nodes, and it is not sensitive to the semantic representation of the sentence.

[0063] To address the shortcomings of the traditional TextRank algorithm, an ELMO model and a convolutional neural network model are incorporated into the graph network model construction. Firstly, after establishing the graph network model, the ELMO model is introduced to calculate the feature vectors of the words corresponding to the nodes, thereby calculating the weights between word nodes. ELMO, by incorporating semantic information, compensates for the shortcomings of the TextRank bag-of-words model. The bag-of-words model considers the co-occurrence of words within a window regardless of word order; therefore, the graph network model is not a bag-of-words model, making the final calculation results more accurate. Secondly, the TextRank bag-of-words model cannot connect images and text. In this graph network model, image nodes are added. The image nodes, processed by the convolutional neural network model, provide feature vectors, allowing for the calculation of weights between image nodes and word nodes. In summary, the graph network model constructed in step 120 includes both text and images, and the weights between nodes also incorporate semantics, resulting in high accuracy in extracting key information from the text and images, ensuring the completeness of the extracted key information.

[0064] Step 130: Construct a graph network model using the text and image information in the document as nodes and the relationships between the text and image information as edges;

[0065] In step 130, see Figure 1 and Figure 2 The graph-text information relationships include word relationships and word-graph relationships. The construction of a graph network model using graph-text information in the document as nodes and graph-text information relationships as edges includes:

[0066] Use words and images as nodes in a graph network model;

[0067] Edges are set for nodes with word relationships or word-graph relationships to construct a graph network model; among them, Figure 3 Chinese w i Nodes representing words, Gi representing image nodes, t i This represents the weight of the edge between any two nodes.

[0068] Step 140: Calculate the weight scores of nodes in the graph network model based on the feature vectors of words and images;

[0069] Step 140, which calculates the weight scores of nodes in the graph network model based on the feature vectors of words and images, includes:

[0070] The inner product of the feature vectors of words between nodes and the inner product of the feature vectors of words and images are used as the weights of the edges between nodes in the graph network model.

[0071] Set the initial values ​​for the weights of edges between nodes and the weight scores of nodes; the weights of edges between nodes are set randomly.

[0072] Using the weights of the edges between nodes and the initial weight scores of the nodes, the weight scores of the nodes are calculated iteratively; the specific process of iteratively calculating the weight scores of the nodes is as follows:

[0073] The initial weight scores of the nodes are used to form the first weight score matrix, and the weights of the edges between the nodes are used to form the weight matrix. The first weight score matrix and the weight matrix are multiplied to obtain the iterative matrix containing the weight scores of the nodes. The next iteration is calculated by multiplying the iterative matrix obtained in the previous iteration with the weight matrix.

[0074] Determine whether the weight score obtained in each iteration converges to the preset threshold. If so, stop the next iteration calculation of the node's weight score and output the final weight score of the node; otherwise, execute the next iteration calculation of the node's weight score until the weight score obtained in the iteration calculation converges to the preset threshold.

[0075] The weight score obtained in each iteration is derived from the iteration matrix. The final node weight score is determined by checking whether the weight scores in the iteration matrix converge to a preset threshold. The specific formula for calculating the node weight score during the iterative calculation process is as follows:

[0076]

[0077] Among them, PR(V i ), PR(V j ) represent nodes V in the graph network model. i V j The weight scores, the graph network model is a directed graph, In(V i ) represents node V in the graph network model i The set of predecessor nodes, that is, pointing to node V in the graph network model. i Let j represent the set of predecessor nodes In(V) i The element of Out(V) j ) represents node V in the graph network model j The set of successor nodes, i.e., node V in the graph network model. j The set of points to other nodes, where D represents the Zuni coefficient, typically 0.85. Essentially, each node in a graph network model has an initial weight, and the edges connecting nodes in different graph network models also have different weights. The weight of each node is iteratively calculated through edge transmission until it stabilizes, thus obtaining the weight score for each node.

[0078] Step 150: Sort the weight scores of the nodes, and take the image and text information corresponding to the top-n nodes as the image and text key information of the document, where n is a positive integer.

[0079] The key text and image information includes key images and keywords; the step of using the text and image information corresponding to the top-n nodes by weight score as the key text and image information of the document includes:

[0080] The words corresponding to the top-n nodes by weight score are used as the keywords, and the images corresponding to the top-n nodes by weight score are used as the key images. Specifically, the weight scores of the nodes are sorted in two ways: either by sorting the weight scores from largest to smallest in a forward order, and extracting the image and text information corresponding to the top-n nodes before the weight score sort as key image and text information; or by sorting the weight scores from smallest to largest in a reverse order, and extracting the image and text information corresponding to the bottom n nodes as key image and text information. In practice, the weight score sorting method needs to be selected according to the actual situation.

[0081] In the method for extracting key information from images and text, the document is first preprocessed to obtain image and text information composed of images and words, achieving the goal of simultaneously obtaining images and words and avoiding the drawback of missing information due to extracting only words or images from the document. Then, a word vector extraction model and a graph vector extraction model are used to receive the image and text information, facilitating the extraction of feature vectors of words and images from the image and text information. Next, a graph network model is constructed using the image and text information in the document as nodes and the relationships between image and text information as edges, enabling the graph network model to associate images and words in the document. Finally, based on the feature vectors of words and images, the key information is calculated... The method involves ranking the weight scores of nodes in a graph network model and extracting the textual and graphical information corresponding to the top-n nodes as key textual and graphical information. This ensures that the extracted key textual and graphical information can help readers quickly understand the central idea and basic structure of a document. The method has a high accuracy rate and guarantees the completeness of the extracted key textual and graphical information. It should be noted that the key textual and graphical information extraction method provided in this embodiment can be applied to other application scenarios besides academic journals, such as e-magazines, e-book reading, news reports, and animation documents. The application scenarios of the key textual and graphical information extraction method provided in this embodiment are not limited here.

[0082] Figure 4 A schematic diagram of the structure of a first embodiment of a graphic key information extraction device provided by the present invention is shown. Figure 4As shown, the device 400 includes: a document processing module 410, a feature vector extraction module 420, a graph network model construction module 430, a weight calculation unit module 440, and a key information extraction module for images and text 450;

[0083] The document processing module 410 is used to acquire a document, preprocess the document, and obtain graphic information composed of images and words;

[0084] The feature vector extraction module 420 is used to call the pre-trained word vector extraction model and graph vector extraction model to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively.

[0085] The graph network model construction module 430 is used to construct a graph network model using the text and image information in the document as nodes and the text and image information relationships as edges.

[0086] The weight calculation unit module 440 is used to calculate the weight scores of nodes in the graph network model based on the feature vectors of words and the feature vectors of images.

[0087] The image and text key information extraction module 450 is used to sort the weight scores of nodes and take the image and text information corresponding to the top-n nodes as the image and text key information of the document, where n is a positive integer.

[0088] In one alternative approach, preprocessing the document to obtain graphic information consisting of images and words includes:

[0089] The document is divided to obtain the original text and original images within it;

[0090] The original text is divided into several sentences, each sentence is segmented into words, and stop words are removed from each sentence to obtain the words in the document.

[0091] The original image is resized to a preset size to obtain the image in the document.

[0092] In one alternative approach, the graph-text information relationships include word relationships and word-graph relationships. The step of constructing a graph network model using graph-text information in the document as nodes and graph-text information relationships as edges includes:

[0093] Use words and images as nodes in a graph network model;

[0094] Edges are set for nodes that have word relationships or word-graph relationships to construct a graph network model.

[0095] In one alternative approach, the calculation of weight scores for nodes in the graph network model based on word feature vectors and image feature vectors includes:

[0096] The inner product of the feature vectors of words between nodes in the graph network model, and the inner product of the feature vectors of words and images are used as the weights of the edges in the graph network model.

[0097] Set the initial values ​​for the weights of the edges between nodes and the weight scores of the nodes;

[0098] Using the weights of the edges between nodes and the initial weight scores of the nodes, the weight scores of the nodes are calculated iteratively.

[0099] Determine whether the weight score obtained in each iteration converges to the preset threshold. If so, stop the next iteration calculation of the node's weight score and output the final weight score of the node. Otherwise, execute the next iteration calculation of the node's weight score until the weight score obtained in the iteration calculation converges to the preset threshold.

[0100] In one optional approach, the key image and text information includes key images and keywords; the step of using the image and text information corresponding to the top-n nodes by weight score as the key image and text information of the document includes:

[0101] The words corresponding to the top-n nodes in terms of weight score are used as the keywords, and the images corresponding to the top-n nodes in terms of weight score are used as the key images.

[0102] In one alternative approach, the word vector extraction model is an ELMO model, which consists of a bidirectional LSTM network structure.

[0103] In one alternative approach, the graph vector extraction model is a convolutional neural network model, which consists of sequentially connected convolutional layers, pooling layers, and fully connected layers.

[0104] In this embodiment, the document processing module first preprocesses the document to obtain image and text information composed of images and words, achieving the goal of simultaneously obtaining images and words, avoiding the drawback of missing information due to extracting only words or images from the document. Then, the feature vector extraction module uses word vector extraction model and graph vector extraction model to receive image and text information, facilitating the extraction of feature vectors of words and images from the image and text information. Next, the graph network model construction module uses the image and text information in the document as nodes and the relationships between image and text information as edges to construct a graph network model, enabling the graph network model to associate images and words in the document. Further, the weight calculation unit module calculates and ranks the weight scores of nodes in the graph network model based on the feature vectors of words and images. Finally, the image and text key information extraction module extracts the image and text information corresponding to the top-n nodes with the top weight scores as image and text key information, so that the extracted image and text key information can completely help readers quickly understand the central idea and basic ideas of the document. The accuracy of image and text key information extraction is high, ensuring the completeness of the extracted image and text key information.

[0105] Figure 5 The diagram illustrates a structural schematic of an embodiment of a key information extraction device for images and text provided by the present invention. The specific embodiments of the present invention do not limit the specific implementation of such a device. Figure 5 As shown, the image and text key information extraction device may include: processor 502, communication interface 504, memory 506, and communication bus 508.

[0106] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508. Communication interface 504 is used to communicate with other network elements such as clients or other servers. The processor 502 executes program 510, specifically performing the relevant steps described in the embodiment of the image and text key information extraction method.

[0107] Specifically, program 510 may include program code, which includes computer-executable instructions. Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The image and text key information extraction device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0108] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0109] Specifically, program 510 can be called by processor 502 to cause the image and text key information extraction device to perform the following operations:

[0110] Obtain the document, preprocess the document to obtain graphic and textual information composed of images and words;

[0111] The pre-trained word vector extraction model and graph vector extraction model are invoked to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively.

[0112] A graph network model is constructed using the text and image information in the document as nodes and the relationships between the text and image information as edges.

[0113] Calculate the weight scores of nodes in the graph network model based on the feature vectors of words and images;

[0114] The nodes are sorted by their weight scores, and the image and text information corresponding to the top-n nodes by weight score is taken as the image and text key information of the document, where n is a positive integer.

[0115] In one alternative approach, preprocessing the document to obtain graphic information consisting of images and words includes:

[0116] The document is divided to obtain the original text and original images within it;

[0117] The original text is divided into several sentences, each sentence is segmented into words, and stop words are removed from each sentence to obtain the words in the document.

[0118] The original image is resized to a preset size to obtain the image in the document.

[0119] In one alternative approach, the graph-text information relationships include word relationships and word-graph relationships. The step of constructing a graph network model using graph-text information in the document as nodes and graph-text information relationships as edges includes:

[0120] Use words and images as nodes in a graph network model;

[0121] Edges are set for nodes that have word relationships or word-graph relationships to construct a graph network model.

[0122] In one alternative approach, the calculation of weight scores for nodes in the graph network model based on word feature vectors and image feature vectors includes:

[0123] The inner product of the feature vectors of words between nodes in the graph network model, and the inner product of the feature vectors of words and images are used as the weights of the edges in the graph network model.

[0124] Set the initial values ​​for the weights of the edges between nodes and the weight scores of the nodes;

[0125] Using the weights of the edges between nodes and the initial weight scores of the nodes, the weight scores of the nodes are calculated iteratively.

[0126] Determine whether the weight score obtained in each iteration converges to the preset threshold. If so, stop the next iteration calculation of the node's weight score and output the final weight score of the node. Otherwise, execute the next iteration calculation of the node's weight score until the weight score obtained in the iteration calculation converges to the preset threshold.

[0127] In one optional approach, the key image and text information includes key images and keywords; the step of using the image and text information corresponding to the top-n nodes by weight score as the key image and text information of the document includes:

[0128] The words corresponding to the top-n nodes in terms of weight score are used as the keywords, and the images corresponding to the top-n nodes in terms of weight score are used as the key images.

[0129] In one alternative approach, the word vector extraction model is an ELMO model, which consists of a bidirectional LSTM network structure.

[0130] In one alternative approach, the graph vector extraction model is a convolutional neural network model, which consists of sequentially connected convolutional layers, pooling layers, and fully connected layers.

[0131] According to another aspect of the present invention, a device for extracting key information from images and text is provided, comprising: a document processing module, a feature vector extraction module, a graph network model construction module, a weight calculation unit module, and a key information extraction module from images and text;

[0132] The document processing module is used to acquire documents, preprocess the documents, and obtain graphic information composed of images and words;

[0133] The feature vector extraction module is used to call the pre-trained word vector extraction model and graph vector extraction model to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively.

[0134] The graph network model construction module is used to construct a graph network model using the text and image information in the document as nodes and the relationships between the text and image information as edges.

[0135] The weight calculation unit module is used to calculate the weight scores of nodes in the graph network model based on the feature vectors of words and images.

[0136] The image and text key information extraction module is used to sort the weight scores of nodes and take the image and text information corresponding to the top-n nodes as the image and text key information of the document, where n is a positive integer.

[0137] In this embodiment, the document is first preprocessed to obtain image and text information composed of images and words, achieving the goal of simultaneously obtaining images and words and avoiding the drawback of missing information due to extracting only words or images from the document. Then, a trained word vector extraction model and graph vector extraction model are called to receive the image and text information, facilitating the extraction of feature vectors of words and images from the image and text information. Next, a graph network model is constructed using the image and text information in the document as nodes and the relationships between image and text information as edges, enabling the graph network model to associate images and words in the document. Furthermore, based on the feature vectors of words and images, the weight scores of nodes in the graph network model are calculated and ranked, and the image and text information corresponding to the top-n nodes with the top weight scores is extracted as key image and text information. This ensures that the extracted key image and text information can completely help readers quickly understand the central idea and basic ideas of the document, achieving high accuracy in extracting key image and text information and guaranteeing the completeness of the extracted key image and text information.

[0138] This invention provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on a graphic and textual key information device / app, the graphic and textual key information device / app performs the graphic and textual key information method in any of the above method embodiments.

[0139] Specifically, the executable instructions can be used to cause the device / app that displays the key graphic information to perform the following operations:

[0140] Obtain the document, preprocess the document to obtain graphic and textual information composed of images and words;

[0141] The pre-trained word vector extraction model and graph vector extraction model are invoked to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively.

[0142] A graph network model is constructed using the text and image information in the document as nodes and the relationships between the text and image information as edges.

[0143] Calculate the weight scores of nodes in the graph network model based on the feature vectors of words and images;

[0144] The nodes are sorted by their weight scores, and the image and text information corresponding to the top-n nodes by weight score is taken as the image and text key information of the document, where n is a positive integer.

[0145] In one alternative approach, preprocessing the document to obtain graphic information consisting of images and words includes:

[0146] The document is divided to obtain the original text and original images within it;

[0147] The original text is divided into several sentences, each sentence is segmented into words, and stop words are removed from each sentence to obtain the words in the document.

[0148] The original image is resized to a preset size to obtain the image in the document.

[0149] In one alternative approach, the graph-text information relationships include word relationships and word-graph relationships. The step of constructing a graph network model using graph-text information in the document as nodes and graph-text information relationships as edges includes:

[0150] Use words and images as nodes in a graph network model;

[0151] Edges are set for nodes that have word relationships or word-graph relationships to construct a graph network model.

[0152] In one alternative approach, the calculation of weight scores for nodes in the graph network model based on word feature vectors and image feature vectors includes:

[0153] The inner product of the feature vectors of words between nodes in the graph network model, and the inner product of the feature vectors of words and images are used as the weights of the edges in the graph network model.

[0154] Set the initial values ​​for the weights of the edges between nodes and the weight scores of the nodes;

[0155] Using the weights of the edges between nodes and the initial weight scores of the nodes, the weight scores of the nodes are calculated iteratively.

[0156] Determine whether the weight score obtained in each iteration converges to the preset threshold. If so, stop the next iteration calculation of the node's weight score and output the final weight score of the node. Otherwise, execute the next iteration calculation of the node's weight score until the weight score obtained in the iteration calculation converges to the preset threshold.

[0157] In one optional approach, the key image and text information includes key images and keywords; the step of using the image and text information corresponding to the top-n nodes by weight score as the key image and text information of the document includes:

[0158] The words corresponding to the top-n nodes in terms of weight score are used as the keywords, and the images corresponding to the top-n nodes in terms of weight score are used as the key images.

[0159] In one alternative approach, the word vector extraction model is an ELMO model, which consists of a bidirectional LSTM network structure.

[0160] In one alternative approach, the graph vector extraction model is a convolutional neural network model, which consists of sequentially connected convolutional layers, pooling layers, and fully connected layers.

[0161] According to another aspect of the present invention, a device for extracting key information from images and text is provided, comprising: a document processing module, a feature vector extraction module, a graph network model construction module, a weight calculation unit module, and a key information extraction module from images and text;

[0162] The document processing module is used to acquire documents, preprocess the documents, and obtain graphic information composed of images and words;

[0163] The feature vector extraction module is used to call the pre-trained word vector extraction model and graph vector extraction model to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively.

[0164] The graph network model construction module is used to construct a graph network model using the text and image information in the document as nodes and the relationships between the text and image information as edges.

[0165] The weight calculation unit module is used to calculate the weight scores of nodes in the graph network model based on the feature vectors of words and images.

[0166] The image and text key information extraction module is used to sort the weight scores of nodes and take the image and text information corresponding to the top-n nodes as the image and text key information of the document, where n is a positive integer.

[0167] In this embodiment, the document is first preprocessed to obtain image and text information composed of images and words, achieving the goal of simultaneously obtaining images and words and avoiding the drawback of missing information due to extracting only words or images from the document. Then, a trained word vector extraction model and graph vector extraction model are called to receive the image and text information, facilitating the extraction of feature vectors of words and images from the image and text information. Next, a graph network model is constructed using the image and text information in the document as nodes and the relationships between image and text information as edges, enabling the graph network model to associate images and words in the document. Furthermore, based on the feature vectors of words and images, the weight scores of nodes in the graph network model are calculated and ranked, and the image and text information corresponding to the top-n nodes with the top weight scores is extracted as key image and text information. This ensures that the extracted key image and text information can completely help readers quickly understand the central idea and basic ideas of the document, achieving high accuracy in extracting key image and text information and guaranteeing the completeness of the extracted key image and text information.

[0168] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Furthermore, the embodiments of this invention are not directed to any particular programming language.

[0169] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. Similarly, for the sake of brevity and to aid in understanding one or more aspects of the invention, in the description of exemplary embodiments of the invention above, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0170] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

[0171] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for extracting key information from images and text, characterized in that, The method includes: Obtain the document, preprocess the document to obtain graphic and textual information composed of images and words; The pre-trained word vector extraction model and graph vector extraction model are invoked to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively. A graph network model is constructed using the text and image information in the document as nodes and the relationships between the text and image information as edges. Calculate the weight scores of nodes in the graph network model based on the feature vectors of words and images; The nodes are sorted by their weight scores, and the image and text information corresponding to the top-n nodes by weight score is taken as the image and text key information of the document, where n is a positive integer.

2. The method according to claim 1, characterized in that, The preprocessing of the document to obtain graphic information composed of images and words includes: The document is divided to obtain the original text and original images within it; The original text is divided into several sentences, each sentence is segmented into words, and stop words are removed from each sentence to obtain the words in the document. The original image is resized to a preset size to obtain the image in the document.

3. The method according to claim 2, characterized in that, The graph-text information relationships include word relationships and word-graph relationships. The construction of a graph network model using graph-text information in the document as nodes and graph-text information relationships as edges includes: Use words and images as nodes in a graph network model; Edges are set for nodes that have word relationships or word-graph relationships to construct a graph network model.

4. The method according to claim 2, characterized in that, The calculation of weight scores for nodes in the graph network model based on word-based feature vectors and image-based feature vectors includes: The inner product of the feature vectors of words between nodes in the graph network model, and the inner product of the feature vectors of words and images are used as the weights of the edges in the graph network model. Set the initial values ​​for the weights of the edges between nodes and the weight scores of the nodes; Using the weights of the edges between nodes and the initial weight scores of the nodes, the weight scores of the nodes are calculated iteratively. Determine whether the weight score obtained in each iteration converges to the preset threshold. If so, stop the next iteration calculation of the node's weight score and output the final weight score of the node. Otherwise, execute the next iteration calculation of the node's weight score until the weight score obtained in the iteration calculation converges to the preset threshold.

5. The method according to claim 2, characterized in that, The key text and image information includes key images and keywords; the step of using the text and image information corresponding to the top-n nodes by weight score as the key text and image information of the document includes: The words corresponding to the top-n nodes in terms of weight score are used as the keywords, and the images corresponding to the top-n nodes in terms of weight score are used as the key images.

6. The method for extracting key information from images and text according to claim 1, characterized in that, The word vector extraction model is the ELMO model, which consists of a bidirectional LSTM network structure.

7. The method for extracting key information from images and text according to claim 1, characterized in that, The graph vector extraction model is a convolutional neural network model, which consists of sequentially connected convolutional layers, pooling layers, and fully connected layers.

8. A device for extracting key information from images and text, characterized in that, include: The module includes a document processing module, a feature vector extraction module, a graph network model construction module, a weight calculation unit module, and a key information extraction module for text and images. The document processing module is used to acquire documents, preprocess the documents, and obtain graphic information composed of images and words; The feature vector extraction module is used to call the pre-trained word vector extraction model and graph vector extraction model to extract the feature vectors of words in the text and image information and the feature vectors of images in the text and image information, respectively. The graph network model construction module is used to construct a graph network model using the text and image information in the document as nodes and the relationships between the text and image information as edges. The weight calculation unit module is used to calculate the weight scores of nodes in the graph network model based on the feature vectors of words and images. The image and text key information extraction module is used to sort the weight scores of nodes and take the image and text information corresponding to the top-n nodes as the image and text key information of the document, where n is a positive integer.

9. A device for extracting key information from images and text, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation of the image and text key information extraction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on the image and text key information extraction device, causes the image and text key information extraction device to perform the operation of the image and text key information extraction method as described in any one of claims 1-7.

Images