Keyword extraction method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application provide a keyword extraction method and device, electronic equipment and computer readable storage medium, relating to the field of artificial intelligence. The method comprises: extracting feature information corresponding to a to-be-processed image, and then extracting a keyword from the to-be-processed image based on the feature information. The keyword extraction method, device, electronic equipment and computer readable storage medium provided by the embodiments of the present application realize the extraction of a keyword from a to-be-processed image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular, the present application relates to a keyword extraction method and device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the development of information technology, image processing technology has also developed, and now more and more information can be displayed through images, and images are also used more and more widely in our life, for example, mobile phone screenshots, posters, bills and the like are in the form of images.

[0003] Among them, since these images contain a lot of important information, we usually need to obtain key information from these images to help users edit picture tags and quickly search for pictures, for example, if a user inputs "The Hobbit" to find images containing "The Hobbit" from multiple images, if the keywords can be extracted from these images, the target image can be quickly found from these images by inputting "The Hobbit" by the user.

[0004] Therefore, how to extract keywords from images has become an important problem. SUMMARY

[0005] The present application provides a keyword extraction method, device, electronic device and computer readable storage medium, which can solve at least one of the above technical problems. The technical solution is as follows:

[0006] In a first aspect, a keyword extraction method is provided, the method comprising:

[0007] extracting feature information corresponding to a to-be-processed image;

[0008] extracting a keyword from the to-be-processed image based on the feature information.

[0009] In a second aspect, a keyword extraction device is provided, the device comprising:

[0010] an extraction module configured to extract feature information corresponding to a to-be-processed image;

[0011] an extraction module configured to extract a keyword from the to-be-processed image based on the feature information.

[0012] In a third aspect, an electronic device is provided, comprising:

[0013] one or more processors;

[0014] a memory;

[0015] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs are configured to perform operations corresponding to the keyword extraction method shown in the first aspect.

[0016] In a fourth aspect, a computer readable storage medium, having stored thereon a computer program, wherein the program is executed by a processor to implement the keyword extraction method shown in the first aspect.

[0017] The technical scheme provided by the present application has the beneficial effects that:

[0018] The present application provides a keyword extraction method, device, electronic equipment and computer readable storage medium, compared with the prior art, in the present application, the feature information corresponding to the to-be-processed image is extracted, and then the keyword is extracted from the to-be-processed image based on the feature information. That is, in the present application, the feature information extracted from the to-be-processed image can be used to extract the keyword from the to-be-processed image. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced.

[0020] Figure 1 It is a traditional keyword extraction-based deep learning model framework schematic diagram;

[0021] Figure 2 It is a traditional picture keyword extraction process schematic diagram;

[0022] Figure 3 It is a schematic diagram of the present application embodiment for visual information loss and optical character recognition (Optical Character Recognition, OCR) recognition text error keyword extraction;

[0023] Figure 4a It is a visual keyword extraction framework schematic diagram provided by the present application embodiment;

[0024] Figure 4b It is a traditional screenshot keyword phrase extraction model framework schematic diagram;

[0025] Figure 5 It is a flowchart of a keyword extraction method provided by the present application embodiment;

[0026] Figure 6 It is a flowchart of a text and its visual information acquisition method based on an OCR framework provided by the present application embodiment;

[0027] Figure 7A schematic diagram of another specific architecture for keyword extraction provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of obtaining a text line representation provided in an embodiment of the present application;

[0029] Figure 9 A schematic diagram of obtaining a text background representation provided in an embodiment of the present application;

[0030] Figure 10 A schematic diagram of obtaining text structure information and topic information provided in an embodiment of the present application;

[0031] Figure 11 Schematic diagram of keyword prediction during the decoding process in an embodiment of the present application;

[0032] Figure 12 Schematic diagram of synthesizing corpus based on model training in an embodiment of the present application;

[0033] Figure 13 This is a schematic diagram of a keyword extraction model obtained by model training in an embodiment of the present application;

[0034] Figure 14 This is an example diagram of image tag editing based on keyword extraction in an embodiment of the present application;

[0035] Figure 15 This is an example diagram of intelligent search based on keyword extraction in an embodiment of the present application;

[0036] Figure 16 Schematic diagram of the Transformer network module in an embodiment of the present application;

[0037] Figure 17 This is a schematic diagram of the structure of the keyword extraction device provided in the embodiment of the present application;

[0038] Figure 18 This is a schematic diagram of the device structure of the electronic device provided in the embodiments of the present application. DETAILED DESCRIPTION

[0039] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present invention.

[0040] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0041] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0042] The method of extracting keywords from images can be implemented through a deep learning model based on keyword extraction. The traditional deep learning model based on keyword extraction conforms to such a framework, such as Figure 1 As shown, the framework can include: OCR tools and keyword extraction models. The operation of extracting keywords from images based on this framework can be divided into two steps:

[0043] 1) Use OCR tools to identify text lines in the image and string all text lines together as a text document;

[0044] 2) Keyword extraction model, using an encoder-decoder framework based on a recurrent neural network (RNN), namely:

[0045] a) Input document, i.e., OCR text, which is the text document with all the text lines strung together as mentioned above;

[0046] b) Encoder: Maps the input document to a hidden layer representation through RNN;

[0047] c) Decoder: The hidden layer representation of the encoder’s output is input into the RNN-based decoder for keyword prediction.

[0048] The above describes the specific method of extracting keywords from pictures using the traditional deep learning-based keyword extraction model. The following embodiment further introduces an example: Figure 2As shown, an image (a) is used to identify the text lines in the image through an OCR tool to obtain OCR text, as shown in (b), and then the keywords predicted by the keyword extraction model are shown in (c). In an embodiment of the present application, the predicted keywords include multiple keywords and the scores corresponding to each keyword. For example, the keywords predicted from (b) and the scores of the keywords include: "b*ooklyn(6.608)", "river(2.050)", "b*ooklyn bridge park(0.890)", "warmer weather(0.442)", "session(0.436)", "B*ooklyn bridge(0.046)", "bridge park(0.031)", "Jun 02(0.000)", "bridge(-0.515)" and "August 26(-0.620)".

[0049] However, the traditional deep learning-based keyword extraction model introduced in the above embodiment may have the following technical problems in extracting keywords from images:

[0050] 1. Technical problems caused by the traditional method of extracting keywords from images based on OCR:

[0051] (1.1) Traditional methods only rely on OCR text when extracting keywords, ignoring the visual information of the text, which results in low accuracy in extracting keywords from images. For example, Figure 3 middle,

[0052] a) The word "Beauty" is located at the top center of the image, which generally indicates that it is an important word. However, it is difficult to predict that "Beauty" is a keyword using only OCR text;

[0053] b) Since the text line “The M*er Sale…saving” has a prominent red background, the keyword candidate “M*er Sale” should have a higher ranking score in the text line “The M*er Sale…saving”. Therefore, only OCR text cannot indicate that the keyword candidate “M*er Sale” has a higher ranking score in the text line “The M*er Sale…saving”;

[0054] c) The text line "SUPER...MASOUE" is in a very small and light font in the image. Normally, the ranking score of the text line "SUPER...MASOUE" should be low. However, using traditional methods, "PIG HAIR MASOUE" may receive a high ranking score and thus be used as an extracted keyword.

[0055] d) If there is no text content or insufficient text content in the image, the traditional method cannot generate any keywords, so it is impossible to extract keywords from the image through the above traditional method.

[0056] (1.2) OCR text often has a lot of text recognition errors. If we directly process the erroneous text, we will generate inappropriate keywords. That is, the accuracy of extracting keywords from images based on OCR text is low. For example, Figure 3 (b) is the OCR text extracted from image (a), where the text "iffit...nu" in region 1 is the incorrect text generated by the OCR tool. Then, the extracted OCR text (b) is used to predict keywords using the traditional method mentioned above to obtain the predicted keywords, i.e. Figure 3 In the "Keywords predicted by existing methods" section, such as Figure 3 As shown in (c), many wrong keywords are extracted from the text due to OCR errors, such as "iii", "iup", "niunnuimiii", "iffit", etc. Moreover, Figure 3 (d) in FIG. 4 shows the expected keywords, including: "Beauty", "M*er", "S*nd&S*y", "C*co&E*e", "$59.90", "$64.90", "M*er Sale" and "A*stralian P*nk C*ay". Therefore, through Figure 3 Comparing the keywords predicted by traditional technology with the expected keywords, we found that the overlap between the keywords predicted by traditional technology and the expected keywords is low, which means that OCR errors reduce the accuracy of the traditional keyword extraction model;

[0057] 2. Defects of traditional deep learning keyword extraction models:

[0058] In the traditional deep learning keyword extraction model, an encoder and a decoder framework based on RNN is adopted. The encoder obtains word embedding processed content by word embedding on the document D (including: text line 1, text line 2, …, text line m, wherein text line 1, text line 2, …, text line m respectively represent the text content corresponding to each text line, i.e., the OCR text involved above), and then obtains the corresponding text background representation by RNN to realize the encoding process and obtain the encoding result. Then, the encoding result and the historical prediction result are decoded by the decoder to predict the corresponding keyword sequence. When predicting a word at each prediction time step, the decoder utilizes the RNN layer and combines the attention mechanism to compress the information of the text background representation to obtain the hidden layer representation of the decoding layer. Then, the probability distribution Prob(y, g) of all words in the common word table and the probability distribution Prob(y, c) on the uncommon words in the input text are added for prediction, and the prediction result is output. In the embodiment of the present application, y represents a word, and g and c respectively represent a first prediction mode (generation mode) and a second prediction mode (copy mode). Details are shown in Figure 4b

[0059] The above traditional framework has the following defects:

[0060] (2.1) The RNN encoder needs to be executed on sequence data, i.e., all text lines need to be sequentially concatenated together to form a text document as the input of the model, which will cause the loss of the spatial structure information existing in the text lines, such as the layout information of the text lines;

[0061] (2.2) The training and inference speed of the RNN encoder is relatively slow, because the RNN needs to wait for the processing of all previous time steps to end at each prediction time step, which causes the prediction speed to be relatively slow;

[0062] (2.3) The keyword phrases generated by the traditional RNN decoder do not have topic diversity.

[0063] For example, in Figure 3 , the picture contains theme information such as “product”, “website”, “price”, etc. The expected extracted keywords should cover these theme information as much as possible, such as “S*nd&S*y” (product), “M*er” (website), “$59.90” (price), etc. However, the keywords extracted by the existing method do not cover the “price” and “product” themes (see the extraction results listed in part (c) of Figure 3

[0064] ​​(2.4) When the decoder predicts a word at each prediction time step, it adds the probability distribution Prob(y,g) of all words in the common vocabulary and the probability distribution Prob(y,c) of uncommon words in the input text, i.e., Prob(y) = Prob(y,g) + Prob(y,c). This prediction method inevitably introduces prediction noise; when the predicted word is a common word, the uncommon word is noise, and vice versa, when the predicted word is an uncommon word, the common word is noise.

[0065] Based on the technical problems that may exist in the prior art described above, specific implementation methods are proposed in the embodiments of the present application to solve the above-mentioned technical problems:

[0066] 1) Establish a deep learning keyword extraction model that can utilize multimodal information such as text content, text visual information, and image visual information, and enhance keyword extraction through visual information and image information; and simultaneously extract text content, text visual information, and image information through an OCR framework. In an embodiment of the present application, the text visual information can be other observable information for a text line in addition to the text line content, such as at least one of: a text feature map (also referred to as text feature information), text position information, and word position; or the text visual information can be obtained through the above information; the image visual information can include an image feature map, or the image visual information can be obtained through an image feature map.

[0067] Among them, the text content information may include the text content information corresponding to each text line, and the text content information corresponding to each text line may include: the word sequence corresponding to each text line module in the picture; the text feature map is the visual feature of the text line obtained by using the convolutional network (ConvNet) (specifically, the text feature map is obtained by combining RPN and RoI on the basis of the image feature map obtained by the convolutional network (ConvNet), which encodes rich visual information of the text line, including font size, font color, font shape, text line background color, structural information, border information, etc.; the text position information is the position information corresponding to each text line in the picture; the word position is the position corresponding to each word in the corresponding text line; the image feature map, that is, the convolution feature of the image, is the encoding of image information, such as object information, scene information, action information, etc.

[0068] Among them, such as Figure 4aAs shown, in the embodiment of the present application, when extracting keywords from the image to be processed, the method includes: preprocessing the image to be processed to obtain text content, text feature map (also called text feature information), text position information, word position and image feature map, and then obtaining a keyword sequence through a keyword extraction model; specifically, the keyword extraction model includes an encoder and a decoder, wherein the encoder includes a Transformer model, a capsule network (Capsule) and a CNN network, and the decoder includes a Transformer model, a generation mode (Generate Mode) and a copy mode (CopyMode), further, the text content, text feature map (also called text feature information), text position information, word position and image feature map obtained by preprocessing are encoded by the encoder, and then the encoded result is decoded by the decoder to obtain keywords; specifically,

[0069] 2) Build a decoder based on the generation mode and the copy mode, and design a mode selection method based on reinforcement learning to reduce the impact of OCR errors on keyword generation. At the same time, the mode selection avoids mutual interference between the two modes during prediction;

[0070] 3) Establishing a Transformer-based encoding and decoding network model. The Transformer network does not rely on the output of other word representations when calculating the representation of each word, and has better parallelization than RNN.

[0071] 4) In the keyword extraction model, all text lines of the keyword extraction model are encoded with local features and global features in order to better obtain the semantic representation of the text;

[0072] 5) In the encoder of the keyword extraction model, a capsule network is introduced to learn the structure of the text input (also known as the spatial structure information corresponding to the text line) and topic information, helping the model generate keywords with diverse topics.

[0073] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0074] The embodiment of the present application provides a keyword extraction method, which can be executed by an electronic device, which can be a terminal device or a server, such as Figure 5 As shown, the method includes:

[0075] Step S101: extract feature information corresponding to the image to be processed.

[0076] In this embodiment of the present application, before step S101, the process further includes: obtaining an image to be processed. In this embodiment of the present application, the obtained image to be processed may be a single image or multiple images. Furthermore, the image to be processed in this embodiment of the present application may also be a video frame, which is not limited in this embodiment of the present application.

[0077] Furthermore, each image to be processed may be an image containing text content or an image not containing text content, which is not limited in the embodiments of the present application.

[0078] Furthermore, the specific contents of the feature information extracted from the image to be processed are detailed in the following embodiments and will not be repeated here.

[0079] Step S102: extract keywords from the image to be processed based on the feature information.

[0080] For the embodiment of the present application, after the corresponding feature information is extracted from the image to be processed based on the above-mentioned step S101, keywords are extracted from the image to be processed based on the feature information. Furthermore, in the embodiment of the present application, the keywords extracted from the image to be processed are representative words that can summarize the image to be processed. In the embodiment of the present application, the method of extracting keywords from the image to be processed can be called keyword extraction (Keyword Extraction), also known as key phrase extraction (Keyphrase Extraction) or topic tag (HashTag) extraction. In the embodiment of the present application, the method of extracting keywords from the image to be processed is not limited to the two methods mentioned above. Any method of extracting keywords from the image to be processed is within the protection scope of the embodiment of the present application.

[0081] Furthermore, in an embodiment of the present application, at least one keyword can be extracted from the image to be processed, or if the end symbol is extracted in the first prediction time step, it indicates that only the end symbol can be extracted from the image to be processed.

[0082] The present embodiment provides a keyword extraction method. Compared to the prior art, the present embodiment extracts feature information corresponding to the image to be processed, and then extracts keywords from the image to be processed based on the feature information. In other words, the present embodiment can extract keywords from the image to be processed using the feature information extracted from the image to be processed.

[0083] Furthermore, feature information includes at least one of textual representation information and image visual information; and textual representation information includes at least one of textual content and textual visual information. Furthermore, for example, textual visual information can be used to represent: text size, text color, text shape, text relative position, background color, and structural information.

[0084] Furthermore, in an embodiment of the present application, the image visual information may include: an image feature map; the text visual information may include text visual information corresponding to each text line in the image to be processed.

[0085] Furthermore, it can be seen from the above embodiments that: the feature information may include image visual information, and in the embodiment of the present application, the image visual information may include: an image feature map. Further, the feature information may only include image visual information (image feature map), that is, keywords can be extracted from the image to be processed based on the image visual information (image feature map). In the embodiment of the present application, keywords can be extracted from the image to be processed based only on the image visual information (image feature map), that is, in the embodiment of the present application, keywords can be extracted from images that do not contain text information (text content) or images that contain a small amount of text information (text content), thereby reducing the probability of not extracting keywords from images that do not contain text content (text content) or images that contain a small amount of text information (text content), thereby improving the user experience.

[0086] Furthermore, the image feature map is the convolution feature that represents the image. Therefore, the method of extracting the image visual information (image feature map) corresponding to the image to be processed from the image to be processed includes: inputting the image to be processed into a convolutional neural network for convolution processing to obtain the image visual information (image feature map) corresponding to the image to be processed.

[0087] Furthermore, extracting the image visual information (image feature map) corresponding to the image to be processed from the image to be processed can specifically include: performing image scaling on the image to be processed, and extracting the image visual information (image feature map) from the image after the image scaling. In an embodiment of the present application, if the image to be processed is a single image, the single image may be scaled or not; if the image to be processed is at least two images, it is generally necessary to scale the at least two images so that the scaled images have the same size, so as to improve the processing speed when extracting image feature information from the at least two images in parallel. Of course, the at least two images may not be scaled.

[0088] Furthermore, in the above embodiment, regardless of whether the image to be processed is a single image or at least two images, only each image can be resized. Specifically, if the image to be processed is resized, the image to be processed can be resized by directly stretching or scaling, or by upsampling or downsampling. In the embodiment of the present application, if at least two images are resized, each image can be resized according to the above resizing method so that the size of each image after resizing is the same. In the embodiment of the present application, the resizing of the image to be processed is not limited to the above resizing method, and any possible method is within the protection scope of the embodiment of the present application.

[0089] For the specific examples of the above embodiments, Figure 6 As shown, the image to be processed (PXQ) is rescaled (resized) to obtain a rescaled image (MXN), and the rescaled image (MXN) is passed through a convolutional network to obtain corresponding image visual information (also called an image feature map).

[0090] Specifically, the feature information in the above embodiment may further include: text expression information. In the embodiment of the present application, the text expression information includes at least one of text content and text visual information.

[0091] Specifically, for a text line, the text content corresponding to the text line includes the word sequence of the text line.

[0092] Specifically, for a text line, the text visual information corresponding to the text line includes at least one of the following:

[0093] The text feature map corresponding to the text line;

[0094] Position information of the text line in the image to be processed;

[0095] Position information of each word in the text line in the image to be processed;

[0096] The relative position information of each word in the text line.

[0097] Specifically, it can be seen from the above embodiments that the feature information corresponding to the image to be processed may include: text expression information, and the text expression information may also include: text visual information and text content. The text visual information may include: text feature maps corresponding to each text line, the position information corresponding to each text line in the image to be processed, and the position information of each word in each text line in the corresponding text line. The following specifically introduces the method of determining text visual information and text content based on the image to be processed.

[0098] Specifically, the manner of extracting the text feature map and the respective position information of each text line in the to-be-processed image includes: determining a text line candidate region (a text line candidate region corresponding to each text line respectively) based on image visual information (an image feature map); and determining the text feature map and the respective position information of each text line in the to-be-processed image from the image visual information (the image feature map) through the text line candidate region. The text feature map in the embodiment of the present application is a text feature map corresponding to each text line in text content respectively.

[0099] Further, the manner of determining the text line candidate region based on the image visual information (the image feature map), and then determining the text feature map and the respective position information of each text line in the to-be-processed image from the image visual information (the image feature map) through the text line candidate region can be implemented through a neural network trained. Specifically, as shown in FIG. 2, the image feature map of the to-be-processed image is input into a region proposal network (RPN), and a text line candidate region (a text line candidate region corresponding to each text line respectively) is output. Then, the text line candidate region (a text line candidate region corresponding to each text line respectively) output by the RPN and the image feature map obtained above are used to obtain the text feature map (a text feature map corresponding to each text line respectively) and the respective position information of each text line in the to-be-processed image. In the embodiment of the present application, the manner of obtaining the text line candidate region from the image feature map of the to-be-processed image is not limited to the RPN network described above, and any manner that can obtain the text line candidate region from the image feature map of the to-be-processed image is within the protection scope of the embodiment of the present application. Figure 6

[0100] Further, the working process of the RPN includes: adopting an anchor mechanism to generate k anchor boxes at each pixel position on the image feature map, and then performing classification and regression on each anchor box, so as to obtain the text feature map (a text feature map corresponding to each text line respectively) and the respective position information of each text line in the to-be-processed image.

[0101] ​Furthermore, a method for extracting the text content corresponding to each text line (the word sequence corresponding to each text line), the position information of each word in the image to be processed, and the relative position information of each word in the corresponding text line from the image to be processed includes: extracting the text content corresponding to each text line (the word sequence corresponding to each text line) and the position information of each word in the corresponding text line based on the text feature map, wherein the position information of each word in the image to be processed and the relative position information of each word in its corresponding text line can also be estimated based on the word sequence length and text line position of the text line in which it is located.

[0102] Furthermore, in order to improve the extraction speed of the text content corresponding to each text line, the position information of each word in the image to be processed, and the relative position information of each word in the corresponding text line based on the text feature map (the text feature map corresponding to each text line), the text feature map (the text feature map corresponding to each text line) can be sized before extraction, and the text feature maps corresponding to each text line can be converted into the same size.

[0103] It should be noted that the text feature map (the text feature map corresponding to each text line) can be resized in any manner. For example, in the embodiment of the present application, the text feature map (the text feature map corresponding to each text line) can be resized by applying the region of interest (RoI) pooling method. Specifically, Figure 6 As shown in the figure, all text feature maps (text feature maps corresponding to each text line) are converted into the same size through the RoI pooling layer, and then all text lines are recognized through OCR based on the text feature maps corresponding to each text line after the conversion, and the text content corresponding to each text line (the word sequence corresponding to each text line) and the position information of each word in the corresponding text line are obtained.

[0104] In the above embodiment, after obtaining at least one of the image visual information (image feature map) and text expression information (text feature maps corresponding to each text line, position information corresponding to each text line in the image to be processed, text content information corresponding to each text line, position information of each word in the image to be processed, and relative position information of each word in the corresponding text line) of the image to be processed, keywords can be extracted from the image to be processed based on the obtained information. That is to say: in the embodiment of the present application, if image visual information is required to extract keywords from the image to be processed, then after the image visual information of the image to be processed is extracted from the extracted image, keywords are extracted from the image to be processed based on the image visual information; if text expression information is required to extract keywords from the image to be processed, then after the text expression information of the image to be processed is extracted from the extracted image, keywords are extracted from the image to be processed based on the text expression information; if text expression information and image visual information are required to extract keywords from the image to be processed, then after the text expression information and image visual information of the image to be processed are extracted from the extracted image, keywords are extracted from the image to be processed based on the text expression information and image visual information.

[0105] Specifically, another possible implementation method of the embodiment of the present application is to extract keywords from the image to be processed based on feature information, including: encoding the feature information to obtain an encoding result of the feature information; and extracting keywords from the image to be processed based on the encoding result.

[0106] Furthermore, based on the feature information obtained above (at least one of the image feature map, the text feature maps corresponding to each text line, the position information corresponding to each text line in the image to be processed, the text content information corresponding to each text line, the position information of each word in the image to be processed, and the relative position information of each word in the corresponding text line), the method of extracting keywords from the image to be processed can be implemented by a neural network or not. A specific example of the method of extracting keywords from the image to be processed by a neural network is as follows: Figure 6 As shown, the feature information (image feature map, text feature maps corresponding to each text line, position information corresponding to each text line in the image to be processed, text content information corresponding to each text line (also can be called: word sequence corresponding to each text line), position information of each word in the image to be processed and at least one of the relative position information of each word in each text line) is passed through a keyword extraction model to obtain a keyword sequence to extract keywords from the image to be processed.

[0107] Specifically, if the feature information includes: image visual information, the feature information is encoded to obtain an encoding result of the feature information; based on the encoding result, keywords are extracted from the image to be processed, including: encoding the image visual information to obtain an image feature representation; extracting keywords from the image to be processed based on the image feature representation; if the feature information includes: text expression information, the feature information is encoded to obtain an encoding result of the feature information; based on the encoding result, keywords are extracted from the image to be processed, including: encoding the text expression information to obtain an encoding result of the text expression information, and extracting keywords from the image to be processed based on the encoding result of the text expression information; if the feature information includes: image visual information and text expression information, the feature information is encoded to obtain an encoding result of the feature information; based on the encoding result, keywords are extracted from the image to be processed, including: encoding the image visual information and the text expression information to obtain an encoding result corresponding to the image feature representation and the text expression information; extracting keywords from the image to be processed based on the encoding result corresponding to the image feature representation and the text expression information. In the embodiment of the present application, the text expression information involved here is still at least one of the text content and text visual information. The text visual information includes: the text feature map corresponding to each text line, the position information corresponding to each text line in the image to be processed, and the position information of each word in the image to be processed, and at least one of the relative position information of each word in its respective text line; the text content includes: the text content information corresponding to each text line.

[0108] For the embodiment of the present application, the above-mentioned model involving the extraction of keyword sequences from the image to be processed is a keyword extraction model, which may include an encoder and a decoder. That is, in the embodiment of the present application, the feature information (at least one of the image visual information and the text expression information) is encoded by the encoder to obtain a corresponding encoding result, and then the corresponding encoding result is decoded by the decoder to predict keywords, thereby realizing the extraction of keywords from the image to be processed. In the embodiment of the present application, the image visual information (image feature map) and the text expression information (the text feature map corresponding to each text line, the position information corresponding to each text line in the image to be processed, the text content information corresponding to each text line, and the position information of each word in the image to be processed, the relative position information of each word in the corresponding text line) are encoded to obtain the encoding result, wherein the obtained encoding result can also be called a hidden layer representation, so the corresponding encoding result (that is, the above-mentioned hidden layer representation) is decoded to predict keywords.

[0109] In the embodiments of the present application, the encoding manner of the image visual information and the text expression information (at least one of the text feature map corresponding to each text line, the position information of each text line in the to-be-processed image, the text content information corresponding to each text line, the position information of each word in the to-be-processed image, and the relative position information of each word in the corresponding text line) can adopt the existing encoding manner, or can adopt the encoding manner introduced in the embodiments of the present application. Based on the same principle, the decoding of the encoded hidden layer representation can adopt the existing decoding manner, or can adopt the decoding manner introduced in the embodiments of the present application.

[0110] In the embodiments of the present application, the encoding manner of the image visual information and the text expression information (at least one of the text feature map corresponding to each text line, the position information of each text line in the to-be-processed image, the text content information corresponding to each text line, the position information of each word in the to-be-processed image, and the relative position information of each word in the corresponding text line) is described in detail in the following embodiments:

[0111] Specifically, the feature information includes text expression information and image visual information, wherein the text expression information includes at least one of text visual information and text content; the encoding result includes at least one of text background representation, image feature representation, text line structure information and theme information representation. The text background representation is obtained based on the text expression information, the image feature representation is obtained based on the image visual information, and the text line structure information and the theme information representation are obtained based on the text background representation.

[0112] Further, if the feature information includes text expression information (the text expression information includes at least one of text visual information and text content), the feature information is encoded to obtain the encoding result of the feature information, including: encoding the text expression information (the text expression information includes at least one of text visual information and text content) to obtain at least one of text background representation, text line structure information and theme information representation.

[0113] Further, if the feature information includes image visual information, the image visual information is encoded to obtain image feature representation; that is, the image feature map corresponding to the to-be-processed image is encoded to obtain the corresponding image feature representation. The specific encoding manner of the image feature map corresponding to the to-be-processed image can be that the image feature map corresponding to the to-be-processed image is convoluted by CNN to obtain the image feature representation, for example, as shown in the following formula (1). Figure 7 In the encoder, the image feature map is convoluted by CNN to obtain the image feature representation I.

[0114] Specifically, the text expression information (at least one of the text feature map corresponding to each text line, the position information of each text line in the to-be-processed image, the word sequence corresponding to each text line, the position information of each word in the to-be-processed image, and the relative position information of each word in the corresponding text line) corresponding to each text line is encoded to obtain at least one of the text line representation corresponding to each text line, the text background representation corresponding to each text line, and the text line corresponding structure information and the theme information representation.

[0115] In another possible implementation of the embodiment of the application, the text expression information is encoded to obtain at least one of the text line representation corresponding to each text line, the text background representation corresponding to each text line, and the text line corresponding structure information and the theme information representation, including: the text expression information is encoded to obtain the text line representation corresponding to each text line; the text line representation corresponding to each text line is encoded to obtain the text background representation corresponding to each text line, and the text background representation corresponding to each text line includes the local text background representation corresponding to each text line and the global text background representation; and the representation corresponding to each text line is encoded to obtain at least one of the structure information and the theme information representation corresponding to all text lines.

[0116] As known from the above embodiment, in the process of encoding the text expression information corresponding to each text line (the text expression information corresponding to each text line includes text content and text visual information, and the text visual information includes text feature information corresponding to each text line, position information of each text line in the to-be-processed image, position information of each word in the to-be-processed image, and relative position information of each word in the corresponding text line), the text line representation corresponding to each text line is first obtained, then the text background representation corresponding to each text line is obtained based on the text line representation corresponding to each text line, and then at least one of the structure information and the theme information representation corresponding to all text lines is obtained based on the text background representation corresponding to each text line.

[0117] It can be seen from the above embodiments that: the text expression information corresponding to each text line (the text feature map corresponding to each text line, the position information corresponding to each text line in the image to be processed, the word sequence corresponding to each text line, and at least one of the position information of each word in the corresponding text line) is encoded to obtain the text line representation corresponding to each text line, and then the text line representation corresponding to each text line is encoded to obtain the text background representation corresponding to each text line, and then at least one of the structural information and subject information representation corresponding to the text line is encoded based on the text background representation corresponding to each text line. For example, the structural information can be the spatial structural information and triangular structural information formed by the text line. In the embodiment of the present application, the structural information representation is used to represent the spatial structural information and triangular structural information formed by the text line.

[0118] Specifically, if Figure 7 As shown, the text expression information corresponding to each text line is obtained by the text line encoder to obtain the text line representation corresponding to each text line. For example, the text feature information corresponding to text line 1 (which can be called text feature Figure 1 ), the text content information corresponding to text line 1 (which can be called text content 1), the position information corresponding to text line 1 in the image to be processed (which can be called text position 1), and the word position corresponding to each word in text line 1 (which can also be called word position 1), the text line representation T1 is obtained through the text line encoder, and the text feature information corresponding to text line 2 (which can be called text feature Figure 1), text content information corresponding to the text line 2 (which can be referred to as text content 2), position information corresponding to the text line 2 in the image to be processed (which can be referred to as text position 2), and word positions corresponding to respective words in the text line 2 (which can also be referred to as word positions 2), to obtain a text line representation T2 through the text line encoder, …, text feature information corresponding to the text line m (which can be referred to as text feature map m), text content information corresponding to the text line m (which can be referred to as text content m), position information corresponding to the text line m in the image to be processed (which can be referred to as text position m), and respective word positions corresponding to respective words in the text line m (which can also be referred to as word positions m), to obtain a text line representation Tm through the text line encoder; then the text line representations T1, T2, …, Tm, etc. are obtained through N Intra-Inter Transformer modules to obtain text background representations C of the corresponding text lines (i.e., text background representations C1, C2, …, Cm), where C = [C1, C2, …, Cm]; further, the obtained text background representations of the text lines (i.e., the text background representations C1, C2, …, Cm) are encoded through a capsule network to encode spatial structure information and theme information of the text lines, to obtain spatial structure information and theme information representations corresponding to all the text lines.

[0119] In the above embodiments, the manner of obtaining the text line representation corresponding to each text line, the text background representation corresponding to each text line, and the structure information and theme information representation corresponding to all the text lines is introduced. The specific implementation manners of obtaining the text line representation, the text background representation, and the structure information and theme information representation of the text line are introduced in detail as follows.

[0120] In a possible implementation manner in the embodiments of the present application, if the feature information includes image visual information and text expression information, the feature information is encoded to obtain an encoding result of the feature information, including at least one of the text background representation, the text line structure information and the theme information representation, and the image feature representation.

[0121] Specifically, if the feature information includes text expression information, and the text expression information includes at least one of text visual information and text content, the feature information is encoded to obtain an encoding result corresponding to the feature information, including at least one of the following: the text expression information is encoded to obtain a text line representation; the text line representation is encoded to obtain a text background representation; and the text background representation is encoded to obtain text line structure information and theme information representation.

[0122] In another possible implementation of the embodiment of the present application, if the feature information includes: text expression information, the text expression information includes text content, the text content includes: word sequences corresponding to each text line; for a text line, the text content of the text line includes: the word sequence corresponding to the text line;

[0123] Among them, for a text line, the text content is encoded to obtain a text line representation, including: encoding the word sequence corresponding to the text line to obtain a word representation based on the character sequence; determining the text line representation corresponding to the text line based on the word representation based on the character sequence.

[0124] Specifically, based on the text feature information corresponding to each text line, the position information corresponding to each text line in the image to be processed, the word sequence corresponding to each text line and the position information of each word in the corresponding text line, the feature representation corresponding to each word in each text line is determined, which can specifically include: applying CNN to the character embedding sequence of each word in the text line to obtain a word representation based on the character sequence; dividing the text feature map into word feature maps according to the word position, and scaling them to the same size, and applying CNN to the scaled feature map to obtain a feature map representation; determining the word position vector corresponding to each text line based on the position information of each word in the corresponding text line; and obtaining the text position representation corresponding to each text line based on the position information corresponding to each text line in the image to be processed. In an embodiment of the present application, a method of obtaining a character-based word representation based on the word sequence corresponding to each text line, a method of obtaining a position vector corresponding to each text line in the image to be processed based on the position information corresponding to each text line in the image to be processed, a method of obtaining a feature map representation corresponding to each text line based on the text feature information corresponding to each text line, the position information of each word in the image to be processed, and a method of determining the position vector of each word in the corresponding text line based on the relative position information of each word in the corresponding text line can be executed simultaneously, or can be executed at different times, or can be executed partially simultaneously, which is not limited in the embodiment of the present application.

[0125] Furthermore, in an embodiment of the present application, a text line representation corresponding to each text line is obtained based on the obtained representation information, which may specifically include: splicing the character-based word representation, the text features corresponding to each word, the relative position information of each word in the corresponding text line, and the position information corresponding to each text line in the image to be processed, and adding the position code in the text line, such as the Sinusoidal position code, to each word in the text line after the splicing process, so as to add the word position information in the text line, thereby obtaining the text line representation corresponding to each text line. In an embodiment of the present application, after the position information corresponding to each text line in the image to be processed, the obtained position vector is copied, and the copied position information corresponding to each text line in the image to be processed and the character-based word representation, the text features corresponding to each word, the position information of each word in the image to be processed, and the relative position information of each word in the corresponding text line are spliced, and adding the Sinusoidal position code to each word in the text line after the splicing process to add the word position information in the text line, thereby obtaining the text line representation corresponding to each text line. In an embodiment of the present application, after the corresponding position information of each text line in the image to be processed, the obtained position vector is copied so that the obtained position vector is the same as the number of words in each text line, so as to achieve the addition of the text line position to each word in the text line.

[0126] Specifically, Sinusoidal position encoding: For the word position encoding vector PE at position p p ∈R d , the values ​​of different dimensions are calculated using sine and cosine functions of different frequencies:

[0127] PE (p,2i) =sin(p / 10000 2i / d )

[0128] PE (p,2i+1) =cos(p / 10000 2i / d )

[0129] Among them, p represents the position of the word in the text line, i represents the dimension, and d represents the dimension of the encoding vector.

[0130] Specifically, an example of a method for obtaining a text line representation corresponding to each text line based on the text feature map corresponding to each text line, the position information corresponding to each text line in the image to be processed, the word sequence corresponding to each text line, the position information of each word in the image to be processed, and the relative position information of each word in the corresponding text line can be as follows: Figure 8As shown in FIG, the word sequence corresponding to text line 1 is processed by character embedding, and the information after character embedding is input into CNN to obtain character-based word representation; the text feature map of text line 1 is divided into word feature maps, and each word feature map is resized to obtain each word feature map of the same size, and then each resized word feature map of the same size is passed through CNN to obtain the feature map representation corresponding to text line 1; based on the word positions corresponding to each word in text line 1, for example, [x 11 、y 11 、w 11 、h 11 ] and [x 12 、y 12 、w 12 、h 12 ] (where (x, y), w, h represent the center point position, width and height of the word respectively), and obtain the corresponding word position vector; based on the position information [x1, y1, w1, h1] corresponding to the text line 1 in the image to be processed, obtain the corresponding text position representation, where (x1, y1), w1, h1 represent the center point position, width and height of the text line 1 respectively, and copy the obtained text position representation so that the number of obtained text position representations is equal to the number of words in the text line 1, so that the text position representation is subsequently added to each word in the text line 1; then the character-based word representation corresponding to the text line 1, the feature map representation corresponding to the text line 1, the word position vector corresponding to the text line 1, and the text position vector corresponding to the text line 1 are spliced, and then the spliced ​​vector is Sinusoidal position encoded to obtain the text line representation corresponding to the text line 1. The way in which the text line obtains the corresponding text line representation is similar to the way in which the text line 1 obtains the text line representation corresponding to the text line 1, and will not be further elaborated in the embodiments of the present application.

[0131] Furthermore, in the embodiment of the present application, the word sequence corresponding to each text line is replaced by word embedding through character-based word representation. Since each word vector in the embodiment of the present application is established based on character sequence learning, rather than directly using a word embedding vector for each word, and the scale of the character embedding vector is much smaller than that of all word embedding vectors, the former embedding vector matrix (i.e., the scale of the character embedding vector) is a matrix of |C|*d, and the latter word embedding vector matrix is ​​a matrix of |V|*d, where |C| is the scale of all characters, |V| is the dictionary size, d is the dimension, and here |C|<<|V|, so that character-based word representation is used instead of word embedding in the embodiment of the present application, which can reduce the storage space and memory of the text line encoder.

[0132] In another possible implementation of the embodiments of the present application, the text line representation includes text line representations corresponding to at least one text line, and encoding the text line representations to obtain text background representations corresponding to the text lines includes: encoding each text line representation to obtain a local text background representation corresponding to each text line respectively; encoding all the text line representations as a whole to obtain a global text background representation corresponding to all the text lines; and determining a text background representation corresponding to each text line based on the local text background representation corresponding to each text line and the global text background representation corresponding to all the text lines.

[0133] In the above embodiments, the manner of obtaining the text line representation corresponding to each text line is introduced in detail, and the specific implementation of encoding the text line representation corresponding to each text line to obtain the text background representation corresponding to the text line is described in the following embodiments.

[0134] In another possible implementation of the embodiments of the present application, encoding the text line representation corresponding to each text line to obtain a text background representation corresponding to each text line includes: encoding each text line representation corresponding to each text line to obtain a local text background representation corresponding to each text line respectively; and encoding each text line simultaneously to obtain a global text background representation corresponding to all the text lines; and determining a text background representation corresponding to each text line based on the local text background representation corresponding to each text line and the global text background representation corresponding to all the text lines.

[0135] Specifically, the text line representation corresponding to each text line is encoded to obtain the text background representation corresponding to each text line, which can be processed by a background coding module in an encoder. In the embodiments of the present application, the text background representation module (also referred to as the background coding module), that is, the Intra-Inter Transformer module, is designed to locally and globally encode each text line representation corresponding to each text line.

[0136] Specifically, in the embodiments of the present application, the manner of encoding the text line representation corresponding to each text line to obtain the text background representation corresponding to the text line is as shown in Figure 9 The specific processing manner is as follows:

[0137] 1) Given input information T = [T1, T2, …, Tm], wherein T represents text line representations corresponding to each text line in a to-be-processed image, T1 represents a text line representation corresponding to a first text line, T2 represents a text line representation corresponding to a second text line, and Tm represents a text line representation corresponding to an mth text line.

[0138] 2) Using the intra-text-line Transformer network, i.e., the Intra TF network, each text line representation is encoded separately, and the local text background representation L corresponding to each text line is obtained;

[0139] 3) Using the inter-text line TF network, i.e., Inter TF, all text lines are encoded simultaneously to obtain the global text context representation G;

[0140] 4) Concatenate the outputs of Intra-TF and Inter-TF. That is, concatenate the local context representation corresponding to each text line with the obtained global context representation. Specifically, concatenate the local context representation matrix L = [L1, L2, …, Lm] corresponding to each text line with the obtained global context representation matrix G by column. Here, Li represents the local context representation obtained by the i-th Intra-TF. In addition, in addition to concatenating by column, the fusion of the Intra-TF and Inter-TF outputs in step 4) can also be performed by addition, i.e., L + G.

[0141] 5) The concatenated information (or the added information) is input into the linear projection layer for linear transformation to obtain F(T), which has the same dimension as the input T;

[0142] 6) Performing a residual connection between F(T) and the text line representation corresponding to each text line (i.e., C=F(T)+T), the final output result is C=[C1,C2,…,Cm], that is, the text background representation of the text line is obtained, where C represents the text background representation of the text line, C1 represents the text background representation of text line 1, C2 represents the text background representation of text line 2, and Cm represents the text background representation of text line m.

[0143] In the above embodiments, both the local text background representation and the global background representation are obtained by using the Transformer network, wherein the Transformer network module is mainly composed of the following three sub-components: Figure 16As shown, it includes: multi-head self-attention (Multi-head Self-Attention), a convolutional layer, a fully connected layer; the specific processing process is: the input information is processed by attention through the multi-head self-attention layer, and the processed information is processed by convolution through the convolutional layer, and then the information processed by convolution is processed by linear transformation through the fully connected layer, and the linear transformation processing result and the above-mentioned input information are connected through the residual connection (the input of the module and the output of the fully connected layer are added) as the output of the module. In the embodiment of the application, the input information of Intra-TF and Inter-TF is input information T=[T1, T2, …, Tm], and the output of Intra-TF and Inter-TF is local text background representation L and global text background representation G.

[0144] Further, in the embodiment of the application, the local text background representation and the global text background representation of all text lines are obtained through the Intra-Inter Transformer, and the text background representation can present the context information, so that the keyword can be more accurately predicted when the local text background representation and the global text background representation of all text lines are used for keyword prediction.

[0145] Another possible implementation of the embodiment of the application, the text background representation includes the text background representation corresponding to each text line respectively; the text background representation is encoded to obtain at least one of the text line structure information and the theme information representation, including: obtaining the hidden vector corresponding to each text line respectively based on the text background representation corresponding to each text line respectively; determining at least one of the text line structure information and the theme information representation based on the hidden vector corresponding to each text line respectively.

[0146] After obtaining the text background representation corresponding to each text line respectively, the text background representation corresponding to each text line respectively can be encoded for structure and theme to obtain the structure information and the theme information representation corresponding to all text lines.

[0147] Another possible implementation of the embodiment of the application, the text background representation is encoded to obtain at least one of the text line structure information and the theme information representation, which can specifically include: based on the text background representation corresponding to each text line respectively, and through at least one capsule network for encoding processing, to obtain the corresponding capsule vector.

[0148] For the embodiments of the present application, a capsule is a group of neurons, and its activation vector represents instantiation parameters of a specific type of entity (such as a target or a part of a target). Unlike a traditional convolutional neural network, the capsule converts a scalar output of a convolutional neuron into a vector output to express more rich feature information, such as relative position, text semantic information, and the like. Given a group of input vectors, the capsule network learns a plurality of groups of weights through a dynamic routing algorithm, weights the input vectors, and obtains a group of capsule vectors. The length of each capsule vector represents the possibility of the existence of a certain object or component in the input, and the vector value itself reflects the relative structural information and clustering information between the input vectors. In the embodiments of the present application, the capsule vector is used to represent the text line structure information and the theme information corresponding to all text lines. In the embodiments of the present application, the structural information corresponding to all text lines can include layout information of the text, such as spatial structural information of all text lines, triangular structural information, and the like; and the theme information refers to a theme described by the text content.

[0149] In the embodiments of the present application, the capsule network encodes the information into a plurality of capsule vectors to represent the structural and theme information corresponding to all text lines.

[0150] Further, in the embodiments of the present application, if only keywords of theme diversity are needed, the capsule network can be used to learn theme information representation, or a theme model (such as Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA), and the like) can be used to learn a theme vector to replace the theme information representation.

[0151] Further, the text background representations (C=[C1,C2,…,Cm]) corresponding to each text line are encoded to obtain at least one of the structural and theme information representations corresponding to all text lines, such as Figure 10 as shown, including:

[0152] 1) input text background representation C=[C1,…,Cm], wherein Ci represents a text background representation of the i th text line;

[0153] 2) pooling each text background representation Ci to obtain a hidden vector hi;

[0154] 3) applying a capsule network dynamic routing to all hidden vectors [h1,..,hm] to obtain k capsule vectors (including: capsule 1, capsule 2, …, capsule k), and the all capsule vectors are V=[v1,…,vk]=Capsules([h1,…,hm]);

[0155] 4) Output all capsule vectors V(v1, v2, ..., vk).

[0156] It should be noted that the capsule vector V output in the above embodiment is used to represent the text line structure information and topic information corresponding to all text lines.

[0157] Furthermore, the structural information and topic information of all text lines in the image are encoded through the capsule network, where the structural information helps the model focus on the important spatial positions of the text lines, and the topic information helps the model's decoder generate keywords with topic diversity, thereby improving the accuracy of extracting text lines from the image to be processed.

[0158] Another possible implementation of the structure and topic encoding module of the embodiment of the present application is to directly apply the capsule network to the input text background representation C to obtain k capsule vectors (including: capsule 1, capsule 2, ..., capsule k).

[0159] As can be seen from the above embodiment, the image feature representation I, the text background representation C of the text line, and the capsule vector V corresponding to the image to be processed are obtained by the encoder, and the image feature representation I, the text background representation C of the text line, and the capsule vector V are decoded to obtain a keyword sequence. In the embodiment of the present application, in the decoding process, all keywords are predicted by generating a word sequence, for example, [ <s>Beauty, <s>,M*er, <s>S*nd, &, S*y,< / s> ]; "Beauty", "M*er" and "S*nd&S*y" are keywords, <s>and< / s> Respectively represent the starting position of the keyword phrase and the end of the word sequence. In an embodiment of the present application, the image feature representation I is used to represent the image features of the image to be processed, the text background representation C of the text line considers the context information of each word when encoding each word in the input sequence, and the obtained feature representation output is used. The capsule vector V is used to represent the text structure information and the topic information. The input information has topic information, which helps to generate keywords with topic diversity so that the extracted keywords will not be concentrated on a single topic. The spatial structure information is the relative position information between text lines and the position information of words and text lines in the picture. Adding spatial structure information helps the model to focus on words with important positions, such as a word in the top-middle position of the picture or surrounded by other words, which is often more important. It is precisely because of the above-mentioned effects of the image feature representation I, the text background representation C of the text line and the capsule vector V that keyword prediction through the image feature representation I, the text background representation C of the text line and the capsule vector V can improve the accuracy of the predicted keywords.

[0160] Another possible implementation of the embodiment of the present application is to extract keywords from the image to be processed based on the encoding results, including: determining the target prediction mode corresponding to each prediction time step and the prediction word corresponding to the target prediction mode based on the encoding results; outputting the prediction word corresponding to each prediction time step; and obtaining keywords based on the prediction word corresponding to each prediction time step.

[0161] For an embodiment of the present application, the encoding result may include: image feature representation, text background representation, and at least one of text line structure information and subject information representation. Then, extracting keywords from the image to be processed based on the encoding result includes: extracting keywords from the image to be processed based on at least one of image feature representation, text background representation, text line structure information and subject information representation.

[0162] In the embodiment of the present application, when predicting keywords based on the encoding result corresponding to the image to be processed (image feature representation I, text line feature representation C of the text line, and capsule vector V), each prediction word is predicted at each prediction time step to obtain a prediction for the above-mentioned keyword, thereby obtaining a keyword. In the embodiment of the present application, one prediction word is predicted at one prediction time step, and at least one prediction word corresponds to one keyword. For example, for the keyword sequence [ <s>,Beauty, <s>,M*er, <s>S*nd, &, S*y,< / s> ]; Among them, the predicted word corresponding to the first prediction time step is " <s>", the predicted word corresponding to the second prediction time step is "Beauty", and the predicted word corresponding to the third prediction time step is " <s>", the predicted word corresponding to the fourth prediction time step is "M*er", and the predicted word corresponding to the fifth prediction time step is " <s>the predicted word corresponding to the sixth prediction time step is "S*nd", the predicted word corresponding to the seventh prediction time step is "&", the predicted word corresponding to the eighth prediction time step is "S*y", the predicted word corresponding to the ninth prediction time step is< / s> ”, where the three predicted words corresponding to the above prediction time steps are “Beauty”, “M*er” and “S*nd&S*y”.

[0163] Specifically, for a prediction time step, based on the encoding result, determining the target prediction mode corresponding to the prediction time step and the prediction word corresponding to the target prediction mode includes at least one of step Sa (not shown in the figure) and step Sb (not shown in the figure):

[0164] Step Sa, based on the encoding result, determining the prediction words of each preconfigured prediction mode corresponding to the prediction time step and the target prediction mode corresponding to the prediction time step, and obtaining the prediction word corresponding to the target pre-stored mode based on the prediction words of each prediction mode and the target prediction mode corresponding to the prediction time step;

[0165] Step Sb: Based on the encoding result, determine the target prediction mode corresponding to the prediction time step from the pre-configured prediction modes, and obtain the prediction word corresponding to the target prediction mode based on the encoding result.

[0166] Specifically, the preconfigured prediction modes include: a first prediction mode for keyword prediction based on a vocabulary; a second prediction mode for keyword prediction based on the words contained in the text line, wherein, when keyword prediction is performed based on the second prediction mode, the words contained in the text line are obtained based on the encoding results of image feature information, and the image feature information includes the text content of the image to be processed.

[0167] Specifically, in the embodiment of the present application, the dictionary based on which the first prediction mode performs keyword prediction may include a predefined dictionary. Specifically, the first prediction mode, which may also be called a generation mode, is a prediction mode that performs keyword prediction based on a predefined dictionary; the second prediction mode, which may also be called a replication mode, is a prediction mode that performs keyword prediction based on each word contained in a text line.

[0168] Specifically, if the target prediction mode is the second prediction mode, based on the encoding result, the prediction word corresponding to the target prediction mode is determined, including: based on the encoding result, the weight corresponding to each word contained in the text content in the image to be processed is determined; based on the weight of each word, the prediction word corresponding to the target prediction mode is determined. Specifically, the image feature information includes text expression information and image visual information, wherein the text expression information includes at least one of text visual information and text content, the encoding result includes text background representation and image feature representation, the text background representation is obtained based on the text expression information, and the image feature representation is obtained based on the image visual information; wherein, based on the encoding result, the weight corresponding to each word contained in the text content in the image to be processed is determined, including: based on the encoding result, the hidden vector corresponding to the current prediction time step is obtained through feature fusion processing; based on the text background representation and the hidden vector, the weight corresponding to each word contained in the text content in the image to be processed is determined. Specifically, based on the text background representation and the hidden vector, the weight corresponding to each word contained in the text content in the image to be processed is determined, which can specifically include: based on the text background representation and the hidden vector, the score corresponding to each word contained in the text content in the image to be processed is determined; based on the score corresponding to each word, the weight corresponding to each word is obtained. In the embodiment of the present application, as shown in Figure 11 , the feature representation of each word contained in the text content in the image to be processed is the output result of the Transformer (forward), that is, the output result of the Transformer (forward) network module, and then the weight of the feature representation of each word is obtained through the output result of the forward Transformer and the text background representation (C1, C2……Cm) corresponding to each text line through the attention mechanism. In the embodiment of the present application, the Transformer (forward) can also be as shown in Figure 16 , wherein the input information of the Transformer (forward) is the image feature representation I, the text background representation C and the capsule vector V, and the output information is the feature representation of each word contained in the text content in the image to be processed.

[0169] Specifically, based on the encoding result, the feature representation of each word contained in the text content in the image to be processed is determined, including: based on the encoding result and the predicted word, the feature representation of each word contained in the text content in the image to be processed is determined. In the embodiment of the present application, the predicted word is the word predicted before the current prediction time step. For example, in the above embodiment, it is mentioned that for the image to be processed, the corresponding keyword sequence is <s>Beauty, <s>,M*er, <s>S*nd, &, S*y,< / s> ], then at the 4th prediction time step, " <s>”、"Beauty”、" <s>" as the predicted word.

[0170] Furthermore, after obtaining the feature representation of each word contained in the text content in the image to be processed based on the above embodiment, the feature representation of each word is fused based on the weight of the feature representation of each word, and the predicted word is obtained based on the fused feature representation.

[0171] In the following embodiments, each prediction time step is taken as an example to introduce the method of predicting keywords. Further, in the embodiments of the present application, Figure 7 、 Figure 11 As shown, when the image feature representation I, the text background representation C of the text line and the capsule vector V are decoded by the decoder, the image feature representation I, the text background representation C of the text line and the capsule vector V are passed through the attention mechanism to perform weighted averaging of the input features, and then the information processed by the attention mechanism and the predicted words are input into the Transformer (forward) network (i.e., the Transformer (forward) network module) by word embedding, and its output result is passed through the mode selection network to determine whether the word to be predicted is a word copied based on the copy mode or a word generated by the generation mode, thereby outputting the predicted word, and then the corresponding keyword. In an embodiment of the present application, the above-mentioned Transformer (forward) network module can be a single network or can be stacked at least twice. This is not limited in the embodiment of the present application.

[0172] Specifically, based on the encoding results corresponding to the image feature information, the encoding results corresponding to the text information, and the predicted words, the prediction mode of the word sequence corresponding to the current prediction time step is determined, including: processing the text line representation corresponding to each text line, the text background representation corresponding to the text line, and the capsule vector through the attention mechanism to obtain the processed results; and performing feature fusion processing on the processed results and the predicted words to obtain a feature fusion result; based on the feature fusion result, the prediction mode of the predicted word corresponding to the current prediction time step is determined.

[0173] Furthermore, if Figure 11 As shown, at each prediction time step, the image feature representation I, the capsule vector V and the text context representation C (text context representation C1, text context representation C2, …, text context representation Cm) are processed by an attention processing mechanism to obtain a weighted average of the information, and then the information obtained by processing the predicted word through the character-based word embedding (the word embedding is the information obtained by processing the predicted word through the character-based word embedding) and the weighted average of the information are processed through a forward Transformer network for feature fusion, and the fused information (which can also be referred to as the hidden state representation h of the current time step of the forward Transformer) is processed through a mode selection strategy network for mode selection, to generate a decision action action for determining the predicted word to be output as a word generated based on the generation mode, or a word obtained by copying through the copy mode. In the embodiment of the present application, the input in the mode selection network is the fused information (which can also be referred to as the hidden state representation h of the forward Transformer to the forward layer), and then a Sigmoid output activation function is used to calculate the mode selection strategy P(a; h), and the action is sampled from the selection strategy distribution P(a; h), i.e. action ~ P(action; h). In the embodiment of the present application, the action obtained is used to determine whether the prediction mode corresponding to the current prediction time step is the first prediction mode (generation mode) or the second prediction mode (copy mode), that is, the action obtained in the embodiment of the present application is used to determine whether the word predicted at the current prediction time step is predicted through the first prediction mode (generation mode) or the second prediction mode (copy mode). Further, for example, the action can be 0 or 1, action = 0 represents that the prediction mode corresponding to the current prediction time step is the first prediction mode (generation mode), and action = 1 represents that the prediction mode corresponding to the current prediction time step is the second prediction mode (copy mode).

[0174] Further, the word predicted through the first prediction mode (generation mode) is a new word prediction according to a predefined common word dictionary, specifically, the output hidden state h of the forward Transformer at the current time step is calculated to the Softmax layer to obtain the probability distribution on the predefined common word dictionary, i.e. Softmax(Wh), where W ∈ R |V|*d where |V| is the size of the vocabulary, d is the dimension of the hidden state h. The most probable word is taken as the word to be generated. The most probable word is taken as the word to be predicted at the current time step. In the embodiment of the present application, since the key word sequence is predicted by the generation mode, the most probable word sequence is selected from the predefined vocabulary as the predicted sequence, which is not simply copying the word sequence in the text line recognized by OCR. That is, when predicting the word by the generation mode, the word in the text line recognized by OCR is not directly taken as the output word (as the predicted output word), so that the key word predicted by the generation mode can avoid the prediction error of the key word caused by the OCR recognition error of the text content;

[0175] Further, the key word predicted by the copy mode is a word sequence copied from the input word sequence as the key word sequence predicted at the current prediction time step. Specifically, the hidden vector h of the forward Transformer at the current time step and the text background representation C are used to calculate the probability distribution of the input word by using the Softmax combined with the attention mechanism, and the input word with the highest score is taken as the copy word for new word prediction (that is, the hidden vector at the current time step is obtained by feature fusion processing based on the encoding result (here, the feature fusion processing refers to the forward Transformer), the scores of each word contained in the text content in the image to be processed are determined based on the text background representation C and the hidden vector at the current time step by using the attention mechanism, and then the weights (or probability distribution) of each word are obtained by using the Softmax, and then the copy word is obtained). It can be understood that the key word sequence predicted by the copy mode is to determine the word sequence with the highest score from the current text line sequence (which is actually in the OCR text), as the word sequence predicted at the current prediction time step. In the embodiment of the present application, when predicting the key word by the copy mode, the weighted average processing is performed by using the attention mechanism, that is, the weights of the feature representations of each word are adjusted by using the forward Transformer and the text background representation of each text line, which helps to reduce the score of the OCR error word, so as to improve the accuracy of the predicted key word.

[0176] In the above embodiment, the key word is extracted from the image to be processed by using the key word extraction model to obtain the key word corresponding to the image to be processed. The model (key word extraction model) used in the embodiment of the present application is a deep learning network. The model training needs to be based on a large-scale training data set. However, since the data labeling of this type is relatively complex and has a high labeling difference, it is very time-consuming and costly to manually label a large-scale text image key word extraction data. In the embodiment of the present application, a training data acquisition method is provided to acquire the training data corresponding to the key word extraction model:

[0177] (1) Training data preparation:

[0178] The training data is divided into two categories, synthetic screenshot corpus and labeled corpus (the labeled corpus can include: manually labeled corpus and automatically labeled corpus):

[0179] 1) Obtain synthetic screenshot corpus:

[0180] It uses open resources crawled from the network to synthesize image data, and the operation process is as shown in Figure 12 In the embodiments of the present application, Figure 12 the synthetic images in Figure 12 are only exemplary, wherein the contents of the synthetic images in

[0181] are not protected. Figure 12 First, extract text data from the network, such as the abstract and keywords of a paper, here the "title" and "abstract" together as a document, and the abstract keywords as keyword phrases, for example, in

[0182] the "title" and "abstract" parts of a scientific article are taken together as a document, and the "keywords" part is taken as a keyword phrase.

[0183] Second, combine the collected text data (including the abstract and keywords of the paper) and image data to synthesize a large-scale labeled corpus, and here all the keywords of the paper are taken as the keywords of the synthetic image.

[0184] 2) Labeled corpus:

[0185] • First, collect a certain amount of formal image data, such as screenshot data, from smartphone users;

[0186] • Then, label these data, which can be manually labeled by crowdsourcing and the like.

[0187] Model training and prediction method, the method includes three steps, as shown in Figure 13 :

[0188] 1) Pre-train the keyword extraction model using the synthetic screenshot corpus;

[0189] 2) Fine-tune the pre-trained model using the labeled screenshot corpus;

[0190] The above embodiment introduces the extraction of keyword sequences from the image to be processed to obtain keywords, and introduces the training method of the keyword model used for extracting keyword sequences. The method of extracting keyword sequences from the image to be processed can be applied to many application scenarios, such as image label editing and intelligent search. Among them, for image label editing, when the user wants to add a text label to a screenshot, keyword extraction can provide the user with some label candidates, and then the user can select a label as the label of the image, or the user can edit the label himself, such as Figure 14 As shown in the figure, the keywords "#weekly catalogue", "#ooles catalogue", and "#n*wmetro" are extracted from the image to be processed as candidate tags, where "#" represents a keyword symbol (also known as a HashTag symbol). This allows users to select tags from the candidate tags as tags for the image. For smart search, when users want to know information related to an image, the keyword extraction tool can automatically extract keywords from the image and provide them to the user. If the user selects a tag (such as tag 1), the tool will present the relevant search results to the user, that is, the search results corresponding to tag 1, as shown in the figure. Figure 15 shown.

[0191] Furthermore, in the above embodiment, adding visual information (at least one of image visual information and text visual information) to the feature information based on which keywords are extracted from the image to be processed can significantly improve the performance of the keyword extraction model and enhance the keyword extraction effect. The following specific experimental data shows that adding visual information can significantly improve the performance of the keyword extraction model and enhance the keyword extraction effect. Specifically, in the embodiment of the present application, the visual information only considers visual information such as the position of each text line in the image to be processed and the position of each word in each text line in the text line, and 1,700 manually annotated Chinese screenshots are used as experimental data, of which 1,000 screenshots are used as training sets and 700 screenshots are used as test sets. The experimental results are shown in Table 1.

[0192] Table 1

[0193]

[0194] Among them, Prec@5 and Prec@10 represent the average precision of the top 5 and top 10 keyword prediction ranking scores of the keyword extraction model, respectively. And as shown in Table 1, the method of the embodiment of the present application (i.e., extracting keywords based on feature information containing visual information) is superior to the keyword extraction method in the original model without adding visual information in terms of precision, recall rate, F1, Prec@5 and Prec@10. Therefore, the keyword extraction method in the embodiment of the present application can significantly improve the performance of the keyword extraction model and enhance the keyword extraction effect.

[0195] Furthermore, in the embodiment of the present application, the extraction of images with English text content and the extraction of English keywords are used as an example for explanation, but the model proposed in the embodiment of the present application is also applicable to keyword extraction from images with text content in other languages, and the extracted keywords can be keywords of the corresponding language.

[0196] The above embodiment introduces the keyword extraction method from the perspective of the method flow. The following embodiment introduces the keyword extraction device from the perspective of a virtual device. The keyword extraction device described in the following embodiment is applicable to the above method embodiment, as shown below:

[0197] The embodiment of the present application provides a keyword extraction device, such as Figure 17 As shown, the keyword extraction device 1700 includes: an extraction module 1701 and an extraction module 1702, wherein:

[0198] Extraction module 1701, used to extract feature information corresponding to the image to be processed;

[0199] The extraction module 1702 is used to extract keywords from the image to be processed based on feature information.

[0200] In a possible implementation of the embodiment of the present application, the feature information includes at least one of text expression information and image visual information; the text expression information includes at least one of text content and text visual information.

[0201] In another possible implementation of the embodiment of the present application, the text visual information includes text visual information corresponding to each text line in the image to be processed;

[0202] For a text line, the text visual information corresponding to the text line includes at least one of the following:

[0203] The text feature map corresponding to the text line;

[0204] Position information of the text line in the image to be processed;

[0205] Position information of each word in the text line in the image to be processed;

[0206] The relative position information of each word in the text line.

[0207] In another possible implementation of the embodiment of the present application, when the extraction module 1702 extracts keywords from the image to be processed based on feature information, it is specifically configured to:

[0208] Encoding the characteristic information to obtain an encoding result of the characteristic information;

[0209] Based on the encoding results, keywords are extracted from the image to be processed.

[0210] In another possible implementation of the embodiment of the present application, the feature information includes text expression information and image visual information, wherein the text expression information includes at least one of text visual information and text content;

[0211] The encoding result includes at least one of a text background representation, an image feature representation, text line structure information, and a topic information representation;

[0212] Among them, the text background representation is obtained based on the text expression information, the image feature representation is obtained based on the image visual information, and the text line structure information and topic information representation are obtained based on the text background representation.

[0213] In another possible implementation of the embodiment of the present application, when the extraction module 1702 extracts keywords from the image to be processed based on the encoding result, it is specifically configured to include:

[0214] Based on the encoding results, determining the target prediction mode corresponding to each prediction time step and the prediction word corresponding to the target prediction mode;

[0215] Output the predicted word corresponding to each prediction time step;

[0216] Based on the predicted words corresponding to each prediction time step, keywords are obtained.

[0217] In another possible implementation of the embodiment of the present application, for a prediction time step, when the extraction module 1702 determines the target prediction mode corresponding to the prediction time step and the prediction word corresponding to the target prediction mode based on the encoding result, it is specifically configured to perform at least one of the following:

[0218] Determining, based on the encoding result, prediction words corresponding to each preconfigured prediction mode at the prediction time step and a target prediction mode corresponding to the prediction time step, and obtaining a prediction word corresponding to the target pre-stored mode based on the prediction words of each prediction mode and the target prediction mode corresponding to the prediction time step;

[0219] Based on the encoding result, a target prediction mode corresponding to the prediction time step is determined from the preconfigured prediction modes, and based on the encoding result, a prediction word corresponding to the target prediction mode is obtained.

[0220] In another possible implementation of the embodiments of the present application, the preconfigured prediction modes include:

[0221] A first prediction mode of keyword prediction based on a word library;

[0222] A second prediction mode of keyword prediction based on each word included in a text line.

[0223] In another possible implementation of the embodiments of the present application, when the target prediction mode is the second prediction mode, the extraction module 1702 is specifically configured to:

[0224] Based on the encoding result, determine the weight corresponding to each word included in the text content in the image to be processed;

[0225] Based on the weight of each word, determine the prediction word corresponding to the target prediction mode.

[0226] In another possible implementation of the embodiments of the present application, the image feature information includes text expression information and image visual information, wherein the text expression information includes at least one of text visual information and text content, the encoding result includes text background representation and image feature representation, the text background representation is obtained based on the text expression information, and the image feature representation is obtained based on the image visual information;

[0227] When the extraction module 1702 determines the weight corresponding to each word included in the text content in the image to be processed based on the encoding result, the extraction module 1702 is specifically configured to:

[0228] Based on the encoding result, the hidden vector corresponding to the current prediction time step is obtained through feature fusion processing;

[0229] Based on the text background representation and the hidden vector, the weight corresponding to each word included in the text content in the image to be processed is determined.

[0230] In another possible implementation of the embodiments of the present application, when the feature information includes text expression information, the text expression information includes at least one of text visual information and text content;

[0231] When the extraction module 1702 encodes the feature information to obtain the encoding result corresponding to the feature information, the extraction module 1702 is specifically configured to at least one of:

[0232] Encode the text expression information to obtain a text line representation;

[0233] encode the text line representation to obtain a text background representation;

[0234] encode the text background representation to obtain a text line structure information and a topic information representation.

[0235] In another possible implementation of the embodiment of the application, the feature information includes text expression information, and the text expression information includes text content, and the text content includes a word sequence corresponding to each text line;

[0236] For a text line, the text content of the text line includes a word sequence corresponding to the text line;

[0237] For a text line, the extraction module 1702 is specifically configured to, when encoding the text content to obtain a text line representation:

[0238] encode the word sequence corresponding to the text line to obtain a word representation based on a character sequence;

[0239] determine the text line representation corresponding to the text line based on the word representation based on the character sequence.

[0240] In another possible implementation of the embodiment of the application, the text line representation includes a text line representation corresponding to at least one text line, and the extraction module 1702 is specifically configured to, when encoding the text line representation to obtain a text line text background representation:

[0241] encode each text line representation to obtain a local text background representation corresponding to each text line;

[0242] encode all text line representations as a whole to obtain a global text background representation corresponding to all text lines;

[0243] determine a text background representation corresponding to each text line based on the local text background representation corresponding to each text line and the global text background representation corresponding to all text lines.

[0244] The keyword extraction device provided in the embodiment of the application can extract the feature information corresponding to the to-be-processed image, and then extract the keyword from the to-be-processed image based on the feature information. That is, in the embodiment of the application, the keyword can be extracted from the to-be-processed image by using the feature information extracted from the to-be-processed image.

[0245] The keyword extraction device provided in the embodiment of the application can execute the keyword extraction method shown in the method embodiment of the application, and the implementation principle is similar, which will not be described here.

[0246] In the above embodiment, a keyword extraction method is introduced from the perspective of a method flow, and a keyword extraction device is introduced from the perspective of a virtual device. In the following embodiment, an electronic device is introduced, which can be used to execute the keyword extraction method shown in the above method embodiment. In the embodiment of the present application, the electronic device can be a terminal device or a server, and is not limited thereto. Details of the electronic device are described in the following embodiment.

[0247] The embodiment of the present application provides an electronic device, such as Figure 18 as shown in the figure, Figure 18 The electronic device 1800 shown in the figure includes a processor 1801 and a memory 1803. The processor 1801 and the memory 1803 are connected, such as through a bus 1802. Optionally, the electronic device 1800 can also include a transceiver 1804. It should be noted that in actual application, the transceiver 1804 is not limited to one, and the structure of the electronic device 1800 does not constitute a limitation on the embodiment of the present application.

[0248] The processor 1801 can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the present application. The processor 1801 can also be a combination of computing functions, such as one or more microprocessor combinations, DSP and microprocessor combinations, etc.

[0249] The bus 1802 can include a channel for transmitting information between the above components. The bus 1802 can be a PCI bus or an EISA bus, etc. The bus 1802 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 18 only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0250] The memory 1803 can be a ROM or other types of static storage devices that can store static information and instructions, a RAM or other types of dynamic storage devices that can store information and instructions, an EEPROM, a CD-ROM or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto.

[0251] The memory 1803 is configured to store application program codes for implementing the solutions of the present application, and the processor 1801 is configured to control the execution of the application program codes. The processor 1801 is configured to execute the application program codes stored in the memory 1803 to implement the content shown in any of the preceding method embodiments.

[0252] The electronic device in the embodiments of the present application comprises a memory and a processor, and at least one program stored in the memory and used for being executed by the processor. When the program is executed by the processor, the program can realize the following compared with the prior art: in the embodiments of the present application, the feature information corresponding to the to-be-processed image is extracted, and then the keyword is extracted from the to-be-processed image based on the feature information. That is, in the embodiments of the present application, the keyword can be extracted from the to-be-processed image by using the feature information extracted from the to-be-processed image.

[0253] The computer readable storage medium in the embodiments of the present application stores a computer program, and when the computer program is executed on a computer, the computer can execute the corresponding content in the preceding method embodiments. Compared with the prior art, in the embodiments of the present application, the feature information corresponding to the to-be-processed image is extracted, and then the keyword is extracted from the to-be-processed image based on the feature information. That is, in the embodiments of the present application, the keyword can be extracted from the to-be-processed image by using the feature information extracted from the to-be-processed image.

[0254] It should be understood that, although each step in the flowchart of the accompanying drawings is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can comprise multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with other steps or sub-steps or stages of other steps.

[0255] The above only describes some embodiments of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, some improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.< / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s>

Claims

1. A keyword extraction method, characterized in that: include: Obtain feature information corresponding to the image to be processed; The feature information includes text expression information and image visual information, the text expression information includes text content and text visual information corresponding to a text line in the image to be processed, the text visual information includes a text feature map corresponding to the text line, and the text feature map includes text line visual features obtained by encoding visual information related to the font in the text line; Based on the image visual information and text expression information, keywords are extracted from the image to be processed, where the keywords are representative words used to summarize the image to be processed.

2. The method according to claim 1, characterized in that The font-related visual information includes at least one of font size, font color, or font shape.

3. The method according to claim 1, characterized in that For a text line, the text visual information corresponding to the text line also includes at least one of the following: Position information of the text line in the image to be processed; Position information of each word in the text line in the image to be processed; The relative position information of each word in the text line.

4. The method according to claim 1, wherein The extracting keywords from the image to be processed includes: Encoding the characteristic information to obtain an encoding result of the characteristic information; Based on the encoding result, keywords are extracted from the image to be processed.

5. The method according to claim 4, characterized in that The encoding result includes at least one of a text context representation obtained based on the text expression information, an image feature representation obtained based on the image visual information, structural information of a text line obtained based on the text context representation, and a topic information representation; And / or, the method further comprises: decoding at least one of the text context representation, the image feature representation, the structural information of the text line, and the topic information; Based on the decoding result, a keyword sequence including the keywords extracted from the image to be processed is obtained.

6. The method according to claim 5, characterized in that The extracting keywords from the image to be processed includes: Determining, based on the encoding result, a target prediction mode corresponding to each prediction time step and a prediction word corresponding to the target prediction mode; Output the predicted word corresponding to each prediction time step; Based on the predicted word sequence corresponding to all prediction time steps, keywords are obtained.

7. The method according to claim 6, characterized in that Determining, based on the encoding result, a target prediction mode corresponding to the prediction time step and a prediction word corresponding to the target prediction mode, including at least one of the following: Determining, based on the encoding result, prediction words corresponding to each preconfigured prediction mode at the prediction time step and a target prediction mode corresponding to the prediction time step, and obtaining a prediction word corresponding to the target pre-stored mode based on the prediction words of each prediction mode and the target prediction mode corresponding to the prediction time step; Based on the encoding result, a target prediction mode corresponding to the prediction time step is determined from preconfigured prediction modes, and based on the encoding result, a prediction word corresponding to the target prediction mode is obtained.

8. The method according to claim 7, characterized in that Preconfigured forecasting modes include: The first prediction mode for keyword prediction based on the vocabulary; The second prediction mode performs keyword prediction based on each word contained in the text line.

9. The method according to claim 8, characterized in that Determining a prediction word corresponding to the target prediction mode includes: In response to the target prediction mode being the second prediction mode, determining, based on the encoding result, weights corresponding to respective words included in the text content in the image to be processed; Based on the weights of the words, a prediction word corresponding to the target prediction mode is determined.

10. The method according to claim 9, characterized in that Determining weights corresponding to respective words contained in the text content of the image to be processed based on the encoding result includes: Obtaining a latent vector corresponding to the current prediction time step based on the encoding result through feature fusion processing; Based on the text context representation and the latent vector, the weights corresponding to the respective words contained in the text content in the image to be processed are determined.

11. The method according to any one of claims 4 to 10, characterized in that: The encoding of the characteristic information to obtain an encoding result corresponding to the characteristic information includes at least one of the following: Encoding the text expression information to obtain a text line representation; Encoding the text line representation to obtain the text context representation; The text context representation is encoded to obtain the text line structure information and topic information representation.

12. The method according to claim 11, characterized in that The text content includes: word sequences corresponding to each text line; For a text line, the text content of the text line includes: a word sequence corresponding to the text line; For a text line, encoding the text content to obtain a text line representation includes: Encode the word sequence corresponding to the text line to obtain a word representation based on the character sequence; A text line representation corresponding to the text line is determined based on the word representation based on the character sequence.

13. The method according to claim 11, characterized in that The text line representation includes a text line representation corresponding to at least one text line, The step of encoding the text line representation to obtain the text line text context representation includes: Encode each text line representation separately to obtain the local text context representation corresponding to each text line; Encoding all text line representations as a whole to obtain a global text context representation corresponding to all text lines; Based on the local text context representation corresponding to each text line and the global text context representation corresponding to all the text lines, the text context representation corresponding to each text line is determined.

14. An electronic device, characterized in that: It includes: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the method according to any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images