Methods, systems, devices, and storage media for extracting text information

Abstract

This specification discloses a method, system, device, and storage medium for extracting text information. It acquires a target image to be processed and target question text for that target image. The target image carries text information, and the target question text instructs the extraction of target text information from the target image. Text recognition is performed on the target image to obtain image text corresponding to the target image, which includes the text information carried in the target image. The target image, image text, and target question text are input into a multimodal processing model. The multimodal processing model processes the target image, image text, and target question text to obtain the target text information. This achieves intelligent text information extraction from the target image. In the above process, the multimodal understanding capability of the multimodal processing model is utilized, that is, information from different modalities is used for text information extraction, resulting in strong generalization ability.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, system, device, and storage medium for extracting text information. Background Technology

[0002] With the development of computer technology, it is increasingly being used to improve efficiency in various scenarios. For example, computer technology is often introduced in situations where text information needs to be extracted from images.

[0003] In related technologies, extracting text information from images usually employs template matching, which involves matching pre-configured text templates within the image to extract the text information.

[0004] However, since the number of text templates is usually limited, the method of extracting text information from images using template matching has poor generalization ability and cannot meet the growing demand. Summary of the Invention

[0005] This specification provides a method, system, device, and storage medium for extracting text information, which can improve the generalization ability of extracting text information from images. The technical solution is as follows:

[0006] On the one hand, a method for extracting text information is provided, the method comprising:

[0007] Acquire a target image and a target question text for the target image, wherein the target image carries text information and the target question text is used to instruct the extraction of target text information from the target image;

[0008] Perform text recognition on the target image to obtain the image text corresponding to the target image;

[0009] The target image, the image text, and the target question text are input into a multimodal processing model. The multimodal processing model processes the target image, the image text, and the target question text to obtain the target text information.

[0010] On the one hand, a text information extraction system is provided, the system comprising: an acquisition module, a recognition module, an input module, and a multimodal processing model;

[0011] The acquisition module is used to acquire a target image and a target question text for the target image, wherein the target image carries text information and the target question text is used to instruct the extraction of target text information from the target image;

[0012] The recognition module is used to perform text recognition on the target image to obtain the image text corresponding to the target image;

[0013] The input module is used to input the target image, the image text, and the target question text into the multimodal processing model;

[0014] The multimodal processing model is used to process the target image, the image text, and the target question text to obtain the target text information.

[0015] In one possible implementation, the multimodal processing model includes an embedding coding layer, an attention coding layer, and a text information extraction layer;

[0016] The embedding coding layer is used to encode the target image, the image text, and the target question text to obtain the image features of the target image, the image text features of the image text, and the question text features of the target question text.

[0017] The attention encoding layer is used to encode the image features of the target image, the image text features of the image text, and the question text features of the target question text based on an attention mechanism to obtain target attention features;

[0018] The text information extraction layer is used to extract the target text information from the image text based on the target attention features.

[0019] In one possible implementation, the embedding coding layer includes an image embedding coding layer, an image text embedding coding layer, and a question text embedding coding layer;

[0020] The image embedding coding layer is used to divide the target image into blocks to obtain multiple image blocks of the target image; and to encode each image block and its position in the multiple image blocks and the target image to obtain the image features.

[0021] The image text embedding encoding layer is used to segment the image text to obtain multiple first words of the image text; and to encode each first word and the position of each first word in the multiple first words and the image text to obtain the image text features.

[0022] The question text embedding encoding layer is used to segment the target question text into words to obtain multiple second words of the target question text; and to encode each second word and the position of each second word in the multiple second words and the target question text to obtain the target question text features.

[0023] In one possible implementation, the image embedding coding layer includes an image content coding sublayer, a first image position coding sublayer, a second image position coding sublayer, and an image feature splicing sublayer;

[0024] The image content encoding sublayer is used to perform a full connection on any image block among the plurality of image blocks to obtain the image block features of the image block;

[0025] The first image location encoding sublayer is used to embed and encode the sequence number of the image block in the plurality of image blocks to obtain the first image block location feature of the image block;

[0026] The second image location encoding sublayer is used to embed and encode the coordinates of the image block in the target image to obtain the second image block location feature of the image block;

[0027] The image feature splicing sub-layer is used to splice the image block features, the first image block position features, and the second image block position features of each image block to obtain the image block embedding features of each image block, and the image block embedding features of each image block constitute the image features.

[0028] In one possible implementation, the image text embedding coding layer includes an image text content coding sublayer, a first image text position coding sublayer, a second image text position coding sublayer, and an image text feature splicing sublayer;

[0029] The image text content encoding sublayer is used to embed and encode any one of the plurality of first words to obtain the first word features of the first word.

[0030] The first image text location encoding sublayer is used to embed and encode the sequence number of the first word in the plurality of first words to obtain the first word location feature of the first word;

[0031] The second image text location encoding sublayer is used to embed and encode the word set to which the first word belongs in the image text, so as to obtain the second word location feature of the first word;

[0032] The image text feature splicing sub-layer is used to splice the first word features, first word position features and second word position features of each first word to obtain the first word embedding features of each first word, and the first word embedding features of each first word constitute the image text features.

[0033] In one possible implementation, the question text embedding encoding layer includes a question text content encoding sublayer, a first question text position encoding sublayer, a second question text position encoding sublayer, and a question text feature splicing sublayer;

[0034] The question text content encoding sublayer is used to embed and encode any one of the plurality of second words to obtain the second word features of the second word.

[0035] The first question text position encoding sublayer is used to embed and encode the sequence number of the second word in the plurality of second words to obtain the third word position feature of the second word;

[0036] The second question text position encoding sublayer is used to embed and encode the word set to which the second word belongs in the target question text, so as to obtain the fourth word position feature of the second word;

[0037] The problem text feature splicing sub-layer is used to splice the second word features, third word position features and fourth word position features of each second word to obtain the second word embedding features of each second word. The second word embedding features of each second word constitute the target problem text features.

[0038] In one possible implementation, the text information extraction layer includes a decoding layer and an extraction layer;

[0039] The decoding layer is used to perform multiple rounds of iterative decoding on the target attention features to obtain the start and end positions of the target text information in the image text;

[0040] The extraction layer is used to extract text information from the image text according to the start position and the end position to obtain the target text information.

[0041] In one possible implementation, the decoding layer includes a first decoder and a second decoder, the parameters of which are different;

[0042] The first decoder is used to perform multiple rounds of iterative decoding on the target attention features to obtain the starting position of the target text information in the image text;

[0043] The second decoder is used to perform multiple rounds of iterative decoding on the target attention features to obtain the termination position of the target text information in the image text.

[0044] In one possible implementation, the recognition module is configured to convolve the target image to obtain image features of the target image; and to perform sequence encoding and sequence decoding on the image features to obtain image text corresponding to the target image.

[0045] Alternatively, the target image can be encoded using an attention mechanism to obtain image features of the target image; the image features can then be decoded using the attention mechanism to obtain image text corresponding to the target image.

[0046] In one possible implementation, the system further includes a sample data engine and a training module;

[0047] The acquisition module is also used to acquire multiple sample images, the sample images carrying text information;

[0048] The recognition module is also used to perform text recognition on each of the sample images to obtain the sample image text corresponding to each of the sample images;

[0049] The sample data engine is used to determine multiple annotation text information for each sample image and sample question text corresponding to each annotation text information based on the sample image text corresponding to each sample image.

[0050] The training module is used to perform multiple rounds of training on the initial model based on the multiple sample images, the sample image text corresponding to each sample image, the multiple labeled text information of each sample image, and the sample question text corresponding to each labeled text information, to obtain the multimodal processing model.

[0051] In one possible implementation, the sample data engine is used to perform field segmentation on the corresponding sample image text of any sample image among the plurality of sample images to obtain a plurality of field groups, each field group including a field name and a field content; determine the field content in the plurality of field groups as a plurality of labeled text information of the sample image; and convert the field name corresponding to each labeled text information in the plurality of field groups into a question to obtain the sample question text corresponding to each labeled text information.

[0052] In one possible implementation, the training module is configured to, in any training round, input the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text into the initial model, process the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text through the initial model to obtain the predicted text information corresponding to the sample image of the current round; and adjust the model parameters of the initial model based on the difference information between the labeled text information and the predicted text information corresponding to the sample image of the current round.

[0053] In one possible implementation, the training module is used to adjust the model parameters of the initial model based on a first difference information between the annotation start position of the annotation text information corresponding to the sample image of the current round and the prediction start position of the predicted text information corresponding to the sample image of the current round in the sample image text, and a second difference information between the annotation end position of the annotation text information corresponding to the sample image of the current round and the prediction end position of the predicted text information corresponding to the sample image of the current round in the sample image text.

[0054] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the text information extraction method.

[0055] On one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the method for extracting the text information.

[0056] On one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the above-mentioned text information extraction method.

[0057] The technical solution provided in the embodiments of this specification obtains a target image to be processed and target question text for the target image. The target image carries text information, and the target question text is used to indicate the extraction of target text information from the target image. The target question text reflects the need for extracting text information. Text recognition is performed on the target image to obtain the image text corresponding to the target image. The image text includes the text information carried in the target image. The target image, the image text, and the target question text are input into a multimodal processing model. The multimodal processing model processes the target image, the image text, and the target question text to obtain the target text information. This achieves intelligent text information extraction from the target image. In the above process, the multimodal understanding capability of the multimodal processing model is utilized, that is, information from different modalities is used for text information extraction, resulting in strong generalization ability. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the implementation environment of a text information extraction method provided in the embodiments of this specification;

[0060] Figure 2 This is a flowchart of a text information extraction method provided in the embodiments of this specification;

[0061] Figure 3 This is a flowchart of another method for extracting text information provided in the embodiments of this specification;

[0062] Figure 4 This is a schematic diagram of a target image provided in an embodiment of this specification;

[0063] Figure 5 This is a schematic diagram of the structure of a multimodal processing model provided in the embodiments of this specification;

[0064] Figure 6 This is a flowchart of a training method for a multimodal processing model provided in the embodiments of this specification;

[0065] Figure 7 This is a schematic diagram of the structure of a text information extraction system provided in the embodiments of this specification;

[0066] Figure 8This is a schematic diagram of another multimodal processing model provided in the embodiments of this specification;

[0067] Figure 9 This is a schematic diagram of the structure of a text information extraction system provided in the embodiments of this specification;

[0068] Figure 10 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this specification clearer, the embodiments of this specification will be further described in detail below with reference to the accompanying drawings.

[0070] In this manual, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0071] In order to introduce the technical solutions provided in this specification, the terms used in the embodiments of this specification will be introduced below.

[0072] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve better results. Machine learning is the core of AI and the fundamental way to endow computers with intelligence; its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0073] Natural Language Processing (NLP): NLP is an important area within artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it is closely related to linguistics, but also has important differences. NLP is not simply the study of natural language in general, but rather the development of computer systems, especially the software systems, that can effectively achieve natural language communication.

[0074] Normalization: Mapping sequences of values ​​with different ranges to the interval (0, 1) to facilitate data processing. In some cases, normalized values ​​can be directly expressed as probabilities.

[0075] Embedded coding, mathematically speaking, represents a correspondence, that is, mapping data in space X to space Y using a function F. This function F is injective, and the mapping result preserves the structure. An injective function means that the mapped data uniquely corresponds to the original data, and preserving the structure means that the size relationship between the original and mapped data is the same. For example, if there are data X1 and X2 before mapping, after mapping we get Y1 corresponding to X1 and Y2 corresponding to X2. If the original data X1 > X2, then correspondingly, the mapped data Y1 > Y2. For words, this means mapping words to another space to facilitate subsequent machine learning and processing.

[0076] Attention weights represent the importance of a piece of data during training or prediction. Importance indicates the magnitude of the influence of input data on output data. Data with high importance corresponds to higher attention weights, while data with low importance corresponds to lower attention weights. The importance of data varies in different scenarios, and training the model to assign attention weights is essentially the process of determining data importance.

[0077] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this manual are authorized by the user or fully authorized by all parties, and the collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0078] After introducing the terms used in the embodiments of this specification, the implementation environment of the embodiments of this specification will be described below. Figure 1 This is a schematic diagram illustrating the implementation environment of a text information extraction method provided in the embodiments of this specification. See also... Figure 1 The implementation environment may include terminal 110 and server 140.

[0079] Terminal 110 is connected to server 140 via a wireless or wired network. Optionally, terminal 110 may be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. Terminal 110 has an application installed and running that supports extracting text information, allowing users to extract desired text information from images.

[0080] Server 140 is a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. Server 140 provides background services for applications running on terminal 110.

[0081] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more, in which case other terminals may also be included in the above implementation environment. This specification does not limit the number of terminals or the type of devices in the embodiments.

[0082] After introducing the implementation environment of the embodiments of this specification, the application scenarios of the embodiments of this specification will be introduced below.

[0083] The technical solutions provided in the embodiments of this specification can be applied to any scenario where text information needs to be extracted from an image. For example, they can be applied to scenarios where text information needs to be extracted from web page images, or to scenarios where text information needs to be extracted from certificates or licenses, or of course, to scenarios where text information needs to be extracted from captured images.

[0084] Taking the application provided in the embodiments of this specification in the scenario of extracting text information from web page images as an example, the following steps are taken: A web page image to be processed and a target question text for that web page image are obtained. The web page image carries text information, and the target question text is used to instruct the extraction of target text information from the web page image. The target question text reflects the need for extracting text information. Text recognition is performed on the web page image to obtain the image text corresponding to the web page image. This image text includes the text information carried in the web page image. The web page image, the image text, and the target question text are input into a multimodal processing model. The multimodal processing model processes the web page image, the image text, and the target question text to obtain the target text information. This achieves intelligent text information extraction from web page images. By using a multimodal processing model instead of a text template for text information extraction, the generalization ability is strong.

[0085] It should be noted that the above description is based on the application of the technical solution provided in the embodiments of this specification to the scenario of extracting text information from web page images. The application of the technical solution provided in the embodiments of this specification to other scenarios belongs to the same inventive concept as described above, and will not be repeated here.

[0086] After introducing the application scenarios of the embodiments in this specification, the text information extraction method provided in the embodiments of this specification will be described below. See [link to documentation]. Figure 2 Taking the server as the executing entity as an example, the method includes the following steps.

[0087] 201. The server obtains a target image and a target question text for the target image. The target image carries text information, and the target question text is used to instruct the extraction of target text information from the target image.

[0088] The target image is the image to be processed. The target question text is used to instruct the extraction of target text information from the target image; that is, it instructs the extraction of the target question information from the text information carried in the target image. This target question text is sent by the terminal to the server, and the terminal can obtain the target question text input by the user. The target text information is the text information indicated by the target question text.

[0089] 202. The server performs text recognition on the target image to obtain the image text corresponding to the target image.

[0090] Among them, text recognition of the target image is used to convert the text information carried in the target image into the image text corresponding to the target image. In other words, the image text carries all the text information carried in the target image, and the text information can be extracted based on the image text.

[0091] 203. The server inputs the target image, the image text, and the target question text into the multimodal processing model, and processes the target image, the image text, and the target question text through the multimodal processing model to obtain the target text information.

[0092] The multimodal processing model is a model with multimodal understanding capabilities and text information extraction capabilities. Multimodal understanding includes text understanding, image understanding, and / or video understanding.

[0093] The technical solution provided in the embodiments of this specification obtains a target image to be processed and target question text for the target image. The target image carries text information, and the target question text is used to indicate the extraction of target text information from the target image. The target question text reflects the need for extracting text information. Text recognition is performed on the target image to obtain the image text corresponding to the target image. The image text includes the text information carried in the target image. The target image, the image text, and the target question text are input into a multimodal processing model. The multimodal processing model processes the target image, the image text, and the target question text to obtain the target text information. This achieves intelligent text information extraction from the target image. In the above process, the multimodal understanding capability of the multimodal processing model is utilized, that is, information from different modalities is used for text information extraction, resulting in strong generalization ability.

[0094] Figure 3 This is a flowchart of a text information extraction method provided in the embodiments of this specification. See also... Figure 3 Taking the server as the executing entity as an example, the method includes the following steps.

[0095] 301. The server obtains a target image and a target question text for the target image. The target image carries text information, and the target question text is used to instruct the extraction of target text information from the target image.

[0096] The target image is the image to be processed. The target question text is used to instruct the extraction of target text information from the target image; that is, it instructs the extraction of the target question information from the text information carried in the target image. This target question text is sent from the terminal to the server, and the terminal can obtain the target question text input by the user. The target text information is the text information indicated by the target question text. For example, see... Figure 4 Taking the target image as document image 401 (e.g., a scanned copy or photograph of a document) as an example, the text information carried by the target image refers to the text content within the document. In the case of an ID card, this text content describes the identity indicated by the ID card. Correspondingly, the target question text could be "Please extract the address from the image," where the address is the target text information. It should be noted that the same target text information may be displayed in different forms in different target images; for example, see [link to relevant documentation]. Figure 4 There is also a license image 402, which also contains an address. As can be seen from the figure, although both are addresses, the display formats in document image 401 and license image 402 are significantly different. Understanding these differences is necessary to achieve more accurate text information extraction.

[0097] In one possible implementation, in response to a text information extraction request, the server obtains the target image and the target question text for the target image from the text information extraction request.

[0098] In this process, the text information extraction request is sent from the terminal to the server. For example, the user selects the target image and enters the target question text through an application running on the terminal. The terminal then packages the target image and the target question text into the text information extraction request through the application and sends the text information extraction request to the server.

[0099] In one possible implementation, the server retrieves the target image and the target question text for the target image from a database that stores multiple candidate images to be processed and candidate question texts for each candidate image.

[0100] In this embodiment, multiple candidate images and candidate question texts for each candidate image are uploaded to the server by the user. Since the server may not be able to process the multiple candidate images and candidate question texts for each candidate image in a timely manner, the server will store the multiple candidate images and candidate question texts for each candidate image in a database for future retrieval. The server can retrieve the candidate images and candidate question texts from the database in a predetermined order. In this embodiment, the candidate images retrieved from the database are also referred to as target images, and the candidate question texts are also referred to as target question texts.

[0101] 302. The server performs text recognition on the target image to obtain the image text corresponding to the target image.

[0102] The text recognition process for the target image involves converting the text information contained in the target image into corresponding image text. In other words, the image text carries all the text information contained in the target image, enabling subsequent text extraction. In some embodiments, the text recognition process for the target image can be viewed as performing Optical Character Recognition (OCR) on the target image, with the aim of converting the text information contained in the target image into image text for subsequent processing.

[0103] In one possible implementation, the server performs convolution on the target image to obtain its image features. The server then performs sequence decoding on these image features to obtain the corresponding image text.

[0104] Image features are an abstract representation of the target image, providing a higher-dimensional representation. Sequence decoding of image features transforms them into image-text, enabling text recognition of the target image.

[0105] For example, the server inputs the target image into a text recognition model, which then uses multiple convolutional kernels to convolve the target image, obtaining multiple convolutional features. The server then fuses these multiple convolutional features using the text recognition model to obtain the image features of the target image. Finally, the server uses a gating mechanism to perform sequence encoding and decoding of these image features using the text recognition model to obtain the image text corresponding to the target image.

[0106] The text recognition model is specifically designed for text recognition of images. It is trained based on multiple sample images and the labeled text of each sample image, and has the ability to recognize text in images. Accordingly, the text recognition model is an OCR model.

[0107] For example, the server inputs the target image into the text recognition model, which then convolves the target image using multiple convolutional kernels of different sizes, obtaining multiple convolutional features of different sizes. The server upsamples or downsamples these multiple convolutional features of different sizes, obtaining multiple convolutional features of the same size. The server then fuses these multiple convolutional features of the same size using the text recognition model to obtain the image features of the target image. Finally, the server encodes these image features using a gating mechanism within the text recognition model, obtaining the corresponding temporal features and latent features. Finally, the server decodes these temporal features and latent features using the gating mechanism within the text recognition model to obtain the image text corresponding to the target image.

[0108] In one possible implementation, the server encodes the target image based on an attention mechanism to obtain the image features of the target image. The server then decodes these image features based on the attention mechanism to obtain the image text corresponding to the target image.

[0109] For example, the server divides the target image into blocks, obtaining multiple image patches. The server then encodes each image patch and its position within the target image using an attention mechanism, obtaining the image features. Finally, the server iteratively decodes these image features using the attention mechanism to obtain the corresponding image text.

[0110] For example, the server divides the target image into blocks, obtaining multiple image patches. The server performs embedding encoding on each image patch and its position within the target image, obtaining the image patch features and positional features of each image patch. The server concatenates the image patch features and positional features of each image patch, obtaining the image patch embedding features of each image patch. The server encodes the image patch embedding features of each image patch using an attention mechanism, obtaining the image patch attention features of each image patch. The server fuses the attention features of each image patch to obtain the image features of the target image. The server performs multiple rounds of iterative decoding on these image features using an attention mechanism to obtain the image text corresponding to the target image.

[0111] In one possible implementation, the server performs region recognition on the target image to obtain at least one target image region, which is the image region corresponding to the text information in the target image. The server then performs text recognition on the at least one target image region to obtain the image text corresponding to the target image.

[0112] Region recognition is a binary classification process, and the result includes whether the region is a target region or not. Text recognition of at least one target image region can be performed using either of the two methods described above; this specification does not limit the specific implementation of this method.

[0113] 303. The server inputs the target image, the image text, and the target question text into the multimodal processing model.

[0114] The multimodal processing model is a model with multimodal understanding and text information extraction capabilities. Multimodal understanding includes text understanding, image understanding, and / or video understanding. In this embodiment, multimodal understanding includes understanding the target image, the text within the image, and the target question text. In some embodiments, the multimodal processing model is a pre-trained and fine-tuned model. Pre-training is to enable the model to have basic multimodal understanding capabilities, while fine-tuning is to enable the model to meet the requirements of a specific task. In this embodiment, this means extracting text information from an image. Subsequent embodiments will describe the training method of this multimodal understanding model. In some embodiments, in addition to obtaining image text through step 302 described above, an OCR model from related technologies can be integrated into the multimodal processing model, thereby endowing the multimodal processing model with OCR capabilities. This allows the target image to be directly input into the multimodal processing model to obtain the corresponding image text. This design takes into account the parameter efficiency of OCR, i.e., the number of parameters required to achieve the same effect. A dedicated OCR model is superior to a multimodal understanding model, thus reducing the number of parameters in the multimodal understanding model and making it more lightweight. In some embodiments, the multimodal processing model is also referred to as ChatKIE. Chat indicates that text information can be extracted from images in a chat-like manner, and KIE stands for Key Information Extraction.

[0115] 304. The server encodes the target image, the image text, and the target question text using the multimodal processing model to obtain the image features of the target image, the image text features of the image text, and the question text features of the target question text.

[0116] Among them, image features are an abstract representation of the target image, capable of representing the target image in a higher dimension; image-text features are an abstract representation of the image-text, capable of representing the image-text in a higher dimension; and question-text features are an abstract representation of the question-text, capable of representing the question-text in a higher dimension. Extracting the target image, image-text, and target text into image features, image-text features, and question-text features respectively is beneficial for subsequent model processing.

[0117] In one possible implementation, the server uses the multimodal processing model to segment the target image into multiple image blocks. The server then uses the multimodal processing model to encode each image block and its position within the multiple image blocks and the target image, obtaining the image features. The server uses the multimodal processing model to segment the image text into multiple first words, obtaining the image text's features. The server uses the multimodal processing model to encode each first word and its position within the multiple first words and the image text, obtaining the image text features. The server uses the multimodal processing model to segment the target question text into multiple second words, obtaining the target question text's features. The server uses the multimodal processing model to encode each second word and its position within the multiple second words and the target question text, obtaining the target question text features.

[0118] The position of an image patch within multiple image patches refers to its order within those patches, also known as its 1D position. The position of an image patch within the target image refers to its coordinates within that image, also known as its 2D position. Similarly, the position of the first word within multiple first words refers to its order within those first words, also known as its 1D position; the position of the second word within multiple second words refers to its order within those second words, also known as its 2D position. The position of the first word within image text refers to the coordinates of the text box containing the first word within the image text, also known as its 1D position; the position of the second word within the target question text refers to the coordinates of the text box containing the second word within the target question text, also known as its 2D position.

[0119] To provide a clearer explanation of the above embodiments, the following description will be divided into several parts.

[0120] Part 1: The server uses the multimodal processing model to divide the target image into blocks, resulting in multiple image blocks of the target image.

[0121] In one possible implementation, the server uses the multimodal processing model to divide the target image into blocks using a preset partitioning method, thereby obtaining multiple image blocks of the target image.

[0122] The preset division method includes the number of divisions and the division size. The number of divisions is the number of multiple image blocks obtained from the division, and the division size is the size of the image blocks obtained from the division. The preset division method is set by technicians according to the actual situation, and this specification does not limit it in the embodiments.

[0123] The second part involves the server encoding each image block and its position within the multiple image blocks and the target image using the multimodal processing model to obtain the image features.

[0124] In one possible implementation, for any image patch among the plurality of image patches, the server performs a fully connected operation on the image patch using the multimodal processing model to obtain the image patch features of that image patch. The server then uses the multimodal processing model to embed and encode the sequence number of the image patch among the plurality of image patches to obtain the first image patch position feature of that image patch. Next, the server uses the multimodal processing model to embed and encode the coordinates of the image patch in the target image to obtain the second image patch position feature of that image patch. Finally, the server uses the multimodal processing model to concatenate the image patch features, the first image patch position feature, and the second image patch position feature of each image patch to obtain the image patch embedding features of each image patch. The image patch embedding features of all image patches constitute the image feature.

[0125] The fully connected layer on the image patch is equivalent to performing a linear transformation on the image patch, converting it into a sequence for processing. The index of an image patch within a set of image patches reflects its order within the set. This index reflects the relative position of the image patch within the target image. The order of the image patches is determined by their position in the target image; for example, the image patches in the target image can be sorted from left to right and from top to bottom to obtain their indexes. The coordinates of an image patch in the target image can be the coordinates of its center point, its corner points, or a combination of both. This specification does not limit the specific coordinates used in the embodiments.

[0126] In one possible implementation, for any image patch among the plurality of image patches, the server performs convolution on the image patch using the multimodal processing model to obtain the image patch features of that image patch. The server then uses the multimodal processing model to embed and encode the sequence number of the image patch among the plurality of image patches to obtain the first image patch position feature of that image patch. Next, the server uses the multimodal processing model to embed and encode the coordinates of the image patch in the target image to obtain the second image patch position feature of that image patch. Finally, the server uses the multimodal processing model to concatenate the image patch features, the first image patch position feature, and the second image patch position feature of each image patch to obtain the image patch embedding features of each image patch, which together constitute the image feature.

[0127] The third part describes how the server uses the multimodal processing model to segment the image text into words, obtaining multiple first words of the image text.

[0128] In one possible implementation, the server segments the image text using the word segmentation sub-model of the multimodal processing model to obtain multiple first words of the image text.

[0129] The word segmentation model is used to divide the input product title into multiple independent and meaningful reference product description words to facilitate subsequent processing. In some embodiments, the word segmentation sub-model is the jiaba word segmentation model, the HanLP word segmentation model, or the FoolNLTK word segmentation model, etc. Of course, with the development of science and technology, other types of word segmentation models can also be used, and this specification does not limit them.

[0130] Part Four: The server uses the multimodal processing model to encode each first word and its position within the multiple first words and the image text, thereby obtaining the image text features.

[0131] In one possible implementation, for any one of the plurality of first words, the server uses the multimodal processing model to embed and encode the first word to obtain the first word feature. The server uses the multimodal processing model to embed and encode the sequence number of the first word among the plurality of first words to obtain the first word position feature. The server uses the multimodal processing model to embed and encode the word set to which the first word belongs in the image text to obtain the second word position feature. The server uses the multimodal processing model to concatenate the first word features, first word position features, and second word position features of each first word to obtain the first word embedding features of each first word. The first word embedding features of each first word constitute the image text features.

[0132] Part 5: The server uses this multimodal processing model to segment the target question text into words, obtaining multiple second words of the target question text.

[0133] In one possible implementation, the server segments the target question text using the word segmentation sub-model of the multimodal processing model to obtain multiple second words of the image text.

[0134] Part 6: The server uses this multimodal processing model to encode each second word and its position in the multiple second words and the target question text, thereby obtaining the target question text features.

[0135] In one possible implementation, for any one of the plurality of second words, the server uses the multimodal processing model to embed and encode the second word to obtain its second word feature. The second word's sequence number within the plurality of second words is then embedded and encoded to obtain its third word position feature. The word set to which the second word belongs in the target question text is also embedded and encoded to obtain its fourth word position feature. The server then uses the multimodal processing model to concatenate the second word features, third word position features, and fourth word position features of each second word to obtain the second word embedding features of each second word. These second word embedding features constitute the target question text features.

[0136] 305. The server uses this multimodal processing model to encode the image features of the target image, the image text features of the image text, and the question text features of the target question text based on the attention mechanism, thereby obtaining the target attention features.

[0137] Among them, the target attention feature is the feature obtained by fusing multimodal data based on the attention mechanism.

[0138] In one possible implementation, the server uses the multimodal processing model to concatenate the image features of the target image, the image text features of the image text, and the question text features of the target question text to obtain a concatenated feature. This concatenated feature includes multiple sub-features, such as image patch embedding features of each image patch in the target image, first word embedding features of each first word in the image text, and second word embedding features of each second word in the target question text. The server then uses the multimodal processing model to encode each sub-feature based on an attention mechanism to obtain attention features for each sub-feature. Finally, the server uses the multimodal processing model to fuse the attention features of each sub-feature to obtain the target attention feature.

[0139] To provide a clearer explanation of the above implementation methods, the following will be divided into two parts to describe the methods for obtaining the attention features of each sub-feature and the target attention features in the above implementation methods.

[0140] Part 1: The server uses this multimodal processing model to encode each sub-feature based on the attention mechanism, thus obtaining the attention features of each sub-feature.

[0141] In one possible implementation, the server uses the multimodal processing model to perform linear transformations on each sub-feature, obtaining the query matrix, key matrix, and value matrix of each sub-feature. Based on the query matrix and key matrix of each sub-feature, the server determines the attention weight of each sub-feature using the multimodal processing model. For example, the server uses the multimodal processing model to perform linear transformations on each sub-feature using three linear transformation matrices, obtaining the query feature, key feature, and value feature of each sub-feature. For any sub-feature among these multiple sub-features, the server uses the multimodal processing model to multiply the query feature of that sub-feature with the key features among the multiple sub-features, obtaining the attention weights of each sub-feature for that sub-feature. The server uses the multimodal processing model to multiply the value features of each sub-feature using the attention weights of each sub-feature for that sub-feature, obtaining the attention feature of that sub-feature.

[0142] The second part involves the server fusing the attention features of each sub-feature through this multimodal processing model to obtain the target attention feature.

[0143] In one possible implementation, the server uses the multimodal processing model to add or concatenate the attention features of each sub-feature to obtain the target attention feature.

[0144] 306. The server extracts the target text information from the image text based on the target attention feature using the multimodal processing model.

[0145] In one possible implementation, the server uses the multimodal processing model to iteratively decode the target attention features multiple times to obtain the start and end positions of the target text information in the image text. The server then extracts text information from the image text according to these start and end positions to obtain the target text information.

[0146] For example, taking an image of an identification document as the target image, the text information carried by the target image refers to the text content within the document. In the case of an ID card, this text content describes the identity indicated by the ID card. Correspondingly, the target question text could be "Please extract the address from the image," and the target text information is the address "XX Province, XX City, XX District" in the identification image. The starting position of the target text information within the image text refers to the coordinates of the first character of the target text information within the image text, and correspondingly, the ending position of the target text information within the image text refers to the coordinates of the last character of the target text information within the image text. In other words, the target text information belongs to the image text, and the multimodal processing model ultimately outputs the starting and ending positions of the target text information, rather than directly outputting the target text information. This method is also known as the extraction-based answer content method, compared to the generative answer content method in related technologies (where the generated content is freely generated by the model). In the embodiments of this specification, the extraction-based answer generation method is chosen because, with the same number of parameters in the multimodal processing model, the accuracy of the content generated by extraction is higher than that of the generative method. In other words, using extraction can reduce the number of parameters as much as possible while ensuring accuracy, thereby reducing the deployment difficulty of the multimodal processing model and improving the processing speed of the multimodal processing model.

[0147] To provide a clearer explanation of the above embodiments, the following description will be divided into two parts.

[0148] In the first part, the server uses the multimodal processing model to perform multiple rounds of iterative decoding on the target attention features to obtain the start and end positions of the target text information in the image text.

[0149] In one possible implementation, the server inputs the target attention feature into the first decoder of the multimodal processing model, and the first decoder performs multiple rounds of iterative decoding on the target attention feature to obtain the starting position of the target text information in the image text. The server then inputs the target attention feature into the second decoder of the multimodal processing model, and the second decoder performs multiple rounds of iterative decoding on the target attention feature to obtain the ending position of the target text information in the image text. The parameters of the first decoder and the second decoder are different.

[0150] The first decoder corresponds to the task of decoding the start position, which is used to decode the start position of the target text information. The second decoder corresponds to the task of decoding the end position, which is used to decode the end position of the target text information. Accordingly, this multimodal processing model is a multi-task model.

[0151] For example, the server inputs the target attention feature into the first decoder of the multimodal processing model. In the first round of decoding, the server decodes the target attention feature and the start symbol using the first decoder to obtain the first character of the starting position. In the second round of iterative decoding, the server decodes the target attention feature, the start symbol, and the first character using the first decoder to obtain the second character of the starting position, and so on, until the end symbol is decoded, indicating that the decoding of the starting position is complete, and the complete starting position is obtained. The server inputs the target attention feature into the second decoder of the multimodal processing model. In the first round of decoding, the server decodes the target attention feature and the start symbol using the second decoder to obtain the first character of the ending position. In the second round of iterative decoding, the server decodes the target attention feature, the start symbol, and the first character using the second decoder to obtain the second character of the ending position, and so on, until the end symbol is decoded, indicating that the decoding of the ending position is complete, and the complete ending position is obtained.

[0152] The second part involves the server extracting text information from the image text according to the start and end positions to obtain the target text information.

[0153] In one possible implementation, the server extracts characters from the image text starting from the starting position until the ending position, obtaining multiple characters, which are the target text information.

[0154] The following is combined Figure 5 For an explanation of steps 301-306 above, please refer to [link / reference]. Figure 5The server acquires a target image and a target question text 502 for that target image 501. The server performs text recognition on the target image 501 to obtain the corresponding image text 503. The server inputs the target image 501, the target question text 502, and the image text 503 into a multimodal processing model 504. The embedding encoding layer 5041 of the multimodal processing model 504 encodes the target image 501, the target question text 502, and the image text 503 to obtain image features of the target image 501, image text features of the image text 502, and question text features of the target question text 503. The server encodes the image features of the target image 501, the image text features of the image text 502, and the question text features of the target question text 503 based on an attention mechanism using the attention encoding layer 5042 of the multimodal processing model 504 to obtain target attention features. The server iteratively decodes the target attention features through multiple rounds using the first decoder 5043 of the multimodal processing model 504 to obtain the starting position of the target text information. The server uses the second decoder 5044 of the multimodal processing model 504 to perform multiple rounds of iterative decoding on the target attention features to obtain the termination position of the target text information. Based on the start and end positions, the server extracts the target text information from the image text.

[0155] All the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this specification, and will not be described in detail here.

[0156] The technical solution provided in the embodiments of this specification obtains a target image to be processed and target question text for the target image. The target image carries text information, and the target question text is used to indicate the extraction of target text information from the target image. The target question text reflects the need for extracting text information. Text recognition is performed on the target image to obtain the image text corresponding to the target image. The image text includes the text information carried in the target image. The target image, the image text, and the target question text are input into a multimodal processing model. The multimodal processing model processes the target image, the image text, and the target question text to obtain the target text information. This achieves intelligent text information extraction from the target image. In the above process, the multimodal understanding capability of the multimodal processing model is utilized, that is, information from different modalities is used for text information extraction, resulting in strong generalization ability.

[0157] In addition to the text information extraction methods described above, this specification also provides a training method for a multimodal processing model. (See attached document for details.) Figure 6 Taking the server as the executing entity as an example, the method includes the following steps.

[0158] 601. The server acquires multiple sample images, each carrying text information.

[0159] The sample images are those used for model training.

[0160] 602. The server performs text recognition on each sample image to obtain the sample image text corresponding to each sample image.

[0161] Steps 602 and 302 belong to the same inventive concept, and the implementation process is described in the relevant description of step 302 above, and will not be repeated here. In some embodiments, text recognition can also be implemented through GKIE, which is a general image information extraction interface. In some embodiments, the sample image text corresponding to each sample image belongs to the D0 dataset. The D0 dataset is used to store the sample image text obtained after directly performing text recognition on the sample images, and one sample image text corresponds to one sample image. In some embodiments, in order to ensure that the sample image text can be directly found in the sample image, the post-processing of text recognition is canceled. For example, "minus 1 for 30; minus 3 for 50" in the sample image is directly converted to "minus 1 for 30 and minus 3 for 50".

[0162] 603. Based on the sample image text corresponding to each sample image, the server determines multiple annotation text information for each sample image and the sample question text corresponding to each annotation text information.

[0163] In one possible implementation, for any one of the plurality of sample images, the server performs field segmentation on the corresponding sample image text to obtain multiple field groups, each field group including a field name and a field content. The server determines the field content in the plurality of field groups as multiple labeled text information of the sample image. The server performs question conversion on the field names corresponding to each labeled text information in the plurality of field groups to obtain sample question text corresponding to each labeled text information.

[0164] The field name reflects the type of the corresponding field content. For example, a field group might be "Store Discount" or "Discount of 30 RMB off 1 RMB, Discount of 50 RMB off 3 RMB". The field name would be "Store Discount", and the field content would be "Discount of 30 RMB off 1 RMB, Discount of 50 RMB off 3 RMB". Question transformation converts declarative sentences into interrogative sentences. For example, the above "Store Discount" would be transformed into "What is the store discount activity?", thus achieving automated generation of sample question text. In some embodiments, question transformation is also called Instruct modification to adapt the model to natural language input.

[0165] For example, for any one of the multiple sample images, the server performs word segmentation on the corresponding sample image text, obtaining multiple sample words in the sample image text. The server combines and matches these sample words based on their semantics, resulting in multiple field groups. The server determines the field content in these multiple field groups as multiple labeled text information for the sample image, and records the start and end positions of each labeled text information in the sample image text for subsequent training. The server uses the target transformation rule from among various transformation rules to perform question transformation on the field names corresponding to each labeled text information in the multiple field groups, obtaining the sample question text corresponding to each labeled text information.

[0166] Different transformation rules correspond to different transformation methods, so a field name may be transformed into different sample question texts, which helps improve the model's generalization ability. In addition, the sample question text will include the corresponding image type or scene description of the sample image with a preset probability, which helps the model learn the corresponding information extraction methods for different types of images. The target transformation rule is randomly determined from multiple transformation rules. In this embodiment, the combination of field name and field content belongs to the D1 dataset; for example, ["store discount", "discount of 30 for purchases over 50", etc.] belongs to the D1 dataset. The combination of sample question text and labeled text information obtained after question transformation belongs to the D2 dataset. For example, [“What is the store's discount activity?” “Discount of 30 RMB off for purchases over 50 RMB, discount of 1 RMB off for purchases over 50 RMB”]. Of course, to facilitate model training, a start position and an end position of the label can be added to the combination of sample question text and labeled text information, resulting in [“What is the store's discount activity?” “Discount of 30 RMB off for purchases over 50 RMB, discount of 1 RMB off for purchases over 50 RMB”, (52, 61)]. Here, 52 is the label start position, meaning that the first character of the labeled text information is the 52nd character in the corresponding sample image text; 61 is the label end position, meaning that the last character of the labeled text information is the 61st character in the corresponding sample image text. In addition, if there are annotation results of the sample data in the D0 dataset by technicians, they can be directly stored in the D1 dataset. In some embodiments, after obtaining the D2 dataset, the server divides the D2 dataset into a training set and a validation set. The principle of the division is to ensure that there are no completely identical sample data in the training set and the validation set.

[0167] In some embodiments, for different data sources, there will be the aforementioned D0 dataset, D1 dataset, and D2 dataset. After fusing the D2 datasets corresponding to different data sources, the total D3 dataset can be obtained. The sample images from different data sources correspond to different types of pages. For example, multiple data sources may include merchant homepages, product introduction pages, activity pages, and payment pages, etc. The multiple sample images corresponding to the merchant homepage are images from different merchant homepages.

[0168] Furthermore, after obtaining the D3 dataset, the sample data in the D3 dataset is processed to obtain the D4 dataset. The D4 dataset stores sample data in the form of QA pairs, where Q stands for Question, referring to the combination of the sample question text and sample image, and A stands for Answer, referring to the start and end positions of the annotation text information. Of course, after obtaining D4, the server can also perform format conversion on D4 to ensure that D4 meets the format requirements for model training.

[0169] In some embodiments, the transformations from D0 to D1, D1 to D2, D2 to D3, and D3 to D4 described above are implemented through a data engine.

[0170] 604. Based on the multiple sample images, the sample image text corresponding to each sample image, the multiple labeled text information of each sample image, and the sample question text corresponding to each labeled text information, the server performs multiple rounds of training on the initial model to obtain the multimodal processing model.

[0171] In one possible implementation, during any training round, the server inputs the sample image of the current round from the multiple sample images, the corresponding sample image text, and the sample question text into the initial model. The server processes the sample image, the corresponding sample image text, and the sample question text of the current round using the initial model to obtain the predicted text information corresponding to the sample image of the current round. Based on the difference between the labeled text information and the predicted text information corresponding to the sample image of the current round, the server adjusts the model parameters of the initial model.

[0172] The following describes how the server adjusts the model parameters of the initial model based on the difference between the labeled text information and the predicted text information corresponding to the sample image in the current round, as described in the above embodiment.

[0173] In some embodiments, the difference information includes first difference information and second difference information. The server adjusts the model parameters of the initial model based on the first difference information between the annotation start position of the annotation text information corresponding to the sample image of the current round and the prediction start position of the predicted text information corresponding to the sample image of the current round in the sample image text, and the second difference information between the annotation end position of the annotation text information corresponding to the sample image of the current round and the prediction end position of the predicted text information corresponding to the sample image of the current round in the sample image text.

[0174] Figure 7This is a schematic diagram of the structure of a text information extraction system provided in the embodiments of this specification. See also... Figure 7 The system includes: an acquisition module 701, a recognition module 702, an input module 703, and a multimodal processing model 704.

[0175] The acquisition module 701 is used to acquire a target image and a target question text for the target image. The target image carries text information, and the target question text is used to instruct the extraction of target text information from the target image.

[0176] The recognition module 702 is used to perform text recognition on the target image to obtain the image text corresponding to the target image.

[0177] The input module 703 is used to input the target image, the image text, and the target question text into the multimodal processing model 704.

[0178] The multimodal processing model 704 is used to process the target image, the image text, and the target question text to obtain the target text information.

[0179] In one possible implementation, see Figure 8 The multimodal processing model 704 includes an embedding coding layer 7041, an attention coding layer 7042, and a text information extraction layer 7043.

[0180] The embedding coding layer 7041 is used to encode the target image, the image text, and the target question text to obtain the image features of the target image, the image text features of the image text, and the question text features of the target question text.

[0181] The attention encoding layer 7042 is used to encode the image features of the target image, the image text features of the image text, and the question text features of the target question text based on the attention mechanism, so as to obtain the target attention features.

[0182] The text information extraction layer 7043 is used to extract the target text information from the image text based on the target attention features.

[0183] In one possible implementation, see Figure 8 The embedding coding layer 7041 includes an image embedding coding layer 70411, an image text embedding coding layer 70412, and a question text embedding coding layer 70413.

[0184] The image embedding coding layer 70411 is used to divide the target image into blocks, resulting in multiple image blocks of the target image. Each image block and its position within the multiple image blocks and the target image are encoded to obtain the image features.

[0185] The image text embedding coding layer 70412 is used to segment the image text into multiple first words. Each first word and its position within the multiple first words and the image text are encoded to obtain the image text features.

[0186] The question text is embedded in encoding layer 70413, which is used to segment the target question text into multiple second words. Each second word and its position within the multiple second words and the target question text are encoded to obtain the features of the target question text.

[0187] In one possible implementation, see Figure 8 The image embedding coding layer 70411 includes an image content coding sublayer 704111, a first image position coding sublayer 704112, a second image position coding sublayer 704113, and an image feature splicing sublayer 704114.

[0188] The image content encoding sublayer 704111 is used to perform a full connection on any one of the multiple image blocks to obtain the image block features of that image block.

[0189] The first image position encoding sublayer 704112 is used to embed and encode the sequence number of the image block in the multiple image blocks to obtain the first image block position feature of the image block.

[0190] The second image location encoding sublayer 704113 is used to embed and encode the coordinates of the image block in the target image to obtain the second image block location feature of the image block.

[0191] The image feature splicing sublayer 704114 is used to splice the image block features, the first image block position features and the second image block position features of each image block to obtain the image block embedding features of each image block, and the image block embedding features of each image block constitute the image feature.

[0192] In one possible implementation, see Figure 8 The image text embedding coding layer 70412 includes an image text content coding sublayer 704121, a first image text position coding sublayer 704122, a second image text position coding sublayer 704123, and an image text feature splicing sublayer 704124.

[0193] The image text content encoding sublayer 704121 is used to embed and encode any one of the multiple first words to obtain the first word features of the first word.

[0194] The first image text position encoding sublayer 704122 is used to embed and encode the sequence number of the first word in the multiple first words to obtain the first word position feature of the first word.

[0195] The second image text location encoding sublayer 704123 is used to embed and encode the word set to which the first word belongs in the image text, thereby obtaining the second word location feature of the first word.

[0196] The image text feature splicing sublayer 704124 is used to splice the first word features, first word position features and second word position features of each first word to obtain the first word embedding features of each first word, and the first word embedding features of each first word constitute the image text features.

[0197] In one possible implementation, see Figure 8 The question text embedding encoding layer 70413 includes a question text content encoding sublayer 704131, a first question text position encoding sublayer 704132, a second question text position encoding sublayer 704133, and a question text feature splicing sublayer 704134.

[0198] The text content encoding sublayer 704131 is used to embed and encode any one of the multiple second words to obtain the second word features of that second word.

[0199] The first problem text position encoding sublayer 704132 is used to embed and encode the sequence number of the second word in the multiple second words to obtain the third word position feature of the second word.

[0200] The second question text position encoding sublayer 704133 is used to embed and encode the word set to which the second word belongs in the target question text, thereby obtaining the fourth word position feature of the second word.

[0201] The text feature splicing sublayer 704134 is used to splice the second word features, third word position features and fourth word position features of each second word to obtain the second word embedding features of each second word. The second word embedding features of each second word constitute the target text feature.

[0202] In one possible implementation, see Figure 8 The text information extraction layer 7043 includes a decoding layer 70431 and an extraction layer 70432.

[0203] The decoding layer 70431 is used to perform multiple rounds of iterative decoding on the attention features of the target to obtain the start and end positions of the target text information in the image text.

[0204] The extraction layer 70432 is used to extract text information from the image text according to the start position and the end position to obtain the target text information.

[0205] In one possible implementation, see Figure 8 The decoding layer 70431 includes a first decoder 704311 and a second decoder 704312, the parameters of which are different.

[0206] The first decoder 704311 is used to perform multiple rounds of iterative decoding on the attention features of the target to obtain the starting position of the target text information in the image text.

[0207] The second decoder 704312 is used to perform multiple rounds of iterative decoding on the attention features of the target to obtain the termination position of the target text information in the image text.

[0208] In one possible implementation, the recognition module 702 is used to convolve the target image to obtain image features of the target image. The image features are then sequence-encoded and sequence-decoded to obtain the image text corresponding to the target image.

[0209] Alternatively, the target image can be encoded using an attention mechanism to obtain its image features. These image features can then be decoded using the same attention mechanism to obtain the corresponding image text.

[0210] In one possible implementation, see Figure 9 The system also includes a sample data engine 705 and a training module 706.

[0211] The acquisition module 701 is also used to acquire multiple sample images, which carry text information.

[0212] The recognition module 702 is also used to perform text recognition on each of the sample images to obtain the sample image text corresponding to each sample image.

[0213] The sample data engine 705 is used to determine multiple annotation text information for each sample image and the sample question text corresponding to each annotation text information based on the sample image text corresponding to each sample image.

[0214] The training module 706 is used to train the initial model in multiple rounds based on the multiple sample images, the sample image text corresponding to each sample image, the multiple labeled text information of each sample image, and the sample question text corresponding to each labeled text information, to obtain the multimodal processing model 704.

[0215] In one possible implementation, the sample data engine 705 is used to perform field segmentation on the corresponding sample image text for any one of the plurality of sample images, obtaining multiple field groups, each field group including a field name and a field content. The field content in the plurality of field groups is determined as multiple labeled text information of the sample image. The field names corresponding to each labeled text information in the plurality of field groups are converted into questions to obtain sample question text corresponding to each labeled text information.

[0216] In one possible implementation, the training module 706 is configured to, in any training round, input the sample image of the current round, the corresponding sample image text, and the sample question text from the plurality of sample images into the initial model. The initial model processes the sample image of the current round, the corresponding sample image text, and the sample question text to obtain the predicted text information corresponding to the sample image of the current round. Based on the difference between the labeled text information and the predicted text information corresponding to the sample image of the current round, the model parameters of the initial model are adjusted.

[0217] In one possible implementation, the training module 706 is used to adjust the model parameters of the initial model based on a first difference information between the annotation start position of the annotation text information corresponding to the sample image of the current round and the prediction start position of the predicted text information corresponding to the sample image of the current round in the sample image text, and a second difference information between the annotation end position of the annotation text information corresponding to the sample image of the current round and the prediction end position of the predicted text information corresponding to the sample image of the current round in the sample image text.

[0218] It should be noted that the text information extraction system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the text information extraction system and the text information extraction method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0219] The technical solution provided in the embodiments of this specification obtains a target image to be processed and target question text for the target image. The target image carries text information, and the target question text is used to indicate the extraction of target text information from the target image. The target question text reflects the need for extracting text information. Text recognition is performed on the target image to obtain the image text corresponding to the target image. The image text includes the text information carried in the target image. The target image, the image text, and the target question text are input into a multimodal processing model. The multimodal processing model processes the target image, the image text, and the target question text to obtain the target text information. This achieves intelligent text information extraction from the target image. In the above process, the multimodal understanding capability of the multimodal processing model is utilized, that is, information from different modalities is used for text information extraction, resulting in strong generalization ability.

[0220] Figure 10 This is a schematic diagram of the structure of a computer device 1000 provided in the embodiments of this specification. The computer device 1000 can vary considerably due to differences in configuration or performance. It may include one or more processors (Central Processing Units, CPUs) 1001 and one or more memories 1002. The one or more memories 1002 store at least one computer program, which is loaded and executed by the one or more processors 1001 to implement the methods provided in the various method embodiments described above. Of course, the computer device 1000 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device 1000 may also include other components for implementing device functions, which will not be elaborated upon here.

[0221] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the text information extraction method in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0222] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the above-described text information extraction method.

[0223] In some embodiments, the computer program described in this specification may be deployed and executed on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network may constitute a blockchain system.

[0224] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0225] The above are merely optional embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification shall be included within the scope of protection of this specification.

Claims

1. A method for extracting text information, the method comprising: Acquire a target image and a target question text for the target image, wherein the target image carries text information and the target question text is used to instruct the extraction of target text information from the target image; Perform text recognition on the target image to obtain the image text corresponding to the target image; The target image, the image text, and the target question text are input into a multimodal processing model, and the target image, the image text, and the target question text are processed by the multimodal processing model to obtain the target text information; The method further includes: The target attention features are iteratively decoded in multiple rounds to obtain the start and end positions of the target text information in the image text. The start and end positions are used to determine the target text information. The target attention features are features obtained by fusing multimodal data based on an attention mechanism. The process of performing multiple rounds of iterative decoding on the target attention features to obtain the start and end positions of the target text information in the image text includes: The target attention feature is input into the first decoder of the multimodal processing model, and the first decoder performs multiple rounds of iterative decoding on the target attention feature to obtain the starting position of the target text information in the image text; The target attention feature is input into the second decoder of the multimodal processing model. The second decoder performs multiple rounds of iterative decoding on the target attention feature to obtain the termination position of the target text information in the image text. The parameters of the first decoder and the second decoder are different.

2. The method according to claim 1, wherein processing the target image, the image text, and the target question text through the multimodal processing model to obtain the target text information includes: The target image, the image text, and the target question text are encoded using the multimodal processing model to obtain the image features of the target image, the image text features of the image text, and the question text features of the target question text. The multimodal processing model encodes the image features of the target image, the image text features of the image text, and the question text features of the target question text based on an attention mechanism to obtain the target attention features. The target text information is extracted from the image text using the multimodal processing model based on the target attention features.

3. The method according to claim 2, wherein encoding the target image, the image text, and the target question text to obtain image features of the target image, image text features of the image text, and question text features of the target question text includes: The target image is divided into blocks to obtain multiple image blocks of the target image; The image features are obtained by encoding each image block and the position of each image block in the plurality of image blocks and the target image; The image text is segmented to obtain multiple first words; each first word and its position in the multiple first words and the image text are encoded to obtain the image text features. The target question text is segmented to obtain multiple second words; each second word and its position in the multiple second words and the target question text are encoded to obtain the features of the target question text.

4. The method according to claim 3, wherein encoding each of the image blocks and the position of each of the image blocks in the plurality of image blocks and the target image to obtain the image features includes: For any image block among the plurality of image blocks, perform a full connection on the image block to obtain the image block features of the image block; The sequence number of the image block in the plurality of image blocks is embedded and encoded to obtain the first image block position feature of the image block; The coordinates of the image block in the target image are embedded and encoded to obtain the second image block position feature of the image block; The image block features, the first image block position features, and the second image block position features of each image block are concatenated to obtain the image block embedding features of each image block, and the image block embedding features of each image block constitute the image features.

5. The method according to claim 3, wherein encoding each of the first words and the position of each of the first words in the plurality of first words and the image text to obtain the image text features includes: For any one of the plurality of first words, the first word is embedded and encoded to obtain the first word feature of the first word; The first word is embedded and encoded in the sequence number of the first word in the plurality of first words to obtain the first word position feature of the first word; The word set to which the first word belongs in the image text is embedded and encoded to obtain the second word position feature of the first word; The first word features, first word position features, and second word position features of each first word are concatenated to obtain the first word embedding features of each first word. The first word embedding features of each first word constitute the image text features.

6. The method according to claim 3, wherein encoding each of the second words and the position of each of the second words in the plurality of second words and the target question text to obtain the target question text features includes: For any one of the plurality of second words, embedding encoding is performed on the second word to obtain the second word feature of the second word; The second word is embedded and encoded in the sequence number of the plurality of second words to obtain the third word position feature of the second word; The word set to which the second word belongs in the target question text is embedded and encoded to obtain the fourth word position feature of the second word; The second word features, third word position features, and fourth word position features of each second word are concatenated to obtain the second word embedding features of each second word. The second word embedding features of each second word constitute the target question text features.

7. The method according to claim 2, wherein encoding the image features of the target image, the image text features of the image text, and the question text features of the target question text based on an attention mechanism to obtain target attention features includes: The image features of the target image, the image text features of the image text, and the question text features of the target question text are concatenated to obtain concatenated features. The concatenated features include multiple sub-features, including image block embedding features of each image block in the target image, first word embedding features of each first word in the image text, and second word embedding features of each second word in the target question text. Each of the sub-features is encoded based on an attention mechanism to obtain the attention features of each of the sub-features; The attention features of each of the sub-features are fused to obtain the target attention feature.

8. The method according to claim 2, wherein extracting the target text information from the image text based on the target attention feature comprises: Text information is extracted from the image text according to the start position and the end position to obtain the target text information.

9. The method according to claim 1, wherein performing text recognition on the target image to obtain image text corresponding to the target image includes: Convolve the target image to obtain its image features; The image features are sequence encoded and sequence decoded to obtain the image text corresponding to the target image; Alternatively, the target image can be encoded based on an attention mechanism to obtain its image features; The image features are decoded based on an attention mechanism to obtain the image text corresponding to the target image.

10. The method according to claim 1, wherein the training method of the multimodal processing model comprises: Acquire multiple sample images, each carrying text information; Perform text recognition on each of the sample images to obtain the sample image text corresponding to each of the sample images; Based on the sample image text corresponding to each of the sample images, determine multiple annotation text information for each of the sample images and the sample question text corresponding to each of the annotation text information; Based on the multiple sample images, the sample image text corresponding to each sample image, the multiple labeled text information of each sample image, and the sample question text corresponding to each labeled text information, the initial model is trained in multiple rounds to obtain the multimodal processing model.

11. The method according to claim 10, wherein determining multiple annotation text information for each sample image and sample question text corresponding to each annotation text information based on the sample image text corresponding to each sample image includes: For any sample image among the plurality of sample images, the corresponding sample image text of the sample image is segmented by field to obtain multiple field groups, each of which includes a field name and a field content; The field content in the multiple field groups is determined as multiple labeled text information of the sample image; The field names corresponding to each of the labeled text information in the multiple field groups are converted into questions to obtain sample question texts corresponding to each of the labeled text information.

12. The method according to claim 10, wherein the step of training the initial model in multiple rounds based on the plurality of sample images, the sample image text corresponding to each sample image, the plurality of labeled text information of each sample image, and the sample question text corresponding to each labeled text information includes: In any training round, the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text are input into the initial model. The initial model processes the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text to obtain the predicted text information corresponding to the sample image of the current round. Based on the difference between the labeled text information and the predicted text information corresponding to the sample images in the current round, the model parameters of the initial model are adjusted.

13. The method according to claim 12, wherein the difference information includes first difference information and second difference information, and adjusting the model parameters of the initial model based on the difference information between the labeled text information and the predicted text information corresponding to the sample image of the current round includes: Based on the first difference information between the annotation start position of the annotation text information corresponding to the sample image of the current round and the prediction start position of the predicted text information corresponding to the sample image of the current round in the sample image text, and the second difference information between the annotation end position of the annotation text information corresponding to the sample image of the current round and the prediction end position of the predicted text information corresponding to the sample image of the current round in the sample image text, the model parameters of the initial model are adjusted.

14. A text information extraction system, the system comprising an acquisition module, a recognition module, an input module, a decoding layer, and a multimodal processing model; The acquisition module is used to acquire a target image and a target question text for the target image, wherein the target image carries text information and the target question text is used to instruct the extraction of target text information from the target image; The recognition module is used to perform text recognition on the target image to obtain the image text corresponding to the target image; The input module is used to input the target image, the image text, and the target question text into the multimodal processing model; The multimodal processing model is used to process the target image, the image text, and the target question text to obtain the target text information; The decoding layer is used to perform multiple rounds of iterative decoding on the target attention features to obtain the start and end positions of the target text information in the image text. The start and end positions are used to determine the target text information. The target attention features are features obtained by fusing multimodal data based on an attention mechanism. The decoding layer includes a first decoder and a second decoder, the parameters of which are different; The first decoder is used to perform multiple rounds of iterative decoding on the target attention features to obtain the starting position of the target text information in the image text; The second decoder is used to perform multiple rounds of iterative decoding on the target attention features to obtain the termination position of the target text information in the image text.

15. The system according to claim 14, wherein the multimodal processing model includes an embedding coding layer, an attention coding layer, and a text information extraction layer; The embedding coding layer is used to encode the target image, the image text, and the target question text to obtain the image features of the target image, the image text features of the image text, and the question text features of the target question text. The attention encoding layer is used to encode the image features of the target image, the image text features of the image text, and the question text features of the target question text based on an attention mechanism to obtain the target attention features; The text information extraction layer is used to extract the target text information from the image text based on the target attention features.

16. The system according to claim 15, wherein the embedding coding layer comprises an image embedding coding layer, an image text embedding coding layer, and a question text embedding coding layer; The image embedding coding layer is used to divide the target image into blocks to obtain multiple image blocks of the target image; and to encode each image block and its position in the multiple image blocks and the target image to obtain the image features. The image text embedding encoding layer is used to segment the image text to obtain multiple first words of the image text; and to encode each first word and the position of each first word in the multiple first words and the image text to obtain the image text features. The question text embedding encoding layer is used to segment the target question text into words to obtain multiple second words of the target question text; and to encode each second word and the position of each second word in the multiple second words and the target question text to obtain the target question text features.

17. The system according to claim 16, wherein the image embedding coding layer comprises an image content coding sublayer, a first image position coding sublayer, a second image position coding sublayer, and an image feature splicing sublayer; The image content encoding sublayer is used to perform a full connection on any image block among the plurality of image blocks to obtain the image block features of the image block; The first image location encoding sublayer is used to embed and encode the sequence number of the image block in the plurality of image blocks to obtain the first image block location feature of the image block; The second image location encoding sublayer is used to embed and encode the coordinates of the image block in the target image to obtain the second image block location feature of the image block; The image feature splicing sub-layer is used to splice the image block features, the first image block position features, and the second image block position features of each image block to obtain the image block embedding features of each image block, and the image block embedding features of each image block constitute the image features.

18. The system according to claim 16, wherein the image text embedding coding layer comprises an image text content coding sublayer, a first image text position coding sublayer, a second image text position coding sublayer, and an image text feature splicing sublayer; The image text content encoding sublayer is used to embed and encode any one of the plurality of first words to obtain the first word features of the first word. The first image text location encoding sublayer is used to embed and encode the sequence number of the first word in the plurality of first words to obtain the first word location feature of the first word; The second image text location encoding sublayer is used to embed and encode the word set to which the first word belongs in the image text, so as to obtain the second word location feature of the first word; The image text feature splicing sub-layer is used to splice the first word features, first word position features and second word position features of each first word to obtain the first word embedding features of each first word, and the first word embedding features of each first word constitute the image text features.

19. The system according to claim 16, wherein the question text embedding encoding layer comprises a question text content encoding sublayer, a first question text position encoding sublayer, a second question text position encoding sublayer, and a question text feature splicing sublayer; The question text content encoding sublayer is used to embed and encode any one of the plurality of second words to obtain the second word features of the second word. The first question text position encoding sublayer is used to embed and encode the sequence number of the second word in the plurality of second words to obtain the third word position feature of the second word; The second question text position encoding sublayer is used to embed and encode the word set to which the second word belongs in the target question text, so as to obtain the fourth word position feature of the second word; The problem text feature splicing sub-layer is used to splice the second word features, third word position features and fourth word position features of each second word to obtain the second word embedding features of each second word. The second word embedding features of each second word constitute the target problem text features.

20. The system according to claim 15, wherein the text information extraction layer includes an extraction layer; The extraction layer is used to extract text information from the image text according to the start position and the end position to obtain the target text information.

21. The system according to claim 14, wherein the recognition module is configured to convolve the target image to obtain image features of the target image; and to perform sequence encoding and sequence decoding on the image features to obtain image text corresponding to the target image; Alternatively, the target image can be encoded using an attention mechanism to obtain image features of the target image; the image features can then be decoded using the attention mechanism to obtain image text corresponding to the target image.

22. The system of claim 14, further comprising a sample data engine and a training module; The acquisition module is also used to acquire multiple sample images, the sample images carrying text information; The recognition module is also used to perform text recognition on each of the sample images to obtain the sample image text corresponding to each of the sample images; The sample data engine is used to determine multiple annotation text information for each sample image and sample question text corresponding to each annotation text information based on the sample image text corresponding to each sample image. The training module is used to perform multiple rounds of training on the initial model based on the multiple sample images, the sample image text corresponding to each sample image, the multiple labeled text information of each sample image, and the sample question text corresponding to each labeled text information, to obtain the multimodal processing model.

23. The system according to claim 22, wherein the sample data engine is configured to, for any sample image among the plurality of sample images, perform field segmentation on the corresponding sample image text of the sample image to obtain a plurality of field groups, wherein each field group includes a field name and a field content; determine the field content in the plurality of field groups as a plurality of labeled text information of the sample image; and perform question conversion on the field name corresponding to each labeled text information in the plurality of field groups to obtain sample question text corresponding to each labeled text information.

24. The system according to claim 22, wherein the training module is configured to, in any training round, input the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text into the initial model, process the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text through the initial model to obtain the predicted text information corresponding to the sample image of the current round; and adjust the model parameters of the initial model based on the difference information between the labeled text information and the predicted text information corresponding to the sample image of the current round.

25. The system according to claim 24, wherein the training module is configured to adjust the model parameters of the initial model based on a first difference information between the annotation start position of the annotation text information corresponding to the sample image of the current round and the prediction start position of the predicted text information corresponding to the sample image of the current round in the sample image text, and a second difference information between the annotation end position of the annotation text information corresponding to the sample image of the current round and the prediction end position of the predicted text information corresponding to the sample image of the current round in the sample image text.

26. A computer device comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, the computer program being loaded and executed by the one or more processors to implement the text information extraction method as described in any one of claims 1 to 13.

27. A computer-readable storage medium storing at least one computer program, the computer program being loaded and executed by a processor to implement the text information extraction method as described in any one of claims 1 to 13.

Images