Text information extraction method, system and equipment and storage medium
Through the multimodal processing model, images, texts and problem texts are encoded and processed, and the problem of insufficient generalization ability of image text extraction and generalization in the prior art is solved, and intelligent text information extraction and recognition are realized.
Patent Information
- Application Number
- CN202410993484.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-23
Smart Images

Figure CN118982832A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present specification relate to the field of computer technology, and in particular, to a method, system, device and storage medium for extracting text information. Background Art
[0002] With the development of computer technology, more and more scenarios will introduce computer technology to improve efficiency. For example, in scenarios where text information needs to be extracted from images, computer technology is usually introduced.
[0003] In the related art, text information is usually extracted from an image by using a template matching method, that is, a pre-configured text template is used to perform matching in the image to achieve the extraction of text information in the image.
[0004] However, since the number of text templates is usually limited, the method of extracting text information from images using template matching has poor generalization ability and cannot meet the growing demand. Summary of the invention
[0005] The embodiments of this specification provide a method, system, device and storage medium for extracting text information, which can improve the generalization ability of extracting text information from images. The technical solution is as follows:
[0006] On the one hand, a method for extracting text information is provided, the method comprising:
[0007] Acquire a target image and a target question text for the target image, wherein the target image carries text information, and the target question text is used to indicate to extract target text information from the target image;
[0008] Performing text recognition on the target image to obtain image text corresponding to the target image;
[0009] The target image, the image text and the target question text are input into a multimodal processing model, and the target image, the image text and the target question text are processed by the multimodal processing model to obtain the target text information.
[0010] On the one hand, a text information extraction system is provided, the system comprising: an acquisition module, a recognition module, an input module and a multimodal processing model;
[0011] The acquisition module is used to acquire a target image and a target question text for the target image, wherein the target image carries text information, and the target question text is used to indicate to extract target text information from the target image;
[0012] The recognition module is used to perform text recognition on the target image to obtain the image text corresponding to the target image;
[0013] The input module is used to input the target image, the image text and the target question text into the multimodal processing model;
[0014] The multimodal processing model is used to process the target image, the image text and the target question text to obtain the target text information.
[0015] In a possible implementation, the multimodal processing model includes an embedding coding layer, an attention coding layer, and a text information extraction layer;
[0016] The embedded coding layer is used to encode the target image, the image text and the target question text to obtain image features of the target image, image text features of the image text and question text features of the target question text;
[0017] The attention encoding layer is used to encode the image features of the target image, the image text features of the image text, and the question text features of the target question text based on the attention mechanism to obtain target attention features;
[0018] The text information extraction layer is used to extract the target text information from the image text based on the target attention feature.
[0019] In a possible implementation, the embedding coding layer includes an image embedding coding layer, an image text embedding coding layer and a question text embedding coding layer;
[0020] The image embedding coding layer is used to divide the target image into blocks to obtain multiple image blocks of the target image; encode each of the image blocks and the position of each of the image blocks in the multiple image blocks and the target image to obtain the image features;
[0021] The image text embedding coding layer is used to segment the image text to obtain a plurality of first words of the image text; encode each of the first words and the position of each of the first words in the plurality of first words and the image text to obtain the image text features;
[0022] The question text embedding coding layer is used to segment the target question text to obtain multiple second words of the target question text; encode each second word and the position of each second word in the multiple second words and the target question text to obtain the target question text features.
[0023] In a possible implementation, the image embedding coding layer includes an image content coding sublayer, a first image position coding sublayer, a second image position coding sublayer, and an image feature splicing sublayer;
[0024] The image content coding sublayer is used for performing full connection on any image block among the multiple image blocks to obtain image block features of the image block;
[0025] The first image position coding sublayer is used to embed the sequence number of the image block in the multiple image blocks to obtain a first image block position feature of the image block;
[0026] The second image position coding sublayer is used to embed the coordinates of the image block in the target image to obtain a second image block position feature of the image block;
[0027] The image feature stitching sublayer is used to stitch the image block features, the first image block position features and the second image block position features of each of the image blocks to obtain the image block embedding features of each of the image blocks, and the image block embedding features of each of the image blocks constitute the image features.
[0028] In a possible implementation, the image text embedding coding layer includes an image text content coding sublayer, a first image text position coding sublayer, a second image text position coding sublayer, and an image text feature splicing sublayer;
[0029] The image text content encoding sublayer is used to perform embedding encoding on any first word among the multiple first words to obtain a first word feature of the first word;
[0030] The first image text position coding sublayer is used to embed the sequence number of the first word in the plurality of first words to obtain a first word position feature of the first word;
[0031] The second image text position coding sublayer is used to embed the vocabulary set to which the first vocabulary belongs in the image text to obtain a second vocabulary position feature of the first vocabulary;
[0032] The image text feature splicing sublayer is used to splice the first vocabulary features, first vocabulary position features and second vocabulary position features of each of the first words to obtain the first vocabulary embedding features of each of the first words. The first vocabulary embedding features of each of the first words constitute the image text features.
[0033] In a possible implementation, the question text embedding coding layer includes a question text content coding sublayer, a first question text position coding sublayer, a second question text position coding sublayer, and a question text feature splicing sublayer;
[0034] The question text content encoding sublayer is used to perform embedding encoding on any second word among the plurality of second words to obtain a second word feature of the second word;
[0035] The first question text position encoding sublayer is used to embed the sequence number of the second word in the plurality of second words to obtain a third word position feature of the second word;
[0036] The second question text position encoding sublayer is used to embed the vocabulary set to which the second vocabulary belongs in the target question text to obtain a fourth vocabulary position feature of the second vocabulary;
[0037] The question text feature splicing sublayer is used to splice the second vocabulary features, third vocabulary position features and fourth vocabulary position features of each of the second words to obtain the second vocabulary embedding features of each of the second words, and the second vocabulary embedding features of each of the second words constitute the target question text features.
[0038] In a possible implementation, the text information extraction layer includes a decoding layer and an extraction layer;
[0039] The decoding layer is used to perform multiple rounds of iterative decoding on the target attention feature to obtain the starting position and the ending position of the target text information in the image text;
[0040] The extraction layer is used to extract text information from the image text according to the starting position and the ending position to obtain the target text information.
[0041] In a possible implementation manner, the decoding layer includes a first decoder and a second decoder, and parameters of the first decoder and the second decoder are different;
[0042] The first decoder is used to perform multiple rounds of iterative decoding on the target attention feature to obtain the starting position of the target text information in the image text;
[0043] The second decoder is used to perform multiple rounds of iterative decoding on the target attention feature to obtain the end position of the target text information in the image text.
[0044] In a possible implementation, the recognition module is used to perform convolution on the target image to obtain image features of the target image; perform sequence encoding and sequence decoding on the image features to obtain image text corresponding to the target image;
[0045] Alternatively, the target image is encoded based on an attention mechanism to obtain image features of the target image; and the image features are decoded based on an attention mechanism to obtain image text corresponding to the target image.
[0046] In a possible implementation, the system further includes a sample data engine and a training module;
[0047] The acquisition module is further used to acquire a plurality of sample images, wherein the sample images carry text information;
[0048] The recognition module is further used to perform text recognition on each of the sample images to obtain sample image text corresponding to each of the sample images;
[0049] The sample data engine is used to determine a plurality of annotated text information of each of the sample images and a sample question text corresponding to each of the annotated text information based on the sample image text corresponding to each of the sample images;
[0050] The training module is used to perform multiple rounds of training on the initial model based on the multiple sample images, the sample image texts corresponding to each of the sample images, the multiple annotated text information of each of the sample images, and the sample question texts corresponding to each of the annotated text information to obtain the multimodal processing model.
[0051] In a possible implementation, the sample data engine is used to perform field segmentation on the sample image text corresponding to any sample image among the multiple sample images to obtain multiple field groups, where one field group includes a field name and a field content; determine the field content in the multiple field groups as multiple annotated text information of the sample image; and perform question conversion on the field name corresponding to each of the annotated text information in the multiple field groups to obtain a sample question text corresponding to each of the annotated text information.
[0052] In a possible implementation, the training module is used to input the sample images of the current round, the sample image texts corresponding to the sample images of the current round, and the sample question texts from among the multiple sample images into the initial model in any round of training; process the sample images of the current round, the sample image texts corresponding to the sample images of the current round, and the sample question texts through the initial model to obtain the predicted text information corresponding to the sample images of the current round; and adjust the model parameters of the initial model based on the difference information between the annotated text information and the predicted text information corresponding to the sample images of the current round.
[0053] In one possible embodiment, the training module is used to adjust the model parameters of the initial model based on first difference information between the annotation starting position of the annotation text information corresponding to the sample image of the current round and the predicted starting position of the predicted text information corresponding to the sample image of the current round in the sample image text, and second difference information between the annotation ending position of the annotation text information corresponding to the sample image of the current round and the predicted ending position of the predicted text information corresponding to the sample image of the current round in the sample image text.
[0054] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the text information extraction method.
[0055] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the text information extraction method.
[0056] On the one hand, a computer program product or a computer program is provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device executes the above-mentioned text information extraction method.
[0057] Through the technical solution provided in the embodiments of this specification, a target image to be processed and a target question text for the target image are obtained. The target image carries text information. The target question text is used to indicate the extraction of target text information from the target image. The target question text can reflect the need to extract text information. Text recognition is performed on the target image to obtain an image text corresponding to the target image. The image text includes the text information carried in the target image. The target image, the image text and the target question text are input into a multimodal processing model. The target image, the image text and the target question text are processed by the multimodal processing model to obtain the target text information, thereby realizing intelligent text information extraction for the target image. In the above process, the multimodal understanding ability of the multimodal processing model is utilized, that is, information of different modalities is utilized to extract text information, and the generalization ability is strong. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0059] Figure 1 It is a schematic diagram of an implementation environment of a text information extraction method provided in an embodiment of this specification;
[0060] Figure 2 is a flowchart of a method for extracting text information provided by an embodiment of this specification;
[0061] Figure 3 is a flowchart of another method for extracting text information provided by an embodiment of this specification;
[0062] Figure 4 is a schematic diagram of a target image provided by an embodiment of this specification;
[0063] Figure 5 It is a structural diagram of a multimodal processing model provided in an embodiment of this specification;
[0064] Figure 6 is a flowchart of a training method for a multimodal processing model provided in an embodiment of this specification;
[0065] Figure 7 It is a structural diagram of a text information extraction system provided in an embodiment of this specification;
[0066] Figure 8is a structural diagram of another multimodal processing model provided in an embodiment of this specification;
[0067] Fig. 9 It is a structural diagram of a text information extraction system provided in an embodiment of this specification;
[0068] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION
[0069] In order to make the purpose, technical solutions and advantages of this specification more clear, the implementation methods of this specification will be further described in detail below with reference to the accompanying drawings.
[0070] In this specification, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between the "first", "second", and "nth", and there is no limitation on the quantity and execution order.
[0071] In order to introduce the technical solutions provided by the implementation of this specification, the terms involved in the embodiments of this specification are first introduced below.
[0072] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain better results. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.
[0073] Natural language processing: Natural language processing is an important direction in the field of artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics, but there are important differences. Natural language processing is not a general study of natural language, but the development of computer systems that can effectively realize natural language communication, especially the software systems therein.
[0074] Normalization: Mapping sequences with different value ranges to the interval (0, 1) facilitates data processing. In some cases, the normalized values can be directly implemented as probabilities.
[0075] Embedded Coding: Embedded coding mathematically represents a correspondence, that is, mapping data in X space to Y space through a function F, where the function F is an injective function, and the mapping result is structural preservation. The injective function indicates that the data after mapping is uniquely corresponding to the data before mapping, and structural preservation indicates that the size relationship of the data before mapping is the same as the size relationship of the data after mapping. For example, there are data X1 and X2 before mapping, and after mapping, Y1 corresponding to X1 and Y2 corresponding to X2 are obtained. If the data X1 before mapping is greater than X2, then correspondingly, the data Y1 after mapping is greater than Y2. For words, it means mapping the words to another space to facilitate subsequent machine learning and processing.
[0076] Attention weight: It can indicate the importance of a certain data in the training or prediction process. Importance indicates the influence of input data on output data. Data with high importance has a higher corresponding attention weight value, and data with low importance has a lower corresponding attention weight value. In different scenarios, the importance of data is different. The process of training the attention weight of the model is also the process of determining the importance of data.
[0077] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this manual are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0078] After introducing the terms involved in the embodiments of this specification, the implementation environment of the embodiments of this specification is introduced below. Figure 1 is a schematic diagram of an implementation environment of a text information extraction method provided in an embodiment of this specification, see Figure 1 , the implementation environment may include a terminal 110 and a server 140.
[0079] The terminal 110 is connected to the server 140 via a wireless network or a wired network. Optionally, the terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal 110 is installed and runs an application that supports extracting text information, and the user can extract the required text information from the image through the application.
[0080] The server 140 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The server 140 provides background services for applications running on the terminal 110.
[0081] Those skilled in the art will appreciate that the number of the above terminals may be more or less. For example, the above terminal may be only one, or the above terminals may be dozens or hundreds, or a greater number, in which case the above implementation environment may also include other terminals. The embodiments of this specification do not limit the number of terminals and device types.
[0082] After introducing the implementation environment of the embodiments of this specification, the application scenarios of the embodiments of this specification are introduced below.
[0083] The technical solution provided in the embodiments of this specification can be applied in any scenario where text information needs to be extracted from an image. For example, it can be applied in the scenario where text information needs to be extracted from a web page image, it can also be applied in the scenario where text information needs to be extracted from a certificate or license, and of course it can also be applied in the scenario where text information needs to be extracted from a captured image.
[0084] Taking the application provided in the embodiment of this specification in the scenario of extracting text information from a web page image as an example, a web page image to be processed and a target question text for the web page image are obtained. The web page image carries text information. The target question text is used to indicate the extraction of target text information from the web page image. The target question text can reflect the need to extract text information. Text recognition is performed on the web page image to obtain the image text corresponding to the web page image. The image text includes the text information carried in the web page image. The web page image, the image text and the target question text are input into a multimodal processing model. The web page image, the image text and the target question text are processed by the multimodal processing model to obtain the target text information, thereby realizing intelligent text information extraction for the web page image. The multimodal processing model is used instead of a text template to extract text information, and the generalization ability is strong.
[0085] It should be noted that the above is explained by taking the application of the technical solution provided in the embodiment of this specification in the scenario of extracting text information from web page images as an example. The application of the technical solution provided in the embodiment of this specification in other scenarios belongs to the same inventive concept as the above description and will not be repeated here.
[0086] After introducing the application scenario of the embodiment of this specification, the following introduces the method for extracting text information provided by the embodiment of this specification. Figure 2 Taking the execution subject as a server as an example, the method includes the following steps.
[0087] 201. The server obtains a target image and a target question text for the target image, where the target image carries text information, and the target question text is used to indicate to extract target text information from the target image.
[0088] The target image is the image to be processed. The target question text is used to indicate the extraction of target text information from the target image, that is, to indicate the extraction of the target question information from the text information carried by the target image. The target question text is sent by the terminal to the server, and the terminal can obtain the target question text input by the user. The target text information is the text information indicated by the target question text.
[0089] 202. The server performs text recognition on the target image to obtain image text corresponding to the target image.
[0090] Among them, text recognition of the target image is used to convert the text information carried in the target image into the image text corresponding to the target image. In other words, the image text carries all the text information carried in the target image, and the text information can be extracted based on the image text later.
[0091] 203. The server inputs the target image, the image text and the target question text into a multimodal processing model, processes the target image, the image text and the target question text through the multimodal processing model, and obtains the target text information.
[0092] Among them, the multimodal processing model is a model with multimodal understanding capabilities and text information extraction capabilities, and multimodal understanding includes text understanding and image understanding and / or video understanding.
[0093] Through the technical solution provided in the embodiments of this specification, a target image to be processed and a target question text for the target image are obtained. The target image carries text information. The target question text is used to indicate the extraction of target text information from the target image. The target question text can reflect the need to extract text information. Text recognition is performed on the target image to obtain an image text corresponding to the target image. The image text includes the text information carried in the target image. The target image, the image text and the target question text are input into a multimodal processing model. The target image, the image text and the target question text are processed by the multimodal processing model to obtain the target text information, thereby realizing intelligent text information extraction for the target image. In the above process, the multimodal understanding ability of the multimodal processing model is utilized, that is, information of different modalities is utilized to extract text information, and the generalization ability is strong.
[0094] Figure 3 is a flowchart of a method for extracting text information provided in an embodiment of this specification, see Figure 3 Taking the execution subject as a server as an example, the method includes the following steps.
[0095] 301. The server obtains a target image and a target question text for the target image, where the target image carries text information, and the target question text is used to indicate that target text information is to be extracted from the target image.
[0096] The target image is the image to be processed. The target question text is used to indicate the extraction of target text information from the target image, that is, to indicate the extraction of the target question information from the text information carried by the target image. The target question text is sent by the terminal to the server, and the terminal can obtain the target question text input by the user. The target text information is the text information indicated by the target question text. For example, see Figure 4 , taking the target image as the document image 401 (such as a scanned copy of the document or a photo of the document) as an example, the text information carried by the target image refers to the text content in the document. In the case where the document is an identity document, the text content refers to the text content used to describe the identity indicated by the identity document. Accordingly, the target question text can be "Please extract the address from the image", and the address is the target text information. It should be noted that the same target text information has different display forms in different target images. For example, see Figure 4 There is also a license image 402, which also contains an address. As can be seen from the figure, although both addresses are displayed in the certificate image 401 and the license image 402, there is a significant difference. At this time, it is necessary to understand this difference in order to achieve more accurate text information extraction.
[0097] In a possible implementation, in response to the text information extraction request, the server obtains the target image and the target question text for the target image from the text information extraction request.
[0098] Among them, the text information extraction request is sent by the terminal to the server. For example, the user selects the target image and enters the target question text through the application running on the terminal. The terminal packages the target image and the target question text into the text information extraction request through the application, and sends the text information extraction request to the server.
[0099] In a possible implementation, the server obtains the target image and the target question text for the target image from a database, in which a plurality of candidate images to be processed and candidate question texts for respective candidate images are stored.
[0100] Among them, multiple candidate images and candidate question texts for each candidate image are uploaded to the server by the user. Since the server may not be able to process the multiple candidate images and candidate question texts for each candidate image in a timely manner, the server will store the multiple candidate images and candidate question texts for each candidate image in a database for easy retrieval. The server can obtain the candidate images and candidate question texts from the database in a predetermined order. In this embodiment of the specification, the candidate images obtained from the database are also referred to as target images, and the candidate question texts are also referred to as target question texts.
[0101] 302. The server performs text recognition on the target image to obtain image text corresponding to the target image.
[0102] Among them, the text recognition of the target image is used to convert the text information carried in the target image into the image text corresponding to the target image. In other words, the image text carries all the text information carried in the target image, and the text information can be extracted based on the image text. In some embodiments, the process of performing text recognition on the target image can be regarded as the process of performing optical character recognition (OCR) on the target image, the purpose of which is to convert the text information carried in the target image into image text for subsequent processing.
[0103] In a possible implementation, the server performs convolution on the target image to obtain image features of the target image, and performs sequence decoding on the image features to obtain image text corresponding to the target image.
[0104] Among them, the image feature is an abstract expression of the target image, and the image feature can represent the target image in a higher dimension. The purpose of sequential decoding of the image feature is to convert the image feature into image text, thereby realizing text recognition of the target image.
[0105] For example, the server inputs the target image into the text recognition model, and the text recognition model uses multiple convolution kernels to convolve the target image to obtain multiple convolution features of the target image. The server fuses the multiple convolution features through the text recognition model to obtain the image features of the target image. The server uses the text recognition model to perform sequence encoding and sequence decoding on the image features using a gating mechanism to obtain the image text corresponding to the target image.
[0106] Among them, the text recognition model is a model dedicated to text recognition on images. The text recognition model is trained based on multiple sample images and annotated image text of each sample image, and has the ability to recognize text on images. Accordingly, the text recognition model is an OCR model.
[0107] For example, the server inputs the target image into the text recognition model, and the text recognition model uses multiple convolution kernels of different sizes to convolve the target image to obtain multiple convolution features of different sizes for the target image. The server upsamples or downsamples the multiple convolution features of different sizes to obtain multiple convolution features of the same size for the target image. The server fuses the multiple convolution features of the same size through the text recognition model to obtain the image features of the target image. The server encodes the image features through the text recognition model using a gating mechanism to obtain the temporal features and latent features corresponding to the image features. The server decodes the temporal features and the latent features through the text recognition model based on the gating mechanism to obtain the image text corresponding to the target image.
[0108] In a possible implementation, the server encodes the target image based on the attention mechanism to obtain the image features of the target image. The server decodes the image features based on the attention mechanism to obtain the image text corresponding to the target image.
[0109] For example, the server divides the target image into blocks to obtain multiple image blocks of the target image. The server encodes each image block and the position of each image block in the target image based on the attention mechanism to obtain the image feature. The server performs multiple rounds of iterative decoding on the image feature based on the attention mechanism to obtain the image text corresponding to the target image.
[0110] For example, the server divides the target image into blocks to obtain multiple image blocks of the target image. The server embeds and encodes each image block and the position of each image block in the target image to obtain the image block features and position features of each image block. The server splices the image block features and position features of each image block to obtain the image block embedding features of each image block. The server encodes the image block embedding features of each image block based on the attention mechanism to obtain the image block attention features of each image block. The server fuses the attention features of each image block to obtain the image features of the target image. The server performs multiple rounds of iterative decoding on the image features based on the attention mechanism to obtain the image text corresponding to the target image.
[0111] In a possible implementation, the server performs region recognition on the target image to obtain at least one target image region in the target image, where the target image region is an image region corresponding to text information in the target image. The server performs text recognition on the at least one target image region to obtain image text corresponding to the target image.
[0112] The region recognition is a binary classification process, and the results of the region recognition include target regions and non-target regions. The text recognition method for at least one target image region can adopt any of the above two implementations, and the embodiments of this specification are not limited to this.
[0113] 303. The server inputs the target image, the image text and the target question text into a multimodal processing model.
[0114] Among them, the multimodal processing model is a model with multimodal understanding capabilities and text information extraction capabilities. Multimodal understanding includes text understanding and image understanding and / or video understanding. In the embodiments of this specification, multimodal understanding includes understanding the target image, the image text, and the target question text. In some embodiments, the multimodal processing model is a pre-trained and fine-tuned model. The pre-training is to enable the model to have basic multimodal understanding capabilities, and the fine-tuning is to enable the model to meet the requirements of specific tasks. In the embodiments of this specification, that is, to extract text information from images. The subsequent embodiments will explain the training method of the multimodal understanding model. In some embodiments, in addition to obtaining the image text through the above step 302, the OCR model in the related technology can also be integrated into the multimodal processing model, so as to give the multimodal processing model the ability of OCR. In this way, the target image can be directly input into the multimodal processing model to obtain the image text corresponding to the target image. This design takes into account the parameter efficiency for OCR, that is, the amount of parameters required to achieve the same effect. The specialized OCR model is better than the multimodal understanding model, which can reduce the amount of parameters of the multimodal understanding model and make the multimodal understanding model more lightweight. In some embodiments, the multimodal processing model is also called ChatKIE. Chat means that text information can be extracted from the image in the form of chatting. The full name of KIE is KeyInformation Extraction, which means key information extraction.
[0115] 304. The server encodes the target image, the image text and the target question text through the multimodal processing model to obtain image features of the target image, image text features of the image text and question text features of the target question text.
[0116] Among them, the image feature is an abstract expression of the target image, which can represent the target image in a higher dimension; the image text feature is an abstract expression of the image text, which can represent the image text in a higher dimension; the question text feature is an abstract expression of the question text, which can represent the question text in a higher dimension. Extracting the target image, image text and target text into image features, image text features and question text features is conducive to subsequent model processing.
[0117] In a possible implementation, the server segments the target image through the multimodal processing model to obtain a plurality of image blocks of the target image. The server encodes each image block and the position of each image block in the plurality of image blocks and the target image through the multimodal processing model to obtain the image feature. The server segments the image text through the multimodal processing model to obtain a plurality of first words of the image text. The server encodes each first word and the position of each first word in the plurality of first words and the image text through the multimodal processing model to obtain the image text feature. The server segments the target question text through the multimodal processing model to obtain a plurality of second words of the target question text. The server encodes each second word and the position of each second word in the plurality of second words and the target question text through the multimodal processing model to obtain the target question text feature.
[0118] Among them, the position of the image block in multiple image blocks refers to the order of the image block in the multiple image blocks, also known as the 1D position of the image block; the position of the image block in the target image refers to the coordinates of the image block in the target image, also known as the 2D position of the image block. The position of the first word in multiple first words refers to the order of the first word in multiple first words, also known as the 1D position of the first word; the position of the second word in multiple second words refers to the order of the second word in multiple second words, also known as the 2D position of the first word. The position of the first word in the image text refers to the coordinates of the text box to which the first word belongs in the image text, also known as the 1D position of the second word; the position of the second word in the target question text refers to the coordinates of the text box to which the second word belongs in the target question text, also known as the 2D position of the second word.
[0119] In order to explain the above implementation more clearly, the above implementation will be explained in several parts below.
[0120] In the first part, the server divides the target image into blocks through the multimodal processing model to obtain multiple image blocks of the target image.
[0121] In a possible implementation, the server divides the target image into blocks using a preset division method through the multimodal processing model to obtain multiple image blocks of the target image.
[0122] Among them, the preset division method includes the number of divisions and the division size. The number of divisions is the number of multiple image blocks obtained by division, and the division size is the size of the image blocks obtained by division. The preset division method is set by technical personnel according to actual conditions, and the embodiments of this specification do not limit this.
[0123] In the second part, the server encodes each image block and the position of each image block in the multiple image blocks and the target image through the multimodal processing model to obtain the image feature.
[0124] In a possible implementation, for any image block among the multiple image blocks, the server performs full connection on the image block through the multimodal processing model to obtain the image block feature of the image block. The server performs embedding coding on the sequence number of the image block in the multiple image blocks through the multimodal processing model to obtain the first image block position feature of the image block. The server performs embedding coding on the coordinates of the image block in the target image through the multimodal processing model to obtain the second image block position feature of the image block. The server splices the image block features, the first image block position features and the second image block position features of each image block through the multimodal processing model to obtain the image block embedding features of each image block, and the image block embedding features of each image block constitute the image feature.
[0125] Among them, fully connecting the image blocks is equivalent to performing a linear transformation on the image blocks, converting the image blocks into a sequence form for processing. The sequence number of the image block in multiple image blocks is used to reflect the order of the image blocks in multiple image blocks. The sequence number of the image block can reflect the relative position of the image block in the target image. The order of the image blocks is determined by the position of the image blocks in the target image. For example, the image blocks in the target image are sorted from left to right and from top to bottom to obtain the sequence number of each image block in the multiple image blocks. The coordinates of the image block in the target image can be the coordinates of the center point of the image block in the target image, or the coordinates of the corner points of the image block in the target image, or the set of coordinates of the center point and the corner point of the image block in the target image, which is not limited in the embodiments of this specification.
[0126] In a possible implementation, for any image block among the multiple image blocks, the server performs convolution on the image block through the multimodal processing model to obtain the image block feature of the image block. The server performs embedding coding on the sequence number of the image block in the multiple image blocks through the multimodal processing model to obtain the first image block position feature of the image block. The server performs embedding coding on the coordinates of the image block in the target image through the multimodal processing model to obtain the second image block position feature of the image block. The server splices the image block features, the first image block position features and the second image block position features of each image block through the multimodal processing model to obtain the image block embedding features of each image block, and the image block embedding features of each image block constitute the image feature.
[0127] Part three: The server performs word segmentation on the image text through the multimodal processing model to obtain multiple first words of the image text.
[0128] In a possible implementation, the server performs word segmentation on the image text by using the word segmentation sub-model of the multimodal processing model to obtain a plurality of first words of the image text.
[0129] The word segmentation model is used to segment the input product title into multiple independent and meaningful reference product description words for subsequent processing. In some embodiments, the word segmentation sub-model is a jiaba word segmentation model, a HanLP word segmentation model, and a FoolNLTK word segmentation model. Of course, with the development of science and technology, other types of word segmentation models can also be used, and this specification embodiment does not limit this.
[0130] In the fourth part, the server encodes each first word and the position of each first word in the multiple first words and the image text through the multimodal processing model to obtain the image text feature.
[0131] In a possible implementation, for any first word among the multiple first words, the server embeds and encodes the first word through the multimodal processing model to obtain the first word feature of the first word. The server embeds and encodes the sequence number of the first word in the multiple first words through the multimodal processing model to obtain the first word position feature of the first word. The server embeds and encodes the word set to which the first word belongs in the image text through the multimodal processing model to obtain the second word position feature of the first word. The server concatenates the first word feature, the first word position feature and the second word position feature of each first word through the multimodal processing model to obtain the first word embedding feature of each first word, and the first word embedding features of each first word constitute the image text feature.
[0132] Part 5: The server performs word segmentation on the target question text through the multimodal processing model to obtain multiple second words of the target question text.
[0133] In a possible implementation, the server performs word segmentation on the target question text through the word segmentation sub-model of the multimodal processing model to obtain a plurality of second words of the image text.
[0134] Part 6: The server encodes each second word and the position of each second word in the plurality of second words and the target question text through the multimodal processing model to obtain the target question text feature.
[0135] In a possible implementation, for any second word among the multiple second words, the server embeds and encodes the second word through the multimodal processing model to obtain the second word feature of the second word. The sequence number of the second word in the multiple second words is embedded and encoded to obtain the third word position feature of the second word. The vocabulary set to which the second word belongs in the target question text is embedded and encoded to obtain the fourth word position feature of the second word. The server splices the second word feature, the third word position feature and the fourth word position feature of each second word through the multimodal processing model to obtain the second word embedding feature of each second word, and the second word embedding feature of each second word constitutes the target question text feature.
[0136] 305. The server encodes the image features of the target image, the image text features of the image text, and the question text features of the target question text based on the attention mechanism through the multimodal processing model to obtain target attention features.
[0137] Among them, the target attention feature is the feature obtained by fusing multimodal data based on the attention mechanism.
[0138] In a possible implementation, the server uses the multimodal processing model to splice the image features of the target image, the image text features of the image text, and the question text features of the target question text to obtain a spliced feature, which includes multiple sub-features, and the multiple sub-features include image block embedding features of each image block in the target image, first vocabulary embedding features of each first vocabulary in the image text, and second vocabulary embedding features of each second vocabulary in the target question text. The server uses the multimodal processing model to encode each sub-feature based on the attention mechanism to obtain an attention feature of each sub-feature. The server uses the multimodal processing model to fuse the attention features of each sub-feature to obtain the target attention feature.
[0139] In order to explain the above implementation more clearly, the following is divided into two parts to explain the method of obtaining the attention features of each sub-feature and the target attention features in the above implementation.
[0140] In the first part, the server encodes each sub-feature based on the attention mechanism through the multimodal processing model to obtain the attention features of each sub-feature.
[0141] In a possible implementation, the server performs a linear transformation on each sub-feature through the multimodal processing model to obtain a query matrix, a key matrix, and a value matrix of each sub-feature. The server determines the attention weight of each sub-feature based on the query matrix and the key matrix of each sub-feature through the multimodal processing model. For example, the server uses three linear transformation matrices to perform a linear transformation on each sub-feature through the multimodal processing model to obtain the query feature, key feature, and value feature of each sub-feature. For any sub-feature of the multiple sub-features, the server multiplies the query feature of the sub-feature with the key feature of the multiple sub-features through the multimodal processing model to obtain the attention weights of the multiple sub-features to the sub-feature. The server uses the attention weights of the multiple sub-features to the sub-features through the multimodal processing model to multiply the value features of each sub-feature to obtain the attention feature of the sub-feature.
[0142] In the second part, the server fuses the attention features of each sub-feature through the multimodal processing model to obtain the target attention feature.
[0143] In a possible implementation, the server adds or concatenates the attention features of each sub-feature through the multimodal processing model to obtain the target attention feature.
[0144] 306. The server extracts the target text information from the image text based on the target attention feature through the multimodal processing model.
[0145] In a possible implementation, the server performs multiple rounds of iterative decoding on the target attention feature through the multimodal processing model to obtain the starting position and the ending position of the target text information in the image text. The server extracts text information from the image text according to the starting position and the ending position to obtain the target text information.
[0146] For example, taking the target image as a certificate image as an example, the text information carried by the target image refers to the text content in the certificate. When the certificate is an identity document, the text content refers to the text content used to describe the identity indicated by the identity document. Accordingly, the target question text can be "Please extract the address from the image", and the target text information is the address "XX District, XX City, XX Province" in the identity image. The starting position of the target text information in the image text refers to the coordinates of the first character of the target text information in the image text. Accordingly, the ending position of the target text information in the image text refers to the coordinates of the last character of the target text information in the image text. In other words, the target text information belongs to the image text, and the multimodal processing model ultimately outputs the starting position and ending position of the target text information, rather than directly outputting the target text information. This method is also called the extractive answer content method, compared with the generative answer content method in the related art (the generated content is freely played by the model). In the embodiments of this specification, the extractive answer generation method is selected because when the number of parameters of the multimodal processing model is the same, the accuracy of the content generated by the extractive method is higher than that of the generative method. In other words, the use of the extractive method can minimize the number of parameters while ensuring accuracy, thereby reducing the difficulty of deploying the multimodal processing model and improving the processing speed of the multimodal processing model.
[0147] In order to explain the above implementation more clearly, the above implementation will be explained in two parts below.
[0148] In the first part, the server uses the multimodal processing model to perform multiple rounds of iterative decoding on the target attention feature to obtain the starting position and ending position of the target text information in the image text.
[0149] In a possible implementation, the server inputs the target attention feature into the first decoder of the multimodal processing model, and the first decoder performs multiple rounds of iterative decoding on the target attention feature to obtain the starting position of the target text information in the image text. The server inputs the target attention feature into the second decoder of the multimodal processing model, and the second decoder performs multiple rounds of iterative decoding on the target attention feature to obtain the ending position of the target text information in the image text, and the parameters of the first decoder and the second decoder are different.
[0150] Among them, the first decoder corresponds to the task of decoding the starting position, and is used to decode the starting position of the target text information. The second decoder corresponds to the task of decoding the ending position, and is used to decode the ending position of the target text information. Accordingly, the multimodal processing model is a multi-task model.
[0151] For example, the server inputs the target attention feature into the first decoder of the multimodal processing model. In the first round of decoding, the server decodes the target attention feature and the start symbol through the first decoder to obtain the first character of the starting position. In the second round of iterative decoding, the server decodes the target attention feature, the start symbol and the first character through the first decoder to obtain the second character of the starting position, and so on, until the end symbol is decoded, indicating that the decoding of the starting position is completed, and the complete starting position can be obtained at this time. The server inputs the target attention feature into the second decoder of the multimodal processing model. In the first round of decoding, the server decodes the target attention feature and the start symbol through the second decoder to obtain the first character of the ending position. In the second round of iterative decoding, the server decodes the target attention feature, the start symbol and the first character through the second decoder to obtain the second character of the ending position, and so on, until the end symbol is decoded, indicating that the decoding of the ending position is completed, and the complete ending position can be obtained at this time.
[0152] In the second part, the server extracts text information from the image text according to the starting position and the ending position to obtain the target text information.
[0153] In a possible implementation, the server extracts characters from the image text from the starting position until the end position is reached, thereby obtaining a plurality of characters, where the plurality of characters are also the target text information.
[0154] Combine the following Figure 5 For an explanation of the above steps 301-306, see Figure 5, the server obtains a target image and a target question text 502 for the target image 501. The server performs text recognition on the target image 501 to obtain an image text 503 corresponding to the target image. The server inputs the target image 501, the target question text 502 and the image text 503 into a multimodal processing model 504, and encodes the target image 501, the target question text 502 and the image text 503 through the embedded coding layer 5041 of the multimodal processing model 504 to obtain the image features of the target image 501, the image text features of the image text 502 and the question text features of the target question text 503. The server encodes the image features of the target image 501, the image text features of the image text 502 and the question text features of the target question text 503 based on the attention mechanism through the attention coding layer 5042 of the multimodal processing model 504 to obtain the target attention features. The server performs multiple rounds of iterative decoding on the target attention features through the first decoder 5043 of the multimodal processing model 504 to obtain the starting position of the target text information. The server performs multiple rounds of iterative decoding on the target attention feature through the second decoder 5044 of the multimodal processing model 504 to obtain the end position of the target text information. The server extracts the target text information from the image text based on the start position and the end position.
[0155] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of this specification, and will not be described in detail here.
[0156] Through the technical solution provided in the embodiments of this specification, a target image to be processed and a target question text for the target image are obtained. The target image carries text information. The target question text is used to indicate the extraction of target text information from the target image. The target question text can reflect the need to extract text information. Text recognition is performed on the target image to obtain an image text corresponding to the target image. The image text includes the text information carried in the target image. The target image, the image text and the target question text are input into a multimodal processing model. The target image, the image text and the target question text are processed by the multimodal processing model to obtain the target text information, thereby realizing intelligent text information extraction for the target image. In the above process, the multimodal understanding ability of the multimodal processing model is utilized, that is, information of different modalities is utilized to extract text information, and the generalization ability is strong.
[0157] In addition to the above-mentioned text information extraction method, this specification embodiment also provides a multimodal processing model training method, see Figure 6 Taking the execution subject as a server as an example, the method includes the following steps.
[0158] 601. The server obtains multiple sample images, where the sample images carry text information.
[0159] Among them, the sample image is the image used for model training.
[0160] 602. The server performs text recognition on each sample image to obtain a sample image text corresponding to each sample image.
[0161] Among them, step 602 and step 302 belong to the same inventive concept, and the implementation process refers to the relevant description of the above step 302, which will not be repeated here. In some embodiments, text recognition can also be implemented through GKIE, which is a general image information extraction interface. In some embodiments, the sample image text corresponding to each sample image belongs to the D0 data set, and the D0 data set is used to store the sample image text obtained after directly performing text recognition on the sample image, and one sample image text corresponds to one sample image. In some embodiments, in order to ensure that the sample image text can directly find the corresponding content in the sample image, the post-processing of text recognition is cancelled. For example, the "minus 1 for full 30; minus 3 for full 50" in the sample image is directly converted to "minus 1 for full 30, minus 3 for full 50".
[0162] 603. The server determines, based on the sample image texts corresponding to the sample images, a plurality of annotated text information of the sample images and a sample question text corresponding to each annotated text information.
[0163] In a possible implementation, for any sample image among the multiple sample images, the server performs field segmentation on the sample image text corresponding to the sample image to obtain multiple field groups, where each field group includes a field name and field content. The server determines the field content in the multiple field groups as multiple annotated text information of the sample image. The server performs question conversion on the field name corresponding to each annotated text information in the multiple field groups to obtain a sample question text corresponding to each annotated text information.
[0164] The field name is used to reflect the type of the corresponding field content. For example, a field group is "store discount" and "discount 1 for 30 yuan and 3 for 50 yuan", the field name is "store discount", and the field content is "discount 1 for 30 yuan and 3 for 50 yuan". Question conversion is to convert declarative sentences into interrogative sentences. For example, the above "store discount" will be converted into "What is the store discount activity?", so as to realize the automatic generation of sample question text. In some embodiments, question conversion is also called Instruct transformation, so that the model can adapt to natural language input.
[0165] For example, for any sample image among the multiple sample images, the server performs word segmentation on the sample image text corresponding to the sample image to obtain multiple sample words in the sample image text. The server combines and matches the semantics of the multiple sample words to obtain multiple field groups. The server determines the field contents in the multiple field groups as multiple annotated text information of the sample image, and records the annotated start position and the annotated end position of each annotated text information in the sample image text for subsequent training. The server uses the target conversion rule among the multiple conversion rules to convert the field name corresponding to each annotated text information in the multiple field groups into a question sentence to obtain the sample question text corresponding to each annotated text information.
[0166] Among them, different conversion rules correspond to different conversion methods, so that a field name may be converted into different sample question texts, which is conducive to improving the generalization ability of the model. In addition, the image type or scene description of the corresponding sample image will be introduced into the sample question text with a preset probability, which is conducive to the model learning the corresponding information extraction method for different types of images. The target conversion rule is randomly determined from a variety of conversion rules. In an embodiment of the present application, the combination of field name and field content belongs to the D1 data set, for example, ["store discount", "discount of 30 or more and 1 discount, and 50 or more and 3 discount"] belongs to the D1 data set. The combination of the sample question text and the annotated text information obtained after the question transformation belongs to the D2 data set, for example, ["What is the store's discount activity?" "Discount 1 for purchases over 30 and 3 for purchases over 50"]. Of course, in order to facilitate model training, the annotation start position and the annotation end position can also be added to the combination of the sample question text and the annotated text information to obtain ["What is the store's discount activity?" "Discount 1 for purchases over 30 and 3 for purchases over 50", (52, 61)], where 52 is the annotation start position, which means that the first character of the annotated text information is the 52nd character in the corresponding sample image text; 61 is the annotation end position, which means that the last character of the annotated text information is the 61st character in the corresponding sample image text. In addition, if there are annotation results of the sample data in the D0 data set by technicians, they can be directly stored in the D1 data set. In some embodiments, after obtaining the D2 data set, the server divides the D2 data set into a training set and a validation set. The principle of division is to ensure that there are no completely identical sample data in the training set and the validation set.
[0167] In some embodiments, for different data sources, there will be the above-mentioned D0 data set, D1 data set and D2 data set. After the D2 data sets corresponding to different data sources are merged, the total D3 data set can be obtained. Among them, the sample images in different data sources correspond to different types of pages. For example, multiple data sources include merchant homepages, product introduction pages, activity pages, and payment pages, and the multiple sample images corresponding to merchant homepages are images of different merchant homepages.
[0168] In addition, after obtaining the D3 data set, the sample data in the D3 data set is processed to obtain the D4 data set, in which the sample data is stored in the form of QA pairs, where Q is Question, which refers to the combination of the sample question text and the sample image, and A is Answer, which refers to the annotation start position and the annotation end position of the annotated text information. Of course, after obtaining D4, the server can also convert the format of D4 so that D4 meets the format requirements of model training.
[0169] In some embodiments, the above conversions from D0 to D1, D1 to D2, D2 to D3, and D3 to D4 are implemented through a data engine.
[0170] 604. The server performs multiple rounds of training on the initial model based on the multiple sample images, the sample image texts corresponding to each sample image, the multiple annotated text information of each sample image, and the sample question texts corresponding to each annotated text information to obtain the multimodal processing model.
[0171] In a possible implementation, in any round of training, the server inputs the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text from among the multiple sample images into the initial model, and the server processes the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text through the initial model to obtain the predicted text information corresponding to the sample image of the current round. The server adjusts the model parameters of the initial model based on the difference information between the annotated text information and the predicted text information corresponding to the sample image of the current round.
[0172] The following describes a method in which the server in the above implementation method adjusts the model parameters of the initial model based on the difference information between the annotated text information and the predicted text information corresponding to the sample image of the current round.
[0173] In some embodiments, the difference information includes first difference information and second difference information. The server adjusts the model parameters of the initial model based on the first difference information between the annotation starting position of the annotation text information corresponding to the sample image of the current round and the predicted starting position of the predicted text information corresponding to the sample image of the current round in the sample image text, and the second difference information between the annotation ending position of the annotation text information corresponding to the sample image of the current round and the predicted ending position of the predicted text information corresponding to the sample image of the current round in the sample image text.
[0174] Figure 7is a structural diagram of a text information extraction system provided in an embodiment of this specification, see Figure 7 The system includes: an acquisition module 701, a recognition module 702, an input module 703 and a multimodal processing model 704.
[0175] The acquisition module 701 is used to acquire a target image and a target question text for the target image. The target image carries text information, and the target question text is used to indicate to extract target text information from the target image.
[0176] The recognition module 702 is used to perform text recognition on the target image to obtain the image text corresponding to the target image.
[0177] The input module 703 is used to input the target image, the image text and the target question text into the multimodal processing model 704.
[0178] The multimodal processing model 704 is used to process the target image, the image text and the target question text to obtain the target text information.
[0179] In one possible implementation, see Figure 8 , the multimodal processing model 704 includes an embedding coding layer 7041, an attention coding layer 7042 and a text information extraction layer 7043.
[0180] The embedded coding layer 7041 is used to encode the target image, the image text and the target question text to obtain the image features of the target image, the image text features of the image text and the question text features of the target question text.
[0181] The attention encoding layer 7042 is used to encode the image features of the target image, the image text features of the image text, and the question text features of the target question text based on the attention mechanism to obtain the target attention features.
[0182] The text information extraction layer 7043 is used to extract the target text information from the image text based on the target attention feature.
[0183] In one possible implementation, see Figure 8 The embedded coding layer 7041 includes an image embedded coding layer 70411, an image text embedded coding layer 70412 and a question text embedded coding layer 70413.
[0184] The image embedding coding layer 70411 is used to divide the target image into blocks to obtain multiple image blocks of the target image, and encode each of the image blocks and the position of each of the image blocks in the multiple image blocks and the target image to obtain the image features.
[0185] The image text is embedded in the coding layer 70412, which is used to segment the image text to obtain a plurality of first words of the image text, and encode each of the first words and the position of each of the first words in the plurality of first words and the image text to obtain the image text features.
[0186] The question text is embedded in the coding layer 70413, which is used to segment the target question text to obtain multiple second words of the target question text. Each of the second words and the position of each of the second words in the multiple second words and the target question text are encoded to obtain the target question text features.
[0187] In one possible implementation, see Figure 8 The image embedding coding layer 70411 includes an image content coding sublayer 704111, a first image position coding sublayer 704112, a second image position coding sublayer 704113 and an image feature splicing sublayer 704114.
[0188] The image content coding sublayer 704111 is used to perform full connection on any image block among the multiple image blocks to obtain the image block features of the image block.
[0189] The first image position coding sublayer 704112 is used to embed the sequence number of the image block in the multiple image blocks to obtain the first image block position feature of the image block.
[0190] The second image position coding sublayer 704113 is used to embed the coordinates of the image block in the target image to obtain the second image block position feature of the image block.
[0191] The image feature stitching sublayer 704114 is used to stitch the image block features, the first image block position features and the second image block position features of each image block to obtain the image block embedding features of each image block, and the image block embedding features of each image block constitute the image feature.
[0192] In one possible implementation, see Figure 8 The image text embedding coding layer 70412 includes an image text content coding sublayer 704121, a first image text position coding sublayer 704122, a second image text position coding sublayer 704123 and an image text feature splicing sublayer 704124.
[0193] The image text content encoding sublayer 704121 is used to perform embedding encoding on any first word among the multiple first words to obtain a first word feature of the first word.
[0194] The first image text position coding sublayer 704122 is used to embed the sequence number of the first word in the multiple first words to obtain the first word position feature of the first word.
[0195] The second image text position coding sublayer 704123 is used to embed the vocabulary set to which the first vocabulary belongs in the image text to obtain the second vocabulary position feature of the first vocabulary.
[0196] The image text feature splicing sublayer 704124 is used to splice the first vocabulary features, first vocabulary position features and second vocabulary position features of each of the first words to obtain the first vocabulary embedding features of each of the first words. The first vocabulary embedding features of each of the first words constitute the image text feature.
[0197] In one possible implementation, see Figure 8 The question text embedding coding layer 70413 includes a question text content coding sublayer 704131, a first question text position coding sublayer 704132, a second question text position coding sublayer 704133 and a question text feature splicing sublayer 704134.
[0198] The question text content encoding sublayer 704131 is used to perform embedding encoding on any second word among the multiple second words to obtain a second word feature of the second word.
[0199] The first question text position encoding sublayer 704132 is used to embed the sequence number of the second word in the multiple second words to obtain the third word position feature of the second word.
[0200] The second question text position encoding sublayer 704133 is used to embed the vocabulary set to which the second vocabulary belongs in the target question text to obtain a fourth vocabulary position feature of the second vocabulary.
[0201] The question text feature concatenation sublayer 704134 is used to concatenate the second vocabulary features, the third vocabulary position features and the fourth vocabulary position features of each of the second words to obtain the second vocabulary embedding features of each of the second words. The second vocabulary embedding features of each of the second words constitute the target question text features.
[0202] In one possible implementation, see Figure 8 The text information extraction layer 7043 includes a decoding layer 70431 and an extraction layer 70432 .
[0203] The decoding layer 70431 is used to perform multiple rounds of iterative decoding on the target attention feature to obtain the starting position and the ending position of the target text information in the image text.
[0204] The extraction layer 70432 is used to extract text information from the image text according to the starting position and the ending position to obtain the target text information.
[0205] In one possible implementation, see Figure 8 The decoding layer 70431 includes a first decoder 704311 and a second decoder 704312, and the parameters of the first decoder 704311 and the second decoder 704312 are different.
[0206] The first decoder 704311 is used to perform multiple rounds of iterative decoding on the target attention feature to obtain the starting position of the target text information in the image text.
[0207] The second decoder 704312 is used to perform multiple rounds of iterative decoding on the target attention feature to obtain the end position of the target text information in the image text.
[0208] In a possible implementation, the recognition module 702 is used to perform convolution on the target image to obtain image features of the target image, perform sequence encoding and sequence decoding on the image features to obtain image text corresponding to the target image.
[0209] Alternatively, the target image is encoded based on the attention mechanism to obtain the image features of the target image, and the image features are decoded based on the attention mechanism to obtain the image text corresponding to the target image.
[0210] In one possible implementation, see Fig. 9 The system also includes a sample data engine 705 and a training module 706.
[0211] The acquisition module 701 is also used to acquire multiple sample images, where the sample images carry text information.
[0212] The recognition module 702 is further used to perform text recognition on each of the sample images to obtain sample image text corresponding to each of the sample images.
[0213] The sample data engine 705 is used to determine a plurality of annotated text information of each sample image and a sample question text corresponding to each annotated text information based on the sample image text corresponding to each sample image.
[0214] The training module 706 is used to perform multiple rounds of training on the initial model based on the multiple sample images, the sample image texts corresponding to each of the sample images, the multiple annotated text information of each of the sample images, and the sample question texts corresponding to each of the annotated text information to obtain the multimodal processing model 704.
[0215] In a possible implementation, the sample data engine 705 is used to perform field segmentation on the sample image text corresponding to any sample image among the multiple sample images to obtain multiple field groups, wherein each field group includes a field name and a field content. The field content in the multiple field groups is determined as multiple annotated text information of the sample image. The field name corresponding to each of the annotated text information in the multiple field groups is converted into a question sentence to obtain a sample question text corresponding to each of the annotated text information.
[0216] In a possible implementation, the training module 706 is used to input the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text in the initial model in any round of training, and process the sample image of the current round, the sample image text corresponding to the sample image of the current round, and the sample question text through the initial model to obtain the predicted text information corresponding to the sample image of the current round. Based on the difference information between the annotated text information and the predicted text information corresponding to the sample image of the current round, the model parameters of the initial model are adjusted.
[0217] In one possible implementation, the training module 706 is used to adjust the model parameters of the initial model based on first difference information between the annotation starting position of the annotation text information corresponding to the sample image of the current round and the predicted starting position of the predicted text information corresponding to the sample image of the current round in the sample image text, and second difference information between the annotation ending position of the annotation text information corresponding to the sample image of the current round and the predicted ending position of the predicted text information corresponding to the sample image of the current round in the sample image text.
[0218] It should be noted that: the text information extraction system provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when extracting text information. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the text information extraction system provided in the above embodiment and the text information extraction method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0219] Through the technical solution provided in the embodiments of this specification, a target image to be processed and a target question text for the target image are obtained. The target image carries text information. The target question text is used to indicate the extraction of target text information from the target image. The target question text can reflect the need to extract text information. Text recognition is performed on the target image to obtain an image text corresponding to the target image. The image text includes the text information carried in the target image. The target image, the image text and the target question text are input into a multimodal processing model. The target image, the image text and the target question text are processed by the multimodal processing model to obtain the target text information, thereby realizing intelligent text information extraction for the target image. In the above process, the multimodal understanding ability of the multimodal processing model is utilized, that is, information of different modalities is utilized to extract text information, and the generalization ability is strong.
[0220] Fig.10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this specification. The computer device 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1001 and one or more memories 1002, wherein at least one computer program is stored in the one or more memories 1002, and the at least one computer program is loaded and executed by the one or more processors 1001 to implement the methods provided in the above-mentioned various method embodiments. Of course, the computer device 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The computer device 1000 may also include other components for implementing device functions, which will not be described in detail here.
[0221] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program, and the computer program can be executed by a processor to complete the text information extraction method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.
[0222] In an exemplary embodiment, a computer program product or a computer program is also provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device executes the above-mentioned text information extraction method.
[0223] In some embodiments, the computer program involved in the embodiments of this specification may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.
[0224] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0225] The above are only optional embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the protection scope of this specification.
Claims
1. A method for extracting text information, the method comprising: Acquire a target image and a target question text for the target image, wherein the target image carries text information, and the target question text is used to indicate to extract target text information from the target image; Performing text recognition on the target image to obtain image text corresponding to the target image; The target image, the image text and the target question text are input into a multimodal processing model, and the target image, the image text and the target question text are processed by the multimodal processing model to obtain the target text information.
2. The method according to claim 1, wherein the target image, the image text, and the target question text are processed by the multimodal processing model to obtain the target text information, comprising: By means of the multimodal processing model, the target image, the image text and the target question text are encoded to obtain image features of the target image, image text features of the image text and question text features of the target question text; By means of the multimodal processing model, the image features of the target image, the image text features of the image text, and the question text features of the target question text are encoded based on the attention mechanism to obtain target attention features; The target text information is extracted from the image text based on the target attention feature through the multimodal processing model.
3. The method according to claim 2, wherein encoding the target image, the image text, and the target question text to obtain image features of the target image, image text features of the image text, and question text features of the target question text comprises: Dividing the target image into blocks to obtain a plurality of image blocks of the target image; Encoding each of the image blocks and the position of each of the image blocks in the multiple image blocks and the target image to obtain the image features; Segmenting the image text to obtain a plurality of first words of the image text; encoding each of the first words and the position of each of the first words in the plurality of first words and the image text to obtain the image text feature; The target question text is segmented to obtain a plurality of second words of the target question text; each of the second words and the position of each of the second words in the plurality of second words and the target question text is encoded to obtain features of the target question text.
4. The method according to claim 3, wherein encoding each of the image blocks and the position of each of the image blocks in the plurality of image blocks and the target image to obtain the image features comprises: For any image block among the multiple image blocks, fully connect the image block to obtain an image block feature of the image block; Embedding and encoding the sequence number of the image block in the multiple image blocks to obtain a first image block position feature of the image block; Embedding and encoding the coordinates of the image block in the target image to obtain a second image block position feature of the image block; The image block features, the first image block position features and the second image block position features of each of the image blocks are spliced to obtain image block embedding features of each of the image blocks, and the image block embedding features of each of the image blocks constitute the image features.
5. The method according to claim 3, wherein encoding each of the first words and the position of each of the first words in the plurality of first words and the image text to obtain the image text feature comprises: For any first word among the plurality of first words, embedding-encoding the first word to obtain a first word feature of the first word; Embedding the sequence number of the first word in the plurality of first words to obtain a first word position feature of the first word; Performing embedding coding on the vocabulary set to which the first vocabulary belongs in the image text to obtain a second vocabulary position feature of the first vocabulary; The first vocabulary features, first vocabulary position features and second vocabulary position features of each of the first vocabulary are concatenated to obtain the first vocabulary embedding features of each of the first vocabulary, and the first vocabulary embedding features of each of the first vocabulary constitute the image text features.
6. The method according to claim 3, wherein encoding each of the second words and the position of each of the second words in the plurality of second words and the target question text to obtain the target question text features comprises: For any second word among the plurality of second words, performing embedding coding on the second word to obtain a second word feature of the second word; Performing embedding coding on the sequence number of the second word in the plurality of second words to obtain a third word position feature of the second word; Performing embedding coding on the vocabulary set to which the second vocabulary belongs in the target question text to obtain a fourth vocabulary position feature of the second vocabulary; The second vocabulary features, the third vocabulary position features and the fourth vocabulary position features of each of the second vocabulary are concatenated to obtain the second vocabulary embedding features of each of the second vocabulary, and the second vocabulary embedding features of each of the second vocabulary constitute the target question text features.
7. The method according to claim 2, wherein the image features of the target image, the image text features of the image text, and the question text features of the target question text are encoded based on the attention mechanism to obtain the target attention features, comprising: splicing the image features of the target image, the image text features of the image text, and the question text features of the target question text to obtain a splicing feature, wherein the splicing feature includes a plurality of sub-features, and the plurality of sub-features include an image block embedding feature of each image block in the target image, a first vocabulary embedding feature of each first vocabulary in the image text, and a second vocabulary embedding feature of each second vocabulary in the target question text; Encoding each of the sub-features based on an attention mechanism to obtain an attention feature of each of the sub-features; The attention features of each of the sub-features are fused to obtain the target attention feature.
8. A text information extraction system, the system comprising an acquisition module, a recognition module, an input module and a multimodal processing model; The acquisition module is used to acquire a target image and a target question text for the target image, wherein the target image carries text information, and the target question text is used to indicate to extract target text information from the target image; The recognition module is used to perform text recognition on the target image to obtain the image text corresponding to the target image; The input module is used to input the target image, the image text and the target question text into the multimodal processing model; The multimodal processing model is used to process the target image, the image text and the target question text to obtain the target text information.
9. A computer device, comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the text information extraction method according to any one of claims 1 to 7.
10. A computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the text information extraction method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Multi-modal feature fusion method, device, equipment, medium and product
CN114821255A
Format information determination method and device, electronic equipment and storage medium
CN116416638A
Open type question answering method and device and related equipment
CN117725168A
Multi-mode-based bad webpage detection method and device, equipment and medium
CN118013144A
