Information extraction method, training method of information extraction model, and computer program product

Abstract

The embodiment of the invention provides an information extraction method, a training method of an information extraction model and a program product, and relates to the technical field of artificial intelligence. The information extraction method comprises the steps of obtaining multi-modal feature data corresponding to a text image, the multi-modal feature data being feature data generated at least according to text features and image features in the text image; on the basis of prompt information related to the text image and the multi-modal feature data, generating feature data to be matched for matching with the multi-modal feature data; and performing matching processing on the to-be-matched feature data and the multi-modal feature data to obtain matching result feature data, and extracting a text result responding to the prompt information from a text corresponding to the text image according to the matching result feature data. Through the embodiment of the invention, the accuracy of the text result obtained by extracting information from the text image can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an information extraction method, an information extraction model training method, and a computer program product. Background Art

[0002] With the rapid development of big data, extracting valuable information from text-rich visual images is becoming increasingly important. For example, extracting the amount from a bill image or the departure station from a train ticket image is crucial for data analysis and decision-making.

[0003] However, due to the visual complexity and content diversity of text images, accurate and efficient information extraction, especially for text images with a specific typesetting, layout, or structure, has become a huge challenge. One existing extraction method uses a discriminant model to discriminate text information generated by an OCR (Optical Character Recognition) engine to achieve information extraction. This method is sensitive to the order of text generated by the OCR engine, and the accuracy and reliability of information extraction from disordered or misordered text information generated by OCR will be significantly reduced.

[0004] Therefore, there is an urgent need for a more effective way to extract the information of the corresponding text in the text image. Summary of the Invention

[0005] In view of this, an embodiment of the present application provides an information extraction solution to at least partially solve the above-mentioned problem.

[0006] According to a first aspect of an embodiment of the present application, there is provided an information extraction method, comprising: acquiring multimodal feature data corresponding to a text image, wherein the multimodal feature data is feature data generated at least based on text features and image features in the text image; generating feature data to be matched for matching with the multimodal feature data based on prompt information related to the text image and the multimodal feature data; performing matching processing on the feature data to be matched and the multimodal feature data to obtain matching result feature data, and extracting a text result responsive to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0007] According to a second aspect of an embodiment of the present application, a training method for an information extraction model is provided, wherein the information extraction model includes an encoder, a generator and a matcher, and the method includes: obtaining a text image sample for training the information extraction model and text result true value annotation information corresponding to the text image sample, as well as prompt information corresponding to the text image sample; inputting the text image sample into the encoder to obtain corresponding multimodal feature data, wherein the multimodal feature data is feature data generated based on at least text features and image features in the text image sample; based on the multimodal feature data and the prompt information, generating to-be-matched feature data for matching with the multimodal feature data by the generator; matching the to-be-matched feature data and the multimodal feature data by the matcher to obtain matching result feature data, and extracting a text prediction result responsive to the prompt information from the text corresponding to the text image based on the matching result feature data; and training the information extraction model based on the difference between the text prediction result and the text result true value annotation information.

[0008] According to a third aspect of the embodiments of the present application, a computer program product is provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method steps described in the first or second aspect are implemented.

[0009] According to the fourth aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect or the second aspect.

[0010] According to a fifth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first or second aspect is implemented.

[0011] According to the information extraction solution provided in the embodiment of the present application, multimodal feature data of the text image can be extracted, which includes not only text features but also image features (which can effectively represent information such as the typesetting, layout, or structure of the text image), so that the information of the multimodal feature data is richer, and the semantic information contained therein is more complete and rich. When performing information extraction, the prompt information related to the text image is also combined, and the prompt information can effectively represent the text content that needs to be extracted from the text image. Furthermore, in order to obtain text that better matches the prompt information, feature data to be matched will be generated based on the multimodal feature data and the prompt information. Because the feature data to be matched is generated based on the prompt information, when the feature data to be matched is matched with the multimodal feature data, a more accurate text result corresponding to the prompt information can be obtained, thereby ensuring the accuracy and interpretability of the output text result. In particular, for text images with a certain typesetting, layout or structure, this method can be used to match the feature data to be matched with the multimodal features corresponding to the text image, so that the required features can be extracted from the multimodal features more accurately, thereby avoiding dependence on the order of the recognized text and improving the accuracy and reliability of information extraction for text images. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 A schematic diagram of an exemplary system applicable to the embodiments of the present application;

[0014] Figure 2A A schematic diagram of a framework of an information extraction model applicable to an embodiment of the present application;

[0015] Figure 2B Schematic diagram of the structure of an encoder applicable to the information extraction model of the embodiment of the present application;

[0016] Figure 2C A schematic diagram of the structure of a generator of an information extraction model applicable to an embodiment of the present application;

[0017] Figure 2D A schematic diagram of the structure of a matcher for an information extraction model applicable to an embodiment of the present application;

[0018] Figure 2E A schematic diagram of the coordination of an encoder, a generator, and a matcher for an information extraction model applicable to an embodiment of the present application;

[0019] Figure 3 This is a flowchart of the steps of an information extraction method according to the first embodiment of the present application;

[0020] Figure 4 This is a flowchart of the steps of an information extraction method according to the second embodiment of the present application;

[0021] Figure 5 Schematic diagram of a prompt-aware resampler of an information extraction model according to the second embodiment of the present application;

[0022] Figure 6A A schematic diagram of an information extraction model matcher using an autoregressive mode according to the second embodiment of the present application;

[0023] Figure 6B Schematic diagram comparing a traditional matching method according to an embodiment of the present application and a matching method according to an embodiment of the present application;

[0024] Figure 7 This is a flowchart of the steps of an information extraction model training method according to Example 3 of the present application;

[0025] Figure 8 is a schematic diagram of a text image sample according to the third embodiment of the present application;

[0026] Figure 9 Schematic diagram of an information extraction device according to the fourth embodiment of the present application;

[0027] Figure 10 Schematic diagram of an information extraction model training device according to the fifth embodiment of the present application;

[0028] Figure 11 This is a structural diagram of an electronic device according to Example 6 of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0030] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0031] Figure 1 An exemplary system applicable to the embodiment of the present application is shown. Figure 1As shown, the system may include a cloud service 102, a communication network 104 and / or one or more user devices 106. Figure 1 The example in the figure is for multiple user devices.

[0032] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to a distributed storage system, a server cluster, a computing cloud server cluster, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can perform information extraction on a text image. In some embodiments, the cloud server 102 can obtain multimodal feature data corresponding to the text image, where the multimodal feature data includes both text features and image features of the text image. Next, based on the prompt information corresponding to the text image and the multimodal feature data, the cloud server 102 can generate feature data to be matched. Furthermore, the feature data to be matched can be matched with the multimodal feature data to obtain a text result responsive to the prompt information. In some embodiments, the cloud server 102 can be configured with an information extraction model to perform the above-described information extraction. As another example, the cloud server 102 can receive and respond to an information extraction request from the user device 106, generating a corresponding text result in the manner described above and returning it to the user device 106.

[0033] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network.

[0034] The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud service end 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0035] In some embodiments, user device 106 may be any suitable type of device capable of interacting with a user. For example, in some embodiments, user device 106 may generate an information extraction request based on prompt information corresponding to a text image input by the user and send the request to cloud server 102; and receive text results returned by cloud server 102 and display them to the user. In some embodiments, user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a game console, a media player, a vehicle entertainment system, other edge computing devices, and / or any other suitable type of user device.

[0036] It should be noted that although this example takes the cloud server 102 as an example, it is understandable that the cloud server 102 can be replaced by a local device or other server that can provide corresponding functions, and there is no limitation on this.

[0037] Based on the above system, an embodiment of the present application provides an information extraction method, which can be implemented by an information extraction model in a feasible manner. The information extraction model can be deployed on the aforementioned cloud service end 102 or other suitable deployment devices.

[0038] In order to facilitate understanding and clarify the description, before explaining the implementation process of the information extraction method, Figures 2A to 2E , briefly describe the structure of the information extraction model:

[0039] like Figures 2A to 2E As shown, the information extraction model of this embodiment includes an encoder 202, a generator 204 and a matcher 206, wherein the encoder 202 is used to receive text images (such as images of restaurant bills, invoices, tickets, etc., which contain not only text but also other visual information, such as structure, typesetting, layout, etc.), and process the text images, such as text recognition and feature extraction, to output multimodal feature data corresponding to the text images (such as image data) based on the text recognition results and the extracted image features. Figure 2B The "Muilt-modal Embeddings" in Figure 2E The generator 204 receives multimodal feature data and prompt information related to the text image (such as Figure 2C The “Prompt” in the text image may be, for example, “What dishes are there in the text image?”, etc.), and based on the prompt information and the multimodal feature data, the feature data to be matched with the multimodal feature data is generated (such as Figure 2D and Figure 2EThe matcher 206 is used to receive the feature data to be matched and the multimodal feature data, and perform matching processing on the two to extract the feature data to be matched that matches the prompt information from the multimodal feature data, and output the text result (such as the text result) corresponding to the prompt information in the text corresponding to the text image. Figure 2D and Figure 2E It should be noted that the above Figure 2B-2E The specific structure is only briefly described here and will be further described in detail below.

[0040] The information extraction method of the present application is described below through multiple embodiments in combination with the information extraction model.

[0041] Example 1

[0042] Reference Figure 3 , shows a schematic flow chart of the steps of an information extraction method according to the first embodiment of the present application. The method includes:

[0043] Step S302: Acquire multimodal feature data corresponding to the text image.

[0044] The multimodal feature data is feature data generated based on at least text features and image features in the text image.

[0045] In the embodiments of this application, a text image refers to an image containing text information. This can be a pure text image, a text image with a certain structure, typesetting, and layout, or an image containing both text information and other image information. A text image has both textual and image features. In the embodiments of this application, multimodal feature data is generated based on at least these two features to better describe the true form of the text image.

[0046] In one feasible approach, the multimodal feature data corresponding to the text image can be obtained by Figure 2A and Figure 2B The multimodal feature data is obtained by the encoder 202 in the text image, and the text, text order, text layout and visual information (image features) in the text image are integrated into the multimodal feature data. Among them, the text, text order, text layout and the like can be obtained by the OCR engine recognition, and the visual information can be obtained by an image encoder such as (e.g. Figure 2B It is obtained by processing modules, units or model structures such as the "Image Encoder" shown in the figure) that have image feature extraction functions.

[0047] Step S304: Based on the prompt information related to the text image and the multimodal feature data, generate feature data to be matched with the multimodal feature data.

[0048] The prompt information associated with the text image can be a prompt information (also called a prompt word) input by the user. The prompt information can effectively represent the information the user intends to extract from the text image, but is not limited to this. The prompt information can also be one or more preset prompt information. For example, the prompt information can be: "The name of the menu is", "The price of the menu is", etc.

[0049] In one possible approach, the prompt information and multimodal feature data can be input into Figure 2A or Figure 2C In the generator 204 shown, the generator 204 processes it and then outputs the feature data to be matched, and its specific implementation will be described in detail below.

[0050] Step S306: performing matching processing on the feature data to be matched and the multimodal feature data to obtain matching result feature data, and extracting a text result corresponding to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0051] Since the feature data to be matched is generated based on the prompt information and the multimodal feature data, which carries information that can effectively indicate the required features, the feature data to be matched is matched with the multimodal feature data corresponding to the text image, so that the final text result can be matched more accurately. When performing the matching, in one feasible way, the similarity between the feature data to be matched and the multimodal feature data can be calculated. This process can also be understood as entity marking the multimodal feature data through the feature data to be matched to find the entities related to the prompt information in the multimodal feature data. The result of the entity marking can be used as the matching result feature data, and the text result corresponding to the prompt information is extracted from the text corresponding to the text image based on the matching result feature data. In this way, it can be ensured that the output content is extracted from the text recognized from the text image, the extraction result is more accurate, and it can solve the problem of "hallucination" (i.e., the appearance of text that is not in the text image).

[0052] In another feasible approach, an autoregressive method can be used to match the data corresponding to each token in the multimodal feature data with the data of the corresponding token in the feature data to be matched, thereby predicting the output corresponding to the token. This also ensures that the text related to the prompt information is extracted from the text of the text image, and can also solve the "hallucination" problem and achieve the effect of accurate output text results.

[0053] In one example, the Figure 2A or Figure 2D The matcher 206 shown in FIG performs matching processing on the feature data to be matched and the multimodal feature data, and its specific implementation will be described in detail below.

[0054] In this way, according to the information extraction scheme provided in the embodiment of the present application, multimodal feature data of the text image can be extracted, which includes not only text features but also image features (which can effectively represent information such as the typesetting, layout, or structure of the text image), so that the information of the multimodal feature data is richer, and the semantic information contained therein is more complete and rich. When performing information extraction, the prompt information related to the text image is also combined, and the prompt information can effectively represent the text content that needs to be extracted from the text image. Furthermore, in order to obtain text that better matches the prompt information, feature data to be matched will be generated based on the multimodal feature data and the prompt information. Because the feature data to be matched is generated based on the prompt information, when the feature data to be matched is matched with the multimodal feature data, a more accurate text result corresponding to the prompt information can be obtained, thereby ensuring the accuracy and interpretability of the output text result. In particular, for text images with a certain typesetting, layout or structure, this method can be used to match the feature data to be matched with the multimodal features corresponding to the text image, so that the required features can be extracted from the multimodal features more accurately, thereby avoiding dependence on the order of the recognized text and improving the accuracy and reliability of information extraction for text images.

[0055] Example 2

[0056] Reference Figure 4 , shows a schematic flow chart of the steps of an information extraction method according to Embodiment 2 of the present application. The method includes:

[0057] Step S402: Acquire multimodal feature data corresponding to the text image.

[0058] In one feasible manner, this step S402 may include the following sub-steps:

[0059] Sub-step S4021: performing text recognition and image feature extraction on the text image to obtain corresponding text recognition results and image features.

[0060] For example, a text image is input into an OCR engine to obtain a text recognition result. The text recognition result contains text features (such as Figure 2E "Texts" shown in ) and the layout features corresponding to the text features (such as Figure 2E )

[0061] In order to preserve the visual information in the text image as much as possible, the text image is also input into the image encoder to obtain the image features output by the image encoder (such as Figure 2EThe features output after being encoded by the image encoder “Image Encoder”).

[0062] Sub-step S4022: Perform feature fusion on the text recognition results and image features to obtain multimodal feature data.

[0063] Through feature fusion, the two parts of features (i.e., the features of the text recognition results and the image features) can be enriched and enhanced, so that the obtained multimodal feature data can more effectively represent the text image.

[0064] In one possible approach, this step can be accomplished by Figure 2A or Figure 2B The encoder 202 shown is implemented. Exemplarily, the encoder 202 may include an OCR engine ( Figure 2B Indicated as "OCR"), image encoder ( Figure 2B ) and the pre-trained layout encoder ( Figure 2B Indicated as “Pretrained LayoutLMEncoder” in

[15] ).

[0065] When this step is implemented by the encoder 202, after the text image is input to the encoder 202, the encoder 202 will perform text recognition processing on the text image through its OCR engine to obtain the corresponding text features ( Figure 2B Indicated as "Texts") and the layout features corresponding to the text features ( Figure 2B On the other hand, the encoder 202 uses its image encoder to extract image features from the text image to obtain corresponding image features. In one feasible approach, since both involve visual information, the image features can be spliced ​​into the layout features.

[0066] Furthermore, text features, layout features, and image features can be input into the layout encoder to fuse them into multimodal feature data (such as Figure 2B Multi-modal feature data can be formed into feature vectors (such as Figure 2E In this embodiment, the dimension of the feature vector is N Word *d. Where, N Word It represents the number of words corresponding to the multimodal feature data, and d is the encoding length.

[0067] The aforementioned OCR engine can be any engine capable of optical character recognition, and the image encoder can be a model capable of extracting image features, including but not limited to a convolutional neural network model, a ViT model, etc., which is not limited in this embodiment of the present application. The pre-trained layout encoder can also be replaced by a network such as DocFormer or FormNet.

[0068] Step S404: Based on the prompt information related to the text image and the multimodal feature data, generate feature data to be matched with the multimodal feature data.

[0069] In one feasible manner, this step S404 can be implemented through the following sub-steps:

[0070] Sub-step S4041: Obtain prompt information corresponding to the text image, and generate auxiliary query information based on the prompt information and pre-generated feature extraction parameters.

[0071] Feature extraction parameters can be understood as parameters that, combined with the prompt information, further clarify the information required from the multimodal feature data. If a model structure is employed, these feature extraction parameters can be implemented as model parameters learned through model training. Auxiliary query information generated based on the prompt information and feature extraction parameters can then be used in conjunction with the multimodal feature data for computations such as attention calculations to further extract features and enhance the inherent information.

[0072] In one feasible method, in order to improve the ability to perceive the prompt information, the following can be used: Figure 2A or Figure 2C The generator 204 shown in FIG implements step S404. The generator 204 may include a prompt-aware resampler (e.g., Figure 2C "Prompt-aware Resamplar" shown in Figure 2C ). Among them, an exemplary prompt-aware resampler can be implemented using a model structure based on the Cross-Attention mechanism, as shown in FIG. Figure 5 The generating unit includes at least one processing combination connected to each other, each processing combination including a cross attention layer (such as Figure 2C Indicated as "Cross-attention Layer") and layout awareness block ( Figure 2C ). Exemplarily, the generation unit can be obtained by pre-training based on a document pre-training model of Transformer, but is not limited thereto.

[0073] When the generator 204 structure is adopted, the sub-step S4041 can be performed by its prompt-aware resampler. Figure 5 As shown, the pre-generated feature extraction parameters are represented as X in the figure q (LearnableQueries is a model parameter that can be learned and can be used for feature extraction), and the prompt information is represented by X p (meaning "Prompt Tokens"). During the training phase, the prompt information X p It can be preset; in the inference stage, the prompt information can be input by the user or generated based on the content input by the user.

[0074] Furthermore, the feature extraction parameter X q and prompt message X p Spliced ​​into auxiliary query information, which Figure 5 It is expressed as Q = [X p , X q ], which can represent the query vector Q in the cross-attention calculation to be performed.

[0075] Sub-step S4042: performing feature resampling based on the multimodal feature data and the auxiliary query information to obtain enhanced prompt information corresponding to the prompt information and enhanced resampled features corresponding to the feature extraction parameters.

[0076] Exemplarily, feature resampling can be achieved through cross-attention calculation. For example, multimodal feature data can be spliced ​​with auxiliary query information to obtain a spliced ​​result; cross-attention processing is performed on the spliced ​​result to obtain enhanced prompt information corresponding to the prompt information and enhanced resampled features corresponding to the feature extraction parameters.

[0077] When using Figure 5 When the cue-aware resampler shown in Figure 1 implements this substep, the multimodal feature data is Figure 5 Indicated as X m (“Multi-modal Embeddings”). Auxiliary query information Q and multimodal feature data X m Splicing, the splicing result formed by the splicing is [X m , X p , X q ], in the cross attention calculation to be performed, the [X m , X p , X q ] can represent the key vector K and the value vector V at the same time.

[0078] Furthermore, based on the above Q, K and V, the cross attention layer in the hint-aware resampler ( Figure 5Cross-Attention is performed to obtain X′ p , X′ q , respectively representing the enhanced prompt information corresponding to the prompt information and the enhanced resampling features corresponding to the feature extraction parameters.

[0079] It can be seen that through cross attention processing, the prompt information X can be realized p , feature extraction parameters X q and multimodal feature data X m The mutual resampling between them is performed to enhance the corresponding information and obtain an output result ("Enhanced Prompts & Resampled Embeddings"), which includes the enhanced prompt information X' corresponding to the prompt information p , and the enhanced features X′ corresponding to the feature extraction parameters q .

[0080] Sub-step S4043: Generate feature data to be matched with the multimodal feature data according to the enhanced prompt information and the enhanced resampling features.

[0081] In one feasible embodiment, the sub-step S4043 can be implemented as follows: performing cross-attention processing on the enhanced prompt information and the enhanced resampling features to obtain the cross-attention processing results; performing layout-aware feature extraction processing on the cross-attention processing results to generate feature data to be matched with the multimodal feature data.

[0082] For example, again referring to Figure 2C , taking the structure of the aforementioned generation unit as an example, the enhanced prompt information and the enhanced features can be input into the cross-attention layer of the first processing combination, so that the two are subjected to cross-attention processing to obtain a cross-attention processing result. The cross-attention processing result is input into the layout perception block of the first processing combination, and layout perception feature extraction processing is performed in the layout perception block, so as to learn the layout-related information of the text contained in the text image. The layout perception block of the first processing combination inputs the extraction result into the cross-attention layer of the second processing combination, and the cross-attention layer performs cross-attention processing on the extraction result output by the previous processing combination and the enhanced features, and then inputs the cross-attention processing result into the layout perception block corresponding to the processing combination in which it is located for processing to obtain the extraction result corresponding to the processing combination. This is repeated until the processing of multiple processing combinations is completed, and the output result is the feature data to be matched ("matcher vectors"). In this embodiment, the dimension of the feature data to be matched is d*(N BO +1). Where d is the decoding length, N BIIndicates the number of entity tag categories. Carrying the entity tag categories of the entities to be extracted in the feature data to be matched can facilitate subsequent rapid word classification and improve the efficiency of subsequent feature matching.

[0083] Step S406: performing matching processing on the feature data to be matched and the multimodal feature data to obtain matching result feature data, and extracting a text result corresponding to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0084] Among them, the matching processing of the feature data to be matched and the multimodal feature data can be implemented in an appropriate manner, but in order to improve the matching efficiency, two implementation methods are provided in the embodiment of the present application, which are respectively called the original mode and the autoregressive mode. Among them, the original mode calculates the similarity between the feature data to be matched and the multiple features contained in the multimodal feature data, and determines the matching result feature data according to the similarity results corresponding to the multiple features, such as the similarity matrix. In the autoregressive mode, the autoregressive method is used to match the feature data to be matched and the multimodal feature data to obtain the matching result feature data. The autoregressive mode is a method that establishes a multi-layer recursive structure, gradually predicts the next value based on the input data, and continuously iterates to generate the entire sequence. Specifically in this embodiment, the autoregressive mode realizes the entity extraction of the text in the text image by sequentially calculating the similarity matrix of the feature data to be matched and the multimodal feature data.

[0085] For example, in the original mode, the similarity is calculated by performing matrix multiplication between the feature data to be matched and the multimodal feature data. The dimension of the similarity calculation result is N Word *(N BI +1), the similarity calculation result is the matching result feature data.

[0086] Because the feature data to be matched carries the category of the entity label of the entity to be extracted, it can be quickly classified based on the corresponding word elements (also called tokens) in the multimodal feature data to determine the word elements with a higher probability of matching the prompt information, and then the corresponding text can be extracted as the text result.

[0087] For example, when using this method, Figure 2EThe text image shown contains two dishes: "BIHUN GORENG SEAFOOD" and "KWETIAW SEAFOOD SIRAM." The quantity of each dish is 1, and the unit price is 40.456 and 42.274, respectively. If the prompt is "menu.nm is," which asks for the name of the dish, the resulting text output is: "BIHUN GORENG SEAFOOD" and "KWETIAW SEAFOOD SIRAM."

[0088] An autoregressive pattern matching process is as follows Figure 6A As shown, Figure 6A In the encoder 202 ( Figure 6A The diagram shows that "Source Encoder" outputs multimodal feature data ( Figure 6A Indicated as "Source Vector"), generator 204 ( Figure 6A The target generator in the figure outputs the corresponding feature data to be matched ( Figure 6A The matcher 206 performs autoregressive pattern matching on the multimodal feature data and the feature data to be matched, using word units as units, and performs N recursions ( Figure 6A It is indicated as 5 times, namely step 0-step 4), and the result that matches the feature data to be matched in the multimodal feature data is predicted, namely the matching result feature data.

[0089] In practical applications, due to the diversity of text layout in text images, it is not easy to obtain the text order. However, through the autoregressive method, a sequence matcher vector that matches the entire entity in an autoregressive manner can be generated, such as Figure 6A As shown in [TA], BIHUN, GORENG, ... [SEP] (used to separate repeated entities) and [EOS] (used to mark the end of sequence), etc. If the text contains repeated entities, the matcher 206 can match and generate a [SEP] token to separate the repeated entities. Furthermore, the matcher 206 can stop processing when it predicts the end-of-sequence marker [EOS]. Using tokens such as [SEP] and [EOS] can make the text sequence easier to decode.

[0090] In one feasible approach, in order to implement context-aware word classification for information extraction, the matcher 206 can be implemented using a decoder to generate weights for sequence classification. By utilizing these weights, classification weights conditioned on each individual input can be generated, thereby distinguishing it from traditional sequence labeling. For ease of distinction, these weights can be referred to as conditional parameter weights. Specifically, the generator 204 generates a feature data to be matched based on each prompt information and the output of the resampler. Then, two sub-matchers generated by linear projection are included in the matcher 206. Among them, one sub-matcher is used to match the starting token for the feature data to be matched, and the other sub-matcher is used to match the token within the entity for the feature data to be matched. In addition, an additional sub-matcher is introduced to match non-entity tokens.

[0091] For ease of explanation, the following Figure 6B , compare and explain the differences between the above matching method in the embodiment of the present application and the matching method in the related art. Figure 6B As shown, taking the matching of the corresponding "GORENG" in the multimodal feature data as an example, in the matching method of the related art, the weights ( Figure 6B In the matching method of the embodiment of the present application, the classification weights ( Figure 6B Dynamic weights) can effectively improve the ability of the classification layer.

[0092] After the matching result feature data is determined by the matching unit 206 through classification matching, the corresponding text can be extracted from the text corresponding to the text image based on the matching result feature data as the text result in response to the prompt information. This serialized autoregressive model can effectively solve the problem of difficulty in easily obtaining the reading order due to the large changes in document layout, ensuring that the output text results are in the correct order.

[0093] In summary, when the above method is implemented through the information extraction model, the encoder 202 is used to perform text recognition and image feature extraction on the input text image, and output multimodal feature data corresponding to the text image based on the text recognition result and the extracted image features; the generator 204 is used to generate feature data to be matched for matching with the multimodal feature data based on the prompt information and multimodal feature data related to the text image; the matcher 206 is used to match the feature data to be matched and the multimodal feature data to obtain matching result feature data, and extract the text result responsive to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0094] Through this embodiment, multimodal feature data of a text image can be extracted, which includes not only text features but also image features (which can effectively represent information such as the typesetting, layout, or structure of the text image), making the information of the multimodal feature data richer and the semantic information contained therein more complete and rich. When extracting information, prompt information related to the text image is also combined, and the prompt information can effectively represent the text content that needs to be extracted from the text image. Furthermore, in order to obtain text that better matches the prompt information, feature data to be matched will be generated based on the multimodal feature data and the prompt information. Because the feature data to be matched is generated based on the prompt information, when the feature data to be matched is matched with the multimodal feature data, a more accurate text result corresponding to the prompt information can be obtained, thereby ensuring the accuracy and interpretability of the output text result. In particular, for text images with a certain typesetting, layout or structure, this method can be used to match the feature data to be matched with the multimodal features corresponding to the text image, so that the required features can be extracted from the multimodal features more accurately, thereby avoiding dependence on the order of the recognized text and improving the accuracy and reliability of information extraction for text images.

[0095] Example 3

[0096] Reference Figure 7 , shows a schematic flow chart of the steps of a training method for an information extraction model according to the third embodiment of the present application. As mentioned above, the information extraction model of this embodiment includes three parts: an encoder, a generator, and a matcher.

[0097] The method includes:

[0098] Step S700: determining a pre-training strategy to be used from a plurality of preset pre-training strategies for training an information extraction model; and determining prompt information corresponding to a text image sample based on the pre-training strategy.

[0099] It should be noted that this step is optional. For example, when there is more than one pre-training strategy, this step can be performed. However, when a default pre-training strategy is set, this step is not required.

[0100] In this embodiment, different pre-training strategies and corresponding pre-training prompts can be configured to enable the information extraction model to learn different capabilities. The pre-training strategies include at least one of the following: Match to Fill (MTF), Search One Direction (SOD), and Search All Directions (SAD).

[0101] The following are the descriptions:

[0102] 1) Filling matching strategy: a strategy for indicating that, based on at least two given words in the prompt information, text between the given words is matched from the text of the text image sample.

[0103] It can improve the model's ability to match content from contextual text and learn reading order.

[0104] When the pre-training strategy is a filling matching strategy, then based on the pre-training strategy, determining the prompt information corresponding to the text image sample can be implemented as follows: selecting at least two given words from the text corresponding to the text image sample, generating prompt information for indicating matching given words from the text image sample based on the selected given words, and determining the text between the given words.

[0105] In addition, in the embodiment of the present application, the training of the information extraction model adopts a supervised training method. Based on this, in order to achieve efficient and accurate training of the information extraction model, after determining the pre-training strategy and prompt information, the training samples used to train the information extraction model need to be labeled. That is, the text image samples are labeled with the true value of the text result. Then, under the filling matching strategy, the true value annotation information of the text result corresponding to the text image sample can be determined in the following way: based on the text in the text image sample, corresponding to the text between at least two selected given words, the true value annotation information of the text result of the text image sample is generated.

[0106] For example, a text image sample such as Figure 8 As shown in the figure, you can choose any two words as given words. For example, if the given words are "Color" and "Cathode", the corresponding prompt information can be "fill in between'Color'and'Cathode'". The corresponding text result ground truth annotation information is "Band" and "Denotes".

[0107] 2) Direction search strategy: a strategy for matching the K nearest neighboring texts in a specified direction to at least one given word in the prompt information from the text of the text image sample.

[0108] Among them, K is greater than or equal to 1. This strategy can improve the model's spatial perception learning ability.

[0109] Under the directional search strategy, based on the pre-training strategy, determining the prompt information corresponding to the text image sample can be implemented as follows: determining the specified direction and the value of K, and selecting at least one given word from the text image sample, and based on the selected given word, the value of K and the specified direction, generating prompt information for indicating the K nearest neighboring texts that match the given word in the specified direction from the text of the text image sample.

[0110] Furthermore, the text result true value annotation information corresponding to the text image sample is generated in the following way: based on the K texts adjacent to the given word in the specified direction in the text of the text image sample, the text result true value annotation information of the text image sample is generated.

[0111] For example, using the aforementioned text image sample as an example, we can choose any word as the given word. For example, if the given word is "Leads", the value of K is 2, and the direction is right, the corresponding prompt information can be "find two instances to the right of 'Leads'" (match the two words adjacent to the right of "Leads"). The corresponding text result ground truth annotation information is "Solderable" and "per".

[0112] 3) Neighborhood strategy: a strategy for indicating that, based on at least one given word in the prompt information, M texts adjacent to the given word are matched from the text of the text image sample.

[0113] Among them, M is greater than or equal to 1. This strategy can improve the ability of information extraction models to utilize the spatial layout of text.

[0114] Under this proximity strategy, based on the pre-training strategy, determining the prompt information corresponding to the text image sample can be implemented as follows: determining the value of M, and selecting at least one given word from the text image sample, and based on the selected given word and the value of M, generating prompt information for indicating that M texts adjacent to the given word are matched from the text of the text image sample.

[0115] Furthermore, the text result true value annotation information corresponding to the text image sample is generated in the following manner: based on M texts adjacent to the given word in the text of the text image sample, the text result true value annotation information is generated.

[0116] For example, if the word selected is "grams" and the value of M is 3, the corresponding prompt information can be "Find three nearest instances of 'grams'" (match the three words closest to "grams"). The corresponding text result ground truth annotation information is "0.3", "(approx)" and "Band".

[0117] Step S702: Obtain text image samples used to train the information extraction model and text result true value annotation information corresponding to the text image samples, as well as prompt information corresponding to the text image samples.

[0118] The text image samples may be obtained from a preset training set, and the text result true value annotation information and prompt information may be obtained in the manner described in step S700 .

[0119] Step S704: Input the text image sample into the encoder to obtain corresponding multimodal feature data.

[0120] The multimodal feature data is feature data generated based on at least text features and image features in the text image sample. The process of obtaining the multimodal feature data through the encoder can be referred to the description of obtaining multimodal feature data based on text images in the aforementioned embodiment, and will not be repeated here.

[0121] Step S706: Based on the multimodal feature data and the prompt information, a generator generates feature data to be matched with the multimodal feature data.

[0122] The process of generating the feature data to be matched by the generator can refer to the relevant description in the aforementioned embodiment and will not be repeated here.

[0123] Step S708: The matching feature data to be matched and the multimodal feature data are matched by a matcher to obtain matching result feature data, and a text prediction result corresponding to the prompt information is extracted from the text corresponding to the text image based on the matching result feature data.

[0124] In the case where a pre-training strategy is selected, the matching process can be performed by the matcher according to the determined pre-training strategy, matching the feature data to be matched and the multimodal feature data. For example, the matcher can use the original mode to perform matching according to the filling method or matching direction indicated by the pre-training strategy, or use the autoregressive mode to perform matching according to the filling method or matching direction indicated by the pre-training strategy. The specific process can be referred to the relevant description in the above embodiment and will not be repeated here.

[0125] Step S710: training the information extraction model based on the difference between the text prediction result and the text result true value annotation information.

[0126] For example, by comparing the output text prediction results with the text result ground truth annotation information, the difference can be calculated, and then the information extraction model can be trained based on the difference. This difference can be calculated using a preset loss function. In the embodiments of this application, the specific implementation of the loss function is not limited. For example, the loss function may include but is not limited to the cross-entropy loss function.

[0127] By using different pre-training strategies, corresponding prompt information and text result true value annotation information to train the information extraction model, the information extraction model can have corresponding capabilities. For example, using the pre-training strategy in this embodiment can improve the information extraction model's perception of spatial context information, thereby making the text prediction result output more accurate.

[0128] Through this embodiment, based on the pre-training strategy, the information extraction model obtained through training can improve its perception of spatial context information, thereby making the information extracted by the trained information extraction model for text images more accurate and reliable.

[0129] Example 4

[0130] Reference Figure 9 , shows a structural block diagram of an information extraction device according to embodiment 4 of the present application.

[0131] The information extraction device of this embodiment includes: an acquisition module 902, which is used to acquire multimodal feature data corresponding to a text image, wherein the multimodal feature data is feature data generated based on at least text features and image features in the text image; a feature generation module 904, which is used to generate feature data to be matched for matching with the multimodal feature data based on prompt information related to the text image and the multimodal feature data; a matching extraction module 906, which is used to match the feature data to be matched with the multimodal feature data to obtain matching result feature data, and extract a text result responsive to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0132] Optionally, the feature generation module 904 is used to obtain prompt information corresponding to the text image, and generate auxiliary query information based on the prompt information and pre-generated feature extraction parameters; perform feature resampling based on the multimodal feature data and the auxiliary query information to obtain enhanced prompt information corresponding to the prompt information, and enhanced resampling features corresponding to the feature extraction parameters; and generate feature data to be matched for matching with the multimodal feature data based on the enhanced prompt information and the enhanced resampling features.

[0133] Optionally, when the feature generation module 904 performs feature resampling based on the multimodal feature data and the auxiliary query information to obtain enhanced prompt information corresponding to the prompt information and enhanced resampling features corresponding to the feature extraction parameters: the multimodal feature data and the auxiliary query information are spliced ​​to obtain a splicing result; and cross-attention processing is performed on the splicing result to obtain enhanced prompt information corresponding to the prompt information and enhanced resampling features corresponding to the feature extraction parameters.

[0134] Optionally, when the feature generation module 904 generates feature data to be matched with multimodal feature data based on the enhanced prompt information and the enhanced resampling features: cross-attention processing is performed on the enhanced prompt information and the enhanced resampling features to obtain a cross-attention processing result; and layout-aware feature extraction processing is performed on the cross-attention processing result to generate feature data to be matched with the multimodal feature data.

[0135] Optionally, when the matching extraction module 906 performs matching processing on the feature data to be matched and the multimodal feature data to obtain matching result feature data: an autoregressive model is used to perform matching processing on the feature data to be matched and the multimodal feature data to obtain matching result feature data; or, similarities are calculated between the feature data to be matched and multiple features contained in the multimodal feature data, and the matching result feature data is determined based on the similarity calculation results corresponding to the multiple features.

[0136] Optionally, when acquiring multimodal feature data corresponding to a text image, the acquisition module 902 performs text recognition and image feature extraction on the text image respectively to obtain corresponding text recognition results and image features, wherein the text recognition results include text features and layout features corresponding to the text features; and performs feature fusion on the text recognition results and image features to obtain multimodal feature data.

[0137] Optionally, the information extraction device of this embodiment extracts information through an information extraction model, which includes an encoder, a generator and a matcher; wherein: the encoder performs text recognition and image feature extraction on the input text image, and outputs multimodal feature data corresponding to the text image based on the text recognition result and the extracted image features (implementing the function of the acquisition model 902); the generator generates feature data to be matched for matching with the multimodal feature data based on the prompt information and multimodal feature data related to the text image (implementing the function of the feature generation model 904); the matcher matches the feature data to be matched and the multimodal feature data to obtain matching result feature data, and extracts the text result responsive to the prompt information from the text corresponding to the text image based on the matching result feature data (implementing the function of the matching extraction module 906).

[0138] The information extraction device of this embodiment is used to implement the corresponding information extraction method in the aforementioned method embodiment, and has the corresponding beneficial effects of the information extraction method. Its specific implementation can refer to the description in the aforementioned method embodiment and will not be repeated here.

[0139] Example 5

[0140] Reference Figure 10 , shows a structural block diagram of a training device for an information extraction model according to Example 5 of the present application.

[0141] In which, the information extraction model includes an encoder, a generator and a matcher, and the training device includes: an information acquisition module 1002, used to obtain text image samples and text result true value annotation information corresponding to the text image samples for training the information extraction model, as well as prompt information corresponding to the text image samples; an encoding module 1004, used to input the text image samples into the encoder to obtain corresponding multimodal feature data, wherein the multimodal feature data is feature data generated based on at least text features and image features in the text image samples; a data generation module 1006, used to generate, through a generator, feature data to be matched for matching with the multimodal feature data based on the multimodal feature data and the prompt information; a data matching module 1008, used to match the feature data to be matched and the multimodal feature data through a matcher to obtain matching result feature data, and extract a text prediction result responsive to the prompt information from the text corresponding to the text image based on the matching result feature data; a training module 1010, used to train the information extraction model based on the difference between the text prediction result and the text result true value annotation information.

[0142] Optionally, the training device also includes: a determination module 1000, which is used to determine the pre-training strategy to be used from multiple pre-training strategies for training the information extraction model before the information acquisition module 1002 obtains the text image samples for training the information extraction model and the text result true value annotation information corresponding to the text image samples; and based on the pre-training strategy, determine the prompt information corresponding to the text image sample; then, the data matching module 1008 matches the feature data to be matched and the multimodal feature data through the matcher, including: matching the feature data to be matched and the multimodal feature data according to the determined pre-training strategy through the matcher.

[0143] Optionally, the pre-training strategy includes at least one of the following: a filling matching strategy: a strategy for indicating that based on at least two given words in the prompt information, a text between given words is matched from the text of the text image sample; a direction search strategy: a strategy for indicating that based on at least one given word in the prompt information, a K nearest neighboring texts in a specified direction to the given word is matched from the text of the text image sample, where K is greater than or equal to 1; a neighboring strategy: a strategy for indicating that based on at least one given word in the prompt information, a M text adjacent to the given word is matched from the text of the text image sample, where M is greater than or equal to 1.

[0144] Optionally, if the pre-training strategy is a filling matching strategy, the determination module 1000 determines, based on the pre-training strategy, that the prompt information corresponding to the text image sample is implemented as follows: selecting at least two given words from the text corresponding to the text image sample, generating prompt information based on the selected given words to indicate matching given words from the text image sample, and determining the text between the given words; the text result true value annotation information corresponding to the text image sample is determined in the following manner: generating the text result true value annotation information of the text image sample based on the text in the text image sample corresponding to the text between the at least two selected given words.

[0145] Optionally, if the strategy is a directional search strategy, the determination module 1000 determines the prompt information corresponding to the text image sample based on the pre-training strategy as follows: determining the specified direction and the value of K, and selecting at least one given word from the text image sample, and based on the selected given word, the value of K and the specified direction, generating prompt information for indicating the K nearest neighboring texts that match the given word in the specified direction from the text of the text image sample; the text result true value annotation information corresponding to the text image sample is generated in the following manner: based on the K texts in the text of the text image sample that are adjacent to the given word in the specified direction, generating the text result true value annotation information of the text image sample.

[0146] Optionally, if the strategy is a proximity strategy, the determination module 1000 determines the prompt information corresponding to the text image sample based on the pre-training strategy as follows: determining the value of M, and selecting at least one given word from the text image sample, and based on the selected given word and the value of M, generating prompt information for indicating that M texts adjacent to the given word are matched from the text of the text image sample; the text result true value annotation information corresponding to the text image sample is generated in the following manner: based on the M texts adjacent to the given word in the text of the text image sample, generating the text result true value annotation information.

[0147] The information extraction model training device of this embodiment is used to implement the training method of the corresponding information extraction model in the aforementioned method embodiment, and has the corresponding beneficial effects of the training method. Its specific implementation can refer to the relevant description in the aforementioned method embodiment, and will not be repeated here.

[0148] Example 6

[0149] Reference Figure 11 , shows a structural diagram of an electronic device according to embodiment six of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0150] like Figure 11 As shown, the electronic device may include: a processor (processor) 1102 , a communication interface (Communications Interface) 1104 , a memory (memory) 1106 , and a communication bus 1108 .

[0151] in:

[0152] The processor 1102 , the communication interface 1104 , and the memory 1106 communicate with each other via a communication bus 1108 .

[0153] The communication interface 1104 is used to communicate with other electronic devices or servers.

[0154] The processor 1102 is configured to execute the program 1110 , and specifically may execute the relevant steps in the above method embodiment.

[0155] Specifically, the program 1110 may include program codes, which include computer operation instructions.

[0156] The processor 1102 may be a CPU, a Graphics Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0157] The memory 1106 is used to store the program 1110. The memory 1106 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0158] The program 1110 may include multiple computer instructions. Specifically, the program 1110 may enable the processor 1102 to execute operations corresponding to the method described in any of the aforementioned method embodiments through the multiple computer instructions.

[0159] The specific implementation of each step in program 1110 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0160] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the aforementioned method embodiments. The computer storage medium includes, but is not limited to, a compact disc read-only memory (CD-ROM), random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk.

[0161] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0162] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0163] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0164] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., random access memory (RAM), read-only memory (ROM), flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0165] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for specific applications, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0166] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. An information extraction method comprising: Acquiring multimodal feature data corresponding to the text image, wherein the multimodal feature data is feature data generated based on at least text features and image features in the text image; Based on the prompt information related to the text image and the multimodal feature data, generating feature data to be matched for matching with the multimodal feature data; Matching processing is performed on the feature data to be matched and the multimodal feature data to obtain matching result feature data, and a text result corresponding to the prompt information is extracted from the text corresponding to the text image based on the matching result feature data.

2. The method according to claim 1, wherein The step of generating feature data to be matched with the multimodal feature data based on the prompt information related to the text image and the multimodal feature data includes: Acquiring prompt information corresponding to the text image, and generating auxiliary query information based on the prompt information and pre-generated feature extraction parameters; Performing feature resampling based on the multimodal feature data and the auxiliary query information to obtain enhanced prompt information corresponding to the prompt information and enhanced features corresponding to the feature extraction parameters; Feature data to be matched with the multimodal feature data is generated according to the enhanced prompt information and the enhanced resampling feature.

3. The method according to claim 2, wherein: The performing feature resampling based on the multimodal feature data and the auxiliary query information to obtain enhanced prompt information corresponding to the prompt information and enhanced resampled features corresponding to the feature extraction parameters includes: Performing splicing processing on the multimodal feature data and the auxiliary query information, and obtaining a splicing result; Cross-attention processing is performed on the splicing result to obtain enhanced prompt information corresponding to the prompt information and enhanced resampling features corresponding to the feature extraction parameters.

4. The method according to claim 2, wherein: The generating, according to the enhanced prompt information and the enhanced resampling feature, feature data to be matched with the multimodal feature data includes: Performing cross attention processing on the enhanced prompt information and the enhanced resampled features to obtain a cross attention processing result; The cross-attention processing result is subjected to layout-aware feature extraction processing to generate feature data to be matched with the multimodal feature data.

5. The method according to claim 1, wherein The matching process is performed on the feature data to be matched and the multimodal feature data to obtain matching result feature data, including: Using an autoregressive model, matching processing is performed on the feature data to be matched and the multimodal feature data to obtain matching result feature data; or, Similarities are calculated between the feature data to be matched and a plurality of features included in the multimodal feature data, and the matching result feature data is determined based on the similarity calculation results corresponding to the plurality of features.

6. The method according to any one of claims 1 to 5, wherein The acquiring of multimodal feature data corresponding to the text image includes: Performing text recognition and image feature extraction on the text image to obtain corresponding text recognition results and image features, wherein the text recognition results include text features and layout features corresponding to the text features; The text recognition result and the image feature are subjected to feature fusion to obtain the multimodal feature data.

7. The method according to any one of claims 1 to 5, wherein The method is implemented by an information extraction model, which includes an encoder, a generator and a matcher; in: The encoder is used to perform text recognition and image feature extraction on the input text image, and output multimodal feature data corresponding to the text image based on the text recognition result and the extracted image features; The generator is used to generate feature data to be matched with the multimodal feature data based on prompt information related to the text image and the multimodal feature data; The matcher is used to perform matching processing on the feature data to be matched and the multimodal feature data to obtain matching result feature data, and extract a text result corresponding to the prompt information from the text corresponding to the text image based on the matching result feature data.

8. A method for training an information extraction model, wherein: The information extraction model includes an encoder, a generator, and a matcher, and the method includes: Obtaining text image samples used to train the information extraction model and text result true value annotation information corresponding to the text image samples, as well as prompt information corresponding to the text image samples; Inputting the text image sample into the encoder to obtain corresponding multimodal feature data, wherein the multimodal feature data is feature data generated based on at least text features and image features in the text image sample; Based on the multimodal feature data and the prompt information, generating, by the generator, feature data to be matched for matching with the multimodal feature data; performing matching processing on the feature data to be matched and the multimodal feature data by the matcher to obtain matching result feature data, and extracting a text prediction result corresponding to the prompt information from the text corresponding to the text image based on the matching result feature data; The information extraction model is trained based on the difference between the text prediction result and the true value annotation information of the text result.

9. The method according to claim 8, wherein Before obtaining the text image sample for training the information extraction model and the text result true value annotation information corresponding to the text image sample, the method further includes: determining a pre-training strategy to be used from a plurality of pre-training strategies preset for training the information extraction model; and determining prompt information corresponding to the text image sample based on the pre-training strategy; The matching process of the feature data to be matched and the multimodal feature data by the matcher includes: matching process of the feature data to be matched and the multimodal feature data by the matcher according to the determined pre-training strategy.

10. The method according to claim 9, wherein: The pre-training strategy includes at least one of the following: Filling matching strategy: a strategy for indicating, based on at least two given words in the prompt information, matching the text between the given words from the text of the text image sample; Direction search strategy: a strategy for matching, based on at least one given word in the prompt information, K nearest neighboring texts to the given word in a specified direction from the text of the text image sample, where K is greater than or equal to 1; Neighborhood strategy: a strategy for indicating that, based on at least one given word in the prompt information, M texts adjacent to the given word are matched from the text of the text image sample, where M is greater than or equal to 1.

11. The method according to claim 10, wherein: If the pre-training strategy is a filling matching strategy, then determining prompt information corresponding to the text image sample based on the pre-training strategy includes: Selecting at least two given words from the text corresponding to the text image sample, and generating prompt information based on the selected given words for indicating matching the given words from the text image sample and determining the text between the given words; The text result true value annotation information corresponding to the text image sample is determined in the following way: Based on the text in the text image sample corresponding to the text between the at least two selected given words, text result true value annotation information of the text image sample is generated.

12. The method according to claim 10, wherein: If the strategy is a direction search strategy, then determining prompt information corresponding to the text image sample based on the pre-training strategy includes: Determining a specified direction and a value of K, and selecting at least one given word from the text image sample; and generating prompt information indicating a matching of K nearest neighboring texts of the given word in the specified direction from the text of the text image sample based on the selected given word, the value of K, and the specified direction; The text result true value annotation information corresponding to the text image sample is generated in the following way: Based on K texts in the text of the text image sample that are adjacent to a given word in a specified direction, text result true value annotation information of the text image sample is generated.

13. The method according to claim 10, comprising: If the strategy is a proximity strategy, then determining prompt information corresponding to the text image sample based on the pre-training strategy includes: Determining a value of M, and selecting at least one given word from the text image sample, and generating prompt information for indicating matching M texts adjacent to the given word from the text of the text image sample based on the selected given word and the value of M; The text result true value annotation information corresponding to the text image sample is generated in the following way: The text result true value annotation information is generated based on M texts adjacent to the given word in the text of the text image sample.

14. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the method steps described in any one of claims 1 to 13 are implemented.

Images