Information extraction method, information extraction model training method, and computer program product

Abstract

Embodiments of the present application provide an information extraction method, a training method and program product of an information extraction model, and relate to the technical field of artificial intelligence. The information extraction method comprises: obtaining multi-modal feature data corresponding to a text image, wherein the multi-modal feature data is feature data generated at least according to text features and image features in the text image; generating to-be-matched feature data used for matching with the multi-modal feature data based on prompt information related to the text image and the multi-modal feature data; performing matching processing on the to-be-matched feature data and the multi-modal feature data to obtain matching result feature data, and extracting text results responding to the prompt information from text corresponding to the text image according to the matching result feature data. Through the embodiments of the present application, the accuracy of the text results obtained by extracting information from the text image can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an information extraction method, an information extraction model training method, and a computer program product. Background Technology

[0002] With the rapid development of big data, extracting valuable information from visually rich text images has become increasingly important. For example, extracting monetary amounts from invoice images or departure station information from train ticket images is crucial for data analysis and decision-making.

[0003] However, the visual complexity and content diversity of text images make accurate and efficient information extraction, especially for text images with a certain layout or structure, a significant challenge. One existing extraction method uses a discriminative model to discriminate text information generated by an OCR (Optical Character Recognition) engine. This method is sensitive to the order of the text generated by the OCR engine; the accuracy and reliability of information extraction from disordered or misordered text generated by OCR significantly decrease.

[0004] Therefore, there is an urgent need for a more effective way to extract information from text images. Summary of the Invention

[0005] In view of this, embodiments of this application provide an information extraction scheme to at least partially solve the above problems.

[0006] According to a first aspect of the embodiments of this application, an information extraction method is provided, comprising: acquiring multimodal feature data corresponding to a text image, wherein the multimodal feature data is feature data generated at least based on text features and image features in the text image; generating matching feature data for matching with the multimodal feature data based on prompt information related to the text image and the multimodal feature data; performing matching processing on the matching feature data and the multimodal feature data to obtain matching result feature data; and extracting a text result responding to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0007] According to a second aspect of the embodiments of this application, a training method for an information extraction model is provided, wherein the information extraction model includes an encoder, a generator, and a matcher, the method comprising: acquiring text image samples for training the information extraction model and text result truth annotation information corresponding to the text image samples, and prompt information corresponding to the text image samples; inputting the text image samples into the encoder to obtain corresponding multimodal feature data, wherein the multimodal feature data is feature data generated at least based on text features and image features in the text image samples; generating, through the generator, feature data to be matched for matching with the multimodal feature data based on the multimodal feature data and the prompt information; performing matching processing on the feature data to be matched and the multimodal feature data through the matcher to obtain matching result feature data, and extracting a text prediction result responding to the prompt information from the text corresponding to the text image based on the matching result feature data; and training the information extraction model based on the difference between the text prediction result and the text result truth annotation information.

[0008] According to a third aspect of the embodiments of this application, a computer program product is provided, including a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0009] According to a fourth aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first or second aspect.

[0010] According to a fifth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first or second aspect.

[0011] According to the information extraction scheme provided in this application, multimodal feature data of text images can be extracted. This data includes not only text features but also image features (which can effectively represent the layout, structure, or other information of the text image), making the information in the multimodal feature data richer and the semantic information it contains more complete and abundant. During information extraction, relevant prompts for the text image are also incorporated. These prompts effectively represent the text content to be extracted from the text image. Furthermore, to obtain text that better matches the prompts, matching feature data is generated based on the multimodal feature data and the prompts. Since the matching feature data is generated based on the prompts, matching the matching feature data and the multimodal feature data yields a more accurate text result corresponding to the prompts, thus ensuring the accuracy and interpretability of the output text result. In particular, for text images with a certain layout or structure, this method uses the feature data to be matched with the multimodal features corresponding to the text image to extract the required features more accurately from the multimodal features, thereby avoiding dependence on the order of the recognized text and improving the accuracy and reliability of information extraction from text images. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0013] Figure 1 A schematic diagram of an exemplary system to which the embodiments of this application are applicable;

[0014] Figure 2A A schematic diagram of the framework of the information extraction model applicable to the embodiments of this application;

[0015] Figure 2B This is a schematic diagram of the encoder structure of the information extraction model applicable to the embodiments of this application;

[0016] Figure 2C A schematic diagram of the structure of the generator for the information extraction model applicable to the embodiments of this application;

[0017] Figure 2D A schematic diagram of the structure of the matcher for the information extraction model applicable to the embodiments of this application;

[0018] Figure 2E A schematic diagram illustrating the coordination of the encoder, generator, and matcher in the information extraction model applicable to embodiments of this application;

[0019] Figure 3 This is a flowchart illustrating the steps of an information extraction method according to Embodiment 1 of this application;

[0020] Figure 4 This is a flowchart illustrating the steps of an information extraction method according to Embodiment 2 of this application;

[0021] Figure 5 This is a schematic diagram of a prompt-aware resampler for an information extraction model according to Embodiment 2 of this application;

[0022] Figure 6A This is a schematic diagram of an information extraction model according to Embodiment 2 of this application, in which the matcher adopts an autoregressive mode;

[0023] Figure 6B This is a schematic diagram comparing a conventional matching method according to an embodiment of this application with a matching method according to an embodiment of this application;

[0024] Figure 7 This is a flowchart illustrating the steps of an information extraction model training method according to Embodiment 3 of this application;

[0025] Figure 8 This is a schematic diagram of a text image sample according to Embodiment 3 of this application;

[0026] Figure 9 This is a schematic diagram of an information extraction device according to Embodiment 4 of this application;

[0027] Figure 10 This is a schematic diagram of an information extraction model training device according to Embodiment 5 of this application;

[0028] Figure 11 This is a schematic diagram of the structure of an electronic device according to Embodiment Six of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0030] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0031] Figure 1 An exemplary system applicable to embodiments of this application is shown. For example... Figure 1As shown, the system may include a cloud server 102, a communication network 104, and / or one or more user devices 106. Figure 1 The example in the text shows multiple user devices.

[0032] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can extract information from text images. In some embodiments, the cloud server 102 can obtain multimodal feature data corresponding to the text image, which simultaneously contains text features and image features of the text image; then, based on the prompt information corresponding to the text image and the multimodal feature data, it can generate feature data to be matched; furthermore, it matches the feature data to be matched with the multimodal feature data to obtain a text result responding to the prompt information. In some embodiments, the cloud server 102 can be configured with an information extraction model, through which the above-mentioned information extraction is performed. As another example, the cloud server 102 can receive and respond to the information extraction request from the user device 106 to generate the corresponding text result in the above manner and return it to the user device 106.

[0033] In some embodiments, the communication network 104 may be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 may include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN) and / or any other suitable communication network.

[0034] User equipment 106 can be connected to communication network 104 via one or more communication links (e.g., communication link 112), and communication network 104 can be linked to cloud server 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between user equipment 106 and cloud server 102, such as a network link, dial-up link, wireless link, hardwired link, any other suitable communication link, or any suitable combination of such links.

[0035] In some embodiments, user device 106 can be any suitable type of device capable of interacting with the user. For example, in some embodiments, user device 106 can generate an information retrieval request based on prompts corresponding to text images input by the user and send it to cloud server 102; and receive text results returned by cloud server 102 and display them to the user. In some embodiments, user device 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, media players, vehicle entertainment systems, other edge computing devices, and / or any other suitable type of user device.

[0036] It should be noted that although this example uses cloud server 102 as an example, it is understood that cloud server 102 can be replaced by a local device or other server that can provide the corresponding function, and there are no restrictions on this.

[0037] Based on the above system, this application provides an information extraction method. In one feasible approach, this information extraction method can be implemented using an information extraction model. This information extraction model can be deployed on the aforementioned cloud server 102, or on other suitable deployment devices.

[0038] To facilitate understanding and clear explanation, before describing the implementation process of the information extraction method, we will combine... Figures 2A to 2E A brief explanation of the structure of the information extraction model:

[0039] like Figures 2A to 2E As shown, the information extraction model in this embodiment includes an encoder 202, a generator 204, and a matcher 206. The encoder 202 receives text images (such as restaurant bills, invoices, tickets, etc., which contain not only text but also other visual information, such as structure, layout, etc.) and processes the text images, such as text recognition and feature extraction, to output multimodal feature data corresponding to the text image (e.g., based on the text recognition results and extracted image features). Figure 2B The “Multi-modal Embeddings” in this context can be broadly termed “Source Vectors”. Figure 2E (Identified by "Xm"); Generator 204 receives multimodal feature data and text image-related prompts (such as...) Figure 2C The "Prompt" in the text could be something like "What dishes are in the text image?", and based on the prompt and multimodal feature data, it generates matching feature data (such as...) to be matched against the multimodal feature data. Figure 2D and Figure 2EThe "Matcher Vectors" shown in the diagram. Matcher 206 receives the feature data to be matched and the multimodal feature data, performs matching processing on the two, extracts the feature data of the matching result that matches the prompt information from the multimodal feature data, and outputs the text result of the text corresponding to the prompt information in the text image (e.g., the text image). Figure 2D and Figure 2E (The text shown above, etc.). It should be noted that the above... Figures 2B-2E The specific structure of the text is only briefly explained here, and will be further explained in detail below.

[0040] The following describes the information extraction method of this application through multiple embodiments, in conjunction with the information extraction model.

[0041] Example 1

[0042] Reference Figure 3 The diagram illustrates a step-by-step flowchart of an information extraction method according to Embodiment 1 of this application. The method includes:

[0043] Step S302: Obtain the multimodal feature data corresponding to the text image.

[0044] The multimodal feature data is feature data generated based at least on text features and image features in the text image.

[0045] In this embodiment, a text image refers to an image containing text information. It can be a plain text image, a text image with a certain structure, layout, and arrangement, or an image containing both text and other image information. A text image possesses both textual and image features. In this embodiment, multimodal feature data is generated based on at least these two features to better describe the true form of the text image.

[0046] In one feasible approach, the multimodal feature data corresponding to the text image can be obtained through methods such as... Figure 2A and Figure 2B The multimodal feature data is obtained by encoder 202 in the image, which integrates information such as text, text order, text layout, and visual information (image features) from the text image. Text, text order, and text layout can be obtained through OCR engine recognition, while visual information can be obtained through methods such as image encoders (e.g., image encoders). Figure 2B The image features are obtained by processing modules, units, or model structures such as the "Image Encoder" shown in the figure, which have image feature extraction functions.

[0047] Step S304: Based on the prompt information related to the text image and the multimodal feature data, generate the feature data to be matched with the multimodal feature data.

[0048] Text-image related prompts can be user-inputted prompts (also known as prompt words), which effectively represent the information the user intends to extract from the text image. However, they are not limited to this; the prompts can also be one or more pre-set prompts. For example, the prompts could be: "The name of the menu is," "The amount of the menu is," etc.

[0049] In one feasible approach, the prompt information and multimodal feature data can be input into, for example... Figure 2A or Figure 2C In the generator 204 shown, the data is processed by the generator 204 to output the feature data to be matched. Its specific implementation will be described in detail below.

[0050] Step S306: Perform matching processing on the feature data to be matched and the multimodal feature data to obtain the matching result feature data, and extract the text result responding to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0051] Since the feature data to be matched is generated based on the prompt information and multimodal feature data, and carries information that can effectively indicate the required features, matching the feature data to be matched with the multimodal feature data corresponding to the text image can more accurately match the final text result. During matching, one feasible approach is to calculate the similarity between the feature data to be matched and the multimodal feature data. This process can also be understood as using the feature data to be matched to perform entity labeling on the multimodal feature data to find entities related to the prompt information within the multimodal feature data. The entity labeling results can be used as the matching result feature data. Based on the matching result feature data, the text result responding to the prompt information is extracted from the text corresponding to the text image. This ensures that the output content is extracted from the text recognized from the text image, resulting in more accurate extraction and resolving the "illusion" problem (i.e., text appearing that is not in the text image).

[0052] In another feasible approach, an autoregressive method can be used to match the data corresponding to each token in the multimodal feature data with the data corresponding to the token in the feature data to be matched, thereby predicting the output corresponding to the token. This also ensures that the text related to the prompt information can be extracted from the text of the text image, which can also solve the "illusion" problem and achieve the effect of accurate output text results.

[0053] In one example, it can be done as follows: Figure 2A or Figure 2D The matcher 206 shown performs matching processing on the feature data to be matched and the multimodal feature data. Its specific implementation will be detailed below.

[0054] In this way, according to the information extraction scheme provided in the embodiments of this application, multimodal feature data of text images can be extracted. This data includes not only text features but also image features (which can effectively represent the layout, structure, or other information of the text image), making the information in the multimodal feature data richer and the semantic information it contains more complete and abundant. During information extraction, relevant prompts for the text image are also incorporated. These prompts effectively represent the text content to be extracted from the text image. Furthermore, in order to obtain text that better matches the prompts, matching feature data is generated based on the multimodal feature data and the prompts. Because the matching feature data is generated based on the prompts, matching the matching feature data and the multimodal feature data yields a more accurate text result corresponding to the prompts, thereby ensuring the accuracy and interpretability of the output text result. In particular, for text images with a certain layout or structure, this method uses the feature data to be matched with the multimodal features corresponding to the text image to extract the required features more accurately from the multimodal features, thereby avoiding dependence on the order of the recognized text and improving the accuracy and reliability of information extraction from text images.

[0055] Example 2

[0056] Reference Figure 4 The diagram illustrates a step-by-step flowchart of an information extraction method according to Embodiment 2 of this application. The method includes:

[0057] Step S402: Obtain the multimodal feature data corresponding to the text image.

[0058] In one feasible approach, step S402 may include the following sub-steps:

[0059] Sub-step S4021: Perform text recognition and image feature extraction on the text image to obtain the corresponding text recognition results and image features.

[0060] For example, a text image is input into an OCR engine to obtain text recognition results. These results include text features (such as...). Figure 2E The “Texts” shown in the figure) and the layout features corresponding to the text features (such as Figure 2E (See “LayoutEmbeddings”).

[0061] To preserve as much visual information as possible from the text image, the text image is also input into an image encoder to obtain the image features output by the image encoder (such as...). Figure 2EThe features output after being encoded by the image encoder ("Image Encoder").

[0062] Sub-step S4022: Perform feature fusion on the text recognition results and image features to obtain multimodal feature data.

[0063] Feature fusion can enrich and enhance two parts of features (i.e., features of text recognition results and image features), thereby enabling the obtained multimodal feature data to more effectively represent text images.

[0064] In one feasible approach, this step can be achieved by, for example... Figure 2A or Figure 2B The encoder 202 shown is an implementation. Exemplarily, the encoder 202 may include an OCR engine (…). Figure 2B The diagram in the image shows "OCR" and an image encoder. Figure 2B The diagram shows the "Image Encoder" and the pre-trained layout encoder. Figure 2B (Illustrated as "Pretrained LayoutLMEncoder").

[0065] When this step is implemented through encoder 202, after the text image is input into encoder 202, on the one hand, encoder 202 will perform text recognition processing on the text image through its OCR engine to obtain the corresponding text features. Figure 2B (Illustrated as "Texts") and the layout features corresponding to the text features ( Figure 2B (This is illustrated as "Layout Embeddings"); on the other hand, encoder 202 extracts image features from the text image using its image encoder to obtain the corresponding image features. In one feasible approach, since both involve visual information, the image features can be concatenated into the layout features.

[0066] Furthermore, text features, layout features, and image features can be input into the layout encoder and fused into multimodal feature data (such as...). Figure 2B As shown in the “Multi-modal Embeddings” diagram. Multi-modal feature data can be used to form feature vectors (such as...). Figure 2E (As shown in the “Source Vectors” example). In this embodiment, the dimension of the feature vector is N. Word *d. Where N Word This represents the number of tokens corresponding to the multimodal feature data, and d is the encoding length.

[0067] The aforementioned OCR engine can be any engine capable of optical character recognition, and the image encoder can be a model capable of extracting image features, including but not limited to convolutional neural network models, ViT models, etc., and this application embodiment does not impose any limitations on this. The pre-trained layout encoder can also be replaced by networks such as DocFormer and FormNet.

[0068] Step S404: Based on the prompt information related to the text image and the multimodal feature data, generate the feature data to be matched with the multimodal feature data.

[0069] In one feasible approach, step S404 can be implemented through the following sub-steps:

[0070] Sub-step S4041: Obtain the prompt information corresponding to the text image, and generate auxiliary query information based on the prompt information and pre-generated feature extraction parameters.

[0071] In this context, feature extraction parameters can be understood as parameters that, combined with prompts, further clarify the information that needs to be obtained from the multimodal feature data. If a model structure is used, these feature extraction parameters can be implemented as model parameters learned through training. Then, the auxiliary query information generated based on the prompts and feature extraction parameters can be used with the multimodal feature data for calculations such as attention calculations to further extract features and enhance its own information.

[0072] In one feasible approach, to enhance the perception of prompts, methods such as... Figure 2A or Figure 2C The generator 204 shown implements step S404. The generator 204 may include a cue-aware resampler (such as...) Figure 2C The “Prompt-aware Resamplar” shown) and the generation unit (such as Figure 2C (As shown in the dashed box inside). One exemplary cue-aware resampler can be implemented using a model structure based on a cross-attention mechanism, such as... Figure 5 As shown in the diagram. The generation unit includes at least one interconnected processing combination, each processing combination including a cross-attention layer (such as...). Figure 2C The diagram shows the "Cross-attention Layer" and the layout-aware block. Figure 2C (Illustrated as "Layout-AwareLanguage Block"). For example, this generation unit can be obtained by pre-training based on a document pre-training model of Transformer, but is not limited thereto.

[0073] When generator 204 structure is used, this sub-step S4041 can be executed by its cue-aware resampler. For example... Figure 5 As shown in the figure, the pre-generated feature extraction parameters are represented by X. q (These are the learnable model parameters "LearnableQueries", which can be used for feature extraction), and the prompt information is represented as X. p (Meaning "Prompt Tokens"). During the training phase, the prompt message X... p It can be preset; during the reasoning phase, the prompts can be input by the user or generated based on the user's input.

[0074] Furthermore, the feature extraction parameter X q And prompt message X p Combined into auxiliary query information, it is in Figure 5 This is represented as Q = [X] p X q In the cross-attention calculation to be performed, it can represent the query vector Q.

[0075] Sub-step S4042: Based on multimodal feature data and auxiliary query information, perform feature resampling to obtain enhanced prompt information corresponding to the prompt information and enhanced resampled features corresponding to the feature extraction parameters.

[0076] For example, feature resampling can be achieved through cross-attention computation. For instance, multimodal feature data can be concatenated with auxiliary query information to obtain the concatenation result; cross-attention processing can then be applied to the concatenation result to obtain enhanced prompts corresponding to the prompt information and enhanced resampled features corresponding to the feature extraction parameters.

[0077] When adopting such Figure 5 When the cue-aware resampling device shown implements this sub-step, the multimodal feature data is in Figure 5 The middle is represented as X m (“Multi-modal Embeddings”). Auxiliary query information Q and multimodal feature data X. m The splicing process results in [X] m X p X q In the cross-attention calculation to be performed, [X] m X p X q It can simultaneously represent the key vector K and the value vector V.

[0078] Furthermore, based on the aforementioned Q, K, and V, the cross-attention layer in the cue-aware resampler ( Figure 5The diagram illustrates "Cross-Attention" to perform cross-attention calculations and obtain X′. p 、X′ q , respectively represent the enhanced prompt information corresponding to the prompt information and the enhanced resampling features corresponding to the feature extraction parameters.

[0079] As can be seen, by using cross-attention processing, the cue information X can be realized. p Feature extraction parameter X q and multimodal feature data X m The information is mutually resampled to enhance the corresponding information, resulting in an output ("Enhanced Prompts & Resampled Embeddings"), which includes the enhanced prompt information X′ corresponding to the prompt information. p And the enhanced features X′ corresponding to the feature extraction parameters. q .

[0080] Sub-step S4043: Generate matching feature data based on the enhanced prompt information and enhanced resampling features, which is used to match the multimodal feature data.

[0081] In one feasible approach, sub-step S4043 can be implemented as follows: performing cross-attention processing on the enhanced cue information and the enhanced resampling features to obtain the cross-attention processing result; performing layout-aware feature extraction processing on the cross-attention processing result to generate feature data to be matched with the multimodal feature data.

[0082] For example, refer again Figure 2C Taking the aforementioned generation unit structure as an example, the enhanced prompt information and enhanced features can be input into the cross-attention layer of the first processing combination, allowing them to undergo cross-attention processing to obtain the cross-attention processing result. This cross-attention processing result is then input into the layout-aware block of the first processing combination, where layout-aware feature extraction is performed to learn the layout-related information of the text contained in the text image. The layout-aware block of the first processing combination inputs the extraction result into the cross-attention layer of the second processing combination. This cross-attention layer performs cross-attention processing on the extraction result output by the previous processing combination and the enhanced features, and then inputs the cross-attention processing result into the layout-aware block corresponding to its own processing combination for processing to obtain the extraction result corresponding to that processing combination. This process is repeated until multiple processing combinations are completed, and the output result is the feature data to be matched (“matcher vectors”). In this embodiment, the dimension of the feature data to be matched is d*(N). BO +1). Where d is the decoding length, N BIThis indicates the number of entity label categories. Including the entity label categories of the entities to be extracted in the feature data to be matched facilitates subsequent rapid lexical classification and improves the efficiency of feature matching.

[0083] Step S406: Perform matching processing on the feature data to be matched and the multimodal feature data to obtain the matching result feature data, and extract the text result responding to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0084] The matching process between the feature data to be matched and the multimodal feature data can be implemented in an appropriate manner. However, to improve matching efficiency, this application provides two implementation methods, referred to as the original mode and the autoregressive mode. The original mode calculates the similarity between multiple features contained in the feature data to be matched and the multimodal feature data, and determines the matching result feature data based on the similarity results (such as a similarity matrix) corresponding to each feature. The autoregressive mode uses an autoregressive approach to match the feature data to be matched and the multimodal feature data to obtain the matching result feature data. The autoregressive mode is a method that establishes a multi-layered recursive structure, progressively predicts the next value based on the input data, and iteratively generates the entire sequence. Specifically, in this embodiment, the autoregressive mode achieves entity extraction of text from text images by sequentially calculating the similarity matrix between the feature data to be matched and the multimodal feature data.

[0085] For example, in the original pattern, the similarity is calculated by matrix multiplication of the feature data to be matched and the multimodal feature data. The dimension of the similarity calculation result is N. Word *(N BI +1), and the similarity calculation result is the feature data of the matching result.

[0086] Because the feature data to be matched carries the category of the entity label of the entity to be extracted, it can be quickly classified into entities based on the corresponding words (also known as tokens) in the multimodal feature data to determine the words with a high probability of matching the prompt information, and then the corresponding text can be extracted as the text result.

[0087] For example, when using this method, Figure 2EThe text image shown contains two dishes: "BIHUN GORENG SEAFOOD" and "KWETIAW SEAFOOD SIRAM", each with a quantity of 1 and unit prices of 40.456 and 42.274 respectively. If the prompt message is "menu.nm is", which means asking for the names of the dishes, the output text result will be: "BIHUN GORENG SEAFOOD" and "KWETIAW SEAFOOD SIRAM".

[0088] Matching of an autoregressive pattern, such as Figure 6A As shown, Figure 6A In the middle, encoder 202 ( Figure 6A The diagram shows the "Source Encoder" outputting multimodal feature data. Figure 6A (Illustrated as "Source Vector"), Generator 204 ( Figure 6A The diagram shows the "Target Generator" outputting the corresponding feature data to be matched. Figure 6A (Illustrated as "d" in the diagram). Matcher 206 performs autoregressive pattern matching on a per-word basis for multimodal feature data and feature data to be matched, through N recursions ( Figure 6A The diagram shows 5 steps (step 0-step 4), predicting the results in the multimodal feature data that match the feature data to be matched, i.e., the matching result feature data.

[0089] In practical applications, the diverse text layouts within text images make it difficult to easily determine the text order. However, by using an autoregressive approach, a sequence matcher vector can be generated that matches the entire entity in an autoregressive manner, such as... Figure 6A The symbols shown include (TA), BIHUN, GORENG…[SEP] (used to separate repeating entities), and [EOS] (used to mark the end of a sequence). If the text contains repeating entities, the matcher 206 can match and generate a [SEP] token to separate the repeating entities. Furthermore, the matcher 206 can stop processing when predicting the sequence end marker [EOS]. Using symbols such as [SEP] and [EOS] makes text sequences easier to decode.

[0090] In one feasible approach, to achieve context-aware lexical classification for information extraction, matcher 206 can be implemented using a decoder to generate weights for sequence classification. By utilizing these weights, classification weights conditional on each individual input can be generated, thus distinguishing it from traditional sequence tokens. For ease of distinction, these weights can be referred to as conditional parameter weights. Specifically, generator 204 generates a feature data to be matched based on each cue and the output of the resampler. Then, matcher 206 includes two sub-matchers generated by linear projection. One sub-matcher is used to match the starting token for the feature data to be matched, while the other sub-matcher is used to match tokens within entities for the feature data to be matched. In addition, an extra sub-matcher is introduced to match non-entity tokens.

[0091] For ease of explanation, the following is combined with Figure 6B The differences between the matching methods described in the embodiments of this application and the matching methods in related technologies are compared and explained. For example... Figure 6B As shown, taking the matching of "GORENG" in multimodal feature data as an example, in the matching methods of related technologies, the weights obtained through learning are derived from the rule classification layer from sequence markers. Figure 6B The "Learnable weights" shown in the diagram are the same for all inputs. However, in the matching method of this application embodiment, the classification weights are dynamically calculated using a function that simultaneously considers the input text image and prompt information. Figure 6B The diagram shows "Dynamic weights," which can effectively improve the capabilities of the classification layer.

[0092] After determining the matching result feature data through matcher 206 classification matching, the corresponding text can be extracted from the text image based on the matching result feature data as the text result in response to the prompt information. This serialized autoregressive mode can effectively solve the problem of difficulty in easily obtaining the reading order due to huge changes in document layout, ensuring the correct order of the output text results.

[0093] In summary, when implementing the above method through the information extraction model, encoder 202 is used to perform text recognition and image feature extraction on the input text image, and outputs multimodal feature data corresponding to the text image based on the text recognition results and the extracted image features; generator 204 is used to generate matching feature data based on the prompt information related to the text image and the multimodal feature data; matcher 206 is used to perform matching processing on the matching feature data and the multimodal feature data to obtain matching result feature data, and extract the text result responding to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0094] This embodiment extracts multimodal feature data from text images, including not only text features but also image features (effectively representing information such as the text image's layout, structure, etc.). This enriches the information in the multimodal feature data and makes the semantic information it contains more complete and abundant. During information extraction, relevant prompts for the text image are also incorporated. These prompts effectively represent the text content to be extracted from the text image. Furthermore, to obtain text that better matches the prompts, matching feature data is generated based on the multimodal feature data and the prompts. Because the matching feature data is generated from the prompts, matching the matching feature data with the multimodal feature data yields a more accurate text result corresponding to the prompts, thus ensuring the accuracy and interpretability of the output text result. In particular, for text images with a certain layout or structure, this method uses the feature data to be matched with the multimodal features corresponding to the text image to extract the required features more accurately from the multimodal features, thereby avoiding dependence on the order of the recognized text and improving the accuracy and reliability of information extraction from text images.

[0095] Example 3

[0096] Reference Figure 7 The diagram illustrates a step-by-step flowchart of a training method for an information extraction model according to Embodiment 3 of this application. As previously described, the information extraction model of this embodiment includes three parts: an encoder, a generator, and a matcher.

[0097] The method includes:

[0098] Step S700: Determine the pre-training strategy to be used from multiple pre-training strategies for training the information extraction model; and, based on the pre-training strategy, determine the prompt information corresponding to the text image sample.

[0099] It should be noted that this step is optional. For example, it can be performed when there is more than one pre-training strategy. However, if a default pre-training strategy is set, this step is not required.

[0100] In this embodiment, different pre-training strategies and corresponding pre-training prompts can be configured to enable the information extraction model to learn different capabilities. The pre-training strategies include at least one of the following: Match to Fill (MTF), Search One Direction (SOD), and Search All Direction (SAD).

[0101] The following are explanations:

[0102] 1) Fill-in matching strategy: This strategy is used to indicate the strategy of matching the text between the given words in the text image sample based on at least two given words in the prompt information.

[0103] It can improve the model's ability to match content from contextual text and learn the reading order.

[0104] When the pre-training strategy is a fill-in matching strategy, the prompt information corresponding to the text image sample can be determined based on the pre-training strategy as follows: select at least two given words from the text corresponding to the text image sample, generate prompt information based on the selected given words to indicate that the given words are matched from the text image sample, and determine the text between the given words.

[0105] Furthermore, in this embodiment, the information extraction model is trained using supervised training. Therefore, to achieve efficient and accurate training of the information extraction model, after determining the pre-training strategy and prompts, it is also necessary to label the training samples used to train the information extraction model. That is, to perform text result truth value labeling on the text image samples. Then, under the fill-in matching strategy, the text result truth value labeling information corresponding to the text image sample can be determined in the following way: based on the text in the text of the text image sample, corresponding to the text between at least two selected given words, the text result truth value labeling information of the text image sample is generated.

[0106] For example, text image samples such as Figure 8 As shown, you can choose any two words as given words. For example, if the given words are "Color" and "Cathode", the corresponding prompt can be "fill in between 'Color' and 'Cathode'". The corresponding text result truth value annotation information is "Band" and "Denotes".

[0107] 2) Directional search strategy: This strategy is used to indicate the strategy of matching the K nearest neighbor texts in a specified direction from the text of a text image sample based on at least one given word in the prompt information.

[0108] Where K is greater than or equal to 1. This strategy can improve the model's spatial awareness learning ability.

[0109] Under the direction search strategy, based on the pre-training strategy, determining the prompt information corresponding to the text image sample can be achieved by: determining the specified direction and the value of K, selecting at least one given word from the text image sample, and generating prompt information based on the selected given word, the value of K and the specified direction to indicate the matching of the K nearest neighbor texts of the given word in the specified direction from the text of the text image sample.

[0110] Furthermore, the text result truth value annotation information corresponding to the text image sample is generated in the following way: based on the text of the text image sample, the text result truth value annotation information of the text image sample is generated from the K texts adjacent to the given word in a specified direction.

[0111] For example, taking the aforementioned text image sample as an example, we can randomly select a word as the given word. If the given word is "Leads", the value of K is 2, and the direction is to the right, the corresponding prompt information can be "find two instances to right of 'Leads'" (match the two words adjacent to the right of "Leads"). The corresponding text result truth value annotation information is "Solderable" and "per".

[0112] 3) Proximity strategy: A strategy used to indicate how to match M texts adjacent to at least one given word in the prompt information from the text of a text image sample.

[0113] Where M is greater than or equal to 1. This strategy can improve the information extraction model's ability to utilize the spatial layout of text.

[0114] Under this proximity strategy, based on the pre-training strategy, determining the prompt information corresponding to the text image sample can be achieved by: determining the value of M, selecting at least one given word from the text image sample, and generating prompt information based on the selected given word and the value of M to indicate that M texts adjacent to the given word are matched from the text of the text image sample.

[0115] Furthermore, the text result truth value annotation information corresponding to the text image sample is generated in the following way: based on the M texts adjacent to the given word in the text image sample, the text result truth value annotation information is generated.

[0116] For example, if the given word is "grams" and M is 3, the corresponding prompt could be "Find three nearest instances of 'grams'" (match the three words closest to "grams"). The corresponding text result truth value annotations would be "0.3", "(approx)", and "Band".

[0117] Step S702: Obtain text image samples used to train the information extraction model, the text result truth value annotation information corresponding to the text image samples, and the prompt information corresponding to the text image samples.

[0118] The text image samples can be obtained from a preset training set, while the text result truth value annotation information and prompt information can be obtained in the manner described in step S700.

[0119] Step S704: Input the text image sample into the encoder to obtain the corresponding multimodal feature data.

[0120] The multimodal feature data refers to feature data generated based at least on the text features and image features in the text image samples. The process of obtaining multimodal feature data through the encoder can be referred to the description of obtaining multimodal feature data based on text images in the foregoing embodiments, and will not be repeated here.

[0121] Step S706: Based on the multimodal feature data and the prompt information, generate the feature data to be matched with the multimodal feature data through the generator.

[0122] The process of generating the feature data to be matched using a generator can be referred to the relevant description in the foregoing embodiments, and will not be repeated here.

[0123] Step S708: The matching algorithm performs matching processing on the feature data to be matched and the multimodal feature data to obtain the matching result feature data, and based on the matching result feature data, the text prediction result responding to the prompt information is extracted from the text corresponding to the text image.

[0124] In this process, with a pre-training strategy selected, the matching process can be performed by a matcher according to the determined pre-training strategy, matching the feature data and multimodal feature data to be matched. For example, the matcher can use the original mode to perform matching according to the filling method or matching direction indicated by the pre-training strategy, or it can use an autoregressive mode to perform matching according to the filling method or matching direction indicated by the pre-training strategy. The specific process can be referred to the relevant description in the foregoing embodiments, and will not be repeated here.

[0125] Step S710: Train the information extraction model based on the difference between the text prediction result and the text result true value annotation information.

[0126] For example, the difference can be calculated by comparing the output text prediction result with the ground truth annotation information of the text result, and then the information extraction model can be trained based on the difference. This difference can be obtained by calculating a preset loss function. In this embodiment, the specific implementation of the loss function is not limited. Exemplarily, the loss function may include, but is not limited to, the cross-entropy loss function.

[0127] By using different pre-training strategies and corresponding prompts and text result truth annotations to train the information extraction model, the information extraction model can acquire corresponding capabilities. For example, using the pre-training strategy in this embodiment can improve the information extraction model's ability to perceive spatial context information, thereby making the text prediction results more accurate.

[0128] Through this embodiment, based on the pre-training strategy, the ability of the trained information extraction model to perceive spatial context information is improved, thereby enabling the trained information extraction model to extract information from text images with higher accuracy and reliability.

[0129] Example 4

[0130] Reference Figure 9 The diagram shows a structural block diagram of an information extraction device according to Embodiment 4 of this application.

[0131] The information extraction device of this embodiment includes: an acquisition module 902, used to acquire multimodal feature data corresponding to a text image, wherein the multimodal feature data is feature data generated based at least on text features and image features in the text image; a feature generation module 904, used to generate matching feature data for matching with the multimodal feature data based on prompt information related to the text image and the multimodal feature data; and a matching extraction module 906, used to perform matching processing on the matching feature data and the multimodal feature data to obtain matching result feature data, and extract the text result responding to the prompt information from the text corresponding to the text image based on the matching result feature data.

[0132] Optionally, the feature generation module 904 is used to obtain prompt information corresponding to the text image, generate auxiliary query information based on the prompt information and pre-generated feature extraction parameters, perform feature resampling based on multimodal feature data and auxiliary query information to obtain enhanced prompt information corresponding to the prompt information and enhanced resampled features corresponding to the feature extraction parameters, and generate matching feature data to be matched with multimodal feature data based on the enhanced prompt information and enhanced resampled features.

[0133] Optionally, when the feature generation module 904 performs feature resampling based on the multimodal feature data and the auxiliary query information to obtain the enhanced prompt information corresponding to the prompt information and the enhanced resampled features corresponding to the feature extraction parameters, the multimodal feature data and the auxiliary query information are concatenated and the concatenation result is obtained; cross-attention processing is performed on the concatenation result to obtain the enhanced prompt information corresponding to the prompt information and the enhanced resampled features corresponding to the feature extraction parameters.

[0134] Optionally, when the feature generation module 904 generates the feature data to be matched with the multimodal feature data based on the enhanced cue information and the enhanced resampled features, it performs cross-attention processing on the enhanced cue information and the enhanced resampled features to obtain the cross-attention processing result; and performs layout-aware feature extraction processing on the cross-attention processing result to generate the feature data to be matched with the multimodal feature data.

[0135] Optionally, when the matching extraction module 906 performs matching processing on the feature data to be matched and the multimodal feature data to obtain the matching result feature data, it may: use an autoregressive mode to perform matching processing on the feature data to be matched and the multimodal feature data to obtain the matching result feature data; or, calculate the similarity between the feature data to be matched and the multiple features contained in the multimodal feature data respectively, and determine the matching result feature data based on the similarity calculation results corresponding to the multiple features respectively.

[0136] Optionally, when acquiring the multimodal feature data corresponding to the text image, the acquisition module 902 performs text recognition and image feature extraction on the text image to obtain the corresponding text recognition result and image features, wherein the text recognition result contains text features and layout features corresponding to the text features; and performs feature fusion on the text recognition result and image features to obtain multimodal feature data.

[0137] Optionally, the information extraction device in this embodiment extracts information through an information extraction model, which includes an encoder, a generator, and a matcher. The encoder performs text recognition and image feature extraction on the input text image, and outputs multimodal feature data corresponding to the text image based on the text recognition results and the extracted image features (implementing the function of acquisition model 902). The generator generates matching feature data based on prompt information related to the text image and the multimodal feature data (implementing the function of feature generation model 904). The matcher performs matching processing on the matching feature data and the multimodal feature data to obtain matching result feature data, and extracts the text result responding to the prompt information from the text corresponding to the text image based on the matching result feature data (implementing the function of matching extraction module 906).

[0138] The information extraction device in this embodiment is used to implement the information extraction method corresponding to the aforementioned method embodiment, and has the beneficial effects corresponding to the information extraction method. Its specific implementation can be referred to the description in the aforementioned method embodiment, and will not be repeated here.

[0139] Example 5

[0140] Reference Figure 10 The diagram shows a structural block diagram of a training device for an information extraction model according to Embodiment 5 of this application.

[0141] The information extraction model includes an encoder, a generator, and a matcher. The training device includes: an information acquisition module 1002, used to acquire text image samples for training the information extraction model, text result ground truth annotations corresponding to the text image samples, and prompt information corresponding to the text image samples; an encoding module 1004, used to input the text image samples into the encoder to obtain corresponding multimodal feature data, wherein the multimodal feature data is feature data generated at least based on text features and image features in the text image samples; a data generation module 1006, used to generate, based on the multimodal feature data and prompt information, feature data to be matched against the multimodal feature data; a data matching module 1008, used to perform matching processing on the feature data to be matched and the multimodal feature data through the matcher to obtain matching result feature data, and extract text prediction results responding to the prompt information from the text corresponding to the text image based on the matching result feature data; and a training module 1010, used to train the information extraction model based on the difference between the text prediction results and the text result ground truth annotations.

[0142] Optionally, the training device further includes: a determination module 1000, configured to determine a pre-training strategy to be used from a set of pre-training strategies for training the information extraction model before the information acquisition module 1002 acquires the text image samples used for training the information extraction model and the text result truth value annotation information corresponding to the text image samples; and, based on the pre-training strategy, determine the prompt information corresponding to the text image samples; then, the data matching module 1008 performs matching processing on the feature data to be matched and the multimodal feature data through the matcher, including: performing matching processing on the feature data to be matched and the multimodal feature data according to the determined pre-training strategy through the matcher.

[0143] Optionally, the pre-training strategy includes at least one of the following: a fill-in matching strategy: a strategy for matching text between at least two given words in the prompt information from the text of a text image sample; a direction search strategy: a strategy for matching K nearest neighbor texts in a specified direction to at least one given word in the prompt information from the text of a text image sample, where K is greater than or equal to 1; a proximity strategy: a strategy for matching M adjacent texts to at least one given word in the prompt information from the text of a text image sample, where M is greater than or equal to 1.

[0144] Optionally, if the pre-training strategy is a fill-in matching strategy, then the determination module 1000 determines the prompt information corresponding to the text image sample based on the pre-training strategy as follows: select at least two given words from the text corresponding to the text image sample, generate prompt information based on the selected given words to indicate that the given words are matched from the text image sample and to determine the text between the given words; the text result truth labeling information corresponding to the text image sample is determined in the following way: generate the text result truth labeling information of the text image sample based on the text in the text of the text image sample that corresponds to the text between the at least two selected given words.

[0145] Optionally, if the strategy is a direction search strategy, the determination module 1000 determines the prompt information corresponding to the text image sample based on the pre-trained strategy as follows: determine the specified direction and the value of K, and select at least one given word from the text image sample. Based on the selected given word, the value of K, and the specified direction, generate prompt information to indicate that the K nearest neighbor texts of the given word in the specified direction are matched from the text of the text image sample. The text result truth value annotation information corresponding to the text image sample is generated in the following way: based on the K texts adjacent to the given word in the specified direction in the text of the text image sample, generate the text result truth value annotation information of the text image sample.

[0146] Optionally, if the strategy is a proximity strategy, the determination module 1000 determines the prompt information corresponding to the text image sample based on the pre-trained strategy as follows: determine the value of M, select at least one given word from the text image sample, and generate prompt information based on the selected given word and the value of M to indicate that M texts adjacent to the given word are matched from the text of the text image sample; the text result truth value annotation information corresponding to the text image sample is generated in the following way: based on the M texts adjacent to the given word in the text of the text image sample, generate the text result truth value annotation information.

[0147] The information extraction model training device in this embodiment is used to implement the training method of the information extraction model corresponding to the aforementioned method embodiment, and has the beneficial effects corresponding to the training method. Its specific implementation can be referred to the relevant description in the aforementioned method embodiment, and will not be repeated here.

[0148] Example 6

[0149] Reference Figure 11 The diagram shows a structural schematic of an electronic device according to Embodiment Six of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0150] like Figure 11 As shown, the electronic device may include: a processor 1102, a communications interface 1104, a memory 1106, and a communications bus 1108.

[0151] in:

[0152] The processor 1102, communication interface 1104, and memory 1106 communicate with each other via communication bus 1108.

[0153] Communication interface 1104 is used to communicate with other electronic devices or servers.

[0154] The processor 1102 is used to execute program 1110, specifically the relevant steps in the above method embodiments.

[0155] Specifically, program 1110 may include program code that includes computer operation instructions.

[0156] The processor 1102 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0157] Memory 1106 is used to store program 1110. Memory 1106 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0158] Program 1110 may include multiple computer instructions. Specifically, program 1110 may use multiple computer instructions to cause processor 1102 to perform the operation corresponding to any of the methods described in the foregoing multiple method embodiments.

[0159] The specific implementation of each step in program 1110 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0160] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0161] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0162] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0163] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0164] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0165] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0166] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. An information extraction method, comprising: Obtain multimodal feature data corresponding to the text image, wherein the multimodal feature data is feature data generated based at least on the text features and image features in the text image; Based on the prompt information related to the text image and the multimodal feature data, generate feature data to be matched with the multimodal feature data; The matching process is performed on the feature data to be matched and the multimodal feature data to obtain matching result feature data, and the text result responding to the prompt information is extracted from the text corresponding to the text image based on the matching result feature data.

2. The method according to claim 1, wherein, The step of generating matchable feature data based on the prompt information related to the text image and the multimodal feature data includes: Obtain prompt information corresponding to the text image, and generate auxiliary query information based on the prompt information and pre-generated feature extraction parameters; Based on the multimodal feature data and the auxiliary query information, feature resampling is performed to obtain enhanced prompt information corresponding to the prompt information and enhanced features corresponding to the feature extraction parameters; Based on the enhanced prompt information and the enhanced resampling features, feature data to be matched is generated for matching with the multimodal feature data.

3. The method according to claim 2, wherein, The step of performing feature resampling based on the multimodal feature data and the auxiliary query information to obtain enhanced prompt information corresponding to the prompt information and enhanced resampled features corresponding to the feature extraction parameters includes: The multimodal feature data and the auxiliary query information are concatenated, and the concatenation result is obtained. Cross-attention processing is performed on the splicing result to obtain the enhanced prompt information corresponding to the prompt information and the enhanced resampling features corresponding to the feature extraction parameters.

4. The method according to claim 2, wherein, The step of generating matchable feature data for matching the multimodal feature data based on the enhanced prompt information and the enhanced resampling features includes: The enhanced prompt information and the enhanced resampling features are subjected to cross-attention processing to obtain the cross-attention processing result; The cross-attention processing results are subjected to layout-aware feature extraction processing to generate feature data to be matched with the multimodal feature data.

5. The method according to claim 1, wherein, The matching process of the feature data to be matched and the multimodal feature data to obtain the matching result feature data includes: The autoregressive model is used to perform matching processing on the feature data to be matched and the multimodal feature data to obtain the matching result feature data; or, The similarity between the feature data to be matched and the multiple features contained in the multimodal feature data is calculated respectively. Based on the similarity calculation results corresponding to the multiple features respectively, the matching result feature data is determined.

6. The method according to any one of claims 1-5, wherein, The acquisition of multimodal feature data corresponding to the text image includes: The text image is subjected to text recognition and image feature extraction respectively to obtain the corresponding text recognition result and image features, wherein the text recognition result includes text features and layout features corresponding to the text features; The text recognition result and the image features are fused to obtain the multimodal feature data.

7. The method according to any one of claims 1-5, wherein, The method is implemented through an information extraction model, which includes an encoder, a generator, and a matcher. in: The encoder is used to perform text recognition and image feature extraction on the input text image, and outputs multimodal feature data corresponding to the text image based on the text recognition results and the extracted image features; The generator is used to generate feature data to be matched against the multimodal feature data based on prompt information related to the text image and the multimodal feature data. The matcher is used to perform matching processing on the feature data to be matched and the multimodal feature data to obtain matching result feature data, and extract the text result responding to the prompt information from the text corresponding to the text image based on the matching result feature data.

8. A training method for an information extraction model, wherein, The information extraction model includes an encoder, a generator, and a matcher, and the method includes: Obtain text image samples used to train the information extraction model, text result truth value annotation information corresponding to the text image samples, and prompt information corresponding to the text image samples; The text image sample is input into the encoder to obtain corresponding multimodal feature data, wherein the multimodal feature data is feature data generated based at least on the text features and image features in the text image sample; Based on the multimodal feature data and the prompt information, the generator generates feature data to be matched with the multimodal feature data. The matcher performs matching processing on the feature data to be matched and the multimodal feature data to obtain matching result feature data, and based on the matching result feature data, extracts the text prediction result in response to the prompt information from the text corresponding to the text image; The information extraction model is trained based on the difference between the text prediction result and the ground truth annotation information of the text result.

9. The method according to claim 8, wherein, Before acquiring the text image samples used to train the information extraction model and the text result truth value annotation information corresponding to the text image samples, the method further includes: determining a pre-training strategy to be used from a plurality of preset pre-training strategies for training the information extraction model; and determining prompt information corresponding to the text image samples based on the pre-training strategy. The matching process of the feature data to be matched and the multimodal feature data by the matcher includes: matching the feature data to be matched and the multimodal feature data by the matcher according to the determined pre-training strategy.

10. The method according to claim 9, wherein, The pre-training strategy includes at least one of the following: Fill-in matching strategy: a strategy used to indicate the matching of text between at least two given words in the prompt information from the text of the text image sample; Directional search strategy: a strategy for matching the K nearest neighbor texts of the given word in a specified direction from the text of the text image sample based on at least one given word in the prompt information, where K is greater than or equal to 1; Proximity strategy: a strategy used to indicate the matching of M adjacent texts from the text of the text image sample based on at least one given word in the prompt information, where M is greater than or equal to 1.

11. The method according to claim 10, wherein, If the pre-training strategy is a fill-in matching strategy, then determining the prompt information corresponding to the text image sample based on the pre-training strategy includes: Select at least two given words from the text corresponding to the text image sample, and generate prompt information based on the selected given words to indicate that the given words are matched from the text image sample and to determine the text between the given words; The text result truth labeling information corresponding to the text image sample is determined in the following way: Based on the text in the text of the text image sample, corresponding to the text between the at least two selected given words, generate the text result truth value annotation information of the text image sample.

12. The method according to claim 10, wherein, If the strategy is a direction search strategy, then determining the prompt information corresponding to the text image sample based on the pre-trained strategy includes: Determine the specified direction and the value of K, and select at least one given word from the text image sample. Based on the selected given word, the value of K and the specified direction, generate prompt information to indicate that the K nearest neighbor texts of the given word in the specified direction are matched from the text of the text image sample. The text result truth value annotation information corresponding to the text image sample is generated in the following way: Based on the text in the text image sample, generate the text result truth value annotation information of the text image sample from the K text adjacent to the given word in a specified direction.

13. The method of claim 10, comprising: If the strategy is a proximity strategy, then determining the prompt information corresponding to the text image sample based on the pre-trained strategy includes: Determine the value of M, and select at least one given word from the text image sample. Based on the selected given word and the value of M, generate prompt information to indicate that M texts adjacent to the given word are matched from the text of the text image sample. The text result truth value annotation information corresponding to the text image sample is generated in the following way: Based on the M texts adjacent to the given word in the text image sample, the text result truth value annotation information is generated.

14. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-13.

Images