Information extraction method and information extraction model training method

Abstract

Provided in the embodiments of the present disclosure are an information extraction method and an information extraction model training method. The information extraction method comprises: acquiring a task image of an information extraction task; performing image feature extraction on the task image to obtain a task image feature; and inputting the task image feature into an encoding unit in an information extraction model to obtain a task encoding feature, and inputting the task encoding feature into a decoding unit in the information extraction model to obtain a text extraction result of the task image, wherein the information extraction model is obtained by performing training on the basis of a sample image feature, the sample image feature is obtained by performing feature extraction on a sample mask image, and the sample mask image is obtained by performing mask processing on text content in a sample image. By training the model after performing mask processing on the text content in the sample image, the model can understand the image feature and text context content, and thus high-precision information extraction is realized.

Description

Information extraction method and information extraction model training method

[0001] The present disclosure claims priority to Chinese Patent Application No. 202410833769.5, filed on June 25, 2024, with the Chinese Patent Office, entitled "Information extraction method and information extraction model training method", the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to the field of computer technology, in particular to an information extraction method and an information extraction model training method. BACKGROUND

[0003] With the development of computer technology, visual information extraction technology is widely used in document images, card pictures, street view photos and other tasks. Through visual information extraction, information can be extracted from scattered, redundant and unstructured pictures, thereby obtaining focused and simplified structured information.

[0004] Currently, visual information extraction is usually realized by using an optical character recognition engine. However, the optical character recognition engine may not be completely accurate due to reasons such as picture layout and font style, further leading to low visual information extraction accuracy. Therefore, there is an urgent need for a visual information extraction scheme with high accuracy. SUMMARY

[0005] In view of this, the present disclosure provides an information extraction method. One or more embodiments of the present disclosure also relate to an information extraction model training method, an information processing method based on an information extraction model, a model training platform, an information extraction model training device, an information extraction device, an information processing device based on an information extraction model, a computing device, a computer-readable storage medium, and a computer program product, to solve the technical defects in the prior art.

[0006] According to a first aspect of an embodiment of the present disclosure, an information extraction method is provided, comprising: obtaining a task image of an information extraction task; performing image feature extraction on the task image to obtain a task image feature; inputting the task image feature into an encoding unit of an information extraction model to obtain a task encoding feature, and inputting the task encoding feature into a decoding unit of the information extraction model to obtain a text extraction result of the task image, wherein the information extraction model is trained based on a sample image feature, the sample image feature is obtained based on feature extraction on a sample mask image, and the sample mask image is obtained based on mask processing on text content in a sample image.

[0007] The information extraction method provided by one embodiment of the present disclosure improves the precision of the information extraction model by performing model training on the text content in the sample image after mask processing, so that the model can understand the image features and text context content of the task image, thereby achieving high-precision information extraction. BRIEF DESCRIPTION OF DRAWINGS

[0008] FIG. 1 is an architecture diagram of an information extraction model training system according to one embodiment of the present disclosure;

[0009] FIG. 2 is an architecture diagram of an information extraction system according to one embodiment of the present disclosure;

[0010] FIG. 3 is a flowchart of an information extraction model training method according to one embodiment of the present disclosure;

[0011] FIG. 4 is a flowchart of a processing process of an information extraction model training method according to one embodiment of the present disclosure;

[0012] FIG. 5 is a flowchart of a processing process of another information extraction model training method according to one embodiment of the present disclosure;

[0013] FIG. 6 is a flowchart of an information extraction method according to one embodiment of the present disclosure;

[0014] FIG. 7 is a flowchart of an information processing method based on an information extraction model according to one embodiment of the present disclosure;

[0015] FIG. 8 is a schematic diagram of a model training platform according to one embodiment of the present disclosure;

[0016] FIG. 9 is a structural schematic diagram of an information extraction model training device according to one embodiment of the present disclosure;

[0017] FIG. 10 is a structural schematic diagram of an information extraction device according to one embodiment of the present disclosure;

[0018] FIG. 11 is a structural schematic diagram of an information processing device based on an information extraction model according to one embodiment of the present disclosure;

[0019] FIG. 12 is a structural block diagram of a computing device according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In the following description, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, the present disclosure can be practiced in many different ways beyond the specific embodiments described herein, and it is understood that persons having ordinary skill in the art can make similar modifications and changes without departing from the scope of the present disclosure, and therefore the present disclosure is not limited to the specific implementations disclosed below.

[0021] The terminology used in the one or more embodiments of the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the present disclosure. As used in the one or more embodiments of the present disclosure and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in the one or more embodiments of the present disclosure, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0022] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, without departing from the scope of the one or more embodiments of the present disclosure, first can be termed second and, similarly, second can be termed first. The term "if' as used herein can be interpreted as meaning "when" or "in response to determining" depending on the context.

[0023] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the one or more embodiments of the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0024] First, the nomenclature involved in the one or more embodiments of the present disclosure is explained.

[0025] Information Extraction (IE): is an important task in the field of Natural Language Processing (NLP), aiming to automatically identify and extract useful information from unstructured or semi-structured text, and convert it into structured form for further analysis and utilization.

[0026] Visual Information Extraction (VIE): extracts information from scattered, redundant and unstructured pictures, and obtains focused and simplified structured information.

[0027] Masked Image Modeling (MIM): A self-supervised learning method in computer vision. In MIM, parts of an input image are randomly "masked" or covered, and the model's task is to reconstruct the complete image from the unmasked parts or predict the pixels in the masked regions. This process allows the model to learn and understand visual structures, textures, colors, and other visual features in images, improving its overall representation ability.

[0028] Masked Language Modeling (MLM): A pre-training task in NLP. In this task, the model receives a partially processed sentence as input, with some words randomly "masked" or covered by a special symbol (e.g., [MASK]), and the goal is to predict the original words at these masked positions. This process allows the model to learn to understand the context of a sentence and infer missing words based on contextual information, enhancing its language understanding and generation capabilities.

[0029] Optical Character Recognition (OCR): A technology that uses optical techniques and computer algorithms to recognize and convert textual information from physical media (such as paper documents, images) into editable and searchable electronic text format. This process typically includes scanning, image preprocessing, character recognition, and post-processing steps, greatly improving the efficiency and accuracy of document processing, and is widely used in literature digitization, data entry, form processing, and archive management.

[0030] Convolutional Neural Network (CNN) Model: A deep learning algorithm suitable for computer vision tasks such as image recognition, image classification, and object detection. Its core feature is the use of convolutional layers to extract features from input data (such as images), followed by pooling layers to reduce data dimensions, and finally fully connected layers for classification or regression prediction. CNN can achieve high accuracy in various visual recognition tasks by learning local features and their spatial relationships in images.

[0031] Deep self-attention (Transformer) model: a deep learning architecture based on attention mechanism, mainly used for processing sequence data. The Transformer model includes a stack of repeated encoding units (Encoder) and decoding units (Decoder). This design allows the Transformer to efficiently learn long-term dependencies, suitable for a variety of natural language processing tasks including machine translation, text summarization, question answering systems, etc.

[0032] Vision Transformer (VIT) model: a deep learning model, the VIT model divides the input image into multiple fixed-size image blocks (patches), then uses linear embedding to convert these patches into feature vectors, and then sends these feature vectors together with a special classification token into the Transformer architecture.

[0033] Visual information extraction involves the ability of picture text detection, text recognition, extraction, etc. Through visual information extraction, information can be extracted from scattered, redundant, unstructured pictures, so as to obtain focused, simplified structured information. The tasks suitable for visual information extraction include but are not limited to bill information extraction tasks, certificate information extraction tasks, table information extraction tasks, and document information extraction tasks. Therefore, improving the ability of visual information extraction is crucial for optical character recognition.

[0034] Currently, the commonly used information extraction methods mainly include three types: the first type unifies text detection, recognition and extraction into the same framework to realize information extraction; the second type extracts OCR information including text position and content through an open source OCR engine and inputs a language model for extraction; the third type directly extracts content without text detection and recognition by inputting a picture into a model. However, the first type lacks a corresponding framework training scheme, resulting in low precision. The second type relies on the OCR engine, which may cause error propagation. The third type is a black box algorithm, lacking interpretability and decoding is time-consuming.

[0035] To solve the above problems, an embodiment of the present disclosure provides a pre-training scheme for an end-to-end visual information extraction framework, that is, an information extraction model training method, including: obtaining an initial extraction model and a sample image, wherein the initial extraction model includes an encoding unit and a decoding unit, the initial extraction model is pre-configured with model parameters, and the encoding unit is connected with the decoding unit; performing mask processing on text content in the sample image to obtain a sample mask image, and performing image feature extraction on the sample mask image to obtain sample image features; inputting the sample image features into the encoding unit to obtain sample encoding features, and inputting the sample encoding features into the decoding unit to obtain sample prediction results; adjusting the model parameters of the initial extraction model based on the sample prediction results to obtain a trained information extraction model. By adding a pre-training task, the model is trained after the text content in the sample image is processed by mask processing, so that the model can understand the image features and text context content of the sample image, improve the precision of the information extraction model, and thus realize high-precision information extraction task processing.

[0036] In the present disclosure, an information extraction model training method is provided, and the present disclosure simultaneously relates to an information extraction method, an information processing method based on an information extraction model, a model training platform, an information extraction model training device, an information extraction device, an information processing device based on an information extraction model, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0037] Referring to FIG. 1, FIG. 1 shows an architecture diagram of an information extraction model training system according to an embodiment of the present disclosure. The information extraction model training system can include a client 100 and a server 200.

[0038] The client 100 is configured to send an initial extraction model and a sample image to the server 200, wherein the initial extraction model includes an encoding unit and a decoding unit, the initial extraction model is pre-configured with model parameters, and the encoding unit is connected with the decoding unit.

[0039] The server 200 is configured to perform mask processing on text content in the sample image to obtain a sample mask image, perform image feature extraction on the sample mask image to obtain sample image features, input the sample image features into the encoding unit to obtain sample encoding features, input the sample encoding features into the decoding unit to obtain sample prediction results, adjust the model parameters of the initial extraction model based on the sample prediction results to obtain a trained information extraction model, and send the model parameters of the trained information extraction model to the client 100, wherein the information extraction model is configured to process task image features of a task image in an information extraction method to obtain a text extraction result of the task image.

[0040] The client 100 is further configured to receive the model parameters of the trained information extraction model sent by the server 200.

[0041] By applying the scheme of the embodiments of the present disclosure, the model can understand the image features and text context content of the sample image by performing mask processing on the text content in the sample image before model training, which improves the precision of the information extraction model, thereby realizing high-precision information extraction task processing.

[0042] Referring to FIG. 2, FIG. 2 shows an architecture diagram of an information extraction system according to an embodiment of the present disclosure. The information extraction system can include a client 100 and a server 200.

[0043] The client 100 is configured to send a task image of an information extraction task to the server 200.

[0044] The server 200 is configured to perform image feature extraction on the task image to obtain task image features; input the task image features into an encoding unit of an information extraction model to obtain task encoding features, and input the task encoding features into a decoding unit of the information extraction model to obtain a text extraction result of the task image, wherein the information extraction model is trained based on sample image features, the sample image features are obtained based on feature extraction on a sample mask image, and the sample mask image is obtained based on mask processing on text content in a sample image; and send the text extraction result to the client 100.

[0045] The client 100 is further configured to receive the text extraction result sent by the server 200.

[0046] By applying the scheme of the embodiments of the present disclosure, high-precision, end-to-end information extraction task processing is realized by using the information extraction model, which further improves the information extraction efficiency and the accuracy of the text extraction result.

[0047] In practical applications, the client 100 and the server 200 establish a connection through a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc. The data transmitted by the client 100 can need to be processed by encoding, transcoding, compression, etc. before being published to the server 200.

[0048] The client 100 can be any one of a browser, an APP (Application), a web application such as an H5 (HyperText Markup Language 5) application, a light application (also known as a mini-program, a lightweight application program), or a cloud application. The client 100 can be developed based on a software development kit (SDK) of a corresponding service provided by the server 200, such as an RTC (Real Time Communication) SDK. The client 100 can be deployed in an electronic device and run in dependence on the device or an APP in the device. The electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, and the like. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

[0049] The server 200 can include servers that provide various services, such as servers that provide communication services for multiple clients, servers that provide support for models used on clients for background training, servers that process data sent by clients, and the like. It should be noted that the server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server of cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0050] Referring to FIG. 3, FIG. 3 shows a flowchart of an information extraction model training method according to an embodiment of the present disclosure, which specifically includes the following steps:

[0051] Step 302: Obtain an initial extraction model and a sample image, wherein the initial extraction model includes an encoding unit and a decoding unit, and the initial extraction model is pre-configured with model parameters, and the encoding unit is connected to the decoding unit.

[0052] Specifically, the initial extraction model refers to a model with information extraction capability. The initial extraction model can be an untrained Transformer model or a pre-trained Transformer model, which is selected according to actual conditions, and the present disclosure does not make any limitation on this. The sample image is used to adjust the parameters of the initial extraction model. The sample image can be an image of different scenes, such as an image of a financial scene, an image of a traffic scene, an image of a transaction scene, and the like. The sample image includes but is not limited to a bill image, an identification image, and a table image. The encoding unit can be referred to as an encoder, which is used to convert an input sequence into a series of high-order feature representations, i.e., sample encoding features. The decoding unit can be referred to as a decoder, which is used to generate an output sequence, i.e., a sample prediction result, by using the high-order feature representations output by the encoding unit and its own input (when decoding step by step, it is the part of the sequence that has been generated previously). The model parameters refer to the initial weights and bias values given to the initial extraction model in order to start the training process when the initial extraction model is constructed. The model parameters are selected based on a series of theories and practical experiences (such as strategies to avoid saturation and gradient disappearance, and strategies to initialize uniform and normal distribution) to facilitate the model to quickly and effectively converge to a better solution.

[0053] It should be noted that the sample image carries a sample label, and the sample label is used to determine whether the training of the initial extraction model is completed. Generally, the sample image used to train the initial extraction model is large, and the sample label can be generated by using an OCR engine. Since the OCR engine has certain errors, the sample label carried by the sample image can be regarded as a pseudo label. The sample label includes a sample content label and a region position label.

[0054] In actual applications, there are various ways to obtain the initial extraction model and the sample image, which are selected according to actual conditions, and the present disclosure does not make any limitation on this. In a possible implementation manner of the present disclosure, the initial extraction model and the sample image can be read from other data acquisition devices or databases. In another possible implementation manner of the present disclosure, the training data of a target task sent by a client can be received.

[0055] Step 304: performing mask processing on the text content in the sample image to obtain a sample mask image, and performing image feature extraction on the sample mask image to obtain a sample image feature.

[0056] Specifically, the process of masking the text content in the sample image can be regarded as a MIM process. The sample masked image refers to the sample image after the text content is masked. Image feature extraction refers to extracting key information from the sample image that helps to describe the image content, structure or semantics, which is usually converted into numerical values, vectors or other forms convenient for the initial extraction model processing. Image feature extraction can enable the initial extraction model to understand and distinguish different image contents in a more efficient and abstract manner. The sample image feature refers to a set of representations or descriptions obtained from the sample image through the image feature extraction process. The sample image feature can represent the key information in the sample image to facilitate subsequent visual information extraction.

[0057] It should be noted that there are many ways to mask the text content in the sample image to obtain the sample masked image, which are selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation. In a possible implementation manner of the present disclosure, a certain proportion (such as 15%) of the text content in the sample image can be randomly masked or covered into a special symbol to obtain the sample masked image. In another possible implementation manner of the present disclosure, the text content in the sample image can be divided into two types, namely word and character, and different masking proportions can be used for word and character to mask the text content and obtain the sample masked image. For example, the word of the text content is randomly masked and replaced by a learnable word mask token; the character of the text content is randomly masked and replaced by a learnable character mask token. Among them, the word mask is 15%, and the character mask is 30%.

[0058] In actual application, there are many ways to perform image feature extraction on the sample masked image to obtain the sample image feature, which are selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation. In a possible implementation manner of the present disclosure, a CNN model can be used to perform image feature extraction on the sample masked image to obtain the sample image feature. In another possible implementation manner of the present disclosure, a VIT model can be used to perform image feature extraction on the sample masked image to obtain the sample image feature.

[0059] Step 306: input the sample image feature into the encoding unit to obtain the sample encoding feature, and input the sample encoding feature into the decoding unit to obtain the sample prediction result.

[0060] It should be noted that in the encoding unit, the sample image feature is first converted into a higher-dimensional representation by an embedding layer, so that the sample image feature can be compatible with the internal representation of the model. Then, the sample image feature will undergo a multi-head self-attention mechanism, which allows the model to learn complex relationships between features, and the gradient propagation and convergence acceleration are helped by the feedforward neural network and residual connection and layer normalization, and finally the sample encoding feature is output.

[0061] In the decoding unit, the sample encoding feature will undergo a multi-head self-attention mechanism to ensure that the model can only see past position information when generating the output of the current position, avoiding the leakage of future information, and then enter the encoder-decoder attention, so that the decoding unit focuses on different parts of the input sequence. Finally, the feedforward neural network and residual connection and layer normalization are used to enhance the nonlinear expression ability of the model and stabilize the training process, and the sample prediction result is obtained.

[0062] Step 308: Based on the sample prediction result, adjust the model parameters of the initial extraction model to obtain a trained information extraction model, wherein the information extraction model is used to process the task image features of the task image in the information extraction method to obtain the text extraction result of the task image.

[0063] It should be noted that the implementation of the information extraction method can refer to the processing flow of the information extraction method shown in FIG. 6, and the embodiments of the present disclosure will not be described in detail. In actual application, based on the sample prediction result, the model parameters of the initial extraction model can be adjusted according to the content prediction loss calculated based on the sample prediction result and the sample content label, and the model parameters of the initial extraction model are adjusted according to the content prediction loss until the preset stopping condition is reached, and the trained information extraction model is obtained. Wherein, the sample content label includes a text content label and a position label of the text content in the sample image.

[0064] The function of calculating the content prediction loss according to the sample prediction result and the sample content label has many kinds, such as cross-entropy loss function, L1 norm loss function, L2 norm loss function, KL divergence (Kullback-Leibler divergence), etc. The specific selection is based on the actual situation, and the embodiments of the present disclosure do not make any limitation. The preset stopping condition includes but is not limited to that the content prediction loss is less than or equal to the preset threshold, and the iteration number reaches the preset iteration number, wherein the preset threshold and the preset iteration number are selected according to the actual situation, and the embodiments of the present disclosure do not make any limitation.

[0065] In a possible implementation of the present disclosure, after the content prediction loss is calculated, the content prediction loss can be compared with a preset threshold. Specifically, if the content prediction loss is greater than the preset threshold, it indicates that the difference between the sample prediction result and the sample content label is large, and the generation capability of the initial extraction model is poor. At this time, the model parameters of the initial extraction model can be adjusted, and the initial extraction model can be continuously trained until the content prediction loss is less than or equal to the preset threshold, which indicates that the difference between the sample prediction result and the sample content label is small, and a preset stopping condition is reached, and the information extraction model that is completed training is obtained.

[0066] In another possible implementation of the present disclosure, in addition to comparing the size relationship between the content prediction loss and the preset threshold, the number of iterations can also be combined to determine whether the current initial extraction model is trained. Specifically, if the content prediction loss is greater than the preset threshold, the model parameters of the initial extraction model are adjusted, and the initial extraction model is continuously trained until the preset number of iterations is reached, and the iteration is stopped to obtain the information extraction model that is completed training.

[0067] By applying the scheme of the embodiments of the present disclosure, the model parameters of the initial extraction model are adjusted according to the sample prediction result and the sample content label until a preset stopping condition is reached, and the information extraction model that is completed training is obtained. By continuously adjusting the model parameters of the initial extraction model in the case where the preset stopping condition is not met, the information extraction model finally obtained can have accurate text detection and recognition capability.

[0068] In the embodiments of the present disclosure, the visual information extraction includes three stages of text detection, text recognition, and information extraction. The text detection and the text recognition can be combined into a text spotting stage, and the information extraction stage can be divided into a text grouping stage of combining texts into entities and an entity labeling stage of classifying entities. Therefore, the whole visual information extraction process can include three stages of text spotting, text grouping, and entity labeling. By adjusting the model parameters of the initial extraction model based on the sample prediction result, the information extraction model that is completed training has good text spotting capability and can process text extraction tasks. Further, in order to enable the information extraction model to have text grouping and entity labeling capability and process entity extraction and entity category recognition tasks, the information extraction model that is completed training can be further updated in parameters to obtain an updated information extraction model.

[0069] In an optional embodiment of the present disclosure, for the text grouping capability of the model, an entity region prediction unit for predicting the entity region in the image can be constructed after the encoding unit of the trained information extraction model, and then the information extraction model is trained by using the spotting task so that it can recognize the text content and the text position in the image, and the information extraction model is trained by using the entity detection (ETD) task so that it can determine the entity region in the image and the text content (entity) in the region, that is, the information extraction model includes an encoding unit, a decoding unit and an entity region prediction unit, and the entity region prediction unit is connected with the encoding unit; after the model parameters of the initial extraction model are adjusted based on the sample prediction result to obtain the trained information extraction model, the following steps can be further included:

[0070] obtaining a training image of an information extraction task;

[0071] masking the text content in the training image to obtain a training mask image, and performing image feature extraction on the training mask image to obtain a training image feature;

[0072] inputting the training image feature into the encoding unit to obtain a training encoding feature, and inputting the training encoding feature into the decoding unit to obtain a training prediction result;

[0073] inputting the training encoding feature into the entity region prediction unit to obtain a training entity region prediction result;

[0074] adjusting the model parameters of the information extraction model based on the training prediction result and the training entity region prediction result to obtain an updated information extraction model.

[0075] Specifically, the training prediction result refers to the result obtained by the information extraction model by performing extraction prediction on the information in the training image. The training entity region prediction result refers to the result obtained by the entity region prediction unit by predicting the region position of the entity in the text content in the training image in the training image. In order to make the information extraction model trained by using the training image can be directly used for the information extraction task, therefore, the training label carried by the training image can be an accurate real label obtained by manual labeling, and the training label includes a training content label and a training region position label. The architecture of the entity region prediction unit includes but is not limited to a CNN model and a Transformer model, which are specifically selected according to actual conditions, and the present disclosure does not make any limitation on this.

[0076] It should be noted that, when adjusting the model parameters of the information extraction model based on the training prediction result and the training entity region prediction result, the training content prediction loss can be calculated according to the training prediction result and the training content label; the training region prediction loss can be calculated according to the training entity region prediction result and the training region position label; the model parameters of the information extraction model are adjusted according to the training content prediction loss and the training region prediction loss to obtain an updated information extraction model. The adjustment process of the training content prediction loss, the training region prediction loss and the model parameters of the information extraction model can refer to the calculation process of the content prediction loss and the training process of the initial extraction model, and the present embodiment will not be described in detail.

[0077] By using the scheme of the present embodiment, the information extraction model is trained by using the spotting task and the ETD task, so that the information extraction model has the entity region detection capability on the basis of the character detection and recognition capability, the processing capability of the information extraction model is expanded, and the application range of the model is improved.

[0078] In an optional embodiment of the present disclosure, for the entity labeling capability of the model, an entity region prediction unit for predicting the entity region in the image can be constructed after the encoding unit of the trained information extraction model, and a category prediction unit for predicting the entity category can be constructed after the entity region prediction unit, and then the information extraction model is trained by using the Tag task, so that the information extraction model can recognize the category of the entity, that is, the information extraction model further includes the category prediction unit, and the category prediction unit is connected with the entity region prediction unit. After the training encoding feature is input into the entity region prediction unit to obtain the training entity region prediction result, the following steps can be further included:

[0079] The training entity region prediction result is input into the category prediction unit for entity category prediction to obtain an entity category prediction result.

[0080] Based on the training prediction result and the training entity region prediction result, the model parameters of the information extraction model are adjusted to obtain an updated information extraction model, which can include the following steps:

[0081] Based on the training prediction result, the training entity region prediction result and the entity category prediction result, the model parameters of the information extraction model are adjusted to obtain an updated information extraction model.

[0082] Specifically, in addition to the training content label and the training region position label, the training image can also carry an entity category label. The architecture of the category prediction unit includes but is not limited to a CNN model and a Transformer model, which is selected according to actual conditions, and the present embodiment does not make any limitation on this.

[0083] It should be noted that, based on the training prediction result, the training entity region prediction result and the entity category prediction result, the model parameters of the information extraction model are adjusted, the training content prediction loss can be calculated according to the training prediction result and the training content label; the training region prediction loss can be calculated according to the training entity region prediction result and the training region position label; the category prediction loss can be calculated according to the entity category prediction result and the entity category label; the model parameters of the information extraction model are adjusted according to the training content prediction loss, the training region prediction loss and the category prediction loss, and the updated information extraction model is obtained. The adjustment process of the training content prediction loss, the training region prediction loss, the category prediction loss and the model parameters of the information extraction model can refer to the calculation process of the above content prediction loss and the training process of the initial extraction model, and the present embodiment will not be described again.

[0084] By applying the scheme of the present embodiment, the training entity region prediction result is input into the category prediction unit for entity category prediction to obtain the entity category prediction result, which realizes direct classification and prediction of the entity by using the Tag task to obtain the entity category, so that the information extraction model has the text detection and recognition capability, the entity region detection capability and the entity category recognition capability at the same time, the processing capability of the information extraction model is expanded, and the application range of the model is improved.

[0085] In an optional embodiment of the present disclosure, the initial extraction model further comprises a mask prediction unit, and the mask prediction unit is connected with the encoding unit. After the sample image feature is input into the encoding unit to obtain the sample encoding feature, the following steps can be further included:

[0086] The sample encoding feature is input into the mask prediction unit for mask content prediction to obtain the mask prediction result.

[0087] Based on the sample prediction result, the model parameters of the initial extraction model are adjusted to obtain the information extraction model trained, which can include the following steps:

[0088] Based on the sample prediction result and the mask prediction result, the model parameters of the initial extraction model are adjusted to obtain the information extraction model trained.

[0089] It should be noted that the mask prediction result refers to a result obtained by the mask prediction unit predicting the mask content in the sample image. The architecture of the mask prediction unit includes but is not limited to a CNN model and a Transformer model, and is specifically selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this. The mask prediction unit is used to predict the mask content in the sample mask image. When the sample encoding feature is input into the mask prediction unit for mask content prediction, in the case that the word in the sample image is randomly masked and replaced by a learnable word mask token, and the char in the text content is randomly masked and replaced by a learnable char mask token to obtain a sample mask image, the target of the mask content prediction is the word and the char in the text content, and therefore, the mask content prediction can be understood as MIM at the char level and the word level, that is, the masked char and word are predicted and reconstructed. Since the text content before the mask is known, the sample image carries the mask content label.

[0090] In actual application, based on the sample prediction result and the mask prediction result, when the model parameters of the initial extraction model are adjusted, the content prediction loss can be calculated according to the sample prediction result and the sample content label; the mask prediction loss can be calculated according to the mask prediction result and the mask content label; the model parameters of the initial extraction model are adjusted according to the content prediction loss and the mask prediction loss, and the trained information extraction model is obtained. The way of calculating the mask prediction loss is the same as the way of calculating the content prediction loss, and the way of adjusting the model parameters of the initial extraction model according to the content prediction loss and the mask prediction loss is the same as the way of adjusting the model parameters of the initial extraction model according to the content prediction loss, and the embodiments of the present disclosure will not be described again.

[0091] By applying the scheme of the embodiments of the present disclosure, in the model parameter adjustment process, if the initial extraction model can reconstruct the char, it means that the initial extraction model knows which chars a specific word is composed of and clearly understands the shape of the char, and the initial extraction model can understand the context semantics of the text content and learn the basic language knowledge such as grammar, so that the masked word can be reconstructed. In this stage of training, the spotting ability and the information extraction ability of the initial extraction model can be improved.

[0092] In an optional embodiment of the present disclosure, the initial extraction model further includes an entity prediction unit, and the entity prediction unit is connected with the encoding unit. After the sample image feature is input into the encoding unit to obtain the sample encoding feature, the following steps can be further included:

[0093] The sample encoding feature is input into the entity prediction unit for entity prediction to obtain an entity prediction result.

[0094] Based on the sample prediction result, the model parameters of the initial extraction model are adjusted to obtain a trained information extraction model, which can include the following steps:

[0095] Based on the sample prediction result and the entity prediction result, the model parameters of the initial extraction model are adjusted to obtain a trained information extraction model.

[0096] It should be noted that the entity prediction result includes an entity region prediction result, an entity content prediction result, and an entity block prediction result. The entity prediction result refers to the result obtained by the entity prediction unit predicting the entity in the sample image. The architecture of the entity prediction unit includes but is not limited to a CNN model and a Transformer model, which is selected according to actual conditions, and the present disclosure does not make any limitation on this. The entity prediction unit is used to predict the entity region, the entity content, and the entity block in the text content. The entity region refers to the region position of the entity in the sample image, and the entity region includes at least one entity block. The text content in the entity block is the entity.

[0097] By applying the scheme of the present disclosure, the entity recognition of the text content in the sample image is realized by the entity prediction unit, so that the information extraction model has the entity recognition capability.

[0098] In an optional embodiment of the present disclosure, the entity prediction unit includes an entity region prediction unit, an entity content prediction unit, and an entity block prediction unit. The entity region includes an entity block, and the entity block includes entity content. The entity region prediction unit, the entity content prediction unit, and the entity block prediction unit are respectively connected with the encoding unit. The above inputting the sample encoding feature into the entity prediction unit for entity prediction to obtain an entity prediction result can include the following steps:

[0099] The sample encoding feature is input into the entity region prediction unit to obtain an entity region prediction result.

[0100] The sample encoding feature is input into the entity content prediction unit to obtain an entity content prediction result.

[0101] The sample encoding feature is input into the entity block prediction unit to obtain an entity block prediction result.

[0102] Specifically, the entity region prediction unit is configured to predict a region position of an entity in the text content of the sample image in the sample image, and an entity region prediction result refers to a result of the entity region prediction unit predicting the region position of the entity in the text content of the sample image in the sample image. The entity content prediction unit is configured to predict entity content in the text content of the sample image, and an entity content prediction result refers to a result of the entity content prediction unit predicting the entity content in the text content of the sample image. The entity block prediction unit is configured to predict a region block of the entity in the entity region, and an entity block prediction result refers to a result of the entity block prediction unit predicting the region block of the entity in the entity region.

[0103] It should be noted that inputting the sample encoding feature into the entity region prediction unit can be regarded as a top-down ETD task. The ETD task first identifies large regions or blocks (such as titles, paragraphs, tables, etc.) from a high-level structure of the entire text content, and then further refines to find entities in these regions. This method focuses on understanding the overall layout and logical structure of the text content, thereby inferring the position of the entity, and finally detecting the entity region.

[0104] Further, the entity content prediction unit can detect an entity word, and the entity block prediction unit can predict an entity block. Therefore, according to the entity content prediction result and the entity block prediction result, the process of determining the entity mapping prediction relationship can be regarded as a bottom-up entity mapping process (WTB, Word-To-Block). In contrast to ETD, WTB starts from identifying a single word and gradually combines to form larger structures, such as words, phrases, and complete entities or information blocks. This method pays more attention to the microstructure and proximity relationship of the text, and infers the overall structure through aggregation of local features.

[0105] By applying the scheme of the embodiments of the present disclosure, the entity region is directly detected by using ETD, and then WTB is used to realize the mapping from word to entity block, so that a closed loop can be formed, the grouping of entities is realized, and thus the information extraction model has the capability of word grouping.

[0106] In an optional embodiment of the present disclosure, the sample image carries a sample content label, a region position label, and an entity relationship label. The above step of adjusting the model parameters of the initial extraction model based on the sample prediction result and the entity prediction result to obtain the trained information extraction model can include the following steps:

[0107] According to the sample prediction result and the sample content label, a content prediction loss is calculated.

[0108] According to the entity region prediction result and the region position label, a region prediction loss is calculated;

[0109] According to the entity content prediction result and the entity block prediction result, an entity mapping prediction relationship is determined;

[0110] According to the entity mapping prediction relationship and the entity relationship label, a contrastive learning loss is calculated;

[0111] According to the content prediction loss, the region prediction loss and the contrastive learning loss, the model parameters of the initial extraction model are adjusted to obtain a trained information extraction model.

[0112] Specifically, the entity relationship label is used to describe the association relationship between the entity and the entity block where the entity is located. Since the sample image carries the sample content label and the region position label, the sample content includes the entity, so the sample content label and the region position label can be parsed by rules to determine the paragraph information in the text content, and further determine the entity block where the entity is located, that is, the entity relationship label. The content prediction loss is used to describe the difference between the sample prediction result and the sample content label; the region prediction loss is used to describe the difference between the entity region prediction result and the region position label; the entity mapping prediction relationship refers to the corresponding relationship between the entity content and the entity block; and the contrastive learning loss is used to describe the difference between the entity mapping prediction relationship and the entity relationship label.

[0113] It should be noted that the sample encoding feature is input into the entity content prediction unit to obtain the entity content prediction result, and the sample encoding feature is input into the entity block prediction unit to obtain the entity block prediction result, which realizes learning of the embedding of the word and the embedding of the entity block through two branches, and the distance between the <entity, word> pair is narrowed down and the remaining relationship is pushed away through the contrastive learning loss.

[0114] In actual application, the way of adjusting the model parameters of the initial extraction model according to the content prediction loss, the region prediction loss and the contrastive learning loss is the same as the way of adjusting the model parameters of the initial extraction model according to the content prediction loss described above, and the embodiments of the present disclosure will not be described in detail.

[0115] By applying the scheme of the embodiments of the present disclosure, the model parameters of the initial extraction model are adjusted according to the content prediction loss, the region prediction loss and the contrastive learning loss to obtain a trained information extraction model. Through the top-down ETD and the bottom-up WTB, the grouping of the entity is realized, so that the information extraction model has the capability of word grouping.

[0116] In an optional embodiment of the present disclosure, the initial extraction model further comprises a context prediction unit connected with the encoding unit; after the sample image feature is input into the encoding unit to obtain the sample encoding feature, the following steps can be further included:

[0117] The sample encoding feature is input into the context prediction unit for prediction to obtain a context prediction result;

[0118] Based on the sample prediction result, the model parameters of the initial extraction model are adjusted to obtain a trained information extraction model, which can include the following steps:

[0119] Based on the sample prediction result and the context prediction result, the model parameters of the initial extraction model are adjusted to obtain a trained information extraction model.

[0120] It should be noted that the context prediction result refers to a result obtained by the context prediction unit predicting the context semantic relationship and the context position relationship of the text content of the sample image. The architecture of the context prediction unit includes but is not limited to a CNN model and a Transformer model, which is specifically selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation thereon. Since the context semantic and the text spatial position relationship are extremely important for information extraction, the context semantic relationship and the context position relationship of the text content can be predicted by using the context prediction unit. The context position relationship is used to describe the reading order of the text content.

[0121] By using the scheme of the embodiments of the present disclosure, the semantic and text spatial position relationship prediction is realized through the context prediction result, the precision of entity labeling is improved, and thus the information extraction model has good labeling capability.

[0122] In an optional embodiment of the present disclosure, the context prediction unit comprises a fusion encoding unit, a semantic prediction unit and a position prediction unit, the fusion encoding unit is connected with the encoding unit, and the semantic prediction unit and the position prediction unit are respectively connected with the fusion encoding unit; the sample encoding feature is input into the context prediction unit for prediction to obtain the context prediction result, which can include the following steps:

[0123] The sample encoding feature, a text encoding feature and a position encoding feature are fused by the fusion encoding unit to obtain a sample fusion feature, wherein the text encoding feature is extracted from the sample image based on the position of the text content in the sample image;

[0124] The sample fusion feature is input into the semantic prediction unit for context semantic relationship prediction to obtain a first context prediction result;

[0125] The sample fusion feature is input into the position prediction unit for context position relationship prediction to obtain a second context prediction result.

[0126] It should be noted that when the text content encoding feature is extracted from the sample image based on the position of the text content in the sample image, the center position of the text content in the sample image can be determined, and the image content at the center position is subjected to feature extraction to obtain the text content encoding feature. The fusion encoding unit can be a multi-modal Transformer model.

[0127] In actual application, in the fusion encoding unit, the sample encoding feature, the text content encoding feature and the position encoding feature are input into the multi-modal Transformer for encoding to obtain the representation of each token, that is, the sample fusion feature. Then, in the semantic prediction unit, the MLM task is performed, that is, the corresponding token is inferred at the token that is masked to obtain the first context prediction result. Finally, in the position prediction unit, the ROR (Reading Order Reasoning) task is performed, that is, the relationship with the remaining lines is identified at the token at the beginning of each line, and if it is the next line, it is 1, otherwise it is 0, so as to identify the reading order and obtain the second context prediction result.

[0128] By applying the scheme of the embodiments of the present disclosure, the semantic and spatial position layout information of the text content are learned through the MLM task and the ROR task, and the precision of entity labeling is improved, so that the information extraction model has good labeling capability.

[0129] Referring to FIG. 4, FIG. 4 shows a processing process flowchart of an information extraction model training method according to an embodiment of the present disclosure. Referring to FIG. 5, FIG. 5 shows a processing process flowchart of another information extraction model training method according to an embodiment of the present disclosure. As shown in FIG. 4 and FIG. 5, the visual information extraction task is divided into three tasks of character detection and recognition, character combination and entity labeling, and corresponding pre-training tasks are proposed for each task, which specifically include:

[0130] Character detection and recognition: obtaining an initial extraction model and a sample image; performing mask processing on the text content in the sample image to obtain a sample mask image, and performing image feature extraction on the sample mask image to obtain a sample image feature; inputting the sample image feature into an encoding unit in the initial extraction model to obtain a sample encoding feature, and inputting the sample encoding feature into a decoding unit in the initial extraction model to obtain a sample prediction result; inputting the sample encoding feature into a mask prediction unit in the initial extraction model for mask content prediction to obtain a mask prediction result, wherein the mask content prediction can be performed for characters and words respectively, and therefore the mask prediction unit can include a word mask prediction unit and a character mask prediction unit, and the character detection and recognition task can include a character mask pixel reconstruction task and a word mask pixel reconstruction task;

[0131] Text combination: input the sample encoding features into the entity prediction unit in the initial extraction model to perform entity prediction and obtain an entity prediction result; specifically, input the sample encoding features into the entity region prediction unit in the entity prediction unit to obtain an entity region prediction result; input the sample encoding features into the entity content prediction unit in the entity prediction unit to obtain an entity content prediction result; input the sample encoding features into the entity block prediction unit in the entity prediction unit to obtain an entity block prediction result; and determine an entity mapping prediction relationship according to the entity content prediction result and the entity block prediction result. Therefore, the text combination task can include an entity region detection task and an entity content-entity block mapping task.

[0132] Entity labeling: input the sample encoding features into the context prediction unit in the initial extraction model to perform prediction and obtain a context prediction result; specifically, input the sample encoding features, the text encoding features, and the position encoding features into the fusion encoding unit in the context prediction unit to perform fusion and obtain sample fusion features; input the sample fusion features into the semantic prediction unit in the context prediction unit to perform context semantic relationship prediction and obtain a first context prediction result; and input the sample fusion features into the position prediction unit in the context prediction unit to perform context position relationship prediction and obtain a second context prediction result. Therefore, the entity labeling task can include a context semantic relationship prediction task and a context position relationship prediction task.

[0133] Pre-training: pre-train the initial extraction model according to the sample prediction result, the mask prediction result, the entity region prediction result, the entity content prediction result, the entity block prediction result, the first context prediction result, and the second context prediction result to obtain a trained information extraction model.

[0134] Fine-tuning: after pre-training is completed, fine-tune the pre-trained information extraction model to obtain an information extraction model that can be directly used for an information extraction task. In the fine-tuning stage, the encoding unit, the decoding unit, the entity region prediction unit, and the category prediction unit are mainly trained, the model is trained in the category prediction unit by using the labeling task to obtain an information extraction model with entity category recognition capability.

[0135] By using the scheme of the embodiments of the present disclosure, through the text detection and recognition task, the information extraction model can obtain the content and position of the text; through the entity detection task, the information extraction model can obtain the region of the entity; through the labeling task, the information extraction model can directly classify the entity to obtain the entity category, thereby realizing information extraction. Through the training scheme of the information extraction model, the model can understand the bottom-layer image features, entity features, and semantic features, thereby improving the model precision.

[0136] Referring to FIG. 6, FIG. 6 shows a flowchart of a method for information extraction according to an embodiment of the present disclosure, which includes the following steps:

[0137] Step 602: Obtain a task image of an information extraction task.

[0138] Specifically, the task image can be an image containing text content in different scenarios, such as an image in a financial scenario, an image in a traffic scenario, an image in a transaction scenario, and the like. The sample image includes but is not limited to a bill image, an identification image, and a table image. The information extraction task includes but is not limited to a text extraction task, an entity recognition task, and an entity category recognition task, which are selected according to actual conditions, and the present disclosure does not make any limitation on this.

[0139] In actual applications, there are various ways to obtain the task image of the information extraction task, which are selected according to actual conditions, and the present disclosure does not make any limitation on this. In a possible implementation manner of the present disclosure, the task image of the information extraction task sent by a terminal device can be received. In another possible implementation manner of the present disclosure, the task image of the information extraction task can be read from other data acquisition devices or databases.

[0140] Step 604: Perform image feature extraction on the task image to obtain task image features.

[0141] It should be noted that the implementation manner of “performing image feature extraction on the task image to obtain task image features” is the same as the implementation manner of “performing image feature extraction on the sample mask image to obtain sample image features” described above, and the present disclosure will not make any further description.

[0142] Step 606: Input the task image features into an encoding unit of an information extraction model to obtain task encoding features, and input the task encoding features into a decoding unit of the information extraction model to obtain a text extraction result of the task image, wherein the information extraction model is trained based on sample image features, the sample image features are obtained by performing feature extraction on a sample mask image, and the sample mask image is obtained by performing mask processing on text content in a sample image.

[0143] It should be noted that for the text extraction task, the task image feature can be input into the encoding unit of the information extraction model to obtain task encoding features, and the task encoding features can be input into the decoding unit of the information extraction model to obtain the text extraction result of the task image. The implementation manner of "inputting the task image feature into the encoding unit of the information extraction model to obtain the task encoding feature, and inputting the task encoding feature into the decoding unit of the information extraction model to obtain the text extraction result of the task image" is the same as the implementation manner of "inputting the sample image feature into the encoding unit to obtain the sample encoding feature, and inputting the sample encoding feature into the decoding unit to obtain the sample prediction result", and the embodiments of the present disclosure will not be described again.

[0144] By applying the scheme of the embodiments of the present disclosure, since the information extraction model is obtained by adjusting the model parameters of the initial extraction model based on the sample prediction result, the sample prediction result is obtained by decoding the sample encoding feature by the decoding unit in the initial extraction model, the sample encoding feature is obtained by encoding the sample image feature by the encoding unit in the initial extraction model, the sample image feature is obtained by image feature extraction on the sample mask image, and the sample mask image is obtained by mask processing on the text content in the sample image, therefore, the information extraction model is a high-precision model, and the high-precision end-to-end information extraction task processing is realized by using the information extraction model, and the information extraction efficiency and the accuracy of the text extraction result are further improved.

[0145] In an optional embodiment of the present disclosure, an entity region prediction unit can be connected after the encoding unit to process the entity recognition task, that is, the information extraction model further includes an entity region prediction unit; after the task image feature is input into the encoding unit of the information extraction model to obtain the task encoding feature, the following steps can be further included:

[0146] The task encoding feature is input into the entity region prediction unit to obtain the entity region of the task image and the entity extraction result of the task image.

[0147] It should be noted that in the entity region prediction unit, after obtaining the entity region of the task image based on the task encoding feature, the region position of the entity in the task image can be determined, and the text content in the region position can be further identified, and the text content is the entity in the sample image.

[0148] By applying the scheme of the embodiments of the present disclosure, according to a specific information extraction task (entity recognition task), an entity region prediction unit can be additionally added on the basis of the basic information extraction model including the encoding unit and the decoding unit, and the entity region prediction unit is used for region prediction to obtain the entity extraction result of the task image, and end-to-end high-precision entity recognition is realized.

[0149] In an optional embodiment of the present disclosure, the entity category identification task can be processed by connecting the category prediction unit after the entity region prediction unit, that is, the information extraction model further comprises a category prediction unit; after the task encoding feature is input into the entity region prediction unit to obtain the entity region of the task image and the entity extraction result of the task image, the following steps can be further included:

[0150] The entity extraction result is input into the category prediction unit for category prediction to obtain the entity category of the entity extraction result.

[0151] Specifically, the entity category can be understood as the type of the entity, for example, the category of the entity "Zhang San" is a name, and the category of the entity "A City" is a place name.

[0152] By applying the scheme of the embodiments of the present disclosure, according to a specific information extraction task (entity category identification task), on the basis of the basic information extraction model including the encoding unit and the decoding unit, the entity region prediction unit and the category prediction unit are additionally added, the region prediction is performed by using the entity region prediction unit to obtain the entity extraction result of the task image, then the category prediction is performed on the entity extraction result by using the category prediction unit, and the end-to-end high-precision entity category identification is realized.

[0153] In an optional embodiment of the present disclosure, after the task encoding feature is input into the decoding unit in the information extraction model to obtain the text extraction result of the task image, the following steps can be further included:

[0154] The text extraction result is sent to the client;

[0155] The result feedback information sent by the client is received, wherein the result feedback information is information for feeding back the text extraction result based on the task information of the information extraction task;

[0156] According to the result feedback information, the model optimization data is constructed;

[0157] The information extraction model is adjusted in parameters by using the model optimization data.

[0158] Specifically, the result feedback information can be information for feeding back the content, quality and completion degree of the text extraction result, reflecting the real feelings and expectations of the user on the text extraction result, and the result feedback information includes but is not limited to result quality evaluation information, corrected accurate text extraction result and optimization field of the model. The model optimization data refers to accurate optimization sample data for optimizing the information extraction model.

[0159] In actual applications, there are various ways to construct model optimization data according to the result feedback information, which are selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this. In a possible implementation manner of the present disclosure, model optimization data can be automatically constructed according to the result feedback information. In another possible implementation manner of the present disclosure, optimization prompt information can be generated based on the result feedback information, and model optimization data sent by the client based on the optimization prompt information is received.

[0160] It should be noted that, taking the example of directly constructing model optimization data according to the result feedback information, if the result feedback information is the corrected accurate text extraction result, the model optimization data can be constructed according to the task image of the information extraction task and the corrected accurate text extraction result. If the result feedback information is the optimization field of the model, such as the XXX field, the sample image of the XXX field can be obtained, and the sample image of the XXX field is determined as the model optimization data. The process of adjusting the parameters of the information extraction model by using the model optimization data is the same as the training process of the information extraction model, and the embodiments of the present disclosure will not be repeated.

[0161] By collecting and utilizing the result feedback information, the performance of the information extraction model is continuously optimized to more accurately meet the actual needs of the user and improve the quality and accuracy of the final text extraction result by applying the scheme of the embodiments of the present disclosure.

[0162] In an optional embodiment of the present disclosure, the above constructing model optimization data according to the result feedback information can include the following steps:

[0163] Generating optimization prompt information according to the result feedback information, wherein the optimization prompt information is used to guide the client to send model optimization data for optimizing the information extraction model;

[0164] Sending the optimization prompt information to the client and receiving the model optimization data sent by the client based on the optimization prompt information.

[0165] It should be noted that there are various ways to generate optimization prompt information according to the result feedback information, which are selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this.

[0166] In a possible implementation of the present disclosure, the preset prompt information can be directly obtained, and the result feedback information is added in the preset prompt information to obtain the optimized prompt information. For example, the preset prompt information is "I am very sorry to bring you inaccurate information. Please point out where the information is inaccurate or provide the correct extraction result of the related image, and I will correct and optimize my answer as soon as possible to better serve you". The result feedback information is "the result is inaccurate", and the optimized prompt information is "I am very sorry to bring you inaccurate information. Please point out where the information is inaccurate or provide the correct extraction result of the related image, and I will correct and optimize my answer as soon as possible to better serve you".

[0167] In another possible implementation of the present disclosure, the result feedback information can be identified by type to determine the information type of the result feedback information. Further, the information type is matched with the prompt type of each prompt information in the prompt information library, and the prompt information with the same prompt type as the information type is determined as the optimized prompt information.

[0168] By applying the scheme of the present disclosure, the model optimization data is obtained through the interactive guidance mode, the interactivity between the user and the system is improved, and the user satisfaction is further improved.

[0169] In an optional embodiment of the present disclosure, after the task encoding feature input information is input into the decoding unit of the information extraction model to obtain the text extraction result of the task image, the following steps can be further included:

[0170] The key information in the text extraction result is labeled to obtain an updated text extraction result.

[0171] The updated text extraction result is sent to the client.

[0172] Specifically, the key information refers to the content in the text extraction result that is helpful for the user to understand the main idea of the result, capture the core idea, and identify important facts or details. These information is usually the core component of the text extraction result. The key information in the text extraction result includes but is not limited to keywords, topics, and entities, which are selected according to actual conditions, and the present disclosure does not make any limitation on this.

[0173] In actual application, before the key information in the text extraction result is marked to obtain the updated text extraction result, the key information in the text extraction result can be identified to determine the key information in the text extraction result. There are various ways to identify the key information in the text extraction result, which are selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this. In a possible implementation manner of the present disclosure, the text extraction result can be matched with the preset key information, and the information appearing in the text extraction result and being the same as the preset key information is determined as the key information. In another possible implementation manner of the present disclosure, the text extraction result can be input into a key information identification model to obtain the key information, wherein the key information identification model is trained based on multiple sample data and key information labels of the sample data.

[0174] It should be noted that when the key information in the text extraction result is marked, the key information can be subjected to operations such as bolding, highlighting, tilting, adding a border, and the like, and the marking manner of the key information is selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this.

[0175] By applying the scheme of the embodiments of the present disclosure, the key information in the text extraction result is marked to obtain the updated text extraction result, and the updated text extraction result is sent to the client, so as to facilitate the user to view the result and improve the user experience.

[0176] In an optional embodiment of the present disclosure, after the task coding feature input information is input into the decoding unit of the information extraction model to obtain the text extraction result of the task image, the following steps can be further included:

[0177] The text extraction result is sent to the client;

[0178] The adjustment information sent by the client is received, wherein the adjustment information is used to adjust the text extraction result;

[0179] According to the adjustment information, the text extraction result is adjusted to obtain the adjusted text extraction result.

[0180] Specifically, the adjustment information is used to describe the adjustment demand of the user on the text extraction result. The adjustment demand includes but is not limited to the adding annotation demand, the deleting content demand and the replacing content demand. Further, the adjustment demand can also be the adjustment demand for the key information in the text extraction result.

[0181] It should be noted that there are various ways to adjust the text extraction result according to the adjustment information, which are selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this. In a possible implementation manner of the present disclosure, an adjustment template (such as a content replacement template or a content deletion template) corresponding to the adjustment information can be obtained, and the text extraction result is adjusted based on the adjustment template. In another possible implementation manner of the present disclosure, a deep learning model (such as a translation model or a classification model) corresponding to the adjustment information can be determined, and the text extraction result is adjusted by using the deep learning model corresponding to the adjustment information.

[0182] By adjusting the text extraction result according to the adjustment information sent by the user, the scheme of the embodiments of the present disclosure increases the human-computer interaction and improves the adaptability and flexibility of information extraction.

[0183] Referring to FIG. 7, FIG. 7 shows a flowchart of an information processing method based on an information extraction model according to an embodiment of the present disclosure, which specifically includes the following steps:

[0184] Step 702: receiving a task generation request, wherein the task generation request includes request information.

[0185] Specifically, the information processing method based on the information extraction model can be applied to a terminal device or a model training platform. The task generation request is used to request the task information of the information extraction task. The task generation request usually contains the task type, the expected output format, and the request information. For example, when a user selects the “entity recognition” function on the front-end interface of the model training platform and uploads a task image, a task generation request can be constructed, which contains the task image, the task type (i.e., entity recognition), the size of the task image, the storage address, and other information. The request information refers to the parameters or description information related to the information extraction task carried in the task generation request. The request information includes but is not limited to the task scene identifier of the information extraction task, the task model identifier, or the training image of the information extraction task.

[0186] Step 704: obtaining an information extraction model based on the request information, wherein the information extraction model is trained based on sample image features, the sample image features are obtained by performing feature extraction on a sample mask image, and the sample mask image is obtained by performing mask processing on text content in a sample image.

[0187] In actual application, there are various ways to obtain the information extraction model based on the request information, which are selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this.

[0188] In an optional embodiment of the present disclosure, the above-mentioned obtaining the information extraction model based on the request information can include the following steps:

[0189] determine a target scene template from a plurality of preset scene templates based on the task scene identification, and find an information extraction model from a model library based on the target scene template, wherein the model library stores a plurality of deep learning models, and the request information includes a task scene identification of the information extraction task; or,

[0190] find an information extraction model from the model library based on the task model identification, wherein the request information includes a task model identification of the information extraction task.

[0191] Specifically, the task scene identification refers to a unique or specific label used to distinguish different task application scenarios. In the embodiments of the present disclosure, the task scene identification is part of the request information. Through the task scene identification, a target scene template matching the request information can be selected from a series of preset scene templates to generate task information. For example, the task scene identification is "entity recognition", which means that the user wants to perform entity recognition on the uploaded task image, and then an entity recognition scene template can be selected from a plurality of preset scene templates according to the task scene identification. The preset scene template is a standard configuration scene template defined in advance for different visual information extraction task application scenarios, and each template contains model information and task processing flow information matching the task application scenario. Through a series of preset scene templates, different scene task generation requests can be quickly responded. Different preset scene templates correspond to different task types, model information and processing flow. For example, there may be an entity recognition scene template in the preset scene template that is specifically designed for entity recognition tasks, which contains model information and processing flow of the trained entity recognition model. The target scene template refers to the scene template matching the task scene identification. When analyzing the task generation request, the corresponding target scene template can be located based on the task scene identification, and the corresponding information extraction model and other related configuration information can be selected from the model library according to the model information included in the target scene template. For example, when the task scene identification is "entity recognition", the target scene template is a template containing model information and related configuration parameters of the entity recognition model. The model library is a resource library that centrally stores deep learning models, and these models are trained and optimized to solve different information extraction tasks. The deep learning models stored in the model library include but are not limited to information extraction models, intelligent question and answer models, counterfactual processing models, and counterfactual question and answer models. Moreover, the deep learning models in the model library can be divided into different versions according to different applicable tasks, for example, the model library may store multiple versions of information extraction models such as entity recognition models, text extraction models, and entity category recognition models. The task model identification refers to a unique or specific label used to distinguish different models applicable to different tasks. For example, the task model identification can be "entity recognition", and based on the task model identification, an entity recognition model applicable to the entity recognition task can be found from the model library.

[0192] By using the scheme of the embodiments of the present disclosure, the obtaining process of the information extraction model is more flexible, efficient and standard by virtue of the pre-defined task scenario template, task model identifier and model library resource.

[0193] In another optional embodiment of the present disclosure, in addition to selecting a pre-trained information extraction model from the model library, the pre-trained information extraction model can also be fine-tuned according to the training image in the request information, that is, the request information includes a training image of the information extraction task; the obtaining of the information extraction model based on the request information can include the following steps:

[0194] Training the information extraction model corresponding to the information extraction task based on the training image to obtain an updated information extraction model.

[0195] It should be noted that the implementation of "training the information extraction model corresponding to the information extraction task based on the training image to obtain an updated information extraction model" is the same as the training method of the above-mentioned "updated information extraction model", and the embodiments of the present disclosure will not be repeated.

[0196] By using the scheme of the embodiments of the present disclosure, the information extraction model corresponding to the information extraction task is trained based on the training image to obtain an updated information extraction model, which ensures the accuracy of the information extraction model and makes the model more suitable for the task requirements.

[0197] Step 706: generating task information based on the information extraction model, wherein the task information is used to execute the information extraction task.

[0198] Specifically, the task information contains the model configuration and processing flow required for executing the information extraction task. The terminal device or other server components can correctly use the information extraction model to process the information extraction task based on the task information.

[0199] It should be noted that when generating the task information based on the information extraction model, the model parameters of the information extraction model can be directly packaged to obtain the task information. Other model information of the information extraction model can also be obtained, and the task information can be constructed based on the other model information, wherein the other model information is, for example, the processing method of the model input data, the specification of the expected output result, and the possible intermediate steps and other auxiliary information.

[0200] For example, in an entity recognition task, the task information can include address information of the selected entity recognition model, storage location of the input task image, target path of the output entity extraction result, and other environment configurations required for running the information extraction model, etc. These information enables the entity recognition model to be correctly loaded and the entity recognition task to be executed on a local or remote server.

[0201] By means of the scheme of the embodiment of the present disclosure, the task information of the information extraction task is generated, the processing quality and efficiency of the information extraction task can be ensured, the system deployment and operation and maintenance cost can be reduced, and convenient and efficient information extraction task processing services are provided for users.

[0202] Referring to FIG. 8, FIG. 8 shows a schematic diagram of a model training platform provided by an embodiment of the present disclosure. The model training platform includes a request interface unit 802 and a model unit 804.

[0203] The request interface unit 802 is configured to receive a model training request, wherein the model training request includes request information.

[0204] The model unit 804 is configured to obtain an initial extraction model and a sample image based on the request information, wherein the initial extraction model includes an encoding unit and a decoding unit, the initial extraction model is pre-configured with model parameters, and the encoding unit is connected with the decoding unit; the text content in the sample image is subjected to mask processing to obtain a sample mask image, and the sample mask image is subjected to image feature extraction to obtain a sample image feature; the sample image feature is input into the encoding unit to obtain a sample encoding feature, and the sample encoding feature is input into the decoding unit to obtain a sample prediction result; the model parameters of the initial extraction model are adjusted based on the sample prediction result to obtain a trained information extraction model.

[0205] Specifically, the model training platform can be deployed on a terminal device or a cloud device, and is specifically set according to actual conditions, which is not limited in the embodiments of the present disclosure. The model training platform provides large-scale data processing and high-performance computing resources for training, optimizing and deploying various machine learning models, especially deep learning models. On the model training platform, users can upload data, select or customize algorithm models, and efficiently perform model training and verification through distributed computing capabilities. The model training platform can receive a task generation request from a terminal device, obtain a corresponding information extraction model according to request information included in the task generation request, and generate task information based on the information extraction model. The model training platform can also quickly respond to the needs of different tasks, call appropriate information extraction models for model training, and finally generate accurate information extraction models.

[0206] It should be noted that the processing manner of the model unit is the same as the training manner of the information extraction model, and the embodiments of the present disclosure will not be described again.

[0207] In an optional embodiment of the present disclosure, the model training platform further includes a model library.

[0208] The model unit is further configured to store the trained information extraction model to the model library.

[0209] It should be noted that the model unit obtains the initial extraction model and the sample image based on the request information, trains the initial extraction model using the sample image to obtain the trained information extraction model, and then stores the trained information extraction model to the model library, so that the trained information extraction model can be directly obtained from the model library for subsequent task processing without repeated model training. The manner of obtaining the trained information extraction model from the model library can refer to the implementation manners of the above-mentioned “determining a target scene template from a plurality of preset scene templates based on a task scene identifier, and searching for an information extraction model from the model library based on the target scene template; or searching for an information extraction model from the model library based on a task model identifier”. The embodiments of the present disclosure will not be described again.

[0210] Corresponding to the above information extraction model training method embodiment, the present disclosure also provides an information extraction model training device embodiment. FIG. 9 shows a structural schematic diagram of an information extraction model training device according to an embodiment of the present disclosure. As shown in FIG. 9, the device comprises:

[0211] The first obtaining module 902 is configured to obtain an initial extraction model and a sample image, wherein the initial extraction model comprises an encoding unit and a decoding unit, and the initial extraction model is pre-configured with model parameters, and the encoding unit is connected with the decoding unit;

[0212] The processing module 904 is configured to perform mask processing on the text content in the sample image to obtain a sample mask image, and perform image feature extraction on the sample mask image to obtain a sample image feature;

[0213] The first input module 906 is configured to input the sample image feature into the encoding unit to obtain a sample encoding feature, and input the sample encoding feature into the decoding unit to obtain a sample prediction result;

[0214] The adjusting module 908 is configured to adjust the model parameters of the initial extraction model based on the sample prediction result to obtain a trained information extraction model, wherein the information extraction model is used to process the task image feature of the task image in the information extraction method to obtain the text extraction result of the task image.

[0215] Optionally, the initial extraction model further comprises a mask prediction unit, and the mask prediction unit is connected with the encoding unit; the device further comprises a third input module configured to input the sample encoding feature into the mask prediction unit for mask content prediction to obtain a mask prediction result; and the adjusting module 908 is further configured to adjust the model parameters of the initial extraction model based on the sample prediction result and the mask prediction result to obtain the trained information extraction model.

[0216] Optionally, the initial extraction model further comprises an entity prediction unit, the entity prediction unit being connected with the encoding unit; the apparatus further comprises: a fourth input module configured to input the sample encoding features into the entity prediction unit for entity prediction to obtain an entity prediction result; and an adjustment module 908 further configured to adjust the model parameters of the initial extraction model based on the sample prediction result and the entity prediction result to obtain the trained information extraction model.

[0217] Optionally, the entity prediction unit comprises an entity region prediction unit, an entity content prediction unit and an entity block prediction unit, an entity region comprising an entity block, and an entity block comprising entity content, the entity region prediction unit, the entity content prediction unit and the entity block prediction unit being connected with the encoding unit respectively; the fourth input module is further configured to input the sample encoding features into the entity region prediction unit to obtain an entity region prediction result, input the sample encoding features into the entity content prediction unit to obtain an entity content prediction result, and input the sample encoding features into the entity block prediction unit to obtain an entity block prediction result.

[0218] Optionally, the sample image carries a sample content label, a region position label and an entity relationship label; the adjustment module 908 is further configured to calculate a content prediction loss according to the sample prediction result and the sample content label, calculate a region prediction loss according to the entity region prediction result and the region position label, determine an entity mapping prediction relationship according to the entity content prediction result and the entity block prediction result, calculate a contrast learning loss according to the entity mapping prediction relationship and the entity relationship label, and adjust the model parameters of the initial extraction model according to the content prediction loss, the region prediction loss and the contrast learning loss to obtain the trained information extraction model.

[0219] Optionally, the initial extraction model further comprises a context prediction unit, the context prediction unit being connected with the encoding unit; the apparatus further comprises: a fifth input module configured to input the sample encoding features into the context prediction unit for prediction to obtain a context prediction result; and the adjustment module 908 is further configured to adjust the model parameters of the initial extraction model based on the sample prediction result and the context prediction result to obtain the trained information extraction model.

[0220] Optionally, the context prediction unit comprises a fusion encoding unit, a semantic prediction unit and a position prediction unit, the fusion encoding unit is connected with the encoding unit, and the semantic prediction unit and the position prediction unit are connected with the fusion encoding unit; the fifth input module is further configured to fuse the sample coding feature, the text coding feature and the position coding feature through the fusion encoding unit to obtain a sample fusion feature, wherein the text coding feature is obtained by extracting the sample image based on the position of the text content in the sample image; inputting the sample fusion feature into the semantic prediction unit to perform context semantic relationship prediction to obtain a first context prediction result; and inputting the sample fusion feature into the position prediction unit to perform context position relationship prediction to obtain a second context prediction result.

[0221] Optionally, the information extraction model comprises an encoding unit, a decoding unit and an entity region prediction unit, the entity region prediction unit is connected with the encoding unit; the device further comprises a fourth acquisition module configured to acquire a training image of an information extraction task; performing mask processing on the text content in the training image to obtain a training mask image, and performing image feature extraction on the training mask image to obtain a training image feature; inputting the training image feature into the encoding unit to obtain a training coding feature, and inputting the training coding feature into the decoding unit to obtain a training prediction result; inputting the training coding feature into the entity region prediction unit to obtain a training entity region prediction result; and adjusting model parameters of the information extraction model based on the training prediction result and the training entity region prediction result to obtain an updated information extraction model.

[0222] Optionally, the information extraction model further comprises a category prediction unit, the category prediction unit is connected with the entity region prediction unit; the device further comprises a sixth input module configured to input the training entity region prediction result into the category prediction unit to perform entity category prediction to obtain an entity category prediction result; and an adjustment module 908 further configured to adjust model parameters of the information extraction model based on the training prediction result, the training entity region prediction result and the entity category prediction result to obtain an updated information extraction model.

[0223] By applying the scheme of the embodiments of the present disclosure, the model can understand the image features and the text context content of the sample image by performing mask processing on the text content in the sample image and then performing model training, the precision of the information extraction model is improved, and high-precision information extraction task processing is realized.

[0224] The above is a schematic scheme of an information extraction model training device according to the present embodiment. It should be noted that the technical scheme of the information extraction model training device belongs to the same concept as the technical scheme of the information extraction model training method described above, and the details of the technical scheme of the information extraction model training device that are not described in detail can be referred to the description of the technical scheme of the information extraction model training method.

[0225] Corresponding to the information extraction method embodiments described above, the present disclosure also provides information extraction device embodiments. FIG. 10 shows a structural schematic diagram of an information extraction device according to an embodiment of the present disclosure. As shown in FIG. 10, the device comprises:

[0226] The second acquisition module 1002 is configured to acquire a task image of an information extraction task.

[0227] The extraction module 1004 is configured to perform image feature extraction on the task image to obtain task image features.

[0228] The second input module 1006 is configured to input the task image features into an encoding unit of an information extraction model to obtain task encoding features, and input the task encoding features into a decoding unit of the information extraction model to obtain a text extraction result of the task image, wherein the information extraction model is trained based on sample image features, the sample image features are obtained based on feature extraction on a sample mask image, and the sample mask image is obtained based on mask processing on text content in a sample image.

[0229] Optionally, the information extraction model further comprises an entity region prediction unit; and the device further comprises a sixth input module configured to input the task encoding features into the entity region prediction unit to obtain an entity region of the task image and an entity extraction result of the task image.

[0230] Optionally, the information extraction model further comprises a category prediction unit; and the device further comprises a seventh input module configured to input the entity extraction result into the category prediction unit for category prediction to obtain an entity category of the entity extraction result.

[0231] Optionally, the device further comprises a first sending module configured to send the text extraction result to a client, receive result feedback information sent by the client, wherein the result feedback information is information for feeding back the text extraction result based on task information of the information extraction task, construct model optimization data according to the result feedback information, and perform parameter adjustment on the information extraction model by using the model optimization data.

[0232] Optionally, the first sending module is further configured to generate optimization prompt information according to the result feedback information, wherein the optimization prompt information is used to guide the client to send model optimization data for optimizing the information extraction model, send the optimization prompt information to the client, and receive model optimization data sent by the client based on the optimization prompt information.

[0233] Optionally, the device further comprises a labeling module configured to label key information in the text extraction result to obtain an updated text extraction result, and send the updated text extraction result to the client.

[0234] Optionally, the apparatus further comprises a second sending module configured to send the text extraction result to the client; receiving adjustment information sent by the client, wherein the adjustment information is used to adjust the text extraction result; and adjusting the text extraction result according to the adjustment information to obtain an adjusted text extraction result.

[0235] By applying the scheme of the embodiments of the present disclosure, since the information extraction model is a high-precision model, the information extraction model can be used to implement a high-precision end-to-end information extraction task processing, and further improve the information extraction efficiency and the accuracy of the text extraction result.

[0236] The above is a schematic scheme of the information extraction apparatus of the present embodiment. It should be noted that the technical scheme of the information extraction apparatus belongs to the same concept as the technical scheme of the information extraction method described above, and the details of the technical scheme of the information extraction apparatus that are not described in detail can be referred to the description of the technical scheme of the information extraction method.

[0237] Corresponding to the information processing method based on the information extraction model described above, the present disclosure further provides an information processing apparatus based on the information extraction model. FIG. 11 shows a structural schematic diagram of an information processing apparatus based on the information extraction model according to an embodiment of the present disclosure. As shown in FIG. 11, the apparatus comprises:

[0238] The receiving module 1102 is configured to receive a task generation request, wherein the task generation request comprises request information;

[0239] The third obtaining module 1104 is configured to obtain an information extraction model based on the request information, wherein the information extraction model is obtained based on sample image features, the sample image features are obtained based on feature extraction on a sample mask image, and the sample mask image is obtained based on mask processing on text content in a sample image;

[0240] The generating module 1106 is configured to generate task information based on the information extraction model, wherein the task information is used to execute an information extraction task.

[0241] Optionally, the third obtaining module 1104 is further configured to determine a target scene template from a plurality of preset scene templates based on a task scene identifier, and find the information extraction model from a model library based on the target scene template, wherein the model library stores a plurality of deep learning models, and the request information comprises a task scene identifier of the information extraction task; or find the information extraction model from the model library based on a task model identifier, wherein the request information comprises a task model identifier of the information extraction task.

[0242] Optionally, the request information includes a training image of the information extraction task; the third acquisition module 1104 is further configured to train an information extraction model corresponding to the information extraction task based on the training image, and obtain an updated information extraction model.

[0243] By generating the task information of the information extraction task, the scheme of the embodiments of the present disclosure can reduce the system deployment and operation and maintenance costs while ensuring the processing quality and efficiency of the information extraction task, and provide convenient and efficient information extraction task processing services for users.

[0244] The above is a schematic scheme of the information processing device based on the information extraction model according to the present embodiment. It should be noted that the technical scheme of the information processing device based on the information extraction model belongs to the same concept as the technical scheme of the information processing method based on the information extraction model described above. The details of the technical scheme of the information processing device based on the information extraction model that are not described in detail can be seen from the description of the technical scheme of the information processing method based on the information extraction model described above.

[0245] FIG. 12 shows a structural block diagram of a computing device according to an embodiment of the present disclosure. The components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 through a bus 1230, and a database 1250 is used to save data.

[0246] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of such networks, such as the Internet. The access device 1240 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, or the like.

[0247] In one embodiment of the present disclosure, the above-mentioned components of the computing device 1200 and other components not shown in FIG. 12 can also be connected to each other, for example, through a bus. It should be understood that the computing device structure block diagram shown in FIG. 12 is only for the purpose of example, and is not a limitation on the scope of the present disclosure. Those skilled in the art can add or replace other components as needed.

[0248] The computing device 1200 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1200 can also be a mobile or stationary server.

[0249] The processor 1220 is configured to execute computer program / instructions that implement the steps of the above-mentioned information extraction model training method or information extraction method or information processing method based on the information extraction model.

[0250] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the information extraction model training method, the information extraction method, and the information processing method based on the information extraction model belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the information extraction model training method or the information extraction method or the information processing method based on the information extraction model.

[0251] The embodiment of the present disclosure also provides a computer readable storage medium storing computer programs / instructions, which, when executed by a processor, implement the steps of the information extraction model training method or the information extraction method or the information processing method based on the information extraction model.

[0252] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical schemes of the information extraction model training method, the information extraction method, and the information processing method based on the information extraction model belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the information extraction model training method or the information extraction method or the information processing method based on the information extraction model.

[0253] The embodiment of the present disclosure also provides a computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the information extraction model training method or the information extraction method or the information processing method based on the information extraction model.

[0254] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical schemes of the information extraction model training method, the information extraction method, and the information processing method based on the information extraction model belong to the same concept, and the details of the technical scheme of the computer program product that are not described in detail can be referred to the description of the technical scheme of the information extraction model training method or the information extraction method or the information processing method based on the information extraction model.

[0255] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still accomplish desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve desirable results. In some implementations, multitasking and parallel processing can be advantageous.

[0256] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or subtractions according to the requirements of patent practice. For example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0257] It should be noted that, for the foregoing method embodiments, in order to facilitate description, each is described as a combination of a series of acts, but those skilled in the art should appreciate that the disclosed embodiments are not limited to the order of the acts described, because, according to the disclosed embodiments, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should appreciate that the embodiments described in the specification are all preferred embodiments, and the acts and modules involved are not necessarily essential to the disclosed embodiments.

[0258] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0259] The preferred embodiments of the disclosure disclosed above are only used to help explain the disclosure. The alternative embodiments do not describe all the details and do not limit the invention to the specific embodiments described. Obviously, according to the content of the disclosed embodiments, many modifications and changes can be made. The disclosure selects and describes these embodiments in order to better explain the principles and practical applications of the disclosed embodiments, so that those skilled in the art can well understand and utilize the disclosure. The disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. An information extraction method, comprising: The task image for information extraction; Image feature extraction is performed on the task image to obtain task image features; The task image features are input into the encoding unit of the information extraction model to obtain task encoding features, and the task encoding features are input into the decoding unit of the information extraction model to obtain the text extraction result of the task image. The information extraction model is trained based on sample image features, the sample image features are obtained by feature extraction from a sample mask image, and the sample mask image is obtained by masking the text content in the sample image.

2. The method according to claim 1, wherein the information extraction model further includes an entity region prediction unit; After obtaining the task coding features by inputting the task image features into the coding unit of the information extraction model, the method further includes: The task encoding features are input into the entity region prediction unit to obtain the entity regions of the task image and the entity extraction results of the task image.

3. The method according to claim 2, wherein the information extraction model further includes a category prediction unit; After inputting the task encoding features into the entity region prediction unit to obtain the entity regions of the task image and the entity extraction results of the task image, the method further includes: The entity extraction result is input into the category prediction unit for category prediction to obtain the entity category of the entity extraction result.

4. The method according to any one of claims 1 to 3, after inputting the task encoding features into the decoding unit in the information extraction model to obtain the text extraction result of the task image, it further includes: The text extraction results are sent to the client. Receive result feedback information sent by the client, wherein the result feedback information is information that provides feedback on the text extraction result based on the task information of the information extraction task; Based on the feedback information, construct model optimization data; The model is used to optimize the data, and the parameters of the information extraction model are adjusted.

5. The method according to claim 4, wherein constructing model optimization data based on the result feedback information includes: Based on the feedback information, an optimization prompt is generated, wherein the optimization prompt is used to guide the client to send model optimization data for optimizing the information extraction model; The optimization prompt information is sent to the client, and the model optimization data sent by the client based on the optimization prompt information is received.

6. The method according to any one of claims 1 to 5, wherein after inputting the task encoding features into the decoding unit in the information extraction model to obtain the text extraction result of the task image, the method further includes: The key information in the extracted text is labeled to obtain the updated extracted text. The updated text extraction results are sent to the client.

7. The method according to any one of claims 1 to 6, wherein after inputting the task encoding features into the decoding unit in the information extraction model to obtain the text extraction result of the task image, the method further comprises: The text extraction results are sent to the client. The system receives adjustment information sent by the client, wherein the adjustment information is used to adjust the text extraction result. Based on the adjustment information, the text extraction result is adjusted to obtain the adjusted text extraction result.

8. A method for training an information extraction model, comprising: An initial extraction model and sample images are obtained, wherein the initial extraction model includes an encoding unit and a decoding unit, the initial extraction model is pre-configured with model parameters, and the encoding unit is connected to the decoding unit; The text content in the sample image is masked to obtain a sample mask image, and image features are extracted from the sample mask image to obtain sample image features. The sample image features are input into the encoding unit to obtain sample encoding features, and the sample encoding features are input into the decoding unit to obtain sample prediction results; Based on the sample prediction results, the model parameters of the initial extraction model are adjusted to obtain the trained information extraction model, wherein the information extraction model is used to process the task image features of the task image in the information extraction method according to any one of claims 1 to 7 to obtain the text extraction result of the task image.

9. The method according to claim 8, wherein the initial extraction model further comprises an entity prediction unit, the entity prediction unit being connected to the encoding unit; After inputting the sample image features into the encoding unit to obtain the sample encoded features, the method further includes: The sample encoding features are input into the entity prediction unit to perform entity prediction and obtain the entity prediction result. The step of adjusting the model parameters of the initial extraction model based on the sample prediction results to obtain the trained information extraction model includes: Based on the sample prediction results and the entity prediction results, the model parameters of the initial extraction model are adjusted to obtain the trained information extraction model.

10. The method according to claim 9, wherein the entity prediction unit comprises an entity region prediction unit, an entity content prediction unit, and an entity block prediction unit, wherein the entity region includes the entity block, the entity block includes the entity content, and the entity region prediction unit, the entity content prediction unit, and the entity block prediction unit are respectively connected to the encoding unit; The step of inputting the sample encoded features into the entity prediction unit to perform entity prediction and obtain entity prediction results includes: The sample encoding features are input into the entity region prediction unit to obtain the entity region prediction result. The sample encoding features are input into the entity content prediction unit to obtain the entity content prediction result; The sample encoding features are input into the entity block prediction unit to obtain the entity block prediction result.

11. The method according to claim 10, wherein the sample image carries sample content tags, region location tags, and entity relationship tags; The step of adjusting the model parameters of the initial extraction model based on the sample prediction results and the entity prediction results to obtain the trained information extraction model includes: Calculate the content prediction loss based on the sample prediction results and the sample content labels; Calculate the region prediction loss based on the entity region prediction result and the region location label; Based on the entity content prediction results and the entity block prediction results, the entity mapping prediction relationship is determined; Calculate the contrastive learning loss based on the entity mapping prediction relationship and the entity relationship label; Based on the content prediction loss, the region prediction loss, and the contrastive learning loss, the model parameters of the initial extraction model are adjusted to obtain the trained information extraction model.

12. The method according to any one of claims 8 to 11, wherein the initial extraction model further comprises a mask prediction unit, the mask prediction unit being connected to the encoding unit; After inputting the sample image features into the encoding unit to obtain the sample encoded features, the method further includes: The sample encoding features are input into the mask prediction unit to predict the mask content and obtain the mask prediction result. The step of adjusting the model parameters of the initial extraction model based on the sample prediction results to obtain the trained information extraction model includes: Based on the sample prediction results and the mask prediction results, the model parameters of the initial extraction model are adjusted to obtain the trained information extraction model.

13. The method according to any one of claims 8 to 12, wherein the initial extraction model further comprises a context prediction unit, the context prediction unit being connected to the encoding unit; After inputting the sample image features into the encoding unit to obtain the sample encoded features, the method further includes: The sample encoded features are input into the context prediction unit for prediction to obtain the context prediction result; The step of adjusting the model parameters of the initial extraction model based on the sample prediction results to obtain the trained information extraction model includes: Based on the sample prediction results and the context prediction results, the model parameters of the initial extraction model are adjusted to obtain the trained information extraction model.

14. The method according to claim 13, wherein the context prediction unit comprises a fusion coding unit, a semantic prediction unit, and a location prediction unit, the fusion coding unit being connected to the coding unit, and the semantic prediction unit and the location prediction unit being respectively connected to the fusion coding unit; The step of inputting the sample encoded features into the context prediction unit for prediction to obtain the context prediction result includes: The fusion encoding unit fuses the sample encoding features, text encoding features, and position encoding features to obtain sample fusion features, wherein the text encoding features are extracted from the sample image based on the position of the text content in the sample image; The sample fusion features are input into the semantic prediction unit to predict the contextual semantic relationship, and a first contextual prediction result is obtained. The sample fusion features are input into the location prediction unit to predict the contextual location relationship, thereby obtaining a second contextual prediction result.

15. The method according to any one of claims 8 to 14, wherein the information extraction model comprises the encoding unit, the decoding unit, and the entity region prediction unit, and the entity region prediction unit is connected to the encoding unit; After adjusting the model parameters of the initial extraction model based on the sample prediction results to obtain the trained information extraction model, the process further includes: Acquire training images for the information extraction task; The text content in the training image is masked to obtain a training mask image, and image features are extracted from the training mask image to obtain training image features. The training image features are input into the encoding unit to obtain training encoded features, and the training encoded features are input into the decoding unit to obtain training prediction results; The trained encoded features are input into the entity region prediction unit to obtain the trained entity region prediction result; Based on the training prediction results and the training entity region prediction results, the model parameters of the information extraction model are adjusted to obtain the updated information extraction model.

16. The method according to claim 15, wherein the information extraction model further comprises a category prediction unit, the category prediction unit being connected to the entity region prediction unit; After inputting the trained encoded features into the entity region prediction unit to obtain the trained entity region prediction result, the method further includes: The training entity region prediction result is input into the category prediction unit to perform entity category prediction and obtain the entity category prediction result. The step of adjusting the model parameters of the information extraction model based on the training prediction results and the training entity region prediction results to obtain an updated information extraction model includes: Based on the training prediction results, the training entity region prediction results, and the entity category prediction results, the model parameters of the information extraction model are adjusted to obtain the updated information extraction model.

17. An information processing method based on an information extraction model, comprising: Receive a task generation request, wherein the task generation request includes request information; Based on the request information, an information extraction model is obtained, wherein the information extraction model is trained based on sample image features, the sample image features are obtained based on feature extraction of a sample mask image, and the sample mask image is obtained based on masking the text content in the sample image; Based on the information extraction model, task information is generated, wherein the task information is used to perform the information extraction task.

18. The method according to claim 17, wherein obtaining the information extraction model based on the request information comprises: Based on the task scenario identifier, a target scenario template is determined from multiple preset scenario templates. Then, based on the target scenario template, an information extraction model is retrieved from a model library, wherein the model library stores multiple deep learning models, and the request information includes the task scenario identifier of the information extraction task; or, Based on the task model identifier, an information extraction model is searched from the model library, wherein the request information includes the task model identifier of the information extraction task.

19. The method according to claim 17 or 18, wherein the request information includes training images of the information extraction task; The step of obtaining the information extraction model based on the request information includes: Based on the training images, the information extraction model corresponding to the information extraction task is trained to obtain an updated information extraction model.

20. A model training platform, comprising a request interface unit and a model unit; The request interface unit is used to receive model training requests, wherein... The model training request includes request information; The model unit is used to obtain an initial extraction model and sample images based on the request information. The initial extraction model includes an encoding unit and a decoding unit, and the initial extraction model is pre-configured with model parameters. The encoding unit is connected to the decoding unit. The text content in the sample image is masked to obtain a sample mask image, and image features are extracted from the sample mask image to obtain sample image features. The sample image features are input into the encoding unit to obtain sample encoded features, and the sample encoded features are input into the decoding unit to obtain sample prediction results. Based on the sample prediction results, the model parameters of the initial extraction model are adjusted to obtain a trained information extraction model.

21. The model training platform according to claim 20 further includes a model library; The model unit is also used to store the trained information extraction model into the model library.

22. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 7, any one of claims 8 to 16, or any one of claims 17 to 19.

23. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7, any one of claims 8 to 16, or any one of claims 17 to 19.

24. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7, any one of claims 8 to 16, or any one of claims 17 to 19.

Images