A pre-training method and related methods and devices

Abstract

This invention provides a pre-training method and related methods and devices. The pre-training method includes: acquiring a training dataset; performing text detection and recognition on training document images in the training dataset to obtain text detection boxes and their corresponding text; using the training document images as input to a pre-trained model; acquiring features of the text detection boxes based on the pre-trained model; acquiring semantic features of the text corresponding to the text detection boxes based on a text encoder; aiming to make the features of the text detection boxes approximate the semantic features of the text corresponding to the text detection boxes; updating the parameters of the pre-trained model; and using the trained pre-trained model as the target pre-trained model. Since the target pre-trained model takes document images as input, it can mine text semantic information from the document images and output features rich in text semantics without needing to input text. Therefore, when applied to downstream tasks, it eliminates the need for text recognition, thus saving OCR engine overhead.

Description

Technical Field

[0001] This invention relates to the field of unsupervised learning technology, and in particular to a pre-training method and related methods and devices. Background Technology

[0002] The general approach to obtaining an information prediction model that includes the encoding component is as follows: first, an initial information prediction model is constructed based on the target pre-trained model and the prediction module; then, the initial information prediction model is fine-tuned using training data for the specified task, thereby obtaining the final information prediction model for the specified task. The target pre-trained model is obtained by training the initial pre-trained model using unsupervised data.

[0003] Document intelligence technology is widely used in industries such as finance, insurance, energy, logistics, and healthcare. Currently, for tasks involving document images, such as text line detection, document classification, and region of interest classification, the target pre-trained model is usually obtained through the following pre-training method: First, text recognition is performed on the training document image based on an OCR (Optical Character Recognition) engine. Then, part of the recognized text is masked. Finally, the masked text is used in conjunction with a text prediction task (predicting the masked content) to train the initial pre-trained model, thereby obtaining the target pre-trained model.

[0004] After obtaining the target pre-trained model through the above methods, it can be applied to downstream tasks (such as text line detection, document classification, etc.). Understandably, when applying the target pre-trained model obtained through the above methods to downstream tasks, it is still necessary to use an OCR engine for text recognition. This means that if the downstream uses the target pre-trained model obtained through the above methods, it will inevitably incur some OCR engine overhead. Furthermore, for some downstream tasks, such as document detection and document classification, the text content in the document image is not required. Summary of the Invention

[0005] In view of this, the present invention provides a pre-training method and related methods and devices to solve the problem that applying a target pre-trained model obtained through existing pre-training methods to downstream tasks will incur some OCR engine overhead. The technical solution is as follows:

[0006] A pre-training method, comprising:

[0007] Obtain a training dataset, wherein the training dataset includes multiple training document images;

[0008] Text detection and recognition are performed on the training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes;

[0009] The training document image is used as the input of the pre-trained model. The features of the text detection box are obtained based on the pre-trained model. The text corresponding to the text detection box is used as the input of the text encoder. The semantic features of the text corresponding to the text detection box are obtained based on the text encoder. The pre-trained model includes a visual encoder. The text encoder is trained in advance using multiple training texts.

[0010] With the goal of making the features of the text detection box approach the semantic features of the text corresponding to the text detection box, the parameters of the pre-trained model are updated, and the pre-trained model obtained is used as the target pre-trained model.

[0011] Optionally, the step of using the training document image as input to the pre-trained model and obtaining the features of the text detection box based on the pre-trained model includes:

[0012] A visual encoder based on a pre-trained model encodes the training document images to obtain the features of the training document images;

[0013] Based on the features of each text detection box and the training document image, the features of each text detection box are obtained.

[0014] Optionally, the pre-trained model may also include: a first feature processing module and a second feature processing module;

[0015] The step of using the training document image as input to the pre-trained model and obtaining the features of the text detection box based on the pre-trained model includes:

[0016] A visual encoder based on a pre-trained model encodes the training document images to obtain the features of the training document images;

[0017] Based on the features of each text detection box and the training document image, obtain the first feature of each text detection box;

[0018] The first feature processing module based on the pre-trained model processes the first feature of each text detection box to obtain the second feature of each text detection box. The second feature of a text detection box contains the temporal information of the text in the text detection box.

[0019] The second feature processing module based on the pre-trained model processes the second features of each text detection box to obtain the third feature of each text detection box, which serves as the final feature of each text detection box. The third feature of a text detection box contains the dependency information between the text detection box and other text detection boxes.

[0020] Optionally, the second feature processing module based on the pre-trained model processes the second features of each text detection box to obtain the third feature of each text detection box, including:

[0021] For each text detection box:

[0022] The second feature processing module based on the pre-trained model determines the relevance weights of the text detection box to each other, so as to obtain the relevance weights corresponding to each text detection box.

[0023] The third feature of each text detection box is obtained by weighting and summing the second features of each text detection box according to the relevance weights corresponding to each text detection box.

[0024] Optionally, updating the parameters of the pre-trained model with the goal of making the features of the text detection box approximate the semantic features of the text corresponding to the text detection box includes:

[0025] For each text detection box, based on the features of the text detection box and the semantic features of the text corresponding to the text detection box, the feature prediction loss of the pre-trained model on the text detection box is determined.

[0026] The parameters of the pre-trained model are updated based on the feature prediction loss of the pre-trained model on each text detection box.

[0027] Optionally, determining the feature prediction loss of the pre-trained model on the text detection box based on the features of the text detection box and the semantic features of the text corresponding to the text detection box includes:

[0028] Calculate the mean squared error between the features of the text detection box and the semantic features of the text corresponding to the text detection box, and use it as the feature prediction loss of the pre-trained model on the text detection box.

[0029] The step of updating the parameters of the pre-trained model based on the feature prediction loss of the pre-trained model on each text detection box includes:

[0030] The feature prediction losses of the pre-trained model on each text detection box are fused to obtain the fused loss.

[0031] The parameters of the pre-trained model are updated based on the fusion loss.

[0032] A method for obtaining an information prediction model, comprising:

[0033] An initial information prediction model is constructed based on the target pre-trained model and the prediction module for the specified task, wherein the target pre-trained model is trained using any of the pre-training methods described above;

[0034] The initial information prediction model is fine-tuned using training document images with labeled data for the specified task to obtain an information prediction model for the specified task.

[0035] Optionally, the specified task is a detection task for document images;

[0036] The initial information prediction model is constructed based on the target pre-trained model and the prediction module for the specified task, including:

[0037] An initial detection model is constructed based on the visual encoder in the target pre-trained model and the prediction module for the detection task.

[0038] The step of fine-tuning the initial information prediction model using training document images with labeled data for the specified task includes:

[0039] The initial detection model is fine-tuned using training document images with labeled data specific to the detection task.

[0040] Optionally, the specified task is a classification task targeting regions of interest in a document image;

[0041] The target pre-trained model includes a first feature processing module and a second feature processing module;

[0042] The initial information prediction model is constructed based on the target pre-trained model and the prediction module for the specified task, including:

[0043] Based on the visual encoder, the first feature processing module and the second feature processing module in the target pre-trained model, and the prediction module for the classification task, an initial region of interest classification model is constructed.

[0044] The step of fine-tuning the initial information prediction model using training document images with labeled data for the specified task includes:

[0045] The initial region of interest (ROI) classification model is fine-tuned using training document images with ROIs and categories labeled with those ROIs.

[0046] An information prediction method, comprising:

[0047] Retrieve the target document image for a specified task;

[0048] Based on the information prediction model obtained by the information prediction model acquisition method described above, the target document image is processed to obtain the information prediction result corresponding to the target document image on the specified task.

[0049] A pre-training device includes: a training data acquisition module and a model training module;

[0050] The training data acquisition module is used to acquire a training dataset, wherein the training dataset includes multiple training document images;

[0051] The model training module is used to perform text detection and recognition on training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes. The training document images are used as input to the pre-trained model to obtain the features of the text detection boxes. The text corresponding to the text detection boxes is used as input to the text encoder to obtain the semantic features of the text corresponding to the text detection boxes. The goal is to make the features of the text detection boxes approach the semantic features of the text corresponding to the text detection boxes. The pre-trained model is then updated with parameters, and the trained pre-trained model is used as the target pre-trained model. The pre-trained model includes a visual encoder, and the text encoder is pre-trained using multiple training texts.

[0052] A processing device, comprising: a memory and a processor;

[0053] The memory is used to store programs;

[0054] The processor is configured to execute the program to implement each step of the pre-training method described above.

[0055] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the pre-training method described in any of the preceding claims.

[0056] The pre-training method provided by this invention, after obtaining the training dataset, first performs text detection and recognition on the training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes. Then, the training document images are used as input to the pre-training model to obtain the features of the text detection boxes based on the pre-training model. The text corresponding to the text detection boxes is used as input to the text encoder to obtain the semantic features of the text corresponding to the text detection boxes based on the text encoder. Finally, with the goal of making the features of the text detection boxes approach the semantic features of the text corresponding to the text detection boxes, the parameters of the pre-training model are updated to obtain the target pre-training model. The pre-training method provided by this invention obtains the semantic features of the text corresponding to the text detection box based on a text encoder. The semantic features obtained based on the text encoder are used as the learning target of the pre-training model to guide the pre-training model to mine the text semantic information in the input document image. The pre-trained model trained in this way has the ability to mine the text semantic information from the input document image and output features rich in text semantic information. Since the target pre-training model takes the document image as input, it can mine the text semantic information from the document image and output features rich in text semantic information without inputting the text in the document image. Therefore, when applied to downstream tasks, it is not necessary to perform text recognition based on an OCR engine, thus saving the OCR engine overhead.

[0057] Based on the pre-training method provided by this invention, this invention also provides an information prediction model acquisition method and an information prediction method. Since the target pre-trained model obtained by the pre-training method provided by this invention takes a document image as input, it can mine textual semantic information from the document image and then output features rich in textual semantic information without inputting the text in the document image. Therefore, there is no need to perform text recognition based on an OCR engine during the acquisition of the information prediction model. Correspondingly, there is no need to perform text recognition based on an OCR engine when performing information prediction based on the acquired information prediction model, thereby saving OCR engine overhead. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of a hardware architecture according to an embodiment of the present invention;

[0060] Figure 2 A flowchart illustrating the pre-training method provided in an embodiment of the present invention;

[0061] Figure 3 This is another flowchart illustrating the pre-training method provided in an embodiment of the present invention;

[0062] Figure 4 This is an example of training a pre-trained model using a text encoder, as provided in an embodiment of the present invention.

[0063] Figure 5 This is another flowchart illustrating the pre-training method provided in an embodiment of the present invention;

[0064] Figure 6 This is another example of training a pre-trained model using a text encoder, as provided in this embodiment of the invention.

[0065] Figure 7 This is a schematic diagram of the pre-training device provided in an embodiment of the present invention;

[0066] Figure 8 This is a schematic diagram of the processing device provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] In the process of realizing this case, the inventors discovered that there are currently three main types of pre-training methods applied to tasks involving document images: the first is a pre-training method based on plain text, the second is a pre-training method based on plain text and spatial information, and the third is a pre-training method based on plain text, spatial information, and image information.

[0069] The plain text-based pre-training method first performs text recognition on the training document image using an OCR engine, then masks part of the recognized text, inputs the masked text into the pre-trained model to obtain the features of the input text, predicts the masked text based on the features of the input text to obtain the prediction result, determines the prediction loss based on the prediction result and the actual masked text, and updates the parameters of the pre-trained model based on the prediction loss.

[0070] The difference between pre-training methods based on plain text and spatial information and those based on plain text lies in the fact that, in addition to using text recognition obtained from training document images based on an OCR engine, the training process also incorporates spatial information of the text, such as the text's location information.

[0071] The difference between pre-training methods based on plain text, spatial information, and image information and those based on plain text and spatial information is that, in addition to using text obtained from text recognition of training document images based on OCR engines and the spatial information of the text, image-level information, such as image information of some text in the document images, is also incorporated during the training process.

[0072] Analyzing the above pre-training methods, it is not difficult to find that all of them require the use of text in the training document images (the model inputs all contain text), that is, they all require text recognition of the training document images based on the OCR engine.

[0073] Since text is required during pre-training (text needs to be input into the pre-trained model for training), text is also required when applying the trained pre-trained model to downstream tasks. This means that applying the target pre-trained model obtained through any of the above training methods to downstream tasks will inevitably incur some OCR engine overhead. For some downstream tasks, such as document detection and document classification, the text content in the document image is not required.

[0074] In view of this, the inventors of this case attempted to propose a pre-training method that would not incur overhead for downstream OCR engines. To this end, they conducted research and, through continuous research, finally proposed a pre-training method. The overall concept of this pre-training method is as follows: the pre-trained model is used as the student model, and the pre-trained text encoder is used as the teacher model. The text recognized from the document image is encoded based on the text encoder to obtain the semantic features of the text. The semantic features of the text are used as the learning target of the pre-trained model to guide the pre-trained model to mine the semantic information of the text from the document image.

[0075] Before introducing the pre-training method provided by this invention, the hardware architecture involved in this invention will be described first.

[0076] In one possible implementation, such as Figure 1 As shown, the hardware architecture involved in this invention may include: electronic device 101 and server 102.

[0077] For example, electronic device 101 can be any electronic product that can interact with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as personal computer, laptop, tablet, mobile phone, smart TV, etc.

[0078] It should be noted that, Figure 1 This is just one example; there can be many types of electronic devices, not limited to... Figure 1 The laptop in the middle.

[0079] For example, server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center. Server 102 may include processors, memory, and network interfaces, etc.

[0080] For example, electronic device 101 can establish a connection and communicate with server 102 through a wireless communication network; for example, electronic device 101 can establish a connection and communicate with server 102 through a wired network.

[0081] Electronic device 101 can acquire a training dataset and send the training data to server 102. Server 102 uses the acquired training dataset to perform pre-training according to the pre-training method provided by the present invention.

[0082] In another possible implementation, the hardware architecture involved in this invention may include an electronic device. The electronic device is one with strong data processing capabilities.

[0083] For example, an electronic device can be any electronic product that can interact with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as a personal computer, laptop, tablet, mobile phone, smart TV, etc.

[0084] The electronic device can acquire a training dataset and use the acquired training dataset to perform pre-training according to the pre-training method provided in this invention.

[0085] Those skilled in the art should understand that the above-described electronic devices and servers are merely examples, and other existing or future electronic devices or servers that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0086] The pre-training method provided by the present invention will be described in the following embodiments.

[0087] Please see Figure 2 The diagram illustrates a flowchart of a pre-training method provided in an embodiment of the present invention, which may include:

[0088] Step S201: Obtain the training dataset.

[0089] The training dataset includes multiple training document images.

[0090] Step S202: Perform text detection and recognition on the training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes.

[0091] Step S203a: Use the training document image as input to the pre-trained model, and obtain the features of the text detection box based on the pre-trained model.

[0092] The pre-trained model in this embodiment may include only a visual encoder, or it may include a visual encoder and a feature processing module that processes the features output by the visual encoder. In practical applications, the modules included in the pre-trained model depend on the downstream task. For example, if the downstream task is a detection task targeting the entire document image (such as text line detection, title detection, etc.), the pre-trained model may only include a visual encoder. If the downstream task is a classification task targeting regions of interest in a document image, the pre-trained model may include a visual encoder and a feature processing module.

[0093] It should be noted that there may be one or more text detection boxes. If there is only one text detection box, the features of this text detection box are obtained based on the pre-trained model. If there are multiple text detection boxes, the features of each text detection box are obtained based on the pre-trained model.

[0094] Step S203b: Use the text corresponding to the text detection box as the input of the text encoder, and obtain the semantic features of the text corresponding to the text detection box based on the text encoder.

[0095] The text encoder is trained using multiple training texts and has the ability to encode the input text and output its semantic features.

[0096] The text corresponding to the text detection box is input into the text encoder. The text encoder encodes the input text and outputs the semantic features of the text corresponding to the text detection box.

[0097] As mentioned above, there may be one or more text detection boxes. If there is only one text detection box, the semantic features of the text corresponding to this text detection box are obtained based on the text encoder. If there are multiple text detection boxes, the semantic features of the text corresponding to each text detection box are obtained based on the text encoder.

[0098] Step S204: With the goal of making the features of the text detection box approach the semantic features of the text corresponding to the text detection box, update the parameters of the pre-trained model, and use the trained pre-trained model as the target pre-trained model.

[0099] If there is only one text detection box, the parameters of the pre-trained model are updated with the goal of making the features of the text detection box approximate the semantic features of the text corresponding to the text detection box. For example, if there is only one text detection box, named Box1, and the text corresponding to Box1 is Text1, then the parameters of the pre-trained model are updated with the goal of making the features of the text detection box Box1 approximate the semantic features of the text Text1.

[0100] If there are multiple text detection boxes, the parameters of the pre-trained model are updated with the goal of making the features of each text detection box approximate the semantic features of the corresponding text. For example, there are two text detection boxes, Box1 and Box2, where the text corresponding to Box1 is Text1 and the text corresponding to Box2 is Text2. The parameters of the pre-trained model are updated with the goal of making the features of text detection box Box1 approximate the semantic features of text Text1, and the features of text detection box Box2 approximate the semantic features of text Text2.

[0101] This invention uses a pre-trained model as the student model and a text encoder as the teacher model. The semantic features output after encoding the text corresponding to the text detection boxes using the text encoder are used as the learning objective of the pre-trained model. This guides the pre-trained model to mine semantic information from the text detection boxes, enabling it to output features rich in semantic information. Furthermore, since the encoder in the pre-trained model is a visual encoder, the final features output by the pre-trained model are rich not only in semantic information but also in visual information and document layout information.

[0102] The pre-training method provided in this embodiment of the invention, after obtaining the training dataset, first performs text detection and recognition on the training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes. Then, the training document images are used as input to the pre-training model to obtain the features of the text detection boxes based on the pre-training model. The text corresponding to the text detection boxes is used as input to the text encoder to obtain the semantic features of the text corresponding to the text detection boxes based on the text encoder. Finally, with the goal of making the features of the text detection boxes approach the semantic features of the text corresponding to the text detection boxes, the parameters of the pre-training model are updated to obtain the target pre-training model. The pre-training method provided in this invention obtains semantic features of the text corresponding to the text detection box based on a text encoder. The semantic features obtained based on the text encoder are used as the learning target of the pre-training model to guide the pre-training model to mine the text semantic information in the input document image. The pre-trained model trained in this way has the ability to mine text semantic information from the input document image and output features rich in text semantic information. Since the target pre-training model takes the document image as input, it can mine text semantic information from the document image and output features rich in text semantic information without needing to input the text in the document image. Therefore, when applied to downstream tasks, it does not need to perform text recognition based on an OCR engine, thus saving OCR engine overhead.

[0103] As mentioned in the above embodiments, the pre-trained model may include only a visual encoder, or it may include both a visual encoder and a feature processing module. Based on the above embodiments, this embodiment takes a pre-trained model that includes only a visual encoder as an example to introduce the pre-training method.

[0104] Please see Figure 3 The diagram illustrates a specific example of the pre-training method provided in this embodiment, which may include:

[0105] Step S301: Obtain the training dataset.

[0106] The training dataset includes multiple training document images.

[0107] Step S302: Obtain training document images from the training dataset.

[0108] Step S303: Perform text detection and recognition on the training document image to obtain several text detection boxes and the text corresponding to each text detection box.

[0109] The text corresponding to each of the several text detection boxes can be obtained based on the OCR engine.

[0110] For example, such as Figure 4As shown, text detection and recognition are performed on the training document image, resulting in two text detection boxes, Box1 and Box2, and the corresponding texts, Text1 and Text2, for Box1 and Box2, respectively.

[0111] Step S304a1: Using the training document images as input to the pre-trained model, the visual encoder based on the pre-trained model encodes the training document images to obtain the features of the training document images.

[0112] training document images (H and W are the height and width of the image, respectively) Input the image into the pre-trained model. The visual encoder in the pre-trained model encodes the input training document image and outputs the features of the training document image. (C is the number of feature channels.)

[0113] The visual encoder in the pre-trained model can be ResNet. Of course, this embodiment is not limited to this. Other visual encoders, such as Vision-Transformer, can also be used in the pre-trained model.

[0114] Step S304a2: Based on the features of each text detection box and the training document image, obtain the features of each text detection box.

[0115] RoIAlign can be used to perform feature pooling on the features of the training document image based on each text detection box, thereby obtaining the features of each text detection box:

[0116] F Boxi =RoIAlign(F,Boxi),i∈[1,N](1)

[0117] Among them, F Boxi Let N represent the feature of the i-th text detection box, and N represent the total number of text detection boxes.

[0118] like Figure 4 As shown, based on the text detection boxes Box1 and Box2 and the features F of the training document image, the features F of the text detection box Box1 can be obtained. box1 Features F of the text detection box Box2 box2 .

[0119] Step S304b: Using the text corresponding to the text detection box as the input of the text encoder, obtain the semantic features of the text corresponding to each text detection box based on the text encoder.

[0120] The text encoder is pre-trained using multiple training texts and has the ability to encode the input text and output its semantic features. In this embodiment, the text encoder can be, but is not limited to, BERT, GPT, etc.

[0121] like Figure 4 As shown, the text Text1 corresponding to the text detection box Box1 is input into the text encoder. The text encoder encodes the text Text1 corresponding to Box1 and outputs the semantic features F of the text Text1 corresponding to Box1. Text1 The text text2 corresponding to the text detection box Box2 is input into the text encoder. The text encoder encodes the text text2 corresponding to Box2 and outputs the semantic features F of the text text2 corresponding to Box2. Text2 .

[0122] Step S305: Based on the features of each text detection box and the semantic features of the text corresponding to each text detection box, determine the feature prediction loss of the pre-trained model on each text detection box.

[0123] Among them, the prediction loss of the pre-trained model on a text detection box can characterize the feature difference between the features of the text detection box and the semantic features of the text corresponding to the text detection box.

[0124] like Figure 4 As shown, based on the features F of the text detection box Box1 Box1 The semantic features F of the text Text1 corresponding to Box1 Text1 Determine the feature prediction loss of the pre-trained model on the text detection box Box1, and based on the features F of the text detection box Box2. Box2 The semantic features F of the text Text2 corresponding to Box2 Text2 Determine the feature prediction loss of the pre-trained model on the text detection box Box2.

[0125] Optionally, for each text detection box, the mean square error (MSE) between the features of the text detection box and the semantic features of the text corresponding to the text detection box can be calculated, and the calculated mean square error can be used as the prediction loss of the pre-trained model on the text detection box.

[0126] It should be noted that this embodiment does not limit the prediction loss of the pre-trained model on each text detection box to mean squared error loss; other losses that can measure the difference between two features are also applicable to this invention.

[0127] Step S306: Update the parameters of the pre-trained model based on the prediction loss of the pre-trained model on each text detection box.

[0128] Specifically, the prediction losses of the pre-trained model on each text detection box can be fused, and the parameters of the pre-trained model can be updated based on the fused loss. There are several ways to fuse the prediction losses of the pre-trained model on each text detection box. For example, the prediction losses of the pre-trained model on each text detection box can be summed directly, or the prediction losses of the pre-trained model on each text detection box can be weighted and summed.

[0129] Step S307: Determine whether the training termination condition is met. If yes, end the training. If no, proceed with step S302 and subsequent steps.

[0130] The training termination condition can be the convergence of the pre-trained model. However, this embodiment is not limited to this. Other termination conditions can also be met, such as reaching a set number of training iterations.

[0131] The pre-trained model after training is used as the target pre-trained model. The target pre-trained model obtained by the pre-training method provided in this embodiment has the ability to encode the input document image and output features rich in textual semantic information.

[0132] The target pre-trained model obtained by the pre-training method provided in this embodiment can be applied to document image detection tasks, such as text line detection tasks, text paragraph detection tasks, title detection tasks, etc.

[0133] The pre-training method provided in this invention obtains semantic features of the text corresponding to the text detection box based on a text encoder. The semantic features obtained based on the text encoder are used as the learning target of the pre-trained model to guide the visual encoder of the pre-trained model to mine the text semantic information in the input document image. The visual encoder trained in this way has the ability to mine text semantic information from the input document image and output features rich in text semantics. Since the target pre-trained model takes the document image as input, it can encode the input document image to output features rich in text semantics without inputting text. Therefore, when applied to downstream tasks, it does not need to perform text recognition based on an OCR engine, thus saving OCR engine overhead.

[0134] As mentioned in the above embodiments, the pre-trained model may include only a visual encoder, or it may include a visual encoder and a feature processing module. Based on the above embodiments, this embodiment takes a pre-trained model that includes a visual encoder and a feature processing module as an example to introduce the pre-training method.

[0135] Please see Figure 5 The flowchart illustrates another specific example of the pre-training method provided in this embodiment, which may include:

[0136] Step S501: Obtain the training dataset.

[0137] The training dataset includes multiple training document images.

[0138] Step S502: Obtain training document images from the training dataset.

[0139] Step S503: Perform text detection and recognition on the training document image to obtain several text detection boxes and the text corresponding to each text detection box.

[0140] The text corresponding to each of the several text detection boxes can be obtained based on the OCR engine.

[0141] Step S504a1: Using the training document images as input to the pre-trained model, the visual encoder based on the pre-trained model encodes the training document images to obtain the features of the training document images.

[0142] like Figure 6 As shown, the training document images are input into the pre-trained model, and the visual encoder in the pre-trained model encodes the input training document images and outputs the features of the training document images.

[0143] The visual encoder in the pre-trained model can be ResNet. Of course, this embodiment is not limited to this. Other visual encoders, such as Vision-Transformer, can also be used in the pre-trained model.

[0144] Step S504a2: Based on the features of each text detection box and the training document image, obtain the first feature of each text detection box.

[0145] RoIAlign can be used to perform feature pooling on the features of the training document image based on each text detection box, thereby obtaining the features of each text detection box. For a more detailed implementation of step S504a2, please refer to the specific implementation of step S304a2; this embodiment will not elaborate further.

[0146] like Figure 6 As shown, based on the features of the text detection boxes Box1 and Box2 and the training document image, the first feature F1 of the text detection box Box1 can be obtained. Box1 Features F1 of the text detection box Box2 Box2 .

[0147] Step S504a3: The first feature processing module based on the pre-trained model processes the first feature of each text detection box to obtain the second feature of each text detection box.

[0148] The second feature of a text detection box contains the temporal information of the text in the text detection box.

[0149] like Figure 6 As shown, the first feature F1 of the text detection box Box1 is... Box1 The first feature processing module is input into the pre-trained model, and the first feature processing module is based on F1. Box1 Determine the temporal information of the text in the text detection box Box1, and output the feature containing the text temporal information of the text detection box Box1 as the second feature F2 of the text detection box Box1. Box1 The second feature F2 of the text detection box Box2 can be obtained in the same way. Box2 .

[0150] In this embodiment, the first feature processing module can employ a GRU unit (recurrent neural network). It should be noted that this embodiment does not limit the first feature processing module to using a GRU unit; other modules capable of determining text temporal information are also applicable to this invention.

[0151] Step S504a4: The second feature processing module based on the pre-trained model processes the second features of each text detection box to obtain the third feature of each text detection box.

[0152] The third feature of a text detection box contains dependency information between the text detection box and other text detection boxes.

[0153] The second features of all text detection boxes are input into the second feature processing module of the pre-trained model. For each text detection box, the second feature processing module first determines the relevance weight between the text detection box and each other text detection box to obtain the relevance weight corresponding to each text detection box. Then, the second features of each text detection box are weighted and summed according to the weights corresponding to each text detection box, and the weighted summed features are output as the third feature of the text detection box.

[0154] like Figure 6 As shown, the second feature F2 of the text detection box Box1 is... Box1 The second feature F2 of the text detection box Box2 Box2 The second feature processing module of the pre-trained model, for the text detection box Box1, determines the relevance weights α corresponding to the text detection boxes Box1 and Box2 respectively. 11 Box1 and α 12 Box2 According to α 11 Box1 and α 12 Box2 For F2 Box1 and F2 Box2 Weighted summation yields the third feature F3 of Box1. Box1For the text detection box Box2, the second feature processing module determines the relevance weights α corresponding to the text detection boxes Box1 and Box2, respectively. 21 Box1 and α 22 Box2 According to α 21 Box1 and α 22 Box2 For F2 Box1 and F2 Box2 Weighted summation yields the third feature F3 of Box2. box2 .

[0155] In this embodiment, the second feature processing module can employ a Transformer. It should be noted that this embodiment does not limit the second feature processing module to using a Transformer; other modules capable of determining inter-frame dependencies in text detection are also applicable to this invention.

[0156] Step S504b: Use the text corresponding to the text detection box as the input of the text encoder, and obtain the semantic features of the text corresponding to each text detection box based on the text encoder.

[0157] The text encoder is pre-trained using multiple training texts and has the ability to encode the input text and output its semantic features. In this embodiment, the text encoder can be, but is not limited to, BERT, GPT, etc.

[0158] like Figure 6 As shown, the text Text1 corresponding to the text detection box Box1 is input into the text encoder. The text encoder encodes the text Text1 corresponding to Box1 and outputs the semantic features F of the text Text1 corresponding to Box1. Text1 The text text2 corresponding to the text detection box Box2 is input into the text encoder. The text encoder encodes the text text2 corresponding to Box2 and outputs the semantic features F of the text text2 corresponding to Box2. Text2 .

[0159] Step S505: Based on the third feature of each text detection box and the semantic features of the text corresponding to each text detection box, determine the feature prediction loss of the pre-trained model on each text detection box.

[0160] Among them, the prediction loss of the pre-trained model on a text detection box can characterize the feature difference between the features of the text detection box and the semantic features of the text corresponding to the text detection box.

[0161] like Figure 6 As shown, based on the third feature F3 of the text detection box Box1Box1 The semantic features F of the text Text1 corresponding to Box1 Text1 Determine the feature prediction loss of the pre-trained model on the text detection box Box1, and the F3 loss based on the features of the text detection box Box2. Box2 The semantic features F of the text Text2 corresponding to Box2 Text2 Determine the feature prediction loss of the pre-trained model on the text detection box Box2.

[0162] Step S506: Update the parameters of the pre-trained model based on the prediction loss of the pre-trained model on each text detection box.

[0163] Specifically, the prediction losses of the pre-trained model on each text detection box can be fused, and the parameters of the pre-trained model can be updated based on the fused loss.

[0164] For a more detailed implementation of step S506, please refer to the detailed implementation of step S306. This embodiment will not elaborate on it here.

[0165] Step S507: Determine whether the training termination condition is met. If yes, end the training. If no, proceed with step S502 and subsequent steps.

[0166] The training termination condition can be the convergence of the pre-trained model. However, this embodiment is not limited to this. Other termination conditions can also be met, such as reaching a set number of training iterations.

[0167] The pre-trained model after training is used as the target pre-trained model. The target pre-trained model obtained by the pre-training method provided in this embodiment has the ability to process the input document image and output features rich in text semantic information, text temporal information within the text detection box, and dependency information between text detection boxes.

[0168] The target pre-trained model obtained by the pre-training method provided in this embodiment can be applied to classification tasks targeting regions of interest in document images.

[0169] The pre-training method provided in this invention obtains the semantic features of the text corresponding to the text detection box based on a text encoder. The semantic features obtained based on the text encoder are used as the learning target of the pre-training model to guide the pre-training model to mine the text semantic information in the input document image. The pre-trained model trained in this way has the ability to mine the text semantic information from the input document image and output features rich in text semantic information. Since the target pre-training model takes the document image as input, it can process the input document image to output features rich in text semantics without needing to input the text in the document image. Therefore, when applied to downstream tasks, it is not necessary to perform text recognition based on an OCR engine, thereby saving OCR engine overhead.

[0170] Based on the pre-training method provided in the above embodiments, this embodiment provides an information prediction model acquisition method, which may include:

[0171] Step a1: Construct an initial information prediction model based on the target pre-trained model and the prediction module for the specified task.

[0172] The target pre-trained model was trained using the pre-training method provided in the above embodiments.

[0173] Step a2: Use training document images with labeled data for the specified task to fine-tune the initial information prediction model to obtain an information prediction model for the specified task.

[0174] In one possible implementation, the task is specified as a detection task for document images (such as text line detection task, text paragraph detection task, title detection task, etc.), and the target pre-trained model may only include a visual encoder. Then, the process of building an initial information prediction model based on the target pre-trained model and the prediction module for the specified task may include: building an initial detection model based on the visual encoder in the target pre-trained model and the prediction module for the detection task.

[0175] Accordingly, the process of fine-tuning the initial information prediction model using training document images with labeled data for a specified task to obtain an information prediction model for a specified task may include: fine-tuning the initial detection model using training document images with labeled data for a detection task to obtain the final detection model.

[0176] In another possible implementation, the task is specified as a classification task for regions of interest in a document image. The target pre-trained model may include a visual encoder, a first feature processing module, and a second feature processing module. The process of constructing an initial information prediction model based on the target pre-trained model and the prediction module for the specified task may include: constructing an initial region of interest classification model based on the visual encoder, the first feature processing module, the second feature processing module in the target pre-trained model, and the prediction module for the classification task.

[0177] Accordingly, the process of fine-tuning the initial information prediction model using training document images with labeled data for a specific task to obtain an information prediction model for a specific task may include: fine-tuning the initial region of interest classification model using training document images with regions of interest and categories labeled with regions of interest to obtain the final region of interest classification model.

[0178] The information prediction model acquisition method provided in this embodiment of the invention can process the input document image to output features rich in textual semantics without needing to input the text in the document image. Therefore, when fine-tuning the initial information prediction model built based on the target pre-trained model and the prediction module for the specified task, the training document image can be directly input into the model for training without needing to recognize the text from the training document image based on the OCR engine, thus saving the OCR engine overhead.

[0179] Based on the information prediction model acquisition method provided in the above embodiments, this embodiment provides an information prediction method, which may include:

[0180] Step b1: Obtain the target document image for the specified task.

[0181] For example, if the task is specified as a detection task (such as a text line detection task, a text paragraph detection task, a title detection task, etc.), then the target document image is a document image without a region of interest.

[0182] For example, if the task is specified as a region of interest classification task, then the target document image is a document image containing a region of interest.

[0183] Step b2: Based on the information prediction model obtained by the information prediction model acquisition method provided in the above embodiments, the target document image is processed to obtain the information prediction result corresponding to the target document image on the specified task.

[0184] For example, if the specified task is a detection task, the information prediction model includes an encoding part (visual encoder) and a prediction module for the detection task. The information prediction result of the target document image on the specified task is the detection result of the target document image, such as the text line detection result, the text paragraph detection result, the title detection task, etc.

[0185] For example, if the specified task is a region of interest classification task, the information prediction model includes an encoding part (visual encoder, first feature processing module, second feature processing module) and a prediction module for the region of interest classification task. The information prediction result of the target document image on the specified task is the category of the region of interest in the target document image.

[0186] The information prediction method provided in this embodiment of the invention is based on an information prediction model. Since the encoding part of the information prediction model takes document images as input, it can process the input document images to output features rich in textual semantics without using the text in the document images. Therefore, when performing information prediction, there is no need to perform text recognition based on an OCR engine, thereby saving OCR engine overhead.

[0187] This invention also provides a pre-training device. The pre-training device provided in this invention is described below. The pre-training device described below can be referred to in correspondence with the pre-training method described above.

[0188] Please see Figure 7 The diagram shows a pre-training device provided in an embodiment of the present invention. The pre-training device may include a training data acquisition module 701 and a model training module 702.

[0189] The training data acquisition module 701 is used to acquire the training dataset.

[0190] The training dataset includes multiple training document images;

[0191] The model training module 702 is used to perform text detection and recognition on training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes. The training document images are used as input to the pre-trained model to obtain the features of the text detection boxes. The text corresponding to the text detection boxes is used as input to the text encoder to obtain the semantic features of the text corresponding to the text detection boxes. The goal is to make the features of the text detection boxes approach the semantic features of the text corresponding to the text detection boxes. The pre-trained model is then updated with parameters, and the trained pre-trained model is used as the target pre-trained model.

[0192] The pre-trained model includes a visual encoder and a text encoder, which are pre-trained using multiple training texts.

[0193] Optionally, when the model training module 702 uses the training document image as input to the pre-trained model and obtains the features of the text detection box based on the pre-trained model, it is specifically used for:

[0194] A visual encoder based on a pre-trained model encodes the training document images to obtain the features of the training document images;

[0195] Based on the features of each text detection box and the training document image, the features of each text detection box are obtained.

[0196] Optionally, the pre-trained model may also include: a first feature processing module and a second feature processing module;

[0197] When the model training module 702 uses the training document image as input to the pre-trained model and obtains the features of the text detection box based on the pre-trained model, it is specifically used for:

[0198] A visual encoder based on a pre-trained model encodes the training document images to obtain the features of the training document images;

[0199] Based on the features of each text detection box and the training document image, obtain the first feature of each text detection box;

[0200] The first feature processing module based on the pre-trained model processes the first feature of each text detection box to obtain the second feature of each text detection box. The second feature of a text detection box contains the temporal information of the text in the text detection box.

[0201] The second feature processing module based on the pre-trained model processes the second features of each text detection box to obtain the third feature of each text detection box, which serves as the final feature of each text detection box. The third feature of a text detection box contains the dependency information between the text detection box and other text detection boxes.

[0202] Optionally, when the model training module 702 processes the second features of each text detection box in the second feature processing module based on the pre-trained model to obtain the third feature of each text detection box, it is specifically used for:

[0203] For each text detection box:

[0204] The second feature processing module based on the pre-trained model determines the relevance weights of the text detection box to each other, so as to obtain the relevance weights corresponding to each text detection box.

[0205] The third feature of each text detection box is obtained by weighting and summing the second features of each text detection box according to the relevance weights corresponding to each text detection box.

[0206] Optionally, when updating the parameters of the pre-trained model with the goal of making the features of the text detection box approach the semantic features of the text corresponding to the text detection box, the model training module 702 is specifically used for:

[0207] For each text detection box, based on the features of the text detection box and the semantic features of the text corresponding to the text detection box, the feature prediction loss of the pre-trained model on the text detection box is determined.

[0208] The parameters of the pre-trained model are updated based on the feature prediction loss of the pre-trained model on each text detection box.

[0209] Optionally, when determining the feature prediction loss of the pre-trained model on the text detection box based on the features of the text detection box and the semantic features of the text corresponding to the text detection box, the model training module 702 is specifically used for:

[0210] Calculate the mean squared error between the features of the text detection box and the semantic features of the text corresponding to the text detection box, and use it as the feature prediction loss of the pre-trained model on the text detection box.

[0211] The step of updating the parameters of the pre-trained model based on the feature prediction loss of the pre-trained model on each text detection box includes:

[0212] The feature prediction losses of the pre-trained model on each text detection box are fused to obtain the fused loss.

[0213] The parameters of the pre-trained model are updated based on the fusion loss.

[0214] The pre-training device provided in this embodiment of the invention, after obtaining the training dataset, first performs text detection and recognition on the training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes. Then, using the training document images as input to the pre-training model, the features of the text detection boxes are obtained based on the pre-training model. The text corresponding to the text detection boxes is used as input to the text encoder to obtain the semantic features of the text corresponding to the text detection boxes. Finally, with the goal of making the features of the text detection boxes approach the semantic features of the text corresponding to the text detection boxes, the parameters of the pre-training model are updated to obtain the target pre-training model. The pre-training device provided in this embodiment of the invention obtains the semantic features of the text corresponding to the text detection box based on the text encoder, and uses the semantic features obtained based on the text encoder as the learning target of the pre-training model to guide the pre-training model to mine the text semantic information in the input document image. The pre-trained model trained in this way has the ability to mine the text semantic information from the input document image and output features rich in text semantic information. Since the target pre-training model takes the document image as input, it can mine the text semantic information from the document image and output features rich in text semantic information without inputting the text in the document image. Therefore, when it is applied to downstream tasks, there is no need to perform text recognition based on the OCR engine, thereby saving the OCR engine overhead.

[0215] This invention also provides a processing device; please refer to [link / reference]. Figure 8 The diagram shows the structure of the processing device, which may include: a processor 801, a communication interface 802, a memory 803, and a communication bus 804.

[0216] In this embodiment of the invention, the number of processor 801, communication interface 802, memory 803, and communication bus 804 is at least one, and processor 801, communication interface 802, and memory 803 communicate with each other through communication bus 804.

[0217] The processor 801 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0218] The memory 803 may include high-speed RAM, or it may also include non-volatile memory, such as at least one disk storage device;

[0219] The memory stores a program, which the processor can call. The program is used for:

[0220] Obtain a training dataset, wherein the training dataset includes multiple training document images;

[0221] Text detection and recognition are performed on the training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes;

[0222] Using the training document image as input to the pre-trained model, the features of the text detection box are obtained based on the pre-trained model, and the semantic features of the text corresponding to the text detection box are obtained based on the text encoder. The pre-trained model includes a visual encoder, and the text encoder is trained in advance using multiple training texts.

[0223] With the goal of making the features of the text detection box approach the semantic features of the text corresponding to the text detection box, the parameters of the pre-trained model are updated, and the pre-trained model obtained is used as the target pre-trained model.

[0224] Optionally, the refined and extended functions of the program can be found in the description above.

[0225] This invention also provides a readable storage medium that stores a program suitable for execution by a processor, the program being used for:

[0226] Obtain a training dataset, wherein the training dataset includes multiple training document images;

[0227] Text detection and recognition are performed on the training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes;

[0228] Using the training document image as input to the pre-trained model, the features of the text detection box are obtained based on the pre-trained model, and the semantic features of the text corresponding to the text detection box are obtained based on the text encoder. The pre-trained model includes a visual encoder, and the text encoder is trained in advance using multiple training texts.

[0229] With the goal of making the features of the text detection box approach the semantic features of the text corresponding to the text detection box, the parameters of the pre-trained model are updated, and the pre-trained model obtained is used as the target pre-trained model.

[0230] Optionally, the refined and extended functions of the program can be found in the description above.

[0231] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0232] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0233] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pre-training method, characterized in that, include: Obtain a training dataset, wherein the training dataset includes multiple training document images; Text detection and recognition are performed on the training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes; The training document image is used as the input of the pre-trained model. The features of the text detection box are obtained based on the pre-trained model. The text corresponding to the text detection box is used as the input of the text encoder. The semantic features of the text corresponding to the text detection box are obtained based on the text encoder. The pre-trained model includes a visual encoder. The text encoder is trained in advance using multiple training texts. With the goal of making the features of the text detection box approach the semantic features of the text corresponding to the text detection box, the parameters of the pre-trained model are updated, and the pre-trained model obtained is used as the target pre-trained model.

2. The pre-training method according to claim 1, characterized in that, The step of using the training document image as input to the pre-trained model and obtaining the features of the text detection box based on the pre-trained model includes: A visual encoder based on a pre-trained model encodes the training document images to obtain the features of the training document images; Based on the features of each text detection box and the training document image, the features of each text detection box are obtained.

3. The pre-training method according to claim 1, characterized in that, The pre-trained model also includes: a first feature processing module and a second feature processing module; The step of using the training document image as input to the pre-trained model and obtaining the features of the text detection box based on the pre-trained model includes: A visual encoder based on a pre-trained model encodes the training document images to obtain the features of the training document images; Based on the features of each text detection box and the training document image, obtain the first feature of each text detection box; The first feature processing module based on the pre-trained model processes the first feature of each text detection box to obtain the second feature of each text detection box. The second feature of a text detection box contains the temporal information of the text in the text detection box. The second feature processing module based on the pre-trained model processes the second features of each text detection box to obtain the third feature of each text detection box, which serves as the final feature of each text detection box. The third feature of a text detection box contains the dependency information between the text detection box and other text detection boxes.

4. The pre-training method according to claim 3, characterized in that, The second feature processing module based on the pre-trained model processes the second features of each text detection box to obtain the third feature of each text detection box, including: For each text detection box: The second feature processing module based on the pre-trained model determines the relevance weights of the text detection box to each other, so as to obtain the relevance weights corresponding to each text detection box. The third feature of each text detection box is obtained by weighting and summing the second features of each text detection box according to the relevance weights corresponding to each text detection box.

5. The pre-training method according to any one of claims 1 to 4, characterized in that, The step of updating the parameters of the pre-trained model with the goal of making the features of the text detection box approximate the semantic features of the text corresponding to the text detection box includes: For each text detection box, based on the features of the text detection box and the semantic features of the text corresponding to the text detection box, the feature prediction loss of the pre-trained model on the text detection box is determined. The parameters of the pre-trained model are updated based on the feature prediction loss of the pre-trained model on each text detection box.

6. The pre-training method according to claim 5, characterized in that, The step of determining the feature prediction loss of the pre-trained model on the text detection box based on the features of the text detection box and the semantic features of the text corresponding to the text detection box includes: Calculate the mean squared error between the features of the text detection box and the semantic features of the text corresponding to the text detection box, and use it as the feature prediction loss of the pre-trained model on the text detection box. The step of updating the parameters of the pre-trained model based on the feature prediction loss of the pre-trained model on each text detection box includes: The feature prediction losses of the pre-trained model on each text detection box are fused to obtain the fused loss. The parameters of the pre-trained model are updated based on the fusion loss.

7. A method for obtaining an information prediction model, characterized in that, include: An initial information prediction model is constructed based on the target pre-trained model and the prediction module for the specified task, wherein the target pre-trained model is trained using the pre-training method as described in any one of claims 1 to 6; The initial information prediction model is fine-tuned using training document images with labeled data for the specified task to obtain an information prediction model for the specified task.

8. The information prediction model acquisition method according to claim 7, characterized in that, The specified task is a detection task for document images; The initial information prediction model is constructed based on the target pre-trained model and the prediction module for the specified task, including: An initial detection model is constructed based on the visual encoder in the target pre-trained model and the prediction module for the detection task. The step of fine-tuning the initial information prediction model using training document images with labeled data for the specified task includes: The initial detection model is fine-tuned using training document images with labeled data specific to the detection task.

9. The information prediction model acquisition method according to claim 7, characterized in that, The specified task is a classification task targeting regions of interest in document images; The target pre-trained model includes a first feature processing module and a second feature processing module; The initial information prediction model is constructed based on the target pre-trained model and the prediction module for the specified task, including: Based on the visual encoder, the first feature processing module and the second feature processing module in the target pre-trained model, and the prediction module for the classification task, an initial region of interest classification model is constructed. The step of fine-tuning the initial information prediction model using training document images with labeled data for the specified task includes: The initial region of interest (ROI) classification model is fine-tuned using training document images with ROIs and categories labeled with those ROIs.

10. An information prediction method, characterized in that, include: Retrieve the target document image for a specified task; Based on the information prediction model obtained by the information prediction model acquisition method as described in any one of claims 7 to 9, the target document image is processed to obtain the information prediction result corresponding to the target document image on the specified task.

11. A pre-training device, characterized in that, include: Training data acquisition module and model training module; The training data acquisition module is used to acquire a training dataset, wherein the training dataset includes multiple training document images; The model training module is used to perform text detection and recognition on training document images in the training dataset to obtain text detection boxes and the text corresponding to the text detection boxes. The training document images are used as input to the pre-trained model to obtain the features of the text detection boxes. The text corresponding to the text detection boxes is used as input to the text encoder to obtain the semantic features of the text corresponding to the text detection boxes. The goal is to make the features of the text detection boxes approach the semantic features of the text corresponding to the text detection boxes. The pre-trained model is then updated with parameters, and the trained pre-trained model is used as the target pre-trained model. The pre-trained model includes a visual encoder, and the text encoder is pre-trained using multiple training texts.

12. A processing apparatus, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the pre-training method as described in any one of claims 1 to 6.

13. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the pre-training method as described in any one of claims 1 to 6.

Images