Model training method, device, computer equipment and storage medium

Abstract

The embodiments of the present application provide a model training method, apparatus, computer equipment and storage medium, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and Internet of Vehicles; the embodiments of the present application can obtain a document image to be processed, and perform region segmentation processing on the document image to be processed to obtain several candidate image regions corresponding to the document image to be processed; fuse the candidate image regions in a preset order to construct a target document image containing image regions; determine the region relationship information corresponding to each image region in the target document image, and generate a document image sample corresponding to the model training task based on the target document image and the region relationship information; train the document image prediction model based on the model training task and the document image sample to obtain a target document image prediction model; the embodiments of the present application can improve the versatility of the trained target document image prediction model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a model training method, apparatus, computer equipment, and storage medium, wherein the storage medium is a computer-readable storage medium. Background Art

[0002] With the development of artificial intelligence technology, in order to improve the training efficiency of neural network models, neural network models are currently pre-trained. In this way, the pre-trained neural network model obtained based on the pre-training can be fine-tuned, so that the network parameters of the pre-trained neural network model can be quickly updated, thereby improving the training efficiency of the neural network model.

[0003] Pre-training is widely used in document image processing. However, currently, the pre-trained neural network models obtained through pre-training are specialized models and can only be applied to a single business. For example, a pre-trained neural network model can only be used for form recognition business; another example is a pre-trained neural network model can only be used for document sorting business; and so on.

[0004] In summary, the current pre-trained neural network model has the problem of poor versatility. Summary of the Invention

[0005] The embodiments of the present application provide a model pre-training method, apparatus, computer equipment, and storage medium, which can improve the versatility of the target document image prediction model obtained through training.

[0006] A model training method, comprising:

[0007] Acquire a document image to be processed, and perform region segmentation processing on the document image to be processed to obtain a number of candidate image regions corresponding to the document image to be processed;

[0008] Performing fusion processing on the candidate image regions in a preset order to construct a target document image containing the image regions;

[0009] Determine the region relationship information corresponding to each image region in the target document image, and generate a document image sample corresponding to the model training task based on the target document image and the region relationship information;

[0010] The document image prediction model is trained according to the model training task and the document image samples to obtain the target document image prediction model.

[0011] Accordingly, an embodiment of the present application provides a model pre-training device, comprising:

[0012] The acquisition unit can be used to acquire the document image to be processed and perform region segmentation processing on the document image to be processed to obtain a number of candidate image regions corresponding to the document image to be processed;

[0013] A fusion unit may be used to fuse the candidate image regions in a preset order to construct a target document image containing the image regions;

[0014] A generating unit may be used to determine region relationship information corresponding to each image region in a target document image, and generate a document image sample corresponding to a model training task based on the target document image and the region relationship information;

[0015] The training unit can be used to train the document image prediction model according to the model training task and the document image samples to obtain the target document image prediction model.

[0016] In some embodiments, the acquisition unit can be specifically used to determine the number of regions of the required candidate image areas in the document image to be processed; determine the region segmentation position of the document image to be processed based on the number of regions; and perform region segmentation processing on the document image to be processed based on the region segmentation position to obtain several candidate image areas corresponding to the document image to be processed.

[0017] In some embodiments, the acquiring unit may be specifically configured to acquire the size of the document image to be processed; and determine the region segmentation positions of the document image to be processed according to the number and size of the regions.

[0018] In some embodiments, the training unit can be specifically used to predict document image samples based on the model training task through the document image prediction model to obtain prediction data corresponding to the model training task; and train the document image prediction model based on the prediction data and the document image samples to obtain the target document image prediction model.

[0019] In some embodiments, the prediction data includes predicted region relationship information; the training unit can be specifically used to determine each training subtask corresponding to the target training task when the model training task is a target training task; according to each training subtask, the document image prediction model is used to perform region relationship prediction on the document image sample to obtain the predicted region relationship information corresponding to each training subtask.

[0020] In some embodiments, the document image sample carries a label, which includes the region relationship information corresponding to each training subtask; the training unit can be specifically used to calculate the loss value between the predicted region relationship information corresponding to the training subtask and the region relationship information corresponding to the training subtask for each training subtask, so as to obtain the loss value corresponding to each training subtask; according to the loss value corresponding to each training subtask, the model parameters of the document image prediction model are converged to obtain the target document image prediction model.

[0021] In some embodiments, the generation unit can be specifically used to determine the arrangement order corresponding to each image area in the target document image according to the position order of the candidate image areas in the document image to be processed; and determine the area relationship information corresponding to each image area according to the arrangement order.

[0022] In some embodiments, the generation unit can be specifically used to filter out the target image area from the image area according to the arrangement order; determine a first preset value based on the positional relationship information between the image area that is before the target image area in the arrangement order and the target image area; determine a second preset value based on the positional relationship information between the image area that is after the target image area in the arrangement order and the target image area; determine the area relationship information of the target image area based on the first preset value and the second preset value to obtain the area relationship information corresponding to each image area.

[0023] In some embodiments, the generation unit can be specifically used to mask the target text information in the target document image to obtain a masked document image; based on the target text information and regional relationship information, the masked document image is labeled with a model training task to obtain a document image sample corresponding to the model training task.

[0024] In some embodiments, the fusion unit can be specifically used to determine the region separation distance between different candidate image regions; determine the target position of the candidate image region in the target document image based on the region separation distance; and merge the candidate image regions in a preset order based on the target position to obtain a target document image containing the image region.

[0025] In addition, an embodiment of the present application also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute any model training method provided in the embodiment of the present application.

[0026] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for loading by a processor to execute any model training method provided in the embodiment of the present application.

[0027] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which implements any model training method provided in the embodiment of the present application when the computer program is executed by a processor.

[0028] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which implements any model training method provided in the embodiment of the present application when the computer program is executed by a processor.

[0029] The embodiment of the present application can obtain a document image to be processed, and perform region segmentation processing on the document image to be processed to obtain several candidate image regions corresponding to the document image to be processed; perform fusion processing on the candidate image regions in a preset order to construct a target document image containing image regions; determine the region relationship information corresponding to each image region in the target document image, and generate a document image sample corresponding to the model training task based on the target document image and the region relationship information; train the document image prediction model based on the model training task and the document image sample to obtain the target document image prediction model; since the embodiment of the present application can construct the target document image based on the candidate image regions of the document image to be processed, the document image sample corresponding to the model training task can be generated based on the region relationship information corresponding to each image region in the target document image and the target document image, so that the document image prediction model can be trained based on the model training task to improve the versatility of the trained target document image prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 This is a schematic diagram of a scenario of the model pre-training method provided in an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of a document image sample with a table structure provided by an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of a reconstructed table document image provided by an embodiment of the present application;

[0034] Figure 4 is a schematic diagram of a document image sample with text information provided by an embodiment of the present application;

[0035] Figure 5is a schematic diagram of a document image with reading order information provided by an embodiment of the present application;

[0036] Figure 6 A diagram illustrating the process of the model pre-training method provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of a process for performing region segmentation processing on a document image to be processed to obtain several candidate image regions corresponding to the document image to be processed, provided in an embodiment of the present application;

[0038] Figure 8 Figure 2 is a flow chart illustrating the model pre-training method provided in an embodiment of the present application;

[0039] Figure 9 Three diagrams illustrating the process of the model pre-training method provided in the embodiments of the present application;

[0040] Figure 10 A schematic diagram of a document image to be processed provided in an embodiment of the present application;

[0041] Figure 11 A schematic diagram of a target document image provided in an embodiment of the present application;

[0042] Figure 12 A second schematic diagram of a target document image provided in an embodiment of the present application;

[0043] Figure 13 A schematic diagram of the StrucTexT model provided in an embodiment of the present application;

[0044] Figure 14 A schematic diagram of a document image prediction model provided in an embodiment of the present application;

[0045] Figure 15 A schematic diagram illustrating fine-tuning a target document image prediction model according to an embodiment of the present application;

[0046] Figure 16 A second diagram illustrating fine-tuning a target document image prediction model according to an embodiment of the present application;

[0047] Figure 17 Schematic diagram of the structure of the model training device provided in the embodiment of the present application;

[0048] Figure 18 It is a structural diagram of the computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0050] The present invention provides a model pre-training method, apparatus, computer device, and computer-readable storage medium. The model pre-training apparatus can be integrated into a computer device, which can be a server or a terminal.

[0051] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0052] Among them, the embodiments of this application relate to artificial intelligence (AI), which is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a manner similar to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0053] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0054] For example, see Figure 1 Taking the model pre-training device integrated into a computer device as an example, the computer device can obtain a document image to be processed, and perform region segmentation processing on the document image to be processed to obtain several candidate image regions corresponding to the document image to be processed; the candidate image regions are fused in a preset order to construct a target document image containing image regions; the region relationship information corresponding to each image region in the target document image is determined, and a document image sample corresponding to the model training task is generated based on the target document image and the region relationship information; the document image prediction model is trained based on the model training task and the document image sample to obtain the target document image prediction model.

[0055] The document image to be processed may be a table text image, a text image, and so on.

[0056] Region relationship information may refer to information about the relationship between an image region and other image regions in the target document image. Region relationship information may also refer to information about the relationship between an image region and candidate image regions in the document image to be processed. Region relationship information may also refer to information about the relationship between sub-image regions within an image region. For each image region, the relationship information between sub-image regions may refer to information about the relationship between the sub-image region and other sub-image regions within the image region.

[0057] The model training task may include at least one model training task, for example, the model training task may include a table structure recognition training task, a document sorting training task, and a document grouping training task. Specifically, for example, the table structure recognition training task may be: Figure 2 The document image samples with the table structure shown in FIG are input into the document image prediction model for training, so that the trained target document image prediction model can output the following Figure 3 As another example, the document sorting training task can be: Figure 4 The document image sample with text information shown in FIG is input into the document image prediction model for training, so that the trained target document image prediction model can output the following Figure 5 The document image with reading order information is shown, wherein Figure 5 The serial numbers 1, 2, 3, 4, and 5 indicate the reading order.

[0058] In an embodiment of the present application, to improve the versatility of the trained target document image prediction model, the embodiment of the present application may employ at least one model training task to train the document image prediction model. When employing one model training task to train the document image prediction model, the embodiment of the present application may employ the model training task to train an existing relevant document image prediction model, where the existing relevant document image prediction model is a model trained using an existing model training task. When employing at least two model training tasks to train the document image prediction model, the model training tasks may be tasks trained in parallel.

[0059] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0060] This embodiment will be described from the perspective of a model pre-training device, which can be specifically integrated into a computer device, which can be a server or a terminal. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a wearable device, a virtual reality device, or other smart devices that can acquire data.

[0061] like Figure 6 As shown, the specific process of the model pre-training method is as follows: Step S101 to Step S104:

[0062] S101 , obtaining a document image to be processed, and performing region segmentation processing on the document image to be processed to obtain a number of candidate image regions corresponding to the document image to be processed.

[0063] The document image to be processed may be a table text image or a text image. The table text image may refer to an image having table information and text information, and the text image may refer to an image having text information.

[0064] In some application embodiments, the document image to be processed may be an image stored in a computer device database; the document image to be processed may be an image stored in the cloud; or the image to be processed may be an image obtained online in real time.

[0065] In some application embodiments, such as Figure 7 As shown, the method of performing region segmentation processing on the document image to be processed and obtaining several candidate image regions corresponding to the document image to be processed can be specifically as follows: Steps A1 to A3:

[0066] A1. Determine the number of candidate image regions required in the document image to be processed.

[0067] In one embodiment of the present application, the layout format of the document image to be processed can be pre-determined; based on the layout format, the number of candidate image regions required in the document image to be processed is obtained. For example, if the layout format of the document image to be processed is an M×N layout format, M refers to the number of candidate image regions required in the length direction of the document image to be processed, and N refers to the number of candidate image regions required in the width direction of the document image to be processed. Based on this, when the layout format of the document image to be processed is an M×N layout format, the number of candidate image regions required in the document image to be processed is M×N. Wherein, M and N are both positive integers.

[0068] A2. Determine the region segmentation positions of the document image to be processed based on the number of regions.

[0069] In some application embodiments, the method for determining the region segmentation position of the document image to be processed based on the number of regions can be: obtaining the size of the document image to be processed; and determining the region segmentation position of the document image to be processed based on the number and size of the regions.

[0070] In one embodiment of the present application, a coordinate system can be established with the upper left corner vertex of the document image to be processed as the coordinate origin. The coordinate system is, for example, a rectangular coordinate system xOy, where the x-axis is in the right direction and the y-axis is in the downward direction. Each pixel of the document image to be processed has a corresponding coordinate point on the coordinate system xOy. Based on the coordinates of the document image to be processed in the coordinate system xOy, the size of the document image to be processed is determined, where the size of the document image to be processed may include, but is not limited to, at least one of the height, width, and area of the document to be processed.

[0071] In some application embodiments, the method for determining the region segmentation position of the document image to be processed based on the number and size of the regions can be: fusing the number and size of the regions to obtain the fused size; determining the region segmentation coordinates in the coordinate system xOy based on the fused size; and using the region segmentation coordinates as the region segmentation position of the document image to be processed.

[0072] For example, if the number of regions is 4, the size of the document image to be processed is the area size, and the area size of the document image to be processed is 20 square centimeters, the embodiment of the present application divides the document image to be processed into 4 regions, each of which has an area size of 5 square centimeters, that is, the size after fusion is 5 square centimeters. Based on this, the embodiment of the present application can determine the region segmentation coordinates according to the coordinates of each region of 5 square centimeters in the coordinate system xOy; thus, the region segmentation coordinates can be used as the region segmentation positions of the document image to be processed.

[0073] In some application embodiments, the method for determining the region segmentation position of the document image to be processed based on the number and size of the regions can be: determining the initial region segmentation position of the document image to be processed based on the number and size of the regions; determining the initial image area of the document image to be processed based on the initial region segmentation position; adjusting the initial region segmentation position based on the document information in the initial image area to obtain the adjusted region segmentation position; and using the adjusted region segmentation position as the region segmentation position of the document image to be processed.

[0074] In this embodiment of the present application, the method for determining the initial region segmentation position of the document image to be processed based on the number and size of regions can be: fusing the number and size of regions to obtain the fused size; determining the initial region segmentation coordinates in the coordinate system xOy based on the fused size; and using the initial region segmentation coordinates as the initial region segmentation position of the document image to be processed. For details, please refer to the above explanation of "fusing the number and size of regions to obtain the fused size; determining the region segmentation coordinates in the coordinate system xOy based on the fused size; and using the region segmentation coordinates as the region segmentation position of the document image to be processed", which will not be repeated here.

[0075] Since the document image to be processed in this embodiment contains tables or text or both tables and text, it is necessary to adjust the initial region segmentation position to avoid segmentation at the text or table, which would result in incomplete text or table. The document information includes at least one of table information and text information.

[0076] Based on the above, when the document information includes table information, the embodiment of the present application can perform table recognition on the initial image area of the document image to be processed to obtain the table information. In particular, the embodiment of the present application can use a table recognition neural network model to perform table recognition on the document image to be processed.

[0077] Based on the above, when the document information includes text information, the embodiment of the present application can perform text recognition on the initial image area of the document image to be processed to obtain the text information. In particular, the embodiment of the present application can use a text recognition neural network model to perform text recognition on the document image to be processed.

[0078] Based on the above, when the document information includes text information and table information, the embodiment of the present application can perform text recognition on the initial image area of the document image to be processed to obtain text information; and then identify the table information corresponding to the text information from the initial image area.

[0079] In one embodiment of the present application, the initial region segmentation position is adjusted based on the document information in the initial image region. The adjusted region segmentation position can be obtained by: if the document information in the initial image region is incomplete, the initial region segmentation position is adjusted to complete the incomplete document information, thereby obtaining the adjusted region segmentation position. Incomplete document information can refer to incomplete tables or incomplete text, or both incomplete tables and text.

[0080] A3. Performing region segmentation processing on the document image to be processed according to the region segmentation position to obtain several candidate image regions corresponding to the document image to be processed.

[0081] S102: Perform fusion processing on the candidate image regions in a preset order to construct a target document image containing the image regions.

[0082] The preset order may refer to the positional order of the candidate image regions in the document image to be processed. It is understood that the positional order of the candidate image regions in the document image to be processed may represent the reading order of the document information in the candidate image regions in the document image to be processed. The preset order may also refer to the positional order of the typesetting layout of the target document image. It is understood that the positional order of the typesetting layout of the target document image represents the reading order of the document information in the image regions in the target document image.

[0083] In some application embodiments, the candidate image areas are fused in a preset order to construct a target document image containing the image area by: determining the region separation distance between different candidate image areas; determining the target position of the candidate image area in the target document image based on the region separation distance; and merging the candidate image areas in a preset order based on the target position to obtain the target document image containing the image area.

[0084] In order to make the positional distinction of candidate image regions in the target document image more distinct, embodiments of the present application set a region separation distance. The region separation distance can be expressed as the length occupied by a pixel. Each pixel has a coordinate in the coordinate system xOy. For example, n pixels occupy n coordinates in the coordinate system xOy, where n is a natural number.

[0085] The image area may be the same as the candidate image area, and the image area may also include the range occupied by the candidate image area and the area separation distance on the target document image.

[0086] In some application embodiments, the method for determining the target position of the candidate image area in the target document image based on the area separation distance can be: the embodiment of the present application can obtain the original coordinate points of each candidate image area based on the above-mentioned coordinate system xOy; calculate the target coordinate points of the candidate image area based on the number of pixel points in the area separation distance and the original coordinate points of the candidate image area; and generate the target position of the candidate image area in the target document image based on the target coordinate points.

[0087] In some application embodiments, merging the candidate image regions may be performing a splicing process on the candidate image regions.

[0088] In some application embodiments, the method of fusing the candidate image areas in a preset order to construct a target document image containing the image areas can be: splicing the candidate image areas in a preset order to obtain a target document image containing the image areas.

[0089] S103: Determine the region relationship information corresponding to each image region in the target document image, and generate a document image sample corresponding to the model training task based on the target document image and the region relationship information.

[0090] In some application embodiments, the method for determining the region relationship information corresponding to each image area in the target document image can be: determining the arrangement order corresponding to each image area in the target document image based on the position order of the candidate image areas in the document image to be processed; and determining the region relationship information corresponding to each image area based on the arrangement order.

[0091] The positional order of the candidate image regions in the document image to be processed and the arrangement order of each image region in the target document image may be in a one-to-one correspondence. For example, the positional order of the candidate image regions in the document image to be processed may be the first positional order, the second positional order, and the third positional order.

[0092] In some application embodiments, the method for determining the regional relationship information corresponding to each image area according to the arrangement order can be: according to the arrangement order, the target image area is screened out from the image area; according to the positional relationship information between the image area before the target image area in the arrangement order and the target image area, a first preset value is determined; according to the positional relationship information between the image area after the target image area in the arrangement order and the target image area, a second preset value is determined; according to the first preset value and the second preset value, the regional relationship information of the target image area is determined to obtain the regional relationship information corresponding to each image area.

[0093] In this embodiment of the present application, the target image area is filtered out from the image area in sequence according to the arrangement order.

[0094] The first preset value may be 0, and the second preset value may be 1. For example, the image region includes a first image region, a second image region, and a third image region, and the image regions are arranged in the following order: the first image region precedes the second image region, and the second image region precedes the third image region. When the second image region is the target image region, the positional relationship information between the first image region and the second image region is 0, and the positional relationship information between the second image region and the third image region is 1.

[0095] In some application embodiments, the method for generating document image samples corresponding to the model training task based on the target document image and regional relationship information can be: masking the target text information in the target document image to obtain the masked document image; and labeling the masked document image with the model training task based on the target text information and regional relationship information to obtain the document image samples corresponding to the model training task.

[0096] Model training tasks may include text recognition training tasks and region relationship recognition training tasks. The text recognition training task may be a task that, through training, enables the target document image prediction model to have the ability to recognize unclear text information in document images. The region relationship recognition training task may be a task that, through training, enables the target document image prediction model to have the ability to recognize image region relationships in document images.

[0097] Among them, the document image samples of the text recognition training task may include document image samples with target text information in the target document image as labels; the document image samples of the region relationship recognition training task may include document image samples with region relationship information as labels.

[0098] S104: Train the document image prediction model according to the model training task and the document image samples to obtain a target document image prediction model.

[0099] In one application embodiment, the document image prediction model is trained according to the model training task and the document image samples to obtain the target document image prediction model. The method can be: the document image prediction model is used to predict the document image samples according to the model training task to obtain the prediction data corresponding to the model training task; the document image prediction model is trained according to the prediction data and the document image samples to obtain the target document image prediction model.

[0100] Based on the above, in one embodiment of the application, the prediction data includes target text information in the target document image; the document image sample is predicted according to the model training task through the document image prediction model, and the method of obtaining the prediction data corresponding to the model training task can be specifically: when the model training task is not a target training task, for example, it is a text recognition training task, the document image sample is predicted by the document image prediction model to obtain the predicted text information corresponding to the text recognition training task.

[0101] Based on the above, in one embodiment of the application, the prediction data includes predicted regional relationship information; the document image sample is predicted according to the model training task through the document image prediction model, and the method for obtaining the prediction data corresponding to the model training task can be specifically: when the model training task is a target training task, each training subtask corresponding to the target training task is determined; according to each training subtask, the document image prediction model is used to predict the regional relationship of the document image sample to obtain the predicted regional relationship information corresponding to each training subtask.

[0102] Among them, the training subtask may include the task of enabling the target document image prediction model to have the ability to recognize the regional relationship of image areas in the document image through training; the training subtask may also include the task of enabling the target document image prediction model to have the ability to recognize the regional relationship of sub-image areas in the image area in the document image through training.

[0103] Based on the above, in one embodiment of the application, for the regional relationship recognition training task, the document image sample carries a label, and the label includes the regional relationship information corresponding to each training subtask; based on the predicted data and the document image sample, the document image prediction model is trained, and the method for obtaining the target document image prediction model can be specifically as follows: for each training subtask, the loss value between the predicted regional relationship information corresponding to the training subtask and the regional relationship information corresponding to the training subtask is calculated to obtain the loss value corresponding to each training subtask; based on the loss value corresponding to each training subtask, the model parameters of the document image prediction model are converged to obtain the target document image prediction model.

[0104] Among them, the embodiment of the present application can use a loss function to calculate the loss value corresponding to each training subtask. The loss function can be a cross entropy loss function, a square loss function, etc.

[0105] Among them, according to the loss value corresponding to each training subtask, the model parameters of the document image prediction model are converged to obtain the target document image prediction model. The method can be: the loss value corresponding to each training subtask is fused to obtain a first fused loss value; according to the first fused loss value, the model parameters of the document image prediction model are converged to obtain the target document image prediction model.

[0106] Among them, the embodiment of the present application can add the loss values corresponding to each training subtask to obtain a first fused loss value.

[0107] In an embodiment of the present application, all model training tasks can be parallel training tasks. In this embodiment of the present application, a loss function can be used to calculate a loss value corresponding to each model training task. The loss values corresponding to each model training task are then added together to obtain an added loss value. The model parameters of the document image prediction model are converged based on the added loss value to obtain a target document image prediction model.

[0108] Based on the above, for example, in one embodiment of the application, the model training task includes a text recognition training task and a region relationship recognition training task. The loss value of the region relationship recognition training task can be found in the above description and will not be repeated here.

[0109] For text recognition training tasks, document image samples carry labels, and the labels include target text information; based on the predicted data and document image samples, the document image prediction model is trained to obtain the target document image prediction model by calculating the loss value between the target text information and the predicted text information corresponding to the text recognition training task.

[0110] Based on the above, the embodiment of the present application can add the loss value between the target text information and the predicted text information and the above-mentioned first fusion loss value to obtain the target added loss value; according to the target added loss value, the model parameters of the document image prediction model are converged to obtain the target document image prediction model.

[0111] The embodiment of the present application can obtain a document image to be processed, and perform region segmentation processing on the document image to be processed to obtain several candidate image regions corresponding to the document image to be processed; perform fusion processing on the candidate image regions in a preset order to construct a target document image containing image regions; determine the region relationship information corresponding to each image region in the target document image, and generate a document image sample corresponding to the model training task based on the target document image and the region relationship information; train the document image prediction model based on the model training task and the document image sample to obtain the target document image prediction model; since the embodiment of the present application can construct the target document image based on the candidate image regions of the document image to be processed, the document image sample corresponding to the model training task can be generated based on the region relationship information corresponding to each image region in the target document image and the target document image, so that the document image prediction model can be trained based on the model training task to improve the versatility of the trained target document image prediction model.

[0112] The method described in the above embodiment will be further described in detail below with examples.

[0113] In this embodiment, the model pre-training device is specifically integrated into a computer device, which is a server or a terminal.

[0114] like Figure 8 As shown, a model pre-training method, the specific process is shown in steps S201 to S207:

[0115] First, the process of the embodiment of the present application is briefly described. Figure 9 As shown, the embodiment of the present application can obtain large-scale document images to be processed, see step S201 here; construct regional relationship information online based on the document images to be processed, see steps S202 to S204 here, wherein the regional relationship information can be information constructed in real time; then, based on the target document image and the regional relationship information, generate document image samples corresponding to the model training task to train the document image prediction model, and the model training tasks include text recognition training tasks and regional relationship recognition training tasks, see steps S205 to S207 here.

[0116] S201: A computer device obtains an image of a document to be processed.

[0117] The image to be processed may be an image acquired online in real time.

[0118] S202: The computer device performs region segmentation processing on the document image to be processed to obtain a number of candidate image regions corresponding to the document image to be processed.

[0119] In an embodiment of the present application, a method for performing region segmentation processing on a document image to be processed and obtaining a number of candidate image regions corresponding to the document image to be processed can specifically be: determining the number of regions of the required candidate image regions in the document image to be processed; determining the region segmentation positions of the document image to be processed based on the number of regions; and performing region segmentation processing on the document image to be processed based on the region segmentation positions to obtain a number of candidate image regions corresponding to the document image to be processed.

[0120] Among them, the embodiment of the present application can predetermine the typesetting format of the document image to be processed; and obtain the number of regions of the required candidate image regions in the document image to be processed according to the typesetting format.

[0121] Specifically, if Figure 10 As shown, the document image to be processed in the embodiment of the present application can be a single-column document image, and the single-column document image can refer to an image in which the document information in the document image to be processed is arranged in sequence from top to bottom in the longitudinal direction of the document image to be processed. For example, the embodiment of the present application can establish a coordinate system with the upper left corner vertex of the document image to be processed as the coordinate origin. The coordinate system is, for example, a rectangular coordinate system xOy, where the x-axis is the positive direction to the right and the y-axis is the positive direction downward. Each pixel point of the document image to be processed has a corresponding coordinate point on the coordinate system xOy. The document information in the document image to be processed is arranged in sequence from top to bottom in the y-axis direction of the document image to be processed, and there are no two or more areas in the document image to be processed in the x-axis direction where the reading order of the document information is different.

[0122] In the embodiment of the present application, the number of candidate image regions required in the target document image may be M×N. Based on this, the embodiment of the present application may segment the document image to be processed into M×N candidate image regions.

[0123] In some application embodiments, the method for determining the region segmentation position of the document image to be processed based on the number of regions can be: obtaining the size of the document image to be processed; and determining the region segmentation position of the document image to be processed based on the number and size of the regions.

[0124] The size of the document image to be processed is determined based on the coordinates of the document image to be processed in the coordinate system xOy. The size may refer to the height. In the embodiment of the present application, the height of the document image to be processed may be the size measured by the coordinate in the y-axis direction in the coordinate system xOy.

[0125] S203: The computer device performs fusion processing on the candidate image regions in a preset order to construct a target document image containing the image regions.

[0126] In some application embodiments, the candidate image areas are fused in a preset order to construct a target document image containing the image area by: determining the region separation distance between different candidate image areas; determining the target position of the candidate image area in the target document image based on the region separation distance; and merging the candidate image areas in a preset order based on the target position to obtain the target document image containing the image area.

[0127] The region separation distance can be expressed as the length occupied by pixels. Each pixel has a coordinate in the coordinate system xOy. For example, n pixels occupy n coordinates in the coordinate system xOy, where n is a natural number. For example, in the embodiments of the present application, the region separation distance includes n pixels as an example for explanation.

[0128] In some application embodiments, the method for determining the target position of the candidate image area in the target document image based on the area separation distance can be: the embodiment of the present application can obtain the original coordinate points of each candidate image area based on the above-mentioned coordinate system xOy; calculate the target coordinate points of the candidate image area based on the number of pixel points in the area separation distance and the original coordinate points of the candidate image area; and generate the target position of the candidate image area in the target document image based on the target coordinate points.

[0129] In the embodiment of the present application, the width and height of the candidate image region can be determined based on the original coordinate points of the candidate image region. The height of the candidate image region is the size measured by the coordinate of the y-axis of the coordinate system xOy, and the width of the candidate image region is the size measured by the coordinate of the x-axis of the coordinate system xOy.

[0130] Specifically, in an embodiment of the present application, the document image to be processed may have a 2×2 layout format, and the number of candidate image regions required in the document image to be processed is 2×2. Based on this, the document image to be processed may be divided into four candidate image regions according to its height, and the candidate image regions include candidate image region A, candidate image region B, candidate image region C, and candidate image region D. Specifically, the document image to be processed may be divided into four candidate image regions of the same height; the document image to be processed may also be divided into four candidate image regions of different heights; the document image to be processed may also be divided into candidate image regions that partially have the same height and candidate image regions that partially have different heights, for example, two candidate image regions of the same height and two candidate image regions of different heights.

[0131] In the embodiment of the present application, the region separation distance may include n pixels, that is, when the candidate image regions are fused, n pixels are inserted between the candidate image regions as a separation. The target document image constructed by fusion processing the candidate image regions in a preset order has two types of reading order layouts. The first type is: Figure 11 As shown in the figure, the zigzag format is used as the reading order, where the sequence numbers 1, 2, 3 and 4 represent the reading order; the second type: Figure 12 As shown, the typesetting format uses the letter "N" as the reading order, wherein the sequence numbers 1, 2, 3, and 4 represent the reading order.

[0132] like Figure 11 As shown, the embodiment of the present application takes the typesetting format according to the first "Z" shape as the reading order as an example, wherein the area corresponding to sequence number 1 is candidate image area A, the area corresponding to sequence number 2 is candidate image area B, the area corresponding to sequence number 3 is candidate image area C, and the area corresponding to sequence number 4 is candidate image area D. Taking the upper left corner vertex of the candidate image area as the starting coordinate of each candidate image area, the starting coordinate of each candidate image area in the target document image can be obtained through calculation as follows:

[0133] The starting coordinates of candidate image area A are (n, n); the starting coordinates of candidate image area B are (n+X the width of candidate image area A+n, n); the starting coordinates of candidate image area C are (n, n+max(the height of candidate image area A, the height of candidate image area B)+n); the starting coordinates of candidate image area D are (n+the height of candidate image area C+n, n+max(the height of candidate image area A, the height of candidate image area B)+n).

[0134] Based on the above, the embodiment of the present application can calculate the starting point coordinates of each candidate image area in the target document image, and thus, based on the starting point coordinates of the candidate image area in the target document image and the original starting point coordinates of the candidate image area in the document image to be processed, the coordinate translation difference of the candidate image area in the target document image compared to the candidate image area in the document image to be processed can be obtained. Based on the coordinate translation difference, each coordinate point corresponding to the candidate image area in the target document image can be obtained, thereby constructing a target document image containing multiple image areas. Among them, the candidate image areas and the image areas can be one-to-one corresponding, and the image areas corresponding to the candidate image areas can be the same.

[0135] S204: The computer device determines the region relationship information corresponding to each image region in the target document image.

[0136] In some application embodiments, the method for determining the region relationship information corresponding to each image area in the target document image can be: determining the arrangement order corresponding to each image area in the target document image based on the position order of the candidate image areas in the document image to be processed; and determining the region relationship information corresponding to each image area based on the arrangement order.

[0137] Specifically, based on the above-mentioned construction of a target document image containing multiple image areas, the embodiment of the present application can obtain ordered image areas in the target document image, and assign an arrangement order to each image area according to the position order of the candidate image areas in the document image to be processed. For example, the arrangement order of the image area corresponding to candidate image area A is 1, the arrangement order of the image area corresponding to candidate image area B is 2, the arrangement order of the image area corresponding to candidate image area C is 3, and the arrangement order of the image area corresponding to candidate image area D is 4.

[0138] In some application embodiments, the method for determining the regional relationship information corresponding to each image area according to the arrangement order can be: according to the arrangement order, the target image area is screened out from the image area; according to the positional relationship information between the image area before the target image area in the arrangement order and the target image area, a first preset value is determined; according to the positional relationship information between the image area after the target image area in the arrangement order and the target image area, a second preset value is determined; according to the first preset value and the second preset value, the regional relationship information of the target image area is determined to obtain the regional relationship information corresponding to each image area.

[0139] Based on the above, for example, since the arrangement order of the image area corresponding to candidate image area A is 1, the arrangement order of the image area corresponding to candidate image area B is 2, the arrangement order of the image area corresponding to candidate image area C is 3, and the arrangement order of the image area corresponding to candidate image area D is 4, based on this, when the image area corresponding to candidate image area B is the target image area, the positional relationship between the image area corresponding to candidate image area B and the image area corresponding to candidate image area A is 0; the positional relationship between the image area corresponding to candidate image area B and the image area corresponding to candidate image area C is 1; and the positional relationship between the image area corresponding to candidate image area B and the image area corresponding to candidate image area D is 1.

[0140] When the image area corresponding to candidate image area A is the target image area, or when the image area corresponding to candidate image area C is the target image area, or when the image area corresponding to candidate image area D is the target image area, please refer to the above and will not repeat them here.

[0141] Based on the above, the embodiment of the present application can determine the sub-image areas in the target image area for each target image area; determine the target arrangement order of the sub-image areas in the target image area according to the position order of the sub-image areas in the target image area; and determine the area relationship information corresponding to each sub-image area according to the target sorting order.

[0142] The embodiment of the present application can determine the sub-image area in the target image area by: performing text recognition on the target image area to obtain candidate text information in the target image area; and determining the sub-image area in the target image area based on the candidate text information. The embodiment of the present application can use a row of candidate text information recognized in the target image area as a sub-image area, thereby obtaining each image area in the target image area. Figure 12 shown.

[0143] The embodiment of the present application represents the reading order of text information in the sub-image area according to the position order of the sub-image areas in the target image area. The position order of the sub-image areas in the embodiment of the present application corresponds one-to-one to the target arrangement order of the sub-image areas in the target image area.

[0144] Among them, the method for the embodiment of the present application to determine the area relationship information corresponding to each sub-image area according to the target sorting order can be: according to the target arrangement order, the target sub-image area is screened out from the sub-image area; according to the positional relationship information between the sub-image area that is before the target sub-image area in the arrangement order and the target sub-image area, a first preset value is determined; according to the positional relationship information between the sub-image area that is after the target sub-image area in the arrangement order and the target sub-image area, a second preset value is determined; according to the first preset value and the second preset value, the area relationship information of the target sub-image area is determined to obtain the area relationship information corresponding to each sub-image area.

[0145] For example, the sub-image area includes a first sub-image area, a second sub-image area, and a third sub-image area. The arrangement order of the first sub-image area is 1, the arrangement order of the second sub-image area is 2, and the arrangement order of the third sub-image area is 3. Based on this, when the second sub-image area is the target sub-image area, the positional relationship between the second sub-image area and the first sub-image area is 0; the positional relationship between the second sub-image area and the third sub-image area is 1.

[0146] Among them, the embodiment of the present application divides the image area into multiple sub-image areas, which can further improve the diversity of target document images and further improve the diversity of training data.

[0147] S205. The computer device generates a document image sample corresponding to the model training task based on the target document image and the region relationship information.

[0148] In some application embodiments, the method for generating document image samples corresponding to the model training task based on the target document image and regional relationship information can be: masking the target text information in the target document image to obtain the masked document image; and labeling the masked document image with the model training task based on the target text information and regional relationship information to obtain the document image samples corresponding to the model training task.

[0149] In an embodiment of the present application, the model training task includes a text recognition training task and a region relationship recognition training task. Based on this, the document image samples of the text recognition training task may include document image samples labeled with the target text information in the target document image; the document image samples of the region relationship recognition training task may include document image samples labeled with the region relationship information.

[0150] The document image sample may also include text information recognized in the target document image.

[0151] In one embodiment of the present application, target text information in a target document image is masked. Specifically, the masked document image can be obtained by randomly masking a certain percentage of the target text information in the target document image. For example, in one embodiment of the present application, 15% of the characters in a sentence in the target document image are randomly selected for masking. Of the 15% of characters in the sentence, 80% are replaced with masks, 10% are replaced with other random characters, and 10% remain the original characters.

[0152] S206. The computer device predicts the document image sample according to the model training task through the document image prediction model to obtain prediction data corresponding to the model training task.

[0153] The document image prediction model can be a self-attention neural network model, or a CNN neural network model, etc. The embodiment of the present application takes the document image prediction model as the StrucTexT model as an example for explanation. Figure 13As shown, the StrucTexT model is a self-supervised language model. The StrucTexT model can self-supervise feature representation learning on large-scale corpus. The network architecture of the StrucTexT model is a multi-layer bidirectional Transformer encoder. The StrucTexT model specifically includes: 12 layers of Transformer, 768 hidden layer parameters, 12 attention heads, and a total parameter volume of approximately 110M. The input of the StrucTexT model can be a fusion of multiple embedding vectors, such as character-level text embedding vectors and box-level visual embedding vectors. The character-level text embedding vector can refer to the embedding vector of text information; the box-level visual embedding vector can refer to the embedding vector with image area information, specifically, it can refer to the embedding vector with text box information.

[0154] Based on the above, when a document image sample is obtained, the StrucTexT model can recognize the document image sample through the optical character (OCR) recognition module to obtain text information and image area information corresponding to the text information; the feature extraction layer of the StrucTexT model can respectively extract features of the text information and the image area information to obtain a text embedding vector corresponding to the text information and a visual embedding vector corresponding to the image area information; then the Transformer layer of the StrucTexT model is used to encode the text embedding vector and the visual embedding vector respectively to obtain encoded text features corresponding to the text embedding vector and encoded visual features corresponding to the visual embedding vector. The embodiment of the present application can make predictions based on the encoded text features and the encoded visual features; the embodiment of the present application can also make predictions based on the encoded text features; the embodiment of the present application can also make predictions based on the encoded visual features.

[0155] Among them, when the embodiment of the present application performs prediction based on the encoded text features and the encoded visual features, the encoded text features and the encoded visual features can be specifically fused to obtain fused features; then, based on the fused features, prediction processing is performed through the StrucTexT model to obtain the prediction results.

[0156] In one embodiment of the application, the model training task includes a text recognition training task and a region relationship recognition training task.

[0157] For the regional relationship recognition training task, the prediction data includes predicted regional relationship information; when the model training task is a target training task, each training subtask corresponding to the target training task is determined; according to each training subtask, the document image prediction model is used to perform regional relationship prediction on the document image sample to obtain the predicted regional relationship information corresponding to each training subtask.

[0158] It should be noted that the region relationship recognition training task can cover at least two training subtasks. For example, the training subtasks include a first training subtask and a second training subtask. The first training subtask can be a task that enables the target document image prediction model to have the ability to recognize region relationships between image regions in a document image through training; the second training subtask can be a task that enables the target document image prediction model to have the ability to recognize region relationships between sub-image regions within an image region in a document image through training.

[0159] Specifically, based on the above, the embodiment of the present application is further described. The embodiment of the present application inputs the document image sample into the document image prediction model, such as Figure 14 As shown, the document image sample is feature extracted through the Transformer layer of the document image prediction model to obtain the image region feature information of the image region corresponding to the first training subtask; the image region is predicted based on the image region feature information through the image region relationship prediction layer to obtain the predicted region relationship information corresponding to the image region.

[0160] Before the embodiment of the present application extracts features from the document image sample through the Transformer layer of the document image prediction model, see Figure 13 The process of processing the document image sample using the optical character (OCR) recognition module and the feature extraction layer described in will not be repeated here. Figure 14 The specific process of using the Transformer layer to extract features from document image samples can also be found in Figure 13 The processing of the Transformer layer described in

[15] will not be repeated here.

[0161] Among them, the sub-image area in the image area has corresponding sub-image area feature information. Based on this, the sub-image area is predicted according to the sub-image area feature information through the sub-image area relationship prediction layer to obtain the predicted area relationship information corresponding to the sub-image area.

[0162] It should be noted here that, for each image region, the embodiment of the present application may average the sub-image feature regions corresponding to all sub-image regions in the image region to obtain image region feature information corresponding to the image region.

[0163] For the text recognition training task, the prediction data includes the target text information in the target document image; the document image sample is predicted by the document image prediction model to obtain the predicted text information corresponding to the text recognition training task.

[0164] S207. The computer device trains the document image prediction model based on the prediction data and the document image samples to obtain a target document image prediction model.

[0165] For the region relationship recognition training task, document image samples are labeled with the region relationship information corresponding to each training subtask. For each training subtask, the loss value is calculated between the predicted region relationship information corresponding to the training subtask and the region relationship information corresponding to the training subtask to obtain the corresponding loss value for each training subtask. The corresponding loss values for each training subtask are then fused to obtain a first fused loss value.

[0166] For the text recognition training task, the document image sample carries a label, which includes the target text information; the loss value between the target text information and the predicted text information corresponding to the text recognition training task is calculated.

[0167] Based on the above, the loss value between the target text information and the predicted text information is added to the above-mentioned first fusion loss value to obtain the target added loss value; the model parameters of the document image prediction model are converged according to the target added loss value to obtain the target document image prediction model.

[0168] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0169] The embodiment of the present application can obtain a document image to be processed, and perform region segmentation processing on the document image to be processed to obtain several candidate image regions corresponding to the document image to be processed; perform fusion processing on the candidate image regions in a preset order to construct a target document image containing image regions; determine the region relationship information corresponding to each image region in the target document image, and generate a document image sample corresponding to the model training task based on the target document image and the region relationship information; train the document image prediction model based on the model training task and the document image sample to obtain the target document image prediction model; since the embodiment of the present application can construct the target document image based on the candidate image regions of the document image to be processed, the document image sample corresponding to the model training task can be generated based on the region relationship information corresponding to each image region in the target document image and the target document image, so that the document image prediction model can be trained based on the model training task to improve the versatility of the trained target document image prediction model.

[0170] Based on the above, Figure 15As shown, embodiments of the present application can fine-tune a target document image prediction model using a small amount of training data to obtain a fine-tuned document image prediction model. For example, taking a table structure recognition task as an example, the input training data is a table image sample and a target label. The table image sample includes a table image and text information recognized in the table image. The label includes cell relationship information of the table image. The cell relationship information can specifically refer to whether the relationship between cells is in the same row or column.

[0171] Among them, the embodiment of the present application can use the target document image prediction model to predict the table image samples to obtain the predicted cell relationship information; converge the target document image prediction model according to the predicted cell relationship information and the cell relationship information to obtain the fine-tuned document image prediction model.

[0172] For example, Figure 16 As shown, the embodiment of the present application can use the Transformer layer of the target document image prediction model to extract features from the table image sample to obtain the cell feature information corresponding to each cell. The feature information corresponding to each cell is as follows Figure 16 As shown in B1, B2, B3, B4, B5, etc. Based on the cell feature information, the prediction layer of the target document image prediction model is used to perform predictions to obtain the predicted cell relationship information, for example, Figure 16 In the example, the cell corresponding to B3 is "Zhang Er," the cell corresponding to B5 is "Project 2," and the cell corresponding to B6 is "Project 5." After prediction, it can be seen that the predicted cell relationship information between B3 and B5 is in the same row, and the predicted cell relationship information between B5 and B6 is in the same column.

[0173] Among them, the embodiment of the present application can use the cross entropy loss function to calculate the loss value between the predicted cell relationship information and the cell relationship information; according to the loss value between the predicted cell relationship information and the cell relationship information, the target document image prediction model is converged to obtain the fine-tuned document image prediction model.

[0174] In order to better implement the above method, an embodiment of the present application also provides a model pre-training device, which can be integrated into a computer device, such as a server or terminal, and the terminal may include a tablet computer, a laptop computer and / or a personal computer, etc.

[0175] For example, Figure 17 As shown, the model pre-training device may include an acquisition unit 301, a fusion unit 302, a generation unit 303 and a training unit 304, as follows:

[0176] (1) Acquisition unit 301;

[0177] The acquisition unit 301 may be used to acquire a document image to be processed, and perform region segmentation processing on the document image to be processed to obtain a number of candidate image regions corresponding to the document image to be processed.

[0178] In some application embodiments, the acquisition unit 301 can be specifically used to determine the number of regions of the required candidate image areas in the document image to be processed; determine the region segmentation position of the document image to be processed based on the number of regions; and perform region segmentation processing on the document image to be processed based on the region segmentation position to obtain several candidate image regions corresponding to the document image to be processed.

[0179] In some application embodiments, the acquisition unit 301 may be specifically configured to acquire the size of the document image to be processed; and determine the region segmentation positions of the document image to be processed according to the number and size of the regions.

[0180] (2) fusion unit 302;

[0181] The fusion unit 302 may be configured to perform fusion processing on the candidate image regions in a preset order to construct a target document image containing the image regions.

[0182] In some application embodiments, the fusion unit 302 can be specifically used to determine the region separation distance between different candidate image regions; determine the target position of the candidate image region in the target document image based on the region separation distance; and merge the candidate image regions in a preset order based on the target position to obtain a target document image containing the image region.

[0183] (3) generating unit 303;

[0184] The generating unit 303 may be configured to determine region relationship information corresponding to each image region in the target document image, and generate a document image sample corresponding to the model training task based on the target document image and the region relationship information.

[0185] In some application embodiments, the generation unit 303 can be specifically used to determine the arrangement order corresponding to each image area in the target document image based on the position order of the candidate image areas in the document image to be processed; and determine the area relationship information corresponding to each image area based on the arrangement order.

[0186] In some application embodiments, the generation unit 303 can be specifically used to filter out the target image area from the image area according to the arrangement order; determine a first preset value based on the positional relationship information between the image area that is before the target image area in the arrangement order and the target image area; determine a second preset value based on the positional relationship information between the image area that is after the target image area in the arrangement order and the target image area; determine the area relationship information of the target image area based on the first preset value and the second preset value to obtain the area relationship information corresponding to each image area.

[0187] In some application embodiments, the generation unit 303 can be specifically used to mask the target text information in the target document image to obtain a masked document image; based on the target text information and regional relationship information, the masked document image is annotated with a model training task to obtain a document image sample corresponding to the model training task.

[0188] (4) training unit 304;

[0189] The training unit 304 may be configured to train the document image prediction model according to the model training task and the document image samples to obtain a target document image prediction model.

[0190] In some application embodiments, the training unit 304 can be specifically used to predict document image samples based on the model training task through the document image prediction model to obtain prediction data corresponding to the model training task; and train the document image prediction model based on the prediction data and the document image samples to obtain a target document image prediction model.

[0191] In some application embodiments, the prediction data includes predicted region relationship information; the training unit 304 can be specifically used to determine each training subtask corresponding to the target training task when the model training task is a target training task; according to each training subtask, the document image prediction model is used to perform region relationship prediction on the document image sample to obtain the predicted region relationship information corresponding to each training subtask.

[0192] In some application embodiments, the document image samples carry labels, which include region relationship information corresponding to each training subtask; the training unit 304 can be specifically used to calculate the loss value between the predicted region relationship information corresponding to the training subtask and the region relationship information corresponding to the training subtask for each training subtask, so as to obtain the loss value corresponding to each training subtask; according to the loss value corresponding to each training subtask, the model parameters of the document image prediction model are converged to obtain the target document image prediction model.

[0193] As can be seen from the above, the acquisition unit 301 of the embodiment of the present application can be used to acquire the document image to be processed, and perform region segmentation processing on the document image to be processed to obtain several candidate image regions corresponding to the document image to be processed; the fusion unit 302 can be used to perform fusion processing on the candidate image regions in a preset order to construct a target document image containing image regions; the generation unit 303 can be used to determine the region relationship information corresponding to each image region in the target document image, and generate a document image sample corresponding to the model training task based on the target document image and the region relationship information; the training unit 304 can be used to train the document image prediction model based on the model training task and the document image sample to obtain the target document image prediction model; since the embodiment of the present application can construct the target document image based on the candidate image region of the document image to be processed, the document image sample corresponding to the model training task can be generated based on the region relationship information corresponding to each image region in the target document image, and the target document image, so that the document image prediction model can be trained based on the model training task to improve the versatility of the trained target document image prediction model.

[0194] The present application also provides a computer device, such as Figure 18 , which shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0195] The computer device may include one or more processing core processors 401, one or more computer readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will understand that Figure 18 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0196] Processor 401 is the control center of the computer device. It connects all components of the computer device using various interfaces and circuits. It executes software programs and / or modules stored in memory 402 and accesses data stored in memory 402 to perform various computer functions and process data. Optionally, processor 401 may include one or more processing cores. Preferably, processor 401 integrates an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and computer programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0197] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0198] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0199] The computer device may further include an input unit 404, which may be used to receive input digital or character information communications and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0200] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the computer device will load the executable files corresponding to one or more computer program processes into the memory 402 according to the following instructions, and the processor 401 will run the computer program stored in the memory 402 to implement various functions as follows:

[0201] A document image to be processed is obtained, and region segmentation processing is performed on the document image to be processed to obtain several candidate image regions corresponding to the document image to be processed; the candidate image regions are fused in a preset order to construct a target document image containing image regions; the region relationship information corresponding to each image region in the target document image is determined, and a document image sample corresponding to the model training task is generated based on the target document image and the region relationship information; a document image prediction model is trained based on the model training task and the document image samples to obtain a target document image prediction model.

[0202] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0203] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0204] To this end, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program can be loaded by a processor to execute any model pre-training method provided in the embodiment of the present application.

[0205] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0206] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0207] Since the instructions stored in the computer-readable storage medium can execute the steps in any model pre-training method provided in the embodiments of the present application, the beneficial effects that can be achieved by any model pre-training method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0208] According to one aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations provided in the above embodiments.

[0209] The above is a detailed introduction to a model pre-training method, computer device, and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method, characterized in that: include: Acquire a document image to be processed, and perform region segmentation processing on the document image to be processed to obtain several candidate image regions corresponding to the document image to be processed, including: Determining a typesetting format of the document image to be processed; obtaining the number of required candidate image regions in the document image to be processed according to the typesetting format; Determining the region segmentation positions of the document image to be processed according to the number of regions; performing region segmentation processing on the document image to be processed according to the region segmentation position to obtain a plurality of candidate image regions corresponding to the document image to be processed; Performing fusion processing on the candidate image regions in a preset order to construct a target document image containing the image regions, wherein the preset order refers to the position order of the candidate image regions in the document image to be processed; Determining region relationship information corresponding to each image region in the target document image, and generating a document image sample corresponding to a model training task based on the target document image and the region relationship information; The document image prediction model is trained according to the model training task and the document image samples to obtain a target document image prediction model.

2. The model training method according to claim 1, characterized in that Determining the region segmentation positions of the document image to be processed according to the number of regions includes: Obtaining the size of the document image to be processed; The region segmentation positions of the document image to be processed are determined according to the number of regions and the sizes.

3. The model training method according to claim 1, characterized in that The step of training the document image prediction model according to the model training task and the document image sample to obtain a target document image prediction model includes: Predicting the document image sample according to the model training task using the document image prediction model to obtain prediction data corresponding to the model training task; The document image prediction model is trained according to the prediction data and the document image samples to obtain a target document image prediction model.

4. The model training method according to claim 3, characterized in that The prediction data includes prediction area relationship information; the document image sample is predicted according to the model training task by the document image prediction model to obtain the prediction data corresponding to the model training task, including: When the model training task is a target training task, determining each training subtask corresponding to the target training task; According to each training subtask, the document image prediction model is used to perform region relationship prediction on the document image sample to obtain predicted region relationship information corresponding to each training subtask.

5. The model training method according to claim 3, characterized in that: The document image samples carry labels, wherein the labels include region relationship information corresponding to each training subtask; and training the document image prediction model based on the prediction data and the document image samples to obtain a target document image prediction model, including: For each training subtask, calculating the loss value between the predicted region relationship information corresponding to the training subtask and the region relationship information corresponding to the training subtask to obtain the loss value corresponding to each training subtask; According to the loss value corresponding to each training subtask, the model parameters of the document image prediction model are converged to obtain the target document image prediction model.

6. The model training method according to claim 1, characterized in that The determining of the region relationship information corresponding to each image region in the target document image includes: Determining an arrangement order corresponding to each image region in the target document image according to a position order of the candidate image regions in the document image to be processed; According to the arrangement order, the region relationship information corresponding to each image region is determined.

7. The model training method according to claim 6, characterized in that Determining the region relationship information corresponding to each image region according to the arrangement order includes: Filtering a target image area from the image areas according to the arrangement order; determining a first preset value based on positional relationship information between an image region preceding the target image region in arrangement order and the target image region; determining a second preset value based on positional relationship information between an image area that follows the target image area in arrangement order and the target image area; The region relationship information of the target image region is determined according to the first preset value and the second preset value to obtain the region relationship information corresponding to each image region.

8. The model training method according to claim 1, characterized in that Generating a document image sample corresponding to the model training task based on the target document image and the region relationship information includes: performing masking on target text information in the target document image to obtain a masked document image; According to the target text information and the region relationship information, the masked document image is labeled with a model training task to obtain a document image sample corresponding to the model training task.

9. The model training method according to claim 1, characterized in that The fusing process of the candidate image regions in a preset order to construct a target document image containing the image regions includes: determining a region separation distance between different candidate image regions; determining a target position of the candidate image region in the target document image according to the region separation distance; According to the target position, the candidate image regions are merged in a preset order to obtain a target document image containing the image region.

10. A model training device, characterized in that: include: An acquisition unit is used to acquire a document image to be processed and perform region segmentation processing on the document image to be processed to obtain a plurality of candidate image regions corresponding to the document image to be processed, including: Determining a typesetting format of the document image to be processed; obtaining the number of required candidate image regions in the document image to be processed according to the typesetting format; Determining the region segmentation positions of the document image to be processed according to the number of regions; performing region segmentation processing on the document image to be processed according to the region segmentation position to obtain a plurality of candidate image regions corresponding to the document image to be processed; a fusion unit, configured to perform fusion processing on the candidate image regions in a preset order to construct a target document image containing the image regions, wherein the preset order refers to the position order of the candidate image regions in the document image to be processed; a production unit, configured to determine region relationship information corresponding to each image region in the target document image, and generate a document image sample corresponding to a model training task based on the target document image and the region relationship information; A training unit is used to train the document image prediction model according to the model training task and the document image sample to obtain a target document image prediction model.

11. A computer device, characterized in that: It comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the model training method described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for loading by a processor to execute the model training method described in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product stores a computer program, which is suitable for loading by a processor to execute the model training method described in any one of claims 1 to 9.

Images