Multimodal model training method, device, equipment and storage medium

Abstract

The present disclosure provides a multimodal model training method, device, equipment and storage medium, which relates to the field of artificial intelligence technology, especially to the technical fields of computer vision, deep learning, large models, etc. The specific implementation scheme is: using the visual encoding network of the preset multimodal model, feature extraction is performed on the first document image and the second document image to obtain fused feature data, wherein the first document image and the second document image are obtained based on the target document image with the target mask area, and the resolution of the first document image is different from the resolution of the second document image; using the model inference network of the preset multimodal model, model inference is performed on the fused feature data to predict the text content masked by the target mask area; based on the predicted text content masked by the target mask area, the preset multimodal model is trained to obtain the target multimodal model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as computer vision, deep learning, and large models. Background Art

[0002] The in-depth research and development of multimodal large models make it easier to process and analyze different types of data. However, for document images, due to their rich content and diverse modalities, it is challenging for current multimodal models to understand them. Summary of the invention

[0003] The present invention provides a multimodal model training method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, a multimodal model training method is provided, comprising:

[0005] Using a visual encoding network of a preset multimodal model, feature extraction is performed on a first document image and a second document image to obtain fused feature data, wherein the first document image and the second document image are obtained based on a target document image with a target mask area, and a resolution of the first document image is different from a resolution of the second document image;

[0006] Using the model inference network of the preset multimodal model, model inference is performed on the fused feature data to predict the text content covered by the target mask area;

[0007] Based on the predicted text content masked by the target mask area, the preset multimodal model is trained to obtain a target multimodal model.

[0008] According to another aspect of the present disclosure, there is provided a multimodal model training device, comprising:

[0009] A feature extraction unit, configured to extract features from a first document image and a second document image using a visual encoding network of a preset multimodal model to obtain fused feature data, wherein the first document image and the second document image are obtained based on a target document image with a target mask area, and a resolution of the first document image is different from a resolution of the second document image;

[0010] A prediction unit, used to perform model reasoning on the fused feature data using a model reasoning network of a preset multimodal model, so as to predict the text content covered by the target mask area;

[0011] The training unit is used to train the preset multimodal model based on the predicted text content masked by the target mask area to obtain a target multimodal model.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] In this way, the disclosed scheme provides a refined scheme for training a multimodal model. The scheme uses target document images with target mask areas of different resolutions, that is, the first document image and the second document image, to train a preset multimodal model. In this way, the multimodal model's ability to understand document images is improved, so that the multimodal model can make accurate predictions when processing document images of different resolutions, thereby providing strong support for improving user experience.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 This is a schematic flow chart of a multimodal model training method according to an embodiment of the present application. Figure 1 ;

[0022] Figure 2 This is a schematic flow chart of a multimodal model training method according to an embodiment of the present application. Figure 2 ;

[0023] Figure 3 It is a schematic diagram of the feature processing flow of the visual encoding network in the multimodal model training method according to an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of the feature processing flow of the model inference network in the multimodal model training method according to one embodiment of the present application;

[0025] Figure 5 is a schematic diagram of an inference scenario of a target multimodal model in a multimodal model training method according to an embodiment of the present application;

[0026] Figure 6 This is a schematic flow chart of a multimodal model training method according to an embodiment of the present application. Figure 3 ;

[0027] FIG7( a) is a schematic diagram of an initial document image in a multimodal model training method according to an embodiment of the present application;

[0028] FIG7( b) is a schematic diagram of a target document image with a first mask area in a multimodal model training method according to an embodiment of the present application;

[0029] FIG7( c ) is a schematic diagram of a target document image with a second mask area in a multimodal model training method according to an embodiment of the present application;

[0030] FIG7( d ) is a schematic diagram of a target document image with a third mask area in a multimodal model training method according to an embodiment of the present application;

[0031] Figure 8 It is a schematic diagram of a process of performing different levels of mask processing on an initial document image in a multimodal model training method according to an embodiment of the present application;

[0032] Fig. 9 is a structural schematic diagram of a multimodal model training device according to an embodiment of the present application;

[0033] Fig.10 It is a block diagram of an electronic device used to implement the multimodal model training method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0036] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0037] The following is a description of the related technologies of the embodiments of the present disclosure. The following related technologies are optional solutions that can be arbitrarily combined with the technical solutions of the embodiments of the present disclosure, and they all belong to the protection scope of the embodiments of the present disclosure.

[0038] The training of large multimodal models relies on high-quality pre-training data. Based on the pre-training data, the model can align the different modes of the multimodal model, align the visual features to the large language model, and provide world knowledge.

[0039] In practical applications, multimodal large model training usually focuses on two types of data: general data and professional data. General data is generally obtained from the Internet, web pages, books and other channels; professional data includes multilingual data, programming codes, scientific content, etc. The scale and quality of pre-training data have a key impact on the performance of the model. High-quality and rich pre-training data can greatly improve the model effect.

[0040] Based on this, in order to improve the reasoning effect of the large multimodal model, the disclosed scheme provides a model training method, so that the multimodal model obtained after training has stronger perception and understanding capabilities when facing document images of different resolutions, and can accurately predict the text content in the document image.

[0041] Specifically, Figure 1 This is a schematic flow chart of a multimodal model training method according to an embodiment of the present application. Figure 1The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0042] Further, the method includes at least part of the following contents. Figure 1 As shown, including:

[0043] Step S101: using a visual encoding network of a preset multimodal model, extracting features from a first document image and a second document image to obtain fused feature data.

[0044] Here, the first document image and the second document image are obtained based on the target document image with the target mask area.

[0045] It can be understood that the target mask area in the target document image masks at least part of the content.

[0046] Further, the resolution of the first document image is different from the resolution of the second document image. That is, the first document image is a target document image with a first resolution, and the second document image is a target document image with a second resolution, wherein the first resolution and the second resolution are different.

[0047] Step S102: using a model inference network of a preset multimodal model, model inference is performed on the fused feature data to predict the text content masked by the target mask area.

[0048] Step S103: Based on the predicted text content masked by the target mask area, the preset multimodal model is trained to obtain a target multimodal model.

[0049] That is to say, the disclosed solution firstly uses the visual encoding network in the preset multimodal model to extract features from the first document image and the second document image respectively, and obtains fused feature data; secondly, uses the model inference network in the preset multimodal model to process the obtained fused feature data to predict the text content covered by the target mask area; finally, according to the predicted text content covered by the target mask area (also called estimated text content) and the actual text content covered by the target mask area, the preset multimodal model is trained to obtain the target multimodal model. In this way, the document understanding ability of the multimodal model in processing document images of different resolutions is improved.

[0050] In this way, the disclosed scheme provides a refined scheme for training a multimodal model. The scheme uses target document images with target mask areas of different resolutions, that is, the first document image and the second document image, to train a preset multimodal model. In this way, the multimodal model's ability to understand document images is improved, so that the multimodal model can make accurate predictions when processing document images of different resolutions, thereby providing strong support for improving user experience.

[0051] Here, since the disclosed solution uses target document images with target mask areas of different resolutions to train the multimodal model, the multimodal model can better capture the global and local information of the document image, thereby providing strong support for better processing of dense document information.

[0052] It should be noted that, in a specific example, the input of the preset multimodal model may include, in addition to the first document image and the second document image, a target prompt template (Prompt), which is used to guide the preset multimodal model to predict the text content masked by the target mask area (also referred to as the area to be predicted) in the target document image; for example, in an example, the target prompt template may be specifically:

[0053] Prompt: Output the text content in the rectangular box [x0, y0, x1, y1], where [x0, y0] represents the rectangle.

[0054] [x1, y1] represents the coordinates of the upper left corner of the rectangular box, and [x1, y1] represents the coordinates of the lower right corner of the rectangular box.

[0055] Further, in an example, the above-mentioned model inference network using a preset multimodal model to perform model inference on the fused feature data to predict the text content covered by the target mask area (for example, step S102) can also specifically include: using the model inference network of the preset multimodal model, and based on the target prompt template indicating the area to be predicted, perform model inference on the fused feature data to predict the text content covered by the target mask area. In this way, the text content of the target mask area in the target document image can be accurately predicted.

[0056] Further, in a specific example, the preset multimodal model can be trained in the following manner; specifically, based on the predicted text content masked by the target mask area as described above, the preset multimodal model is trained to obtain the target multimodal model (for example, step S103), which can specifically include: based on the predicted text content masked by the target mask area, and the actual text content (that is, the real text content) masked by the target mask area, the loss value of the target loss function (used to measure the difference between the predicted text content and the real text content) is obtained; based on the loss value of the target loss function, some adjustable parameters in the preset multimodal model (for example, some adjustable parameters in the visual encoding network, and some adjustable parameters in the model inference network) are fine-tuned, and the target multimodal model is obtained when the preset training requirements are met. In this way, the multimodal model's ability to perceive and understand document images is improved, so that the multimodal model obtained after training has the ability to process complex document images.

[0057] It should be pointed out that the preset multimodal model may specifically be a large language model (also referred to as a large model (LargeLanguage Model, LLM)), or it may also be other multimodal models, and the present disclosure does not make any specific limitation on this.

[0058] Figure 2 This is a schematic flow chart of a multimodal model training method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0059] Further, the method includes at least part of the following contents. Figure 2 As shown, including:

[0060] Step S201: using a visual encoding network of a preset multimodal model, extracting features from a first document image and a second document image to obtain local feature data of the first document image and global feature data of the second document image.

[0061] Here, the first document image and the second document image are obtained based on the target document image with the target mask area.

[0062] Further, the resolution of the first document image is greater than the resolution of the second document image; that is, the first document image is a target document image with a first resolution, and the second document image is a target document image with a second resolution, wherein the first resolution is greater than the second resolution.

[0063] For example, in one example, the resolution of the first document image is 1024×1024, and the resolution of the second document image is 448×448.

[0064] That is to say, the disclosed solution utilizes a first document image with high resolution to extract local features of the target document image, and at the same time utilizes a second document image with low resolution to extract global features of the target document image, thereby obtaining global and local information of the target document image, thereby providing strong support for improving the perception and understanding capabilities of the multimodal model.

[0065] Step S202: Obtain fused feature data based on the local feature data of the first document image and the global feature data of the second document image.

[0066] Step S203: using the model inference network of the preset multimodal model, model inference is performed on the fused feature data to predict the text content masked by the target mask area.

[0067] Step S204: Based on the predicted text content masked by the target mask area, the preset multimodal model is trained to obtain a target multimodal model.

[0068] In this way, the disclosed scheme can first extract the local feature data of the first document image with high resolution and the global feature data of the second document image with low resolution, so as to obtain the global information and local information of the target document image to the greatest extent, thereby improving the multimodal model's understanding ability on document images, so that the multimodal model can accurately predict when processing document images of different resolutions, thereby providing strong support for improving user experience.

[0069] Further, in a specific example, the following method can be used to obtain local feature data of the first document image and global feature data of the second document image; specifically, the above-mentioned visual encoding network using the preset multimodal model to extract features from the first document image and the second document image to obtain local feature data of the first document image and global feature data of the second document image (for example, the above-mentioned step S201) can specifically include:

[0070] Step S201 - 1 : using a convolution module in a preset multimodal model, extracting local features of the first document image to obtain local feature data of the first document image.

[0071] Step S201 - 2 : Using the visual encoding module in the preset multimodal model, extract the global features of the second document image to obtain global feature data of the second document image.

[0072] Further, in another specific example, the fused feature data may be obtained in the following manner; specifically, the fused feature data (for example, the step S202 described above) may be obtained based on the local feature data of the first document image and the global feature data of the second document image, and may specifically include:

[0073] The feature alignment module in the visual encoding network is used to perform feature fusion on the local feature data of the first document image and the global feature data of the second document image to obtain fused feature data for the target document image.

[0074] That is to say, the disclosed solution presupposes that the visual encoding network of the multimodal model is a mixed resolution network, which includes dual processing branches, wherein a convolution module is provided in the first branch, and at this time, the convolution module in the first branch is used to extract features of the input first document image to obtain local feature data of the first document image; a visual encoding module is provided in the second branch, and at this time, the visual encoding module in the second branch is used to extract features of the input second document image to obtain global feature data of the second document image; finally, the output results of the two branches are feature fused, for example, the feature alignment module in the visual encoding network is used to fuse the local feature data of the first document image with the global feature data of the second document image to obtain fused feature data. In this way, the model can capture the global information and local information of the target document image at the same time, thereby facilitating better processing of dense document information, thereby providing data support for subsequent accurate model reasoning.

[0075] In this way, the disclosed scheme can utilize different modules in the preset multimodal model to perform feature extraction on the input document images of different granularities to obtain fused feature data, and then perform model inference on the fused feature data to train the preset multimodal model. In other words, the disclosed scheme utilizes reasonable feature extraction and feature fusion methods to enable the preset multimodal model to learn more information, such as features of different granularities, thereby improving the perception and understanding ability of the preset multimodal model when processing document images, laying the foundation for subsequent improvement of the model's reasoning quality and reasoning efficiency.

[0076] For example, if Figure 3As shown, in one example, the input of the visual encoding network is two document images of different resolutions, for example, a low-resolution (such as 448×448) document image and a high-resolution (such as 1024×1024) document image; further, the convolutional neural network in the visual encoding network is used to perform feature extraction on the high-resolution document image to capture its local feature data, and the visual recognition network (Vision Transformer, ViT) in the visual encoding network is used to perform feature extraction on the low-resolution document image to obtain its global feature data.

[0077] The processing strategy of the above-mentioned visual encoding network enables the multimodal model to better process dense document information and capture the global and local information of the document image. For example, for low-resolution images, a pre-trained visual recognition network is used to obtain visual tokens (i.e., visual features), and finally obtains a coding token of length L (corresponding to the global feature data described above); for high-resolution images, since convolutional neural networks can better capture local features, convolutional neural networks are used to extract the features of high-resolution images, and through continuous downsampling, the dimensionality reduction and flattening of the coding features of length L are obtained (corresponding to the local feature data described above).

[0078] Furthermore, since the above-mentioned different branches capture features of different granularities, in a specific example, a feature alignment module can also be introduced. Furthermore, the disclosed scheme can also utilize the feature alignment module in the visual encoding network to first align the extracted global feature data with the local feature data, and then fuse the two feature data after feature alignment to obtain fused feature data.

[0079] For example, in one example, the feature alignment module can use the following expression to implement feature fusion:

[0080] F′ vl =F vl +f l (F vl )+g*f h (F vh )

[0081] Here, F′ vl represents the feature after feature fusion, that is, the fusion feature data mentioned above; F vl Represents the global feature data output by the low-resolution branch; F vh represents the local feature data output by the high-resolution branch, f l ,f hBoth represent fully connected layers, which are used for dimension alignment operations; g represents a fusion coefficient, which is a preset value and can be set based on demand.

[0082] It should be noted that, in this example, in addition to using a fully connected layer, a sampling method (such as downsampling) can be used to sample the local feature data to achieve feature alignment, so that the dimension of the sampled local feature data matches the dimension of the global feature data, and then the two are fused, etc. The disclosed solution does not impose any specific restrictions on the method of feature alignment.

[0083] Further, in a specific example, the above-mentioned model inference network using the preset multimodal model to perform model inference on the fused feature data to predict the text content masked by the target mask area (for example, step S203) may specifically include:

[0084] Step S203-1: Using the multi-layer perception module in the preset multimodal model, feature processing is performed on the fused feature data to obtain target perception features.

[0085] Here, in one example, the multi-layer perception module includes multiple linear layers, for example, the multi-layer perception module includes multiple multi-layer perceptron (MLP) networks.

[0086] Step S203 - 2 : Based on the target perception features and the regional features of the target mask region, the content masked by the target mask region is predicted to obtain the text content of the target mask region.

[0087] For example, in one example, Figure 4 As shown, the multi-layer perception module in the model reasoning network of the preset multimodal model performs feature processing on the fused feature data to obtain the processed feature data (corresponding to the above target perception feature), and uses the text encoding module in the model reasoning network to perform text encoding on the target prompt template (such as "output the text content in [x0, y0, x1, y1]") containing the regional features (such as position features) of the target mask area to obtain the prompt encoding feature; then the target perception feature and the prompt encoding feature are spliced ​​and input into the language model module, so as to use the language model module to predict the content masked by the target mask area, and output the text content masked by the target mask area (that is, the predicted text content).

[0088] Furthermore, in one example, the language model module also includes an encoding unit composed of multiple encoding modules (eg, transformer encoders), and a decoding unit composed of multiple decoding modules (eg, transformer decoders).

[0089] In this way, the disclosed scheme provides a refined scheme for model reasoning, which is simple and efficient, and can predict the text content masked by the target mask area based on document images with different resolutions. In this way, the understanding ability of the multimodal model on document images is improved, so that the multimodal model can accurately predict when processing document images with different resolutions, thereby providing strong support for improving user experience.

[0090] For example, after obtaining the target multimodal model using the training method described above, the following method can be used for reasoning: Figure 5 As shown, first, the visual recognition network in the visual encoding network of the target multimodal model is used to extract features of the document image to be predicted (part of the content in the document image to be predicted is masked) to obtain the global visual features of the document image to be predicted, and the convolutional neural network in the visual encoding network of the target multimodal model is used to extract features of the document image to be predicted to obtain the local features of the document image to be predicted; secondly, the feature alignment module in the model inference network of the target multimodal model is used to align the dimensions of the local features of the predicted document image and the global visual features of the document image to be predicted, and then perform feature fusion to obtain to the target fusion features; finally, the target fusion features are processed by multiple MLPs in the model reasoning network of the target multimodal model to obtain perceptual features, and the target prompt template (such as "output the text content in [x0, y0, x1, y1] in the document image to be predicted") is encoded by the text encoding module in the model reasoning network of the target multimodal model to obtain encoding features; and then the language model module in the model reasoning network of the target multimodal model is used to predict the masked content in the document image to be predicted based on the perceptual features and encoding features described above, and then the masked text content is predicted.

[0091] In order to further improve the effect of the multimodal model, the disclosed solution also provides sample data of multiple mask granularities, such as sample data at the text line level, layout paragraph level or length level, so as to further improve the reasoning effect of the multimodal large model.

[0092] Specifically, Figure 6 This is a schematic flow chart of a multimodal model training method according to an embodiment of the present application. Figure 3 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figures 1 to 5 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0093] Further, the method includes at least part of the following contents. Figure 6 As shown, including:

[0094] Step S601: Perform text recognition on the initial document image to obtain a text recognition result.

[0095] Here, since the portable document format (PDF) files of electronic editors have higher data quality and usually include papers, books, journals, etc., which help the model learn general knowledge, the initial document image can specifically be a PDF image of a paper, journal article, book, etc.

[0096] Furthermore, in practical applications, the initial document image may also be data in other formats, and the disclosed solution does not impose any specific limitation on this.

[0097] Furthermore, the text recognition result of the initial document image in this example can be obtained by recognizing the initial document image using model recognition technology. For example, in one example, optical character recognition (OCR) technology is used to perform text recognition on the initial document image to obtain a text recognition result.

[0098] Step S602: Based at least on the text recognition result, masking is performed on part of the content in the initial document image to obtain a target document image with a target mask area.

[0099] Here, the target mask area is used to cover the part of the content in the initial document image.

[0100] It should be noted that, while obtaining the target document image with the target mask area, the regional features (e.g., position information) of the target mask area in the target document image can also be obtained. For example, in one example, the position coordinates [x0, y0, x1, y1] of the target mask area in the target document image are obtained, where [x0, y0] represents the coordinates of the upper left corner of the target mask area, and [x1, y1] represents the coordinates of the lower right corner of the rectangular frame. It can be understood that the above is only an example. In actual applications, the regional features can be represented according to actual needs, and the disclosed solution does not limit this.

[0101] At this point, a large amount of training data can be obtained according to step S601 to step S602. For example, the target document image with the target mask area obtained by the above steps S601 to step S602 and the actual text content of the target mask area can be used as training data, thereby providing data support for subsequent model training of the preset multimodal model.

[0102] Step S603: using a visual encoding network of a preset multimodal model, extracting features from a first document image obtained based on the target document image and a second document image obtained based on the target document image to obtain fused feature data.

[0103] Here, the resolution of the first document image is different from the resolution of the second document image.

[0104] It should be noted that, for relevant examples of the first document image and the second document image, reference may be made to the above description, which will not be repeated here.

[0105] Step S604: using the model inference network of the preset multimodal model, model inference is performed on the fused feature data to predict the text content masked by the target mask area.

[0106] Step S605: Based on the predicted text content masked by the target mask area, the preset multimodal model is trained to obtain a target multimodal model.

[0107] It should be noted that for relevant examples of the preset multimodal model, please refer to the above description and will not be repeated here.

[0108] In this way, the disclosed scheme provides a refined scheme for obtaining training data, which is simple and efficient, provides sufficient training data for subsequent model training, and helps the multimodal model learn general world knowledge. At the same time, the disclosed scheme also provides a refined scheme for using the obtained training data to train a preset multimodal model. In this way, the trained multimodal model has the ability to understand contextual content, laying the foundation for subsequent improvement of the reasoning effect of the multimodal large model on long documents.

[0109] Further, in a specific example, the target document image with the target mask area can be obtained in the following manner; specifically, the above-mentioned masking process is performed on part of the content in the initial document image based on at least the text recognition result to obtain the target document image with the target mask area (for example, step S602), which can specifically include at least one of the following three methods:

[0110] Method 1: Based on the text recognition result, mask processing is performed on some lines in the initial document image to obtain a target document image with a first mask area.

[0111] Here, the first mask area is used to cover the partial line. That is, this method 1 can obtain a mask image at the text line level.

[0112] For example, taking a journal article as an example, as shown in Figure 7(a), it is a document image of the journal article, which includes the document citation format (such as "XXX-1", "XXX-2"), Chinese title (such as "XY"), Chinese author information (such as "X"), Chinese abstract (such as "XXXYYY"), Chinese keywords (such as "YYYXX"), English title (such as "xy"), English author information (such as "x"), English abstract (such as "xxxyyy"), English keywords (such as "yyyxx") and text (such as "YYY"), etc.

[0113] Furthermore, in method 1, first, text recognition is performed on the initial document image shown in Figure 7(a) to obtain a text recognition result; secondly, based on the text recognition result, at least one line of text content in the initial document image is randomly masked. For example, as shown in Figure 7(b), the target document image after masking of "XXX-2" in the document citation format is obtained.

[0114] Alternatively, in another example, the disclosed solution may also pre-use the text line detection capability of OCR to perform text line detection on the initial document image to obtain detection frames of each text line and the text content within each detection frame, and then perform mask processing on the text content within the detection of each text line based on the detection frame of the text line to obtain a target document image with a first mask area.

[0115] Method 2: Based at least on the text recognition result, mask processing is performed on part of the layout content in the initial document image to obtain a target document image with a second mask area.

[0116] Here, the second mask area is used to cover the part of the layout content. That is to say, this method 2 can obtain a mask image at the layout paragraph level.

[0117] Mode 3: Based at least on the text recognition result, mask processing is performed on at least two paragraphs with contextual relationship in the initial document image (it may also be referred to as mask processing on part of the paragraphs) to obtain a target document image with a third mask area.

[0118] Here, the third mask area is used to mask the at least two paragraphs having a contextual relationship. That is, this method 3 can obtain a mask image at the length level.

[0119] It should be pointed out that the multiple target documents in the training data used to train the preset multimodal model can be obtained by any of the above methods, or can be obtained by any two of the above methods, or can be obtained by all three of the above methods, for example, obtaining training data of at least three granularities, such as text line level, layout paragraph level, and length level.

[0120] It should be noted that the target document image with the target mask area used in the above model training may specifically include at least one of the following:

[0121] A target document image with the first mask area described above;

[0122] A target document image with the second mask area described above;

[0123] The target document image with the third mask area described above.

[0124] In this way, the disclosed scheme provides three methods for generating target document images, so that rich and diverse training data with different mask granularities can be constructed, thereby ensuring the data quality of the training data used to train the preset multimodal model, providing data support for the subsequent improvement of the multimodal model's ability to understand the contextual content of the document image, and laying the foundation for the subsequent improvement of the reasoning effect of the preset multimodal model.

[0125] Furthermore, since the disclosed solution can obtain training data of three different granularities using the above method, it helps to improve the model's understanding of the contextual content of the image, thereby enabling the model to learn general world knowledge.

[0126] Furthermore, in a specific example, before obtaining the target document image with the second mask area, the method further includes:

[0127] Perform layout recognition on the initial document image to obtain the layout structure features of the initial document image.

[0128] Furthermore, the above-mentioned method of performing masking on part of the layout content in the initial document image at least based on the text recognition result to obtain the target document image with the second mask area (ie, method 2) may specifically include:

[0129] Based on the text recognition result and the layout structure feature, randomly determining a region corresponding to at least one layout category from the initial document image;

[0130] Masking is performed on the area corresponding to the determined at least one layout category to obtain a target document image with a second mask area, where the second mask area is used to cover the area corresponding to the at least one layout category.

[0131] Here, the layout structure feature may include multiple layout categories that reflect the document structure of the initial document image. For example, continuing to take the initial document image shown in FIG. 7(a) as an example, at this time, the pre-trained layout recognition model may be used to perform layout recognition on the initial document image to obtain multiple layout categories of the document structure of the initial document image, such as paragraph, title, paragraph title and other layout categories. For example, the disclosed scheme may randomly mask different layout categories, allowing the multimodal model to output the text content corresponding to the layout category; for example, the document title may be masked, and the multimodal model may predict the text content in the masked area to summarize the document content in the text and predict the title content; or, the sub-titles of multiple paragraphs may be masked to predict the specific content of the masked paragraph titles.

[0132] It should be noted that the above-mentioned format recognition processing can be performed before the text recognition processing (for example, step S601), or the format recognition processing and the text recognition processing can be performed simultaneously, and the present disclosure does not impose any specific limitation on this.

[0133] Furthermore, it should be noted that the disclosed scheme randomly determines layout content with a granularity greater than the "row" level (that is, the text content in the area corresponding to at least one layout category) from the initial document image based on the layout structure characteristics of the initial document image. At this time, the mask processing of the layout content in the initial document image can be called layout-level-based mask processing. In this way, by utilizing this mask processing strategy, a target document image with a larger granularity in the mask area can be obtained, providing data support for subsequent model training.

[0134] For example, continuing with the initial document image shown in Figure 7(a), when the text recognition result and layout structure features of the initial document image shown in Figure 7(a) are obtained, based on the text recognition result and the layout structure features, at least one area corresponding to the layout category (for example, the area where the English abstract is located) is randomly determined from the initial document image, and then the area where the English abstract is located in the initial document image is masked to obtain the target document image with the second mask area as shown in Figure 7(c).

[0135] In this way, the disclosed scheme provides a specific scheme for obtaining a target document image with a second mask area, that is, using a random masking method based on layout categories to mask the areas corresponding to different layout categories in the initial document to obtain a target document image with a second mask area. In this way, it is convenient to subsequently use the document images obtained above to train the preset multimodal model, so that the trained preset multimodal model can understand and summarize the text content under different layouts, enhance the understanding of the document structure, thereby improving the model's ability to summarize the document and paragraph content, and at the same time, further improve the model's understanding ability.

[0136] Furthermore, in a specific example, before obtaining the target document image with the third mask area, the method further includes:

[0137] Contextual relationship detection is performed on the initial document image to obtain paragraph structural features of the initial document image. For example, reading order detection is performed on the initial document image to obtain paragraph structural features of the initial document image.

[0138] Furthermore, the above-mentioned method of performing mask processing on at least two paragraphs with a contextual relationship in the initial document image based at least on the text recognition result to obtain a target document image with a third mask area (ie, method 3) may specifically include:

[0139] Based on the text recognition result, the layout structure feature and the paragraph structure feature, randomly determining from the initial document image at least two target continuous paragraphs with upper and lower paragraph relationships composed of the layout categories;

[0140] The determined target continuous paragraphs are subjected to mask processing to obtain a target document image with a third mask area, where the third mask area is used to cover the target continuous paragraphs.

[0141] Here, the paragraph structure feature includes multiple paragraphs that reflect the upper and lower paragraph relationship of the initial document image. For example, continuing with the initial document image shown in FIG7(a), after obtaining the text recognition result, layout structure feature and paragraph structure feature of the initial document image shown in FIG7(a), at least two layout categories of target continuous paragraphs with upper and lower paragraph relationship (for example, Chinese abstract, Chinese keywords, English title, English author information, English abstract and paragraphs where English keywords are located, in other words, all paragraphs between the Chinese abstract and the English keywords) can be randomly determined from the initial document image, and further, the randomly determined target continuous paragraphs are masked to obtain the target document image with the third mask area as shown in FIG7(d).

[0142] It should be noted that compared with method 2, the target continuous paragraphs masked by method 3 have larger content in the initial document image. At this time, the masking processing of the target continuous paragraphs in the initial document image can be called masking processing based on the length level. In this way, by using this masking processing, a target document image with a larger granularity of the mask area can be obtained, which provides data support for subsequent model training and helps to further improve the model's reasoning and understanding capabilities.

[0143] In this way, a specific scheme is provided to obtain a target document image with a third mask area. In this way, training data at the paragraph level is obtained. Compared with the target document images of the above two levels, the target document image at this paragraph level can consider longer contextual content, which is conducive to further improving the model's understanding ability and provides strong support for the subsequent improvement of the model's reasoning effect on long documents.

[0144] In summary, the disclosed scheme provides a method for constructing three types of text random mask pre-training data of different granularities and a method for training a multimodal model using the constructed training data. Here, the training data obtained based on the random masking of text lines can enable the multimodal model to improve the reasoning effect based on the least information, while the training data obtained based on the random masking of text layouts can help improve the multimodal model's ability to summarize and understand documents. The training data obtained based on the random masking of the length level enables the multimodal model to predict more continuous text content, which is beneficial to improving the reasoning effect of the model in the creation of long documents.

[0145] For example, for an initial document image, the disclosed solution can be used to construct training data of different granularities, such as a target document image with a mask region based on a text line level (i.e., a first mask region), a target document image with a mask region based on a layout level (i.e., a second mask region), and a target document image with a mask region based on a length level (i.e., a third mask region); specifically, Figure 8As shown, text recognition is performed on the initial document image to obtain a text recognition result, and based on the text recognition result, a mask process is randomly performed on the area where some lines in the initial document image are located to obtain a target document image with a first mask area, and the text content of the first mask area can also be obtained to serve as label data in subsequent model training; and, layout recognition can also be performed on the initial document image to obtain multiple layout categories of the document structure of the initial document image, and then based on the obtained text recognition result and the multiple layout categories, a part of the layout content is randomly determined from the initial document image, and a mask process is performed on the area where the determined part of the layout content is located to obtain a target document image with a second mask area, and the text content of the first mask area can also be obtained. The text content of the second mask area is used as label data in subsequent model training; the reading order of the initial document image is detected to obtain paragraph structure features, and then based on the obtained text recognition results, multiple layout categories and paragraph structure features, the target continuous paragraphs are randomly determined from the initial document image, and the area where the determined target continuous paragraphs are located is masked to obtain a target document image with a third mask area. At the same time, the text content of the third mask area can also be obtained as label data in subsequent model training. In this way, target document images of different granularities can be obtained, which provides data support for the subsequent training of the multimodal model, thereby laying a foundation for improving the reasoning effect of the multimodal model on long documents.

[0146] In summary, in order to improve the reasoning effect of the multimodal model, the disclosed solution proposes a method for constructing pre-training data of document images with multiple granularities and training the multimodal model, that is, according to the three mask granularities of line level, layout level and length level, a mask is added to the text area in the document image, and then the multimodal model predicts the text content in the mask area. Through the above training strategy, the model has the ability to understand the contextual content of the image and can accurately output the text content of the mask area, which helps the model learn general world knowledge. Specifically, the disclosed solution has the following advantages, including:

[0147] First, high-quality training data. The disclosed solution can use documents in electronic editing (such as PDF format files) to obtain high-quality training data. Because the document data of electronic editing has high data quality, based on these document data, training data of different mask granularities such as line level, layout level, and length level can be constructed, thereby providing data support for improving the training effect of the model.

[0148] Second, efficient processing of multi-resolution images is achieved. The disclosed solution proposes a multimodal model based on a mixed-resolution network, and trains it through the constructed training data to obtain a multimodal model for document images of different resolutions. This greatly improves the recognition effect of text content in images of different resolutions and enhances the perception and understanding ability of the multimodal model.

[0149] Third, the ability to handle multiple tasks. Since the disclosed solution can construct training data of different mask granularities, the multimodal model trained using these training data has the ability to handle multiple tasks. For example, the training data obtained using the masking method based on the line level can improve the model's ability to understand the text content; the training data obtained using the masking method based on the layout level enables the model to understand and summarize the text content under different layouts, enhances the understanding of the document structure, and thus improves the ability to summarize the document and paragraph content; the training data obtained using the masking method based on the length level can enhance the model's ability to create long documents. In short, using training data of different mask granularities not only improves the model's text comprehension, but also enhances the model's ability to extract and reconstruct information from complex documents, thereby improving its adaptability in a multimodal environment.

[0150] The disclosed solution also provides a multimodal model training device, such as Fig. 9 As shown, including:

[0151] A feature extraction unit 901 is used to extract features from a first document image and a second document image using a visual encoding network of a preset multimodal model to obtain fused feature data, wherein the first document image and the second document image are obtained based on a target document image with a target mask area, and a resolution of the first document image is different from a resolution of the second document image;

[0152] A prediction unit 902 is used to perform model reasoning on the fused feature data using a model reasoning network of a preset multimodal model to predict the text content masked by the target mask area;

[0153] The training unit 903 is used to train the preset multimodal model based on the predicted text content masked by the target mask area to obtain a target multimodal model.

[0154] In a specific example of the disclosed solution, the feature extraction unit is specifically used to:

[0155] Using a visual encoding network of a preset multimodal model, feature extraction is performed on the first document image and the second document image to obtain local feature data of the first document image and global feature data of the second document image; the resolution of the first document image is greater than the resolution of the second document image;

[0156] Based on the local feature data of the first document image and the global feature data of the second document image, fused feature data is obtained.

[0157] In a specific example of the disclosed solution, the feature extraction unit is specifically used to:

[0158] Using a convolution module in a preset multimodal model, extracting local features of the first document image to obtain local feature data of the first document image;

[0159] The visual encoding module in the preset multimodal model is used to extract the global features of the second document image to obtain the global feature data of the second document image.

[0160] In a specific example of the disclosed solution, the feature extraction unit is specifically used to:

[0161] The feature alignment module in the visual encoding network is used to perform feature fusion on the local feature data of the first document image and the global feature data of the second document image to obtain fused feature data for the target document image.

[0162] In a specific example of the disclosed solution, the prediction unit is specifically used to:

[0163] Using the multi-layer perception module in the preset multimodal model, feature processing is performed on the fused feature data to obtain target perception features;

[0164] Based on the target perception features and the regional features of the target mask area, the content masked by the target mask area is predicted to obtain the text content masked by the target mask area.

[0165] In a specific example of the disclosed solution, the system further includes a preprocessing unit, wherein the preprocessing unit is used to:

[0166] Performing text recognition on the initial document image to obtain a text recognition result;

[0167] At least based on the text recognition result, mask processing is performed on part of the content in the initial document image to obtain a target document image with a target mask area; wherein the target mask area is used to cover up the part of the content in the initial document image.

[0168] In a specific example of the disclosed solution, the preprocessing unit is specifically used for at least one of the following:

[0169] Based on the text recognition result, masking is performed on some lines in the initial document image to obtain a target document image with a first mask area; the first mask area is used to cover the some lines;

[0170] At least based on the text recognition result, masking is performed on part of the layout content in the initial document image to obtain a target document image with a second mask area; the second mask area is used to cover up the part of the layout content;

[0171] At least based on the text recognition result, at least two paragraphs with contextual relationship in the initial document image are masked to obtain a target document image with a third mask area; the third mask area is used to cover the at least two paragraphs with contextual relationship.

[0172] In a specific example of the disclosed solution, the preprocessing unit is specifically used to:

[0173] Performing layout recognition on the initial document image to obtain layout structure features of the initial document image; the layout structure features include multiple layout categories that reflect the document structure of the initial document image;

[0174] Based on the text recognition result and the layout structure feature, randomly determining a region corresponding to at least one layout category from the initial document image;

[0175] Masking is performed on the area corresponding to the determined at least one layout category to obtain a target document image with a second mask area, where the second mask area is used to cover the area corresponding to the at least one layout category.

[0176] In a specific example of the disclosed solution, the preprocessing unit is specifically used to:

[0177] Performing context relationship detection on the initial document image to obtain paragraph structure features of the initial document image, wherein the paragraph structure features include a plurality of paragraphs reflecting the upper and lower paragraph relationships of the initial document image;

[0178] Based on the text recognition result, the layout structure feature and the paragraph structure feature, randomly determining from the initial document image at least two target continuous paragraphs with upper and lower paragraph relationships composed of the layout categories;

[0179] The determined target continuous paragraphs are subjected to mask processing to obtain a target document image with a third mask area, where the third mask area is used to cover the target continuous paragraphs.

[0180] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0181] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0182] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0183] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0184] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0185] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0186] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a multimodal model training method. For example, in some embodiments, the multimodal model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the multimodal model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the multimodal model training method in any other suitable manner (e.g., by means of firmware).

[0187] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0189] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0191] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0192] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0193] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0194] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multimodal model training method, comprising: Using a visual encoding network of a preset multimodal model, feature extraction is performed on a first document image and a second document image to obtain fused feature data, wherein the first document image and the second document image are obtained based on a target document image with a target mask area, and a resolution of the first document image is different from a resolution of the second document image; Using the model inference network of the preset multimodal model, model inference is performed on the fused feature data to predict the text content covered by the target mask area; Based on the predicted text content masked by the target mask area, the preset multimodal model is trained to obtain a target multimodal model.

2. The method according to claim 1, wherein: The method of using a visual encoding network of a preset multimodal model to extract features from the first document image and the second document image to obtain fused feature data includes: Using a visual encoding network of a preset multimodal model, feature extraction is performed on the first document image and the second document image to obtain local feature data of the first document image and global feature data of the second document image; the resolution of the first document image is greater than the resolution of the second document image; Based on the local feature data of the first document image and the global feature data of the second document image, fused feature data is obtained.

3. The method according to claim 2, wherein: The method of using a visual encoding network of a preset multimodal model to extract features from the first document image and the second document image to obtain local feature data of the first document image and global feature data of the second document image includes: Using a convolution module in a preset multimodal model, extracting local features of the first document image to obtain local feature data of the first document image; The visual encoding module in the preset multimodal model is used to extract the global features of the second document image to obtain the global feature data of the second document image.

4. The method according to claim 2 or 3, wherein: The method of obtaining fused feature data based on the local feature data of the first document image and the global feature data of the second document image includes: The feature alignment module in the visual encoding network is used to perform feature fusion on the local feature data of the first document image and the global feature data of the second document image to obtain fused feature data for the target document image.

5. The method according to any one of claims 1 to 4, wherein: The model inference network of the preset multimodal model is used to perform model inference on the fused feature data to predict the text content covered by the target mask area, including: Using the multi-layer perception module in the preset multimodal model, feature processing is performed on the fused feature data to obtain target perception features; Based on the target perception features and the regional features of the target mask area, the content masked by the target mask area is predicted to obtain the text content masked by the target mask area.

6. The method according to any one of claims 1 to 5, further comprising: Performing text recognition on the initial document image to obtain a text recognition result; At least based on the text recognition result, mask processing is performed on part of the content in the initial document image to obtain a target document image with a target mask area; wherein the target mask area is used to cover up the part of the content in the initial document image.

7. The method according to claim 6, wherein: The step of performing mask processing on part of the content in the initial document image at least based on the text recognition result to obtain a target document image with a target mask area includes at least one of the following: Based on the text recognition result, masking is performed on some lines in the initial document image to obtain a target document image with a first mask area; The first mask area is used to mask the part of rows; At least based on the text recognition result, masking is performed on part of the layout content in the initial document image to obtain a target document image with a second mask area; The second mask area is used to cover the part of the layout content; At least based on the text recognition result, masking is performed on at least two paragraphs with a contextual relationship in the initial document image to obtain a target document image with a third mask area; The third mask area is used to mask the at least two paragraphs having a contextual relationship.

8. The method according to claim 7, further comprising: Performing layout recognition on the initial document image to obtain layout structure features of the initial document image; the layout structure features include multiple layout categories that reflect the document structure of the initial document image; The step of performing masking on part of the layout content in the initial document image at least based on the text recognition result to obtain a target document image with a second mask area includes: Based on the text recognition result and the layout structure feature, randomly determining a region corresponding to at least one layout category from the initial document image; Masking is performed on the area corresponding to the determined at least one layout category to obtain a target document image with a second mask area, where the second mask area is used to cover the area corresponding to the at least one layout category.

9. The method according to claim 7, further comprising: Performing context relationship detection on the initial document image to obtain paragraph structure features of the initial document image, wherein the paragraph structure features include a plurality of paragraphs reflecting the upper and lower paragraph relationships of the initial document image; The step of performing masking on at least two paragraphs with a contextual relationship in the initial document image based at least on the text recognition result to obtain a target document image with a third mask area includes: Based on the text recognition result, the layout structure feature and the paragraph structure feature, randomly determining from the initial document image at least two target continuous paragraphs with upper and lower paragraph relationships composed of the layout categories; The determined target continuous paragraphs are subjected to mask processing to obtain a target document image with a third mask area, where the third mask area is used to cover the target continuous paragraphs.

10. A multimodal model training device, comprising: A feature extraction unit, configured to extract features from a first document image and a second document image using a visual encoding network of a preset multimodal model to obtain fused feature data, wherein the first document image and the second document image are obtained based on a target document image with a target mask area, and a resolution of the first document image is different from a resolution of the second document image; A prediction unit, used to perform model reasoning on the fused feature data using a model reasoning network of a preset multimodal model, so as to predict the text content covered by the target mask area; The training unit is used to train the preset multimodal model based on the predicted text content masked by the target mask area to obtain a target multimodal model.

11. The device according to claim 10, wherein: The feature extraction unit is specifically used for: Using a visual encoding network of a preset multimodal model, feature extraction is performed on the first document image and the second document image to obtain local feature data of the first document image and global feature data of the second document image; the resolution of the first document image is greater than the resolution of the second document image; Based on the local feature data of the first document image and the global feature data of the second document image, fused feature data is obtained.

12. The device according to claim 11, wherein The feature extraction unit is specifically used for: Using a convolution module in a preset multimodal model, extracting local features of the first document image to obtain local feature data of the first document image; The visual encoding module in the preset multimodal model is used to extract the global features of the second document image to obtain the global feature data of the second document image.

13. The device according to claim 11 or 12, wherein: The feature extraction unit is specifically used for: The feature alignment module in the visual encoding network is used to perform feature fusion on the local feature data of the first document image and the global feature data of the second document image to obtain fused feature data for the target document image.

14. The device according to any one of claims 10 to 13, wherein: The prediction unit is specifically used for: Using the multi-layer perception module in the preset multimodal model, feature processing is performed on the fused feature data to obtain target perception features; Based on the target perception features and the regional features of the target mask area, the content masked by the target mask area is predicted to obtain the text content masked by the target mask area.

15. The device according to any one of claims 10 to 14, further comprising: A preprocessing unit; wherein the preprocessing unit is used to: Performing text recognition on the initial document image to obtain a text recognition result; At least based on the text recognition result, mask processing is performed on part of the content in the initial document image to obtain a target document image with a target mask area; wherein the target mask area is used to cover up the part of the content in the initial document image.

16. The device according to claim 15, wherein: The preprocessing unit is specifically configured to perform at least one of the following: Based on the text recognition result, masking is performed on some lines in the initial document image to obtain a target document image with a first mask area; the first mask area is used to cover the some lines; At least based on the text recognition result, masking is performed on part of the layout content in the initial document image to obtain a target document image with a second mask area; The second mask area is used to cover the part of the layout content; At least based on the text recognition result, masking is performed on at least two paragraphs with a contextual relationship in the initial document image to obtain a target document image with a third mask area; The third mask area is used to mask the at least two paragraphs having a contextual relationship.

17. The device according to claim 16, wherein: The preprocessing unit is specifically used for: Performing layout recognition on the initial document image to obtain layout structure features of the initial document image; the layout structure features include multiple layout categories that reflect the document structure of the initial document image; Based on the text recognition result and the layout structure feature, randomly determining a region corresponding to at least one layout category from the initial document image; Masking is performed on the area corresponding to the determined at least one layout category to obtain a target document image with a second mask area, where the second mask area is used to cover the area corresponding to the at least one layout category.

18. The device according to claim 16, wherein: The preprocessing unit is specifically used for: Performing context relationship detection on the initial document image to obtain paragraph structure features of the initial document image, wherein the paragraph structure features include a plurality of paragraphs reflecting the upper and lower paragraph relationships of the initial document image; Based on the text recognition result, the layout structure feature and the paragraph structure feature, randomly determining from the initial document image at least two target continuous paragraphs with upper and lower paragraph relationships composed of the layout categories; The determined target continuous paragraphs are subjected to mask processing to obtain a target document image with a third mask area, where the third mask area is used to cover the target continuous paragraphs.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images