Layout extraction model training method and device and related equipment

Abstract

The invention provides a training method and device for a layout extraction model and related equipment. The method comprises the steps that a first training data set is acquired; performing image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of pieces of second layout layout information, and performing illumination dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of pieces of third layout layout information; and based on the plurality of second images, the plurality of pieces of second layout information, the plurality of third images and the plurality of pieces of third layout information, training a layout extraction model to obtain a target layout extraction model. According to the method and the device, the plurality of second images and the plurality of third images are respectively obtained after the plurality of first images in the first training data set are subjected to the image distortion processing and the illumination dispersion processing, so that the layout extraction model is trained through more layout images under different conditions; and the identification accuracy of the layout image by the layout extraction model is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a training method, apparatus and related equipment for a page layout extraction model. Background Art

[0002] With the rapid development of the Internet and the Internet of Things, electronic documents have gradually become a carrier of information. Among them, the Portable Document Format (PDF) is a file format developed by Adobe and is widely used for document exchange and storage. In the prior art, the layout of the PDF document to be detected is generally detected by using a layout extraction model. However, when used for PDF documents, there will generally be problems such as uneven light distribution and image distortion of text and images, which leads to the problem that the layout extraction model in the prior art has low recognition accuracy for PDF documents. Summary of the invention

[0003] The embodiments of the present application provide a training method, device and related equipment for a layout extraction model to solve the problem of low accuracy of layout image recognition by layout extraction models in the prior art.

[0004] To solve the above problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for training a layout extraction model, the method comprising:

[0006] Acquire a first training data set, wherein the first training data set includes a plurality of first images and a plurality of first layout information corresponding to the plurality of first images, wherein the plurality of first images are layout images without distortion and with uniform illumination;

[0007] Performing image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and performing light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images;

[0008] Based on the multiple second images, the multiple second layout information, the multiple third images and the multiple third layout information, a layout extraction model is trained to obtain a target layout extraction model, and the layout extraction model is used to extract layout information from layout images.

[0009] Optionally, the training of a layout extraction model based on the plurality of second images, the plurality of second layout information, the plurality of third images, and the plurality of third layout information to obtain a target layout extraction model includes:

[0010] Generate a second training data set, wherein the second training data set includes the plurality of first images, the plurality of first layout information, the plurality of second images, the plurality of second layout information, the plurality of third images, and the plurality of third layout information;

[0011] Inputting the second training data set into a target image distortion model for feature extraction to obtain a plurality of image distortion features, and inputting the second training data set into a target illumination detection model for feature extraction to obtain a plurality of image illumination features;

[0012] The layout extraction model is trained based on the multiple image distortion features and the multiple image illumination features to obtain the target layout extraction model.

[0013] Optionally, the method further includes:

[0014] Training the image distortion model based on the plurality of second images and the plurality of second layout information to obtain the target image distortion model; and / or

[0015] The illumination detection model is trained based on the plurality of third images and the plurality of third layout information to obtain the target illumination detection model.

[0016] Optionally, performing image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and performing light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images, comprises:

[0017] Inputting the plurality of first images and the plurality of first layout information into an image distortion model for image distortion processing to obtain a third training data set, wherein the third training data set includes the plurality of second images and the plurality of second layout information;

[0018] The plurality of first images and the plurality of first layout information are input into an illumination dispersion model for illumination dispersion processing to obtain a fourth training data set, wherein the fourth training data set includes the plurality of third images and the plurality of third layout information.

[0019] Optionally, the step of inputting the plurality of first images and the plurality of first layout information into an image distortion model to perform image distortion processing to obtain a third training data set includes:

[0020] Inputting the plurality of first images into the image distortion model, and performing rotation distortion on the plurality of first images respectively to obtain the plurality of second images;

[0021] Calculating a plurality of second layout information corresponding to the plurality of second images according to the plurality of first layout information, wherein the second layout information is used to indicate a distortion angle of the corresponding second image;

[0022] The third training data set is generated based on the plurality of second images and the plurality of second layout information.

[0023] Optionally, the inputting the plurality of first images and the plurality of first layout information into an illumination dispersion model to perform illumination dispersion processing to obtain a fourth training data set includes:

[0024] Inputting the plurality of first images into a light dispersion model, generating random occlusions on the plurality of first images respectively, to obtain a plurality of fifth images;

[0025] Performing blur processing and illumination adjustment on the plurality of fifth images respectively to obtain the plurality of third images;

[0026] A plurality of third layout information corresponding to the plurality of third images is calculated according to the plurality of first layout information, wherein the third layout information is used to indicate the blur degree and illumination of the corresponding third images.

[0027] In a second aspect, an embodiment of the present application provides a method for extracting a layout, the method comprising:

[0028] Obtaining a target image to be processed;

[0029] The target image to be processed is input into the target layout extraction model described in any one of the first aspects to perform layout extraction, so as to obtain layout information of the target image to be processed.

[0030] In a third aspect, an embodiment of the present application further provides a training device for a layout extraction model, the device comprising:

[0031] A first acquisition module is used to acquire a first training data set, wherein the first training data set includes a plurality of first images and a plurality of first layout information corresponding to the plurality of first images, wherein the plurality of first images are layout images without distortion and with uniform illumination;

[0032] a processing module, configured to perform image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and to perform light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images;

[0033] A training module is used to train a layout extraction model based on the multiple second images, the multiple second layout information, the multiple third images and the multiple third layout information to obtain a target layout extraction model, wherein the layout extraction model is used to extract layout information from layout images.

[0034] In a fourth aspect, the present application also provides a layout extraction device, comprising:

[0035] A second acquisition module is used to acquire a target image to be processed;

[0036] The extraction module is used to input the target image to be processed into any of the target layout extraction models mentioned above to perform layout extraction and obtain the layout information of the target image to be processed.

[0037] In the fifth aspect, an embodiment of the present application also provides an electronic device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0038] In a sixth aspect, an embodiment of the present application further provides a readable storage medium for storing a program, wherein when the program is executed by a processor, the program implements the steps in the method described in the first aspect above, or implements the steps in the method described in the second aspect above.

[0039] In the seventh aspect, an embodiment of the present application also provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps in the method described in the first aspect, or to implement the steps in the method described in the second aspect.

[0040] The present application provides a training method, device and related equipment for a layout extraction model, the method comprising: obtaining a first training data set, the first training data set comprising a plurality of first images and a plurality of first layout information corresponding to the plurality of first images, the plurality of first images being layout images without distortion and with uniform illumination; performing image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and performing illumination dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images; based on the plurality of second images, the plurality of second layout information, the plurality of third images and the plurality of third layout information, training a layout extraction model to obtain a target layout extraction model, the layout extraction model being used to extract layout information from layout images. The present application obtains multiple second images and multiple third images respectively after performing image distortion processing and light dispersion processing on multiple first images in a first training data set, thereby training the layout extraction model with layout images in more different situations, thereby improving the recognition accuracy of the layout extraction model for layout images. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a method for training a layout extraction model provided in an embodiment of the present application;

[0042] Figure 2 It is one of the processing schematic diagrams provided in the embodiment of the present application;

[0043] Figure 3 This is the second processing schematic diagram provided in the embodiment of the present application;

[0044] Figure 4 It is a flow chart of a method for extracting layout provided in an embodiment of the present application;

[0045] Figure 5 It is a structural schematic diagram of a training device for a page layout extraction model provided in an embodiment of the present application;

[0046] Figure 6 It is a structural schematic diagram of a layout extraction device provided in an embodiment of the present application;

[0047] Figure 7 It is a schematic diagram of the structure of an electronic device provided by the implementation of this application.

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The terms "first", "second" etc. in the embodiments of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. In addition, the terms "include" and "have" and any variation thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. In addition, "and / or" is used in the present application to represent at least one of connected objects, such as A and / or B and / or C, representing the inclusion of separate A, separate B, separate C, and A and B all exist, B and C all exist, A and C all exist, and 7 situations that A, B and C all exist.

[0051] See also Figure 1 , Figure 1 is a flow chart of a training method for a layout extraction model provided in an embodiment of the present application, such as Figure 1 As shown, the training method of the layout extraction model may include the following steps:

[0052] Step 101: Acquire a first training data set, where the first training data set includes a plurality of first images and a plurality of first layout information corresponding to the plurality of first images, wherein the plurality of first images are layout images without distortion and with uniform illumination.

[0053] In this embodiment, the layout image is described by taking a PDF image as an example, and in other embodiments, it can also be other images. The first training data set includes multiple first images and multiple first layout information, the multiple first images are PDF images without distortion and uniform illumination, and the first layout information is label information, indicating the layout conditions of the multiple first images, such as the typesetting conditions, paragraph conditions, etc. in the PDF image.

[0054] It should be further explained that, in this embodiment, basic data sets of various layouts are prepared, for example, obtained from open source data PubLayNet, DocLayNet and M6Doc.

[0055] Step 102: perform image distortion processing on the multiple first images to obtain multiple second images and multiple second layout information corresponding to the multiple second images, and perform light dispersion processing on the multiple first images to obtain multiple third images and multiple third layout information corresponding to the multiple third images.

[0056] In this embodiment, in order to enrich the diversity of model training, multiple first images are processed to generate different multiple second images and multiple third images, and multiple second layout information corresponding to the multiple second images and multiple third layout information corresponding to the multiple third images are determined.

[0057] Specifically, multiple second images are generated by performing image distortion processing on multiple first images, converting the angle of the PDF image or adjusting the angle of the elements contained in the PDF image. After the multiple second images are generated, multiple second layout information corresponding to the multiple second images needs to be determined based on the multiple second images. Similar to the second image, multiple third images are generated by performing light dispersion processing on multiple first images, that is, by illuminating a part of the PDF image without light or with weak light. After the multiple third images are generated, multiple third layout information corresponding to the multiple third images needs to be determined based on the multiple third images.

[0058] Step 103: Based on the multiple second images, the multiple second layout information, the multiple third images and the multiple third layout information, a layout extraction model is trained to obtain a target layout extraction model, and the layout extraction model is used to extract layout information from layout images.

[0059] In this embodiment, the layout extraction model is trained based on the acquired multiple second images, multiple second layout information, multiple third images and multiple third layout information, and finally a trained target layout extraction model is obtained. It should be noted that the layout extraction model is a model for analyzing and extracting layout structures in documents, web pages or other types of visual content. Its main goal is to identify different layout elements, such as text boxes, images, tables, titles, paragraphs, etc., and extract the location information and attributes of these elements.

[0060] Through the above steps, the accurate analysis and processing of the PDF document layout can be achieved, and the accuracy and robustness of the model recognition of the PDF layout in real scenes facing image rotation distortion and uneven lighting conditions can be improved.

[0061] The present application obtains multiple second images and multiple third images respectively after performing image distortion processing and light dispersion processing on multiple first images in a first training data set, thereby training the layout extraction model with layout images in more different situations, thereby improving the recognition accuracy of the layout extraction model for layout images.

[0062] In some feasible implementations, optionally, training a layout extraction model based on the plurality of second images, the plurality of second layout information, the plurality of third images, and the plurality of third layout information to obtain a target layout extraction model includes:

[0063] Generate a second training data set, wherein the second training data set includes the plurality of first images, the plurality of first layout information, the plurality of second images, the plurality of second layout information, the plurality of third images, and the plurality of third layout information;

[0064] Inputting the second training data set into a target image distortion model for feature extraction to obtain a plurality of image distortion features, and inputting the second training data set into a target illumination detection model for feature extraction to obtain a plurality of image illumination features;

[0065] The layout extraction model is trained based on the multiple image distortion features and the multiple image illumination features to obtain the target layout extraction model.

[0066] In this embodiment, optionally, by expanding the layout data, the basic layout data, the distorted image data set and the non-uniform illumination data set are integrated to form expanded layout data, i.e., a second training data set, wherein the second training data set includes multiple first images, multiple first layout information, multiple second images, multiple second layout information, multiple third images and multiple third layout information.

[0067] Specifically, the image distortion features in the expanded layout data are extracted using the image distortion model, and the image illumination features in the expanded layout data are extracted using the illumination detection model. Thus, the image distortion features and the image illumination features are combined with the features in the original image and input into the layout model for training, and finally the prediction result of the layout model is output, and the layout extraction model is updated according to the prediction result, and finally a trained target layout extraction model is obtained.

[0068] Optionally, the method further includes:

[0069] Training the image distortion model based on the plurality of second images and the plurality of second layout information to obtain the target image distortion model; and / or

[0070] The illumination detection model is trained based on the plurality of third images and the plurality of third layout information to obtain the target illumination detection model.

[0071] In this embodiment, the image distortion model is used to train the plurality of second images and the plurality of second layout information to generate a target image distortion model, wherein the image distortion model is a mathematical model for describing and processing image deformation or distortion. Such models are usually used in the fields of computer vision, image processing, computer graphics, etc. to help study and implement tasks such as image transformation, registration, and enhancement.

[0072] The target illumination detection model is generated by training multiple third images and multiple third layout information through the illumination detection model, wherein the illumination detection model is an algorithm or model used to analyze and evaluate the illumination conditions in an image or scene. Such models are usually used in the fields of computer vision, image processing, robot vision and augmented reality, aiming to identify and understand the position, intensity, direction of the light source and the effect of illumination on the appearance of the object.

[0073] Optionally, performing image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and performing light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images, comprises:

[0074] Inputting the plurality of first images and the plurality of first layout information into an image distortion model for image distortion processing to obtain a third training data set, wherein the third training data set includes the plurality of second images and the plurality of second layout information;

[0075] The plurality of first images and the plurality of first layout information are input into an illumination dispersion model for illumination dispersion processing to obtain a fourth training data set, wherein the fourth training data set includes the plurality of third images and the plurality of third layout information.

[0076] In this embodiment, multiple first images and multiple first layout information are input into the image distortion model for image distortion processing to obtain a third training data set. It should be noted that the image distortion model is the same as the image distortion model in the above embodiment.

[0077] The plurality of first images and the plurality of first layout information are input into the light diffusion model for image distortion processing to obtain a fourth training data set. It should be noted that the light diffusion model is a mathematical model used to describe and simulate the scattering and diffusion behavior of light on the surfaces of different materials. Such models are widely used in computer graphics, computer vision, optical engineering, materials science and other fields.

[0078] Optionally, the step of inputting the plurality of first images and the plurality of first layout information into an image distortion model to perform image distortion processing to obtain a third training data set includes:

[0079] Inputting the plurality of first images into the image distortion model, and performing rotation distortion on the plurality of first images respectively to obtain the plurality of second images;

[0080] Calculating a plurality of second layout information corresponding to the plurality of second images according to the plurality of first layout information, wherein the second layout information is used to indicate a distortion angle of the corresponding second image;

[0081] The third training data set is generated based on the plurality of second images and the plurality of second layout information.

[0082] In this embodiment, if Figure 2 As shown in FIG. 1 , the undistorted dataset is subjected to a 0-360 degree rotation distortion process to generate a distorted image. The distortion angle of each distorted image is recorded and paired with the original image to form a distorted image dataset.

[0083] Specifically, taking a text box in the layout as an example, assuming its coordinates are (x0, y0, x1, y1), its coordinates after rotation around the origin (0, 0) can be obtained through the spatial rotation formula. The rotation formula is as follows:

[0084]

[0085] Optionally, the inputting the plurality of first images and the plurality of first layout information into an illumination dispersion model to perform illumination dispersion processing to obtain a fourth training data set includes:

[0086] Inputting the plurality of first images into a light dispersion model, generating random occlusions on the plurality of first images respectively, to obtain a plurality of fifth images;

[0087] Performing blur processing and illumination adjustment on the plurality of fifth images respectively to obtain the plurality of third images;

[0088] A plurality of third layout information corresponding to the plurality of third images is calculated according to the plurality of first layout information, wherein the third layout information is used to indicate the blur degree and illumination of the corresponding third images.

[0089] In this embodiment, if Figure 3 As shown in Figure 1, an uneven lighting effect is added to the undistorted dataset to generate an image with dispersed lighting.

[0090] The images with dispersed illumination are paired with the original images to form a dataset of non-uniform image illumination.

[0091] The lighting dispersion method is as follows: first, polygons are randomly generated on the white picture background, and these polygons are filled with black to represent the areas where light does not reach; then blur processing is performed, and the mask M is blurred using a Gaussian filter to soften the lighting boundary and make the light and shadow effect more natural; the illumination adjustment, the application of the lighting effect is mathematically achieved through pixel multiplication between the image I and the mask M.

[0092] Where I′(x,y) is the adjusted image pixel value, I(x,y) is the original image pixel value, M(x,y) is the mask pixel value. V is the lighting scaling factor, which varies depending on the lighting effect to be simulated.

[0093] This application proposes a new layout analysis model, which introduces two feature extraction models (distortion detection and light distribution detection) to pre-extract PDF features, enrich the features of the existing layout recognition model, and thus improve the recognition ability and robustness of the layout model. It effectively solves the problem of recognition accuracy caused by image distortion and uneven light in PDF files, and ensures the correctness and robustness of the analysis.

[0094] like Figure 4 As shown, Figure 4 A method for extracting a page layout is provided, the method comprising:

[0095] Step 401: Acquire a target image to be processed.

[0096] Step 402: Input the target image to be processed into the target layout extraction model described in any one of the above embodiments to perform layout extraction, and obtain layout information of the target image to be processed.

[0097] The present application obtains multiple second images and multiple third images respectively after performing image distortion processing and light dispersion processing on multiple first images in a first training data set, thereby training the layout extraction model with layout images in more different situations, thereby improving the recognition accuracy of the layout extraction model for layout images.

[0098] See also Figure 5 , Figure 5 is a structural diagram of a training device for a layout extraction model provided in an embodiment of the present application. Figure 5 As shown, the training device 500 for the layout extraction model includes:

[0099] A first acquisition module 510 is used to acquire a first training data set, wherein the first training data set includes a plurality of first images and a plurality of first layout information corresponding to the plurality of first images, wherein the plurality of first images are layout images without distortion and with uniform illumination;

[0100] The processing module 520 is used to perform image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and to perform light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images;

[0101] The training module 530 is used to train the layout extraction model based on the multiple second images, the multiple second layout information, the multiple third images and the multiple third layout information to obtain a target layout extraction model, and the layout extraction model is used to extract the layout information in the layout image.

[0102] Optionally, the training module 530 includes:

[0103] A generating submodule, configured to generate a second training data set, wherein the second training data set includes the plurality of first images, the plurality of first layout information, the plurality of second images, the plurality of second layout information, the plurality of third images, and the plurality of third layout information;

[0104] an extraction submodule, configured to input the second training data set into a target image distortion model for feature extraction to obtain a plurality of image distortion features, and input the second training data set into a target illumination detection model for feature extraction to obtain a plurality of image illumination features;

[0105] The first training submodule is used to train the layout extraction model based on the multiple image distortion features and the multiple image illumination features to obtain the target layout extraction model.

[0106] Optionally, also include:

[0107] A second training submodule is configured to train the image distortion model based on the plurality of second images and the plurality of second layout information to obtain the target image distortion model; and / or

[0108] The third training submodule is used to train the illumination detection model based on the plurality of third images and the plurality of third layout information to obtain the target illumination detection model.

[0109] Optionally, the processing module 520 includes:

[0110] A first processing submodule is used to input the plurality of first images and the plurality of first layout information into an image distortion model to perform image distortion processing, so as to obtain a third training data set, wherein the third training data set includes the plurality of second images and the plurality of second layout information;

[0111] The second processing submodule is used to input the multiple first images and the multiple first layout information into the illumination dispersion model for illumination dispersion processing to obtain a fourth training data set, wherein the fourth training data set includes the multiple third images and the multiple third layout information.

[0112] Optionally, the first processing submodule includes:

[0113] A rotation unit, used for inputting the plurality of first images into the image distortion model, and performing rotation distortion on the plurality of first images respectively, to obtain the plurality of second images;

[0114] A first calculation unit, configured to calculate a plurality of second layout information corresponding to the plurality of second images according to the plurality of first layout information, wherein the second layout information is used to indicate a distortion angle of the corresponding second image;

[0115] A generating unit is used to generate the third training data set based on the plurality of second images and the plurality of second layout information.

[0116] Optionally, the second processing submodule includes:

[0117] An input unit, used for inputting the plurality of first images into a light dispersion model, generating random occlusions on the plurality of first images respectively, to obtain a plurality of fifth images;

[0118] an adjusting unit, configured to perform blur processing and illumination adjustment on the plurality of fifth images respectively, to obtain the plurality of third images;

[0119] The second calculation unit is used to calculate a plurality of third layout information corresponding to the plurality of third images according to the plurality of first layout information, wherein the third layout information is used to indicate the blur degree and illumination of the corresponding third images.

[0120] The present application obtains multiple second images and multiple third images respectively after performing image distortion processing and light dispersion processing on multiple first images in a first training data set, thereby training the layout extraction model with layout images in more different situations, thereby improving the recognition accuracy of the layout extraction model for layout images.

[0121] See also Figure 6 , Figure 6 is a structural diagram of a layout extraction device provided in an embodiment of the present application. Figure 6 As shown, the layout extraction device 600 includes:

[0122] A second acquisition module is used to acquire a target image to be processed;

[0123] An extraction module is used to input the target image to be processed into the target layout extraction model according to any one of claims 1 to 6 to perform layout extraction, so as to obtain the layout information of the target image to be processed.

[0124] The present application obtains multiple second images and multiple third images respectively after performing image distortion processing and light dispersion processing on multiple first images in a first training data set, thereby training the layout extraction model with layout images in more different situations, thereby improving the recognition accuracy of the layout extraction model for layout images.

[0125] The present application also provides an electronic device. Figure 7 , the electronic device may include a processor 701, a memory 702, and a program 7021 stored in the memory 702 and executable on the processor 701.

[0126] When program 7021 is executed by processor 701, it can achieve Figure 1 or Figure 4 Any step in the corresponding method embodiment:

[0127] Acquire a first training data set, wherein the first training data set includes a plurality of first images and a plurality of first layout information corresponding to the plurality of first images, wherein the plurality of first images are layout images without distortion and with uniform illumination;

[0128] Performing image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and performing light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images;

[0129] Based on the multiple second images, the multiple second layout information, the multiple third images and the multiple third layout information, a layout extraction model is trained to obtain a target layout extraction model, and the layout extraction model is used to extract layout information from layout images.

[0130] Optionally, the training of a layout extraction model based on the plurality of second images, the plurality of second layout information, the plurality of third images, and the plurality of third layout information to obtain a target layout extraction model includes:

[0131] Generate a second training data set, wherein the second training data set includes the plurality of first images, the plurality of first layout information, the plurality of second images, the plurality of second layout information, the plurality of third images, and the plurality of third layout information;

[0132] Inputting the second training data set into a target image distortion model for feature extraction to obtain a plurality of image distortion features, and inputting the second training data set into a target illumination detection model for feature extraction to obtain a plurality of image illumination features;

[0133] The layout extraction model is trained based on the multiple image distortion features and the multiple image illumination features to obtain the target layout extraction model.

[0134] Optionally, the method further includes:

[0135] Training the image distortion model based on the plurality of second images and the plurality of second layout information to obtain the target image distortion model; and / or

[0136] The illumination detection model is trained based on the plurality of third images and the plurality of third layout information to obtain the target illumination detection model.

[0137] Optionally, performing image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and performing light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images, comprises:

[0138] Inputting the plurality of first images and the plurality of first layout information into an image distortion model for image distortion processing to obtain a third training data set, wherein the third training data set includes the plurality of second images and the plurality of second layout information;

[0139] The plurality of first images and the plurality of first layout information are input into an illumination dispersion model for illumination dispersion processing to obtain a fourth training data set, wherein the fourth training data set includes the plurality of third images and the plurality of third layout information.

[0140] Optionally, the step of inputting the plurality of first images and the plurality of first layout information into an image distortion model to perform image distortion processing to obtain a third training data set includes:

[0141] Inputting the plurality of first images into the image distortion model, and performing rotation distortion on the plurality of first images respectively to obtain the plurality of second images;

[0142] Calculating a plurality of second layout information corresponding to the plurality of second images according to the plurality of first layout information, wherein the second layout information is used to indicate a distortion angle of the corresponding second image;

[0143] The third training data set is generated based on the plurality of second images and the plurality of second layout information.

[0144] Optionally, the inputting the plurality of first images and the plurality of first layout information into an illumination dispersion model to perform illumination dispersion processing to obtain a fourth training data set includes:

[0145] Inputting the plurality of first images into a light dispersion model, generating random occlusions on the plurality of first images respectively, to obtain a plurality of fifth images;

[0146] Performing blur processing and illumination adjustment on the plurality of fifth images respectively to obtain the plurality of third images;

[0147] A plurality of third layout information corresponding to the plurality of third images is calculated according to the plurality of first layout information, wherein the third layout information is used to indicate the blur degree and illumination of the corresponding third images.

[0148] or

[0149] Obtaining a target image to be processed;

[0150] The target image to be processed is input into the target layout extraction model described in any one of the first aspects to perform layout extraction, so as to obtain layout information of the target image to be processed.

[0151] The present application obtains multiple second images and multiple third images respectively after performing image distortion processing and light dispersion processing on multiple first images in a first training data set, thereby training the layout extraction model with layout images in more different situations, thereby improving the recognition accuracy of the layout extraction model for layout images.

[0152] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the training method of the layout extraction model or each process of the layout extraction embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0153] An embodiment of the present application further provides a computer program product, which is stored in a storage medium. The computer program product is executed by at least one processor to implement the above-mentioned training method of the layout extraction model or the various processes of the layout extraction embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0154] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0155] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0156] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A training method for a layout extraction model, characterized in that: The method comprises: Acquire a first training data set, wherein the first training data set includes a plurality of first images and a plurality of first layout information corresponding to the plurality of first images, wherein the plurality of first images are layout images without distortion and with uniform illumination; Performing image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and performing light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images; Based on the multiple second images, the multiple second layout information, the multiple third images and the multiple third layout information, a layout extraction model is trained to obtain a target layout extraction model, and the layout extraction model is used to extract layout information from layout images.

2. The method according to claim 1, characterized in that The method of training a layout extraction model based on the plurality of second images, the plurality of second layout information, the plurality of third images, and the plurality of third layout information to obtain a target layout extraction model comprises: Generate a second training data set, wherein the second training data set includes the plurality of first images, the plurality of first layout information, the plurality of second images, the plurality of second layout information, the plurality of third images, and the plurality of third layout information; Inputting the second training data set into a target image distortion model for feature extraction to obtain a plurality of image distortion features, and inputting the second training data set into a target illumination detection model for feature extraction to obtain a plurality of image illumination features; The layout extraction model is trained based on the multiple image distortion features and the multiple image illumination features to obtain the target layout extraction model.

3. The method according to claim 2, characterized in that The method further comprises: Training the image distortion model based on the plurality of second images and the plurality of second layout information to obtain the target image distortion model; and / or The illumination detection model is trained based on the plurality of third images and the plurality of third layout information to obtain the target illumination detection model.

4. The method according to claim 1, characterized in that: The performing image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and performing light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images, comprises: Inputting the plurality of first images and the plurality of first layout information into an image distortion model for image distortion processing to obtain a third training data set, wherein the third training data set includes the plurality of second images and the plurality of second layout information; The plurality of first images and the plurality of first layout information are input into an illumination dispersion model for illumination dispersion processing to obtain a fourth training data set, wherein the fourth training data set includes the plurality of third images and the plurality of third layout information.

5. The method according to claim 4, characterized in that The step of inputting the plurality of first images and the plurality of first layout information into an image distortion model to perform image distortion processing to obtain a third training data set includes: Inputting the plurality of first images into the image distortion model, and performing rotation distortion on the plurality of first images respectively to obtain the plurality of second images; Calculating a plurality of second layout information corresponding to the plurality of second images according to the plurality of first layout information, wherein the second layout information is used to indicate a distortion angle of the corresponding second image; The third training data set is generated based on the plurality of second images and the plurality of second layout information.

6. The method according to claim 4, characterized in that The step of inputting the plurality of first images and the plurality of first layout information into the illumination dispersion model for illumination dispersion processing to obtain a fourth training data set includes: Inputting the plurality of first images into a light dispersion model, generating random occlusions on the plurality of first images respectively, to obtain a plurality of fifth images; Performing blur processing and illumination adjustment on the plurality of fifth images respectively to obtain the plurality of third images; A plurality of third layout information corresponding to the plurality of third images is calculated according to the plurality of first layout information, wherein the third layout information is used to indicate the blur degree and illumination of the corresponding third images.

7. A method for extracting page layout, characterized in that: The method comprises: Obtaining a target image to be processed; The target image to be processed is input into the target layout extraction model according to any one of claims 1 to 6 to perform layout extraction, so as to obtain layout information of the target image to be processed.

8. A training device for a layout extraction model, characterized in that: The device comprises: A first acquisition module is used to acquire a first training data set, wherein the first training data set includes a plurality of first images and a plurality of first layout information corresponding to the plurality of first images, wherein the plurality of first images are layout images without distortion and with uniform illumination; a processing module, configured to perform image distortion processing on the plurality of first images to obtain a plurality of second images and a plurality of second layout information corresponding to the plurality of second images, and to perform light dispersion processing on the plurality of first images to obtain a plurality of third images and a plurality of third layout information corresponding to the plurality of third images; A training module is used to train a layout extraction model based on the multiple second images, the multiple second layout information, the multiple third images and the multiple third layout information to obtain a target layout extraction model, wherein the layout extraction model is used to extract layout information from layout images.

9. A layout extraction device, characterized in that: The device comprises: A second acquisition module is used to acquire a target image to be processed; An extraction module is used to input the target image to be processed into the target layout extraction model according to any one of claims 1 to 6 to perform layout extraction, so as to obtain the layout information of the target image to be processed.

10. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is used to read the program in the memory to implement the steps in the training method of the layout extraction model as described in any one of claims 1 to 6, or to implement the steps in the layout extraction method as described in claim 7.

11. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps in the training method of the layout extraction model as described in any one of claims 1 to 6 are implemented, or the steps in the layout extraction method as described in claim 7 are implemented.

12. A computer program product, characterized in that The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps in the training method of the layout extraction model as described in any one of claims 1 to 6, or to implement the steps in the layout extraction method as described in claim 7.

Images