A document layout element detection method, device, storage medium and equipment

Abstract

The application discloses a document layout element detection method and device, a storage medium and equipment. The method comprises the following steps: firstly, obtaining a target image in which a target document to be detected is located; then, constructing a preset coding vector corresponding to a preset layout element type; next, inputting the target image and the coding vector into a pre-constructed document layout element detection model to predict a layout element detection result corresponding to the target document; wherein, the document layout element detection model is trained according to a preset document mixed element by using a pre-training mode of contrast learning and mask prediction. It can be seen that, since the document layout element detection model trained according to the preset document mixed element is used to detect the layout element of the target document, the detection efficiency and accuracy of the layout element can be effectively improved, and the self-defined detection can be performed according to the preset layout element type on demand in the detection process, thereby improving the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a document layout element detection method and device, a storage medium and equipment. BACKGROUND

[0002] With the advent of the information age, the amount of information to be processed is growing at a geometric rate. And documents are everywhere in people's daily life, such as teaching aids, homework and test papers in the education industry, etc. In this case, how to realize more rapid and accurate detection and analysis processing of document layout elements to obtain useful information has become an important research topic.

[0003] At present, the target detection scheme is usually used to directly detect each layout element in the image of the document, such as the header, footer, title, page number, etc., by using the optical character recognition (OCR) recognition technology. However, this detection scheme only detects according to the text semantic information in the image of the document, and the detection basis is not sufficient enough. When there are many layout elements in the image of the document, the detection accuracy will be greatly reduced, and the final layout element detection result of the image of the document will not be accurate enough. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a document layout element detection method and device, a storage medium and equipment, which can improve the detection efficiency and accuracy of document layout elements.

[0005] The embodiments of the present application provide a document layout element detection method, which comprises:

[0006] Obtaining a target image in which a target document to be detected is located;

[0007] According to a preset layout element type, an encoding vector corresponding to the preset layout element is constructed;

[0008] The target image and the encoding vector are input into a pre-constructed document layout element detection model to predict a layout element detection result corresponding to the target document. The document layout element detection model is trained according to a preset document mixed element by using a pre-training method of contrast learning and mask prediction.

[0009] In a possible implementation manner, the step of inputting the target image and the encoding vector into the pre-constructed document layout element detection model to predict the layout element detection result corresponding to the target document comprises:

[0010] input the target image into a pre-constructed document layout element detection model, and extract image features of the target image;

[0011] input the encoding vector into a pre-constructed document layout element detection model, and extract a feature vector corresponding to the encoding vector;

[0012] fuse the feature vector corresponding to the encoding vector and the image features of the target image by using a preset image style migration processing algorithm, to obtain fused features;

[0013] decode the fused features to obtain a decoding result of the target image, and perform target layout element detection processing on the decoding result to predict a layout element detection result corresponding to the target document.

[0014] In a possible implementation manner, the layout element detection result corresponding to the target document includes position information and attribute information of each layout element in the target document.

[0015] In a possible implementation manner, the method further includes:

[0016] obtaining a sample image in which a document without annotation is located, and randomly selecting a first sample image, a second sample image, and a third sample image from the sample image;

[0017] performing page extraction on the first sample image to obtain a fourth sample image;

[0018] training an initial image feature extraction model by using the first sample image, the second sample image, the fourth sample image, and a target loss function in a contrast learning training manner, and training the initial image feature extraction model by using the third sample image in a mask prediction training manner to generate an image feature extraction model, and using the image feature extraction model as an initialization structure of an image feature extraction part of the document layout element detection model.

[0019] In a possible implementation manner, the target loss function is a triplet loss function, and the triplet loss function is used to shorten a distance between image features of the first sample image and image features of the fourth sample image extracted by the model and to lengthen a distance between image features of the second sample image and image features of the fourth sample image.

[0020] In a possible implementation manner, the document layout element detection model is constructed in the following manner:

[0021] obtaining a fifth sample image in which a sample document containing a preset document mixed element is located;

[0022] construct a sample code vector corresponding to the preset document mixed element according to the preset document mixed element type;

[0023] train an initial document layout element detection model according to the fifth sample image and the sample code vector, to generate the document layout element detection model;

[0024] The initial document layout element detection model includes an image feature extraction part, a code vector feature extraction part, an image style migration processing part, and a decoding part.

[0025] In a possible implementation, the method further includes:

[0026] obtaining a verification image in which a verification document containing a verification document mixed element is located;

[0027] constructing a verification code vector corresponding to the verification document mixed element according to the verification document mixed element type;

[0028] inputting the verification image and the verification code vector into the document layout element detection model to predict a verification layout element detection result corresponding to the verification document;

[0029] when the verification layout element detection result is inconsistent with a layout element corresponding to the verification document, reusing the verification image as the fifth sample image to update the document layout element detection model.

[0030] In a possible implementation, the preset layout element includes at least one of a header, a title, a picture, a question number, an answer text line, and a stem text line.

[0031] In a possible implementation, the code vector is a one-hot code vector.

[0032] Embodiments of the present application further provide a document layout element detection device, which includes:

[0033] a first obtaining unit configured to obtain a target image in which a target document to be detected is located;

[0034] a first constructing unit configured to construct a code vector corresponding to a preset layout element according to a preset layout element type;

[0035] a first predicting unit configured to input the target image and the code vector into a pre-constructed document layout element detection model to predict a layout element detection result corresponding to the target document; the document layout element detection model is trained according to a preset document mixed element by using a pre-training manner of contrast learning and mask prediction.

[0036] In a possible implementation manner, the first prediction unit comprises:

[0037] The first input subunit is configured to input the target image into a pre-constructed document layout element detection model, and extract image features of the target image.

[0038] The second input subunit is configured to input the encoding vector into the pre-constructed document layout element detection model, and extract a feature vector corresponding to the encoding vector.

[0039] The fusion subunit is configured to perform fusion processing on the feature vector corresponding to the encoding vector and the image features of the target image by using a preset image style migration processing algorithm, to obtain fusion features.

[0040] The prediction subunit is configured to perform decoding processing on the fusion features, to obtain a decoding result of the target image, and perform target layout element detection processing on the decoding result, to obtain the layout element detection result corresponding to the target document.

[0041] In a possible implementation manner, the layout element detection result corresponding to the target document comprises position information and attribute information of each layout element in the target document.

[0042] In a possible implementation manner, the apparatus further comprises:

[0043] The second acquisition unit is configured to acquire a sample image in which the unlabeled document is located, and randomly select a first sample image, a second sample image and a third sample image from the sample image.

[0044] The extraction unit is configured to perform page extraction on the first sample image, to obtain a fourth sample image.

[0045] The training unit is configured to train an initial image feature extraction model by using the first sample image, the second sample image, the fourth sample image and a target loss function, by using a training manner of contrast learning, and train the initial image feature extraction model by using the third sample image, by using a training manner of mask prediction, to generate an image feature extraction model, and use the image feature extraction model as an initialization structure of an image feature extraction part of the document layout element detection model.

[0046] In a possible implementation manner, the target loss function is a triplet loss function; and the triplet loss function is used to shorten a distance between image features of the first sample image and image features of the fourth sample image, and lengthen a distance between image features of the second sample image and the image features of the fourth sample image.

[0047] In a possible implementation manner, the apparatus further includes:

[0048] a third obtaining unit, configured to obtain a fifth sample image in which a sample document containing a preset document mixed element is located;

[0049] a second constructing unit, configured to construct a sample encoding vector corresponding to the preset document mixed element according to the preset document mixed element type;

[0050] a generating unit, configured to train an initial document layout element detection model according to the fifth sample image and the sample encoding vector, and generate the document layout element detection model;

[0051] The initial document layout element detection model includes an image feature extraction part, an encoding vector feature extraction part, an image style migration processing part and a decoding part.

[0052] In a possible implementation manner, the apparatus further includes:

[0053] a third obtaining unit, configured to obtain a verification image in which a verification document containing a verification document mixed element is located;

[0054] a third constructing unit, configured to construct a verification encoding vector corresponding to the verification document mixed element according to the verification document mixed element type;

[0055] a second predicting unit, configured to input the verification image and the verification encoding vector into the document layout element detection model, and predict a verification layout element detection result corresponding to the verification document;

[0056] an updating unit, configured to, when the verification layout element detection result is inconsistent with a layout element corresponding to the verification document, re-use the verification image as the fifth sample image, and update the document layout element detection model.

[0057] In a possible implementation manner, the preset layout element includes at least one of a header, a title, a picture, a question number, an answer text line and a stem text line.

[0058] In a possible implementation manner, the encoding vector is a one-hot encoding vector.

[0059] Embodiments of the present application further provide a document layout element detection device, including: a processor, a memory, a system bus;

[0060] The processor and the memory are connected through the system bus;

[0061] The memory is configured to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform any one of the implementation manners of the document layout element detection method.

[0062] The embodiment of the present application further provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions run on a terminal device, the terminal device executes any one of the implementation manners of the document layout element detection method.

[0063] The embodiment of the present application further provides a computer program product, and when the computer program product runs on a terminal device, the terminal device executes any one of the implementation manners of the document layout element detection method.

[0064] The embodiment of the present application provides a document layout element detection method, device, storage medium and equipment, first, the target image where the target document to be detected is obtained; then, according to the preset layout element type, the encoding vector corresponding to the preset layout element is constructed; then, the target image and the encoding vector are input into the pre-constructed document layout element detection model, and the layout element detection result corresponding to the target document is predicted; wherein the document layout element detection model is trained according to the preset document mixed element by using the pre-training mode of contrast learning and mask prediction. It can be seen that since the document layout element detection model trained according to the preset document mixed element is used to detect the layout element of the target document, the detection efficiency and accuracy of the layout element can be effectively improved, and the self-defined detection according to the preset layout element type can be performed as needed in the detection process, thereby improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings without creative labor based on these drawings.

[0066] Figure 1 The flowchart of the document layout element detection method provided by the embodiment of the present application is shown in the figure;

[0067] Figure 2 The example diagram of the document layout element provided by the embodiment of the present application is shown in the figure;

[0068] Figure 3 The composition diagram of the document layout element detection model provided by the embodiment of the present application is shown in the figure;

[0069] Figure 4 A schematic diagram of a determination process of an initialization structure of an image feature extraction part of a document layout element detection model provided by an embodiment of the present application;

[0070] Figure 5 A schematic diagram of a training process of a document layout element detection model provided by an embodiment of the present application;

[0071] Figure 6 A schematic diagram of a composition of a document layout element detection device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0072] The analysis and processing of a document can refer to an operation of extracting and structuring element information in the document. The document layout elements can include, but are not limited to, various titles of the document, a header, a footer, pictures contained in the document, and the like. Most of the existing document processing technologies only perform OCR processing on the document to extract text lines and semantics to directly detect various layout elements in the image where the document is located. However, the layout information of the document is easily lost in the processing, the detection basis is not sufficient, the detection effect is poor when there are many layout elements in the image where the document is located, and the user experience is not good.

[0073] Therefore, how to improve the document layout element detection effect and further improve the user experience is a technical problem to be solved at present.

[0074] To solve the above defects, the present application provides a document layout element detection method. First, a target image where a target document to be detected is located is obtained. Then, according to a preset layout element type, a coding vector corresponding to the preset layout element is constructed. Next, the target image and the coding vector are input into a pre-constructed document layout element detection model to predict a layout element detection result corresponding to the target document. The document layout element detection model is trained according to a preset document mixed element by using a pre-training method of contrast learning and mask prediction. It can be seen that, since the document layout element detection model trained according to the preset document mixed element is used to detect the layout elements of the target document, the detection efficiency and accuracy of the layout elements can be effectively improved, and the self-defined detection according to the preset layout element type can be performed as needed during the detection process, thereby improving the user experience.

[0075] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0076] First embodiment

[0077] See Figure 1 This is a flowchart illustrating a document layout element detection method provided in this embodiment. The method includes the following steps:

[0078] S101: Obtain the target image containing the target document to be detected.

[0079] In this embodiment, any document used for layout element detection is defined as the target document to be detected, and the image containing the target document is defined as the target image. Furthermore, this embodiment does not limit the language type of the target document; for example, the target document can be a Chinese document or an English document. This embodiment also does not limit the source of the target document; for example, the target document can be teaching materials, homework, and test papers from the education industry, or it can be from other industries such as medicine and food. This embodiment also does not limit the type of the target document; for example, the target document can be a company's articles of association or a student's subject exercise book.

[0080] Furthermore, this embodiment does not limit the type of the target image. For example, the target image can be a color image composed of the three primary colors of red (R), green (G), and blue (B), or it can be a grayscale image. Also, this embodiment does not limit the resolution of the target image. For example, the target image can be a 720*480 RGB image or a high-resolution 1920*1280 RGB image. This embodiment also does not limit the method of acquiring the target image; for example, it can be obtained by taking a picture of the document using a mobile phone camera.

[0081] S102: Construct the encoding vector corresponding to the preset layout element type.

[0082] In this embodiment, after obtaining the target image of the target document to be detected through step S101, in order to improve the detection efficiency and accuracy of the document layout elements of the target document, it is further necessary to construct the encoding vector corresponding to the preset layout elements according to the preset layout element type, so as to execute the subsequent step S103.

[0083] The types of preset layout elements can be set according to the user's actual needs. The specific content is not limited in this application. For example, one possible implementation is that the preset layout elements can include, but are not limited to, at least one of the following: header, title, image, question number, answer text line, and question stem text line.

[0084] For example: Figure 2 As shown in the diagram, the text line is used as the basic unit, and the layout element to which this text line belongs is also identified. It should be noted that...Figure 2 Each layout element (such as "answer text line") shown in the figure is only exemplarily shown by a box and an arrow and a corresponding text line of the text, and all layout elements and corresponding text lines are not all shown, and the position information of each layout element can be identified by a box, and can also be identified by other marks, which will not be described here.

[0085] Furthermore, the specific format of the encoding vector corresponding to the preset layout element is not limited in the present application, and can be set according to actual conditions and experience values. A preferred implementation manner is that the encoding vector can be set as a one-hot encoding vector. That is, the preset layout element can be one-hot encoded according to the type of the preset layout element. Specifically, the one-hot encoding vector corresponding to the preset layout element can be represented by the number of types of the preset layout element, for example, when the preset layout element contains "page header, title, picture, question number, answer text line, stem text line", the dimension of the corresponding one-hot encoding vector can be 6.

[0086] For example, still taking Figure 2 Taking the six layout elements shown as preset layout elements as an example, if single layout element encoding is performed, when the preset layout element is only "page header", the corresponding one-hot encoding vector is (1, 0, 0, 0, 0, 0); the one-hot encoding vector corresponding to "title" is (0, 1, 0, 0, 0, 0); the encoding of "stem text line" is (0, 0, 0, 0, 0, 1), and so on. Alternatively, multi-layout element encoding can also be performed, for example, when the preset layout element is "page header, title", the corresponding one-hot encoding vector is (1, 1, 0, 0, 0, 0), and so on.

[0087] S103: input the target image and the encoding vector into the pre-constructed document layout element detection model to predict the layout element detection result corresponding to the target document; wherein the document layout element detection model is trained according to the preset document mixed element by using the pre-training manner of contrast learning and mask prediction.

[0088] In the present embodiment, after the target image in which the target document to be detected is acquired by step S101, and the encoding vector corresponding to the preset layout element is constructed by step S102, in order to improve the detection efficiency and accuracy of the document layout element of the target document, the target image and the encoding vector can be further input into the pre-constructed document layout element detection model, so as to predict the layout element detection result of the target document.

[0089] It should be noted that the present application does not limit the specific content type contained in the layout element detection result of the target document. An optional implementation is that the layout element detection result corresponding to the target document can include, but is not limited to, the position information and attribute information of each layout element in the target document. For example, taking the "title" layout element as an example, the corresponding position information can represent the specific position of the "title" in the target document, and the corresponding attribute information can represent the font size, whether it has an underline, whether it is bold, whether it is centered, etc.

[0090] Specifically, an optional implementation is that after obtaining the target image where the target document to be detected is located, and constructing the encoding vector corresponding to the preset layout element, further, first, the target image can be input into the pre-constructed document layout element detection model to extract the image features of the target image through the model to represent the overall structure content information and the local structure content information of the target image. At the same time, the encoding vector corresponding to the preset layout element can be input into the pre-constructed document layout element detection model to extract the feature vector corresponding to the encoding vector through the model. Then, by using a preset image style transfer processing algorithm, the feature vector of the extracted encoding vector and the image features of the target image are fused, the feature vector of the encoding vector is fused into the image features of the target image, and a fused feature is obtained. Next, the existing or future decoding method can be used to decode the fused feature to detect the target layout element according to the obtained processing result, and further obtain the detection result of the position information and attribute information of the layout element corresponding to the target document.

[0091] In this application, the specific algorithm content of the preset image style transfer processing algorithm is not limited, which can be selected according to the actual situation, such as using AdaIN as the preset image style transfer processing algorithm. And the function realized by the preset image style transfer processing algorithm can be to transfer the style, texture, color and other features in an image (i.e. style image) to another image (i.e. content image), while keeping the main structure of the content image unchanged.

[0092] It should be noted that in order to improve the accuracy of the layout element detection result, the present application will use a large amount of document data containing document mixed elements to pre-train the model by using the pre-training method of contrast learning and mask prediction, so as to construct a document layout element detection model with better detection effect, and the specific network structure of the document layout element detection model is not limited, which can be selected and set according to the actual situation. An optimal implementation is that, as shown in FIG. 6, the document layout element detection model can include an input layer, a feature extraction layer, a feature fusion layer, a feature decoding layer, and an output layer. Figure 3As shown, the pre-constructed document layout element detection model can include but is not limited to an image feature extraction part, an encoding vector feature extraction part, an image style migration processing part, and a decoding part.

[0093] In this application, the specific structure of the image feature extraction part is not limited, which can be trained by the pre-training method described later to obtain an initial image feature extraction model (the specific result is not limited, for example, it can be a convolutional neural network (CNN) and the like). And the specific structure of the encoding vector feature extraction part is not limited, which can be set according to the actual situation, such as using a deep neural network (DNN) as the structure of the encoding vector feature extraction part. And the specific structure of the decoding part is not limited, which can be selected according to the actual situation, so as to perform the detection processing operation of the target layout element after the deconvolution processing.

[0094] In this way, by first training the model according to the preset document mixed element training, the accuracy of the model in detecting the mixed layout elements contained in the document can be effectively improved, so that when the document layout element detection model is used to detect the layout elements of the target document, the accuracy of the detection result can be effectively improved, and the detection effect is improved.

[0095] For example, as shown in Figure 3 Suppose Figure 3 The left image is the target image; the preset layout elements include the header and the title, and the encoding vector corresponding to the preset layout elements is (1, 1, 0, 0, 0, 0), so after the target image is input into the image feature extraction part of the model, the image features of the target image can be extracted. And the encoding vector (1, 1, 0, 0, 0, 0) is input into the encoding vector feature extraction part (DNN) of the model, and the feature vector corresponding to the encoding vector can be obtained, and then the feature vector is fused into the image features of the target image through AdaIN, and the fusion features as shown in Figure 3 After that, the decoding part of the model is used to perform deconvolution processing on the fusion features, and the detection processing operation of the target layout element is performed, and the page and title contained in the target image can be obtained as shown in Figure 3 The identification of the box and the text in the right image.

[0096] Next, the embodiment will introduce the determination process of the initialization structure of the "image feature extraction part" in the document layout element detection model. As shown in Figure 4As shown in FIG. 1, it shows a process diagram for determining the initialization structure of the image feature extraction part of the document layout element detection model provided by the embodiment of the present application. The specific determination process can include the following steps A1-A3:

[0097] Step A1: Obtain a sample image where the unlabeled document is located, and randomly select a first sample image, a second sample image and a third sample image from the sample image.

[0098] In this embodiment, in order to construct the document layout element detection model and determine the initialization structure of the image feature extraction part, a large amount of preparation work needs to be done in advance. First, a large number of training images where training documents are located need to be collected, for example, 100 images where different training documents are located can be collected in advance as sample images, and three different images are randomly selected from them as the first sample image, the second sample image and the third sample image, as shown in FIG. 2. Figure 4 As shown in FIG. 3, it is used to perform subsequent steps A2-A3 in order to determine the initialization structure of the image feature extraction part.

[0099] Step A2: Perform page extraction on the first sample image to obtain a fourth sample image.

[0100] In this embodiment, in order to construct the document layout element detection model and determine the initialization structure of the image feature extraction part, after obtaining the first sample image through step A1, further, the existing or future page feature extraction method can be used to perform page extraction processing on the first sample image, and the extracted image is obtained as the fourth sample image. For example, the four vertices of the "question type" corresponding to the page in the first sample image can be detected first, and then based on the four points, the page content extraction processing is performed to obtain the "rectangular" fourth sample image, as shown in FIG. 4. Figure 4 As shown in FIG. 5, it is used to perform subsequent step A3 in order to determine the initialization structure of the image feature extraction part.

[0101] Step A3: using the first sample image, the second sample image, the fourth sample image and the target loss function, training the initial image feature extraction model through the contrastive learning training mode; and using the third sample image, training the initial image feature extraction model through the mask prediction training mode, generating the image feature extraction model, and taking the image feature extraction model as the initialization structure of the image feature extraction part of the document layout element detection model.

[0102] It should be noted that the initial text encoding model used by the present application can be any existing or future image feature extraction model, such as CNN model or Transformer model, etc., as shown in FIG. 6. Figure 4 ​

[0103] It also needs to be explained that in order to construct the document layout element detection model, the initialization structure of the image feature extraction part is determined, after the third sample image is obtained through step A1, further, a single word detection and recognition processing can be performed on the third sample image, to randomly select a single word from the document in the third sample image for mask processing, and the image after mask is input into the initial text encoding model (such as the CNN network shown in Figure 4 The mask feature is extracted, and the feature is used to predict the mask text content.

[0104] Specifically, in the embodiment, in order to construct the document layout element detection model, the initialization structure of the image feature extraction part is determined, after the first sample image, the second sample image and the fourth sample image are obtained through steps A1 and A2, further, first, the training mode of contrast learning and mask prediction is used, the first sample image, the second sample image, the fourth sample image and the third sample image after mask are input into the initial image feature extraction model for image feature extraction, and the preset target loss function is used for model training, and in the training process, the model parameters of the model can be continuously updated according to the change of the target loss function value, until the target loss function value meets the requirement, such as reaching the minimum value and the change amplitude is very small (basically unchanged), or reaching the preset maximum iteration number (such as 100 times), then stop updating the model parameters, complete the training of the image feature extraction model, and the image feature extraction model is used as the initialization structure of the image feature extraction part of the document layout element detection model.

[0105] Among them, the specific content of the target loss function is not limited in the application, which can be selected according to the actual situation, and one optional implementation mode is that the target loss function can be set as a triplet loss function (TripletLoss), which is used to narrow the distance between the image features of the first sample image and the fourth sample image and to widen the distance between the image features of the second sample image and the fourth sample image.

[0106] In this way, since the training mode of contrast learning can make the image feature extraction model better model the overall structure of the image, and the mask content prediction training mode can also make the image feature extraction model better model the local structure of the image, so that the image feature extraction model obtained after training can efficiently extract the structure content information of the image, which is used as the initialization structure of the image feature extraction part of the document layout element detection model, to extract more accurate feature maps of the target image, which can more favorably improve the accuracy of subsequent document layout element detection.

[0107] Next, the construction process of the document layout element detection will be introduced in this embodiment, and an optional implementation manner is that the construction process of the document layout element detection model can include the following steps B1-B3:

[0108] Step B1: Obtain a fifth sample image containing a sample document with a preset document mixed element.

[0109] In this embodiment, in order to construct the document layout element detection model, a large amount of preparation work needs to be done in advance. First, a large number of training images containing various training documents of various preset document mixed elements (such as page header, title, picture, question number, answer text line, stem text line, etc.) in different fields (such as education, medical treatment, etc.) need to be collected. For example, 500 images containing different training documents of various preset document mixed elements can be collected in advance as the fifth sample image to perform the subsequent step B2.

[0110] Step B2: Construct a sample encoding vector corresponding to the preset document mixed element according to the type of the preset document mixed element.

[0111] In this embodiment, after obtaining the fifth sample image containing the sample document with the preset document mixed element through step B1, a sample encoding vector corresponding to the preset document mixed element can be further constructed according to the type of the preset document mixed element to perform the subsequent step B3. The specific construction manner can be realized by referring to the related introduction of the above step S102, and will not be described here.

[0112] Step B3: Train the initial document layout element detection model according to the fifth sample image and the sample encoding vector to generate a document layout element detection model.

[0113] An optional implementation manner is that the initial document layout element detection model can include but is not limited to an image feature extraction part, an encoding vector feature extraction part, an image style migration processing part, and a decoding part.

[0114] In this embodiment, after the fifth sample image and the sample encoding vector are obtained through steps B1 and B2, further, first, the fifth sample image can be input into the image feature extraction part of the initial document layout element detection model to obtain the sample image feature of the fifth sample image, and then the sample encoding vector is input into the encoding vector feature extraction part of the document layout element detection model to obtain the predicted sample feature vector, then, through the image style migration processing part, the predicted sample feature vector is fused into the sample image feature to obtain the sample fusion feature, and then the sample fusion feature is input into the decoding part of the initial document layout element detection model to predict the layout element detection result corresponding to the sample document in the fifth sample image, as shown in Figure 5

[0115] Then, the predicted result is compared with the real label result of the preset document mixed element corresponding to the sample document in the fifth sample image, so as to constantly update the model parameters of the detection model according to the comparison result, until the preset requirement is met, such as the difference value in the comparison result reaches the minimum value and the change amplitude is very small (basically unchanged), or the preset maximum iteration number (such as 100 times) is reached, then the update of the model parameters is stopped, and the training of the document layout element detection model is completed.

[0116] In this way, any combination of several layout elements (such as Figure 5 selected in the preset document mixed element) can be randomly selected, and then in the corresponding element detection result, only the position information and attribute information of the selected several layout elements need to be predicted, and through such mixed element training, not only the utilization rate of the training data can be greatly improved, but also the user experience can be effectively improved, because the user can customize which layout elements to detect and train, and after the training is completed, the corresponding position and attribute information of the corresponding layout elements in the image can be accurately detected through the trained detection model.

[0117] On this basis, after the document layout element detection model is trained and generated according to the fifth sample image containing the sample document mixed with the preset document mixed element, further, the generated document layout element detection model can also be verified by using the verification image containing the verification document mixed with the verification document mixed element. The specific verification process can include the following steps C1-C4:

[0118] Step C1: obtaining a verification image containing a verification document mixed element.

[0119] ​In this embodiment, in order to realize the verification of the document layout element detection model, first, the verification image in which the verification document containing the verification document mixed element is located is needed to be obtained, so as to execute the subsequent step C2, wherein the verification document in the verification image refers to the text information which can be used to verify the document layout element detection model.

[0120] Step C2: constructing the verification coding vector corresponding to the verification document mixed element according to the type of the verification document mixed element.

[0121] After the verification image in which the verification document containing the verification document mixed element is located is obtained through step C1, further, the verification coding vector corresponding to the verification document mixed element can be constructed according to the type of the verification document mixed element, so as to execute the subsequent step C3, and the specific construction method can be realized by referring to the related introduction of the above-mentioned step S102, which will not be described here.

[0122] Step C3: inputting the verification image and the verification coding vector into the document layout element detection model to predict the verification layout element detection result corresponding to the verification document.

[0123] After the verification image in which the verification document containing the verification document mixed element is located and the verification coding vector corresponding to the verification document mixed element are obtained through steps C1 and C2 respectively, further, the verification image and the verification coding vector can be inputted into the document layout element detection model to predict the verification layout element detection result of the verification document, so as to execute the subsequent step C4.

[0124] Step C4: when the verification layout element detection result is inconsistent with the layout element corresponding to the verification document, the verification image is re-used as the fifth sample image to update the document layout element detection model.

[0125] After the verification layout element detection result corresponding to the verification document is predicted through step C3, if the predicted verification layout element detection result is consistent with the layout element (such as the layout element result marked by artificial) corresponding to the verification document, it can be verified that the detection result of the trained document layout element detection model is relatively accurate; otherwise, if the predicted verification layout element detection result is inconsistent with the layout element (such as the layout element result marked by artificial) corresponding to the verification document, the verification image and the verification document are re-used as the fifth sample image and the sample document respectively to update the parameters of the document layout element detection model, so as to obtain the updated document layout element detection model.

[0126] Through the above embodiment, the document layout element detection model can be effectively verified by using the verification image in which the verification document containing the verification document mixed element is located. When the verification layout element detection result corresponding to the verification document predicted by the model is inconsistent with the layout element corresponding to the verification document (such as the layout element result marked by a person), the document layout element detection model can be timely adjusted and updated, thereby helping to improve the detection precision and accuracy of the model.

[0127] To sum up, the document layout element detection method provided in the embodiment first acquires a target image in which a target document to be detected is located; then constructs an encoding vector corresponding to a preset layout element type; next, inputs the target image and the encoding vector into a pre-constructed document layout element detection model to predict a layout element detection result corresponding to the target document; wherein the document layout element detection model is trained according to a preset document mixed element by using a pre-training manner of contrast learning and mask prediction. It can be seen that since the document layout element detection model trained according to the preset document mixed element is used to detect the layout element of the target document, the detection efficiency and accuracy of the layout element can be effectively improved, and the self-defined detection according to the preset layout element type can also be performed during the detection process, thereby improving the user experience.

[0128] Second embodiment

[0129] The embodiment will introduce a document layout element detection device, and the related content can be referred to the above method embodiment.

[0130] Reference is made to Figure 6 A composition schematic diagram of a document layout element detection device provided in the embodiment is shown in FIG. 6. The device 600 includes:

[0131] A first acquisition unit 601 is configured to acquire a target image in which a target document to be detected is located.

[0132] A first construction unit 602 is configured to construct an encoding vector corresponding to a preset layout element type.

[0133] A first prediction unit 603 is configured to input the target image and the encoding vector into a pre-constructed document layout element detection model to predict a layout element detection result corresponding to the target document. The document layout element detection model is trained according to a preset document mixed element by using a pre-training manner of contrast learning and mask prediction.

[0134] In an implementation manner of the embodiment, the first prediction unit 601 includes:

[0135] The first input subunit is configured to input the target image into a pre-constructed document layout element detection model, and extract image features of the target image.

[0136] The second input subunit is configured to input the encoding vector into the pre-constructed document layout element detection model, and extract a feature vector corresponding to the encoding vector.

[0137] The fusion subunit is configured to perform fusion processing on the feature vector corresponding to the encoding vector and the image features of the target image by using a preset image style migration processing algorithm, to obtain fusion features.

[0138] The prediction subunit is configured to perform decoding processing on the fusion features, to obtain a decoding result of the target image, and perform target layout element detection processing on the decoding result, to predict a layout element detection result corresponding to the target document.

[0139] In an implementation form of the embodiment, the layout element detection result corresponding to the target document includes position information and attribute information of each layout element in the target document.

[0140] In an implementation form of the embodiment, the apparatus further includes:

[0141] The second acquisition unit is configured to acquire a sample image in which the unlabeled document is located, and randomly select a first sample image, a second sample image, and a third sample image from the sample image.

[0142] The extraction unit is configured to perform page extraction on the first sample image, to obtain a fourth sample image.

[0143] The training unit is configured to train an initial image feature extraction model by using the first sample image, the second sample image, the fourth sample image, and a target loss function, through a training manner of contrast learning, and train the initial image feature extraction model by using the third sample image, through a training manner of mask prediction, to generate an image feature extraction model, and use the image feature extraction model as an initialization structure of an image feature extraction part of the document layout element detection model.

[0144] In an implementation form of the embodiment, the target loss function is a triplet loss function; and the triplet loss function is configured to shorten a distance between image features of the first sample image and image features of the fourth sample image, and lengthen a distance between image features of the second sample image and image features of the fourth sample image.

[0145] In an implementation form of the embodiment, the apparatus further includes:

[0146] The third acquisition unit is configured to acquire a fifth sample image in which a sample document containing a preset document mixed element is located.

[0147] The second construction unit is configured to construct a sample encoding vector corresponding to the preset document mixed element according to the preset document mixed element type.

[0148] The generation unit is configured to train an initial document layout element detection model according to the fifth sample image and the sample encoding vector, and generate the document layout element detection model.

[0149] The initial document layout element detection model includes an image feature extraction part, an encoding vector feature extraction part, an image style migration processing part, and a decoding part.

[0150] In an implementation form of the embodiment, the apparatus further includes:

[0151] The third acquisition unit is configured to acquire a verification image in which a verification document containing a verification document mixed element is located.

[0152] The third construction unit is configured to construct a verification encoding vector corresponding to the verification document mixed element according to the verification document mixed element type.

[0153] The second prediction unit is configured to input the verification image and the verification encoding vector into the document layout element detection model, and predict a verification layout element detection result corresponding to the verification document.

[0154] The update unit is configured to, when the verification layout element detection result is inconsistent with a layout element corresponding to the verification document, re-use the verification image as the fifth sample image, and update the document layout element detection model.

[0155] In an implementation form of the embodiment, the preset layout element includes at least one of a header, a title, a picture, a question number, an answer text line, and a stem text line.

[0156] In an implementation form of the embodiment, the encoding vector is a one-hot encoding vector.

[0157] Further, the embodiment of the application further provides a document layout element detection device, including a processor, a memory, and a system bus.

[0158] The processor and the memory are connected through the system bus.

[0159] The memory is configured to store one or more programs, and the one or more programs include instructions which, when executed by the processor, cause the processor to perform any implementation method of the above-mentioned document layout element detection method.

[0160] Further, the embodiment of the present application further provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions are run on a terminal device, the terminal device executes any one of the implementation methods of the document layout element detection method.

[0161] Further, the embodiment of the present application further provides a computer program product, wherein when the computer program product is run on a terminal device, the terminal device executes any one of the implementation methods of the document layout element detection method.

[0162] From the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software and necessary universal hardware platforms. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) execute the methods described in the various embodiments or some parts of the embodiments of the present application.

[0163] It should be noted that the various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be mutually referred to. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts are referred to the method part.

[0164] It should also be noted that in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0165] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of document layout element detection, characterized by, The method comprises the following steps: obtaining a target image in which a target document to be detected is located; constructing an encoding vector corresponding to a preset layout element type according to the preset layout element type; inputting the target image and the encoding vector into a pre-constructed document layout element detection model to predict a layout element detection result corresponding to the target document; the document layout element detection model is trained according to a preset document mixed element by using a pre-training mode of contrast learning and mask prediction; the step of inputting the target image and the encoding vector into the pre-constructed document layout element detection model to predict the layout element detection result corresponding to the target document comprises the following steps: inputting the target image into the pre-constructed document layout element detection model to extract image features of the target image; inputting the encoding vector into the pre-constructed document layout element detection model to extract a feature vector corresponding to the encoding vector; using a preset image style transfer processing algorithm to perform fusion processing on the feature vector corresponding to the encoding vector and the image features of the target image to obtain fused features; decoding the fused features to obtain a decoding result of the target image; and performing target layout element detection processing on the decoding result to predict the layout element detection result corresponding to the target document.

2. The method of claim 1, wherein, The layout element detection result corresponding to the target document comprises position information and attribute information of each layout element in the target document.

3. The method of claim 1, wherein, The method further comprises the following steps: obtaining a sample image in which a document without annotation is located, and randomly selecting a first sample image, a second sample image and a third sample image from the sample image; performing page extraction on the first sample image to obtain a fourth sample image; training an initial image feature extraction model by using the first sample image, the second sample image, the fourth sample image and a target loss function through a contrast learning training mode; and training the initial image feature extraction model by using the third sample image through a mask prediction training mode to generate an image feature extraction model, and using the image feature extraction model as an initialization structure of an image feature extraction part of the document layout element detection model.

4. The method of claim 3, wherein, The target loss function is a triplet loss function; the triplet loss function is used to shorten the distance between the image features of the first sample image and the fourth sample image extracted by the model and to lengthen the distance between the image features of the second sample image and the fourth sample image.

5. The method of claim 3, wherein, The document layout element detection model is constructed in the following manner: obtaining a fifth sample image in which a sample document containing a preset document mixed element is located; constructing a sample encoding vector corresponding to the preset document mixed element according to the preset document mixed element type; training an initial document layout element detection model according to the fifth sample image and the sample encoding vector to generate the document layout element detection model; wherein the initial document layout element detection model comprises an image feature extraction part, an encoding vector feature extraction part, an image style transfer processing part and a decoding part.

6. The method of claim 5, wherein, The method further comprises the following steps: acquire a verification image in which a verification document containing a verification document mixed element is located; construct a verification code vector corresponding to the verification document mixed element according to the verification document mixed element type; input the verification image and the verification code vector into the document layout element detection model to predict a verification layout element detection result corresponding to the verification document; when the verification layout element detection result is inconsistent with a layout element corresponding to the verification document, re-acquire the verification image as the fifth sample image, and update the document layout element detection model.

7. A document layout element detection apparatus characterized by comprising: comprise: a first acquisition unit configured to acquire a target image in which a target document to be detected is located; a first construction unit configured to construct a code vector corresponding to a preset layout element according to a preset layout element type; a first prediction unit configured to input the target image and the code vector into a pre-constructed document layout element detection model to predict a layout element detection result corresponding to the target document; the document layout element detection model is trained according to a preset document mixed element by using a pre-training manner of contrast learning and mask prediction; the first prediction unit comprises: a first input sub-unit configured to input the target image into the pre-constructed document layout element detection model to extract image features of the target image; a second input sub-unit configured to input the code vector into the pre-constructed document layout element detection model to extract a feature vector corresponding to the code vector; a fusion sub-unit configured to fuse the feature vector corresponding to the code vector and the image features of the target image by using a preset image style migration processing algorithm to obtain fused features; a prediction sub-unit configured to decode the fused features to obtain a decoding result of the target image, and perform target layout element detection processing on the decoding result to predict the layout element detection result corresponding to the target document.

8. A document layout element detection apparatus characterized by comprising: comprise: a processor, a memory, and a system bus; the processor and the memory are connected through the system bus; the memory is configured to store one or more programs, the one or more programs comprising instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, when the instructions run on the terminal device, make the terminal device execute the method of any one of claims 1-6.

Images