Page analysis and model training method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a layout analysis and model training method and device, electronic equipment and storage medium. The layout analysis method comprises: performing text recognition on a picture to be analyzed to obtain a plurality of texts; based on the context semantic relationship between the plurality of texts, a plurality of layout element labels of the plurality of texts are labeled respectively; a plurality of pixel gray values corresponding to the plurality of layout element labels are determined respectively; a plurality of pixel positions of the plurality of texts in the picture to be analyzed are matched with the plurality of pixel gray values; based on the matching relationship between the plurality of pixel positions and the plurality of pixel gray values, input into a pre-trained target detection model to obtain a layout element region of the picture to be analyzed. The scheme of the embodiments of the present application improves the compatibility of the existing target detection model, and makes the target detection model have strong expansion ability.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a layout analysis and model training method, apparatus, electronic device and storage medium. Background Technology

[0002] Resources combining text and images constitute a significant portion of the overall resource landscape, such as textbooks in the education industry, legal documents in the legal field, and contracts for businesses. To effectively extract information from these vast resources by dividing the space into sections, we can utilize layout analysis techniques.

[0003] Page layout analysis can be categorized as an object detection task. Page layout analysis solutions in the industry include, but are not limited to: projection-based page layout analysis methods, computer vision-based page layout analysis methods, and natural language processing-based page layout analysis methods.

[0004] Among the various layout analysis methods currently available, multimodal layout analysis, which integrates visual, semantic, and locational elements, is an effective solution for combining text and images into electronically structured resources.

[0005] However, the existing multimodal layout analysis schemes use complex models with poor scalability. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a layout analysis and model training method, apparatus, electronic device, and storage medium to at least partially solve the above-mentioned problems.

[0007] According to a first aspect of the present invention, a layout analysis method is provided, comprising: performing text recognition on an image to be analyzed to obtain multiple texts; labeling multiple layout element tags of the multiple texts based on the contextual semantic relationships between the multiple texts; determining multiple pixel grayscale values ​​corresponding to the multiple layout element tags; matching multiple pixel positions of the multiple texts in the image to be analyzed with the multiple pixel grayscale values; and inputting the matching relationship between the multiple pixel positions and the multiple pixel grayscale values ​​into a pre-trained target detection model to obtain the layout element regions of the image to be analyzed.

[0008] In another implementation of the present invention, the step of labeling multiple layout element tags of the multiple texts based on the contextual semantic relationship between the multiple texts includes: inputting the text sequence formed by the multiple texts into a pre-trained sequence labeling model to obtain multiple layout element tags of the multiple texts.

[0009] In another implementation manner of the present application, the determining the multiple pixel gray scale values corresponding to the multiple layout element labels comprises: determining multiple pixel gray scale value ranges corresponding to the multiple layout element labels, the multiple pixel gray scale value ranges being non-overlapping with each other; and selecting multiple pixel gray scale values in the multiple pixel gray scale value ranges respectively.

[0010] In another implementation manner of the present application, the determining the multiple pixel gray scale value ranges corresponding to the multiple layout element labels comprises: determining a correlation degree between the multiple layout element labels; and determining the multiple pixel gray scale value ranges corresponding to the multiple layout element labels respectively, so that the correlation degree between the multiple layout element labels is inversely related to a distance between the multiple pixel gray scale value ranges.

[0011] In another implementation manner of the present application, the determining the multiple pixel gray scale value ranges corresponding to the multiple layout element labels comprises: dividing a pixel gray scale value range based on a number of the multiple layout element labels to obtain multiple pixel gray scale value ranges; determining an order of the multiple layout element labels based on a correlation degree between the multiple layout element labels; and determining the multiple pixel gray scale value ranges corresponding to the multiple layout element labels based on the order of the multiple layout element labels.

[0012] In another implementation manner of the present application, the matching the multiple pixel positions of the multiple texts in the to-be-analyzed picture with the multiple pixel gray scale values comprises: constructing a pixel coordinate system of the to-be-analyzed picture, a first direction and a second direction of the pixel coordinate system being orthogonal to each other; determining a first pixel coordinate in the first direction and a second pixel coordinate in the second direction of each text as a pixel position of the text; and determining the first pixel coordinate, the second pixel coordinate, and a gray scale value corresponding to the pixel position of the text as three parameter dimensions of the text.

[0013] In another implementation manner of the present application, the multiple layout element labels comprise at least two of a list, a title, a body, an author, and an illustration.

[0014] According to a second aspect of the embodiments of the present application, a model training method is provided, comprising: obtaining a training sample, the training sample comprising a layout feature and a layout element region of a sample picture, the layout feature comprising multiple pixel positions of multiple texts of the sample picture and corresponding gray scale values thereof; and performing further training on a pre-trained object detection model based on the layout feature of the sample picture as input and the layout element region as a supervision condition.

[0015] According to a third aspect of the embodiments of the present application, a layout analysis device is provided, comprising: an identification module, which performs text identification on a picture to be analyzed to obtain a plurality of texts; a labeling module, which labels a plurality of layout element labels of the plurality of texts based on context semantic relationships between the plurality of texts; a determination module, which determines a plurality of pixel gray values corresponding to the plurality of layout element labels; a matching module, which matches a plurality of pixel positions of the plurality of texts in the picture to be analyzed with the plurality of pixel gray values; and an analysis module, which inputs a matching relationship between the plurality of pixel positions and the plurality of pixel gray values into a pre-trained target detection model to obtain a layout element region of the picture to be analyzed.

[0016] According to a fourth aspect of the embodiments of the present application, a model training device is provided, comprising: an acquisition module, which acquires a training sample, the training sample comprising a layout feature and a layout element region of a sample picture, the layout feature comprising a plurality of pixel positions of a plurality of texts of the sample picture and corresponding gray values thereof; and a training module, which performs further training on a pre-trained target detection model based on the layout feature of the sample picture as input and based on the layout element region as a supervision condition.

[0017] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface performing communication with each other through the communication bus; the memory is used to store at least one executable instruction, the executable instruction causing the processor to perform operations corresponding to the method of the first aspect.

[0018] According to a sixth aspect of the embodiments of the present application, a computer storage medium is provided, which stores a computer program, the program being executed by a processor to implement the method of the first aspect.

[0019] In the scheme of the embodiments of the present application, the plurality of pixel gray values correspond to the plurality of layout element labels, and the matching between the pixel positions of the texts and the pixel value gray levels reflects the matching relationship between the layout element labels and the pixel positions, thereby reliably constructing the layout feature and improving the compatibility of the existing target detection model, and making the target detection model have strong expansion capability. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0021] Figure 1 A step flowchart of a layout analysis method according to an embodiment of the present application.

[0022] Figure 2 A step flowchart of a model training method according to another embodiment of the present application.

[0023] Figure 3 A structural block diagram of a layout analysis device according to another embodiment of the present application.

[0024] Figure 4 A structural block diagram of a model training device according to another embodiment of the present application.

[0025] Figure 5 A structural schematic diagram of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and in detail below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art should belong to the scope of protection of the present application.

[0027] The specific implementation of the embodiments of the present application will be further described below in combination with the drawings of the embodiments of the present application.

[0028] Figure 1 A step flowchart of a layout analysis method according to an embodiment of the present application. The scheme of the present embodiment can be applied to any appropriate electronic device with data processing capability, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.), and PC machines, etc. For example, in the model training phase, a computing device (for example, a data center) configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can be used to train an encoder-decoder model based on training samples. The computing device such as the data center can be deployed in a cloud server such as a private cloud, a private cloud, or a hybrid cloud. Accordingly, in the inference phase of layout analysis, a computing device configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can also be used for inference operation.

[0029] The layout analysis method of the present embodiment includes:

[0030] S110: performing text recognition on the picture to be analyzed to obtain a plurality of texts.

[0031] It should be understood that the picture to be analyzed can be a color picture, for example, each pixel position of the picture to be analyzed has channel values of R channel, G channel and B channel.

[0032] It should also be understood that text recognition can be performed on the picture to be analyzed, such as OCR technology. For example, a plurality of characters (such as Latin or Sino-Tibetan characters, for example, English letters or Chinese characters) can be recognized first, and then the positional relationship between the characters is connected to form a string of characters. For example, a string of characters includes one or more characters. Each text in the plurality of texts can be a string of characters.

[0033] S120: based on the context semantic relationship between the plurality of texts, respectively labeling a plurality of layout element labels of the plurality of texts.

[0034] It should be understood that each text can be input into a pre-trained sequence labeling model, for example, a layout element label labeling model. The layout element label labeling model can be based on RNN, LSTM, BERT, Transformer, etc. The layout element label labeling model is used to build the association between the semantics of each text and the layout element label.

[0035] It should also be understood that the layout element label includes but is not limited to title, illustration or image, list, body, table, author, page, etc., and the plurality of layout element labels are at least two of the above labels. The same layout element label has similar graphic features or graphic formats, and different layout element labels have different graphic features or graphic formats.

[0036] S130: respectively determining a plurality of pixel gray values corresponding to the plurality of layout element labels.

[0037] It should be understood that the picture to be analyzed having channel values of R channel, G channel and B channel is processed to obtain a plurality of pixel positions corresponding to each text.

[0038] In one example, the picture to be analyzed can be first processed in gray scale, that is, the picture to be analyzed having channel values of R channel, G channel and B channel is converted into a gray scale picture having gray scale values, for example, for each pixel position, the channel values of R channel, G channel and B channel of the pixel can be normalized.

[0039] Then, based on the gray scale picture (the picture to be analyzed after gray scale processing), a plurality of texts are extracted by technologies such as OCR, etc. Since the gray scale features and color features have similar reference values in recognizing texts, the accuracy of text recognition is ensured.

[0040] Then, the identified pixel position range of each text can be marked. Alternatively, each pixel position of the gray scale image can be directly marked, and then the pixel positions of the multiple texts are obtained while identifying the multiple texts.

[0041] S140: match the pixel positions of the multiple texts in the picture to be analyzed with the multiple pixel gray scale values.

[0042] Then, the multiple pixel positions corresponding to each text are associated with the pixel gray scale values corresponding to the layout element labels. That is, after identifying the multiple texts based on the real gray scale values, the gray scale values are adjusted or changed for marking the layout element labels, and the gray scale values at this time utilize the format of the gray scale value data itself without changing the data format of the picture to be analyzed.

[0043] For example, a pixel coordinate system of the picture to be analyzed can be constructed, and the first direction and the second direction of the pixel coordinate system are orthogonal to each other, for example, the horizontal direction and the vertical direction are orthogonal. In the coordinate system of the picture to be analyzed, the pixel positions corresponding to a text are the 10th-510th pixel in the horizontal direction and the 10th-40th pixel in the vertical direction, and the gray scale values of the pixels in this range of pixel positions are marked as a value between 0 and 255, for example, 50. Different layout element labels correspond to different pixel gray scale values.

[0044] In an example, the multiple pixel gray scale value ranges do not overlap with each other, for example, the interval of 0-255 can be equally divided based on the categories of the multiple layout element labels. For example, in the case of four labels of title, text, image, and table, the gray scale values corresponding to the title, text, image, and table are set as 63, 127, 191, and 255, respectively, or the value intervals of the gray scale values corresponding to the image and the table are [0-63], [64-127], [128-191], and [192, 255], respectively.

[0045] In another example, for two layout element labels with similar graphic-text features, the gray scale values of the two layout element labels can be set to have a small difference, that is, the correlation between the multiple layout element labels is inversely related to the distance between the multiple pixel gray scale value ranges.

[0046] For example, for two layout element labels with small graphic-text feature similarity, the gray scale values of the two layout element labels can be set to have a large difference. For example, the similarity between the text and the title is large, and the similarity between the image and the text such as the title and the text is small, the gray scale value corresponding to the text can be set to 100, the gray scale value corresponding to the title can be set to 200, the gray scale value corresponding to the image can be set to 210, and the gray scale value corresponding to the table can be set to 255, so as to overall consider the distinguishability between various layout element labels.

[0047] In another example, for the two layout element labels with the smallest similarity in the plurality of layout element labels, one gray value is set based on the minimum value (i.e., 0), and the other gray value is set based on the maximum value (i.e., 255), thereby further improving the distinction between the two layout element labels.

[0048] S150: Based on the matching relationship between the plurality of pixel positions and the plurality of pixel gray values, input into the pre-trained target detection model to obtain the layout element region of the picture to be analyzed.

[0049] In the traditional layout analysis technology, image processing needs to be performed on the picture to be analyzed. Each pixel position of the picture to be analyzed has channel values of R, G, and B channels. The color features of the picture-text format of the picture to be analyzed can reflect the layout elements to some extent. By labeling the layout elements, the machine learning model can learn the association between the color features of the picture-text format and the layout elements. The structure of such a machine learning model (for example, the target detection model Cascade Mask Rcnn) can be designed for R, G, and B channel data, effectively improving the training efficiency of the model. For different training tasks, only different picture color data and layout element labels can be used to efficiently train a model with layout analysis function.

[0050] Specifically, by labeling the layout element label of the pixel through the gray value, and by training the target detection model, the layout element label, the text, and the pixel position can be reliably and effectively associated, and the analysis capability of the trained layout analysis model is improved.

[0051] In the scheme of the embodiment of the present application, the plurality of pixel gray values correspond to the plurality of layout element labels, and the matching between the pixel position of the text and the pixel value gray reflects the matching relationship between the layout element label and the pixel position, thereby reliably constructing the layout feature. At the same time, the channel values of the three color channels (R, G, and B channels) are compatible with the traditional machine learning model, that is, the compatibility of the existing target detection model is improved, and the target detection model has strong expansion capability.

[0052] In other words, each pixel of the traditional data of the picture to be analyzed stores the channel values of the corresponding RGB channels, that is, (R channel value, G channel value, B channel value).

[0053] In the embodiments of the present application, the gray value corresponding to the first pixel coordinate, the second pixel coordinate, and the pixel position of the text is determined as the three parameter dimensions of the text. For example, the pixel corresponding to the text pixel of the picture to be analyzed stores (gray value, X pixel position, Y pixel position), wherein the gray value indicates the layout element label corresponding to the position of the pixel, and the X pixel position and the Y pixel position respectively represent the pixel position of the pixel in the coordinate system of the picture to be analyzed. That is, the layout element label and the position information are ingeniously and lightly integrated into the three channels of the image, and the information integration process of the three channels of the image belongs to data processing in essence. Without the need to make too many changes to the target detection model, the decoupling of the information integration process and the downstream target detection model is achieved.

[0054] In some examples, the multiple layout element labels of the multiple texts are respectively labeled based on the context semantic relationship between the multiple texts, and the method comprises the following steps.

[0055] In some examples, the multiple pixel gray values corresponding to the multiple layout element labels are respectively determined, and the method comprises the following steps.

[0056] In some examples, the multiple pixel gray value ranges corresponding to the multiple layout element labels are determined, and the method comprises the following steps.

[0057] In some examples, the multiple pixel gray value ranges corresponding to the multiple layout element labels are respectively determined, and the method comprises the following steps.

[0058] In some examples, matching the plurality of texts with the plurality of pixel gray values at the plurality of pixel positions in the picture to be analyzed comprises: constructing a pixel coordinate system of the picture to be analyzed, a first direction and a second direction of the pixel coordinate system being orthogonal to each other; determining a first pixel coordinate of each text in the first direction and a second pixel coordinate of the text in the second direction as a pixel position of the text; and determining the first pixel coordinate, the second pixel coordinate, and a gray value corresponding to the pixel position of the text as three parameter dimensions of the text.

[0059] In some examples, the plurality of layout element labels comprises at least two of a list, a title, a body, an author, and an illustration.

[0060] Figure 2 A step flowchart of a model training method according to another embodiment of the present application. The scheme of the present embodiment can be applied to any appropriate electronic device with data processing capability, including but not limited to servers, mobile terminals (such as mobile phones, PADs, etc.), and PC machines, etc. For example, in the model training stage, a computing device (for example, a data center) configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can be used to train an encoder-decoder model based on training samples. The computing device such as a data center can be deployed in a cloud server such as a private cloud, a private cloud, or a hybrid cloud. Accordingly, in the inference stage of layout analysis, a computing device configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can also be used for inference operations.

[0061] The model training method of the present embodiment comprises:

[0062] S210: obtaining a training sample, the training sample comprising layout features and layout element regions of a sample picture, the layout features comprising a plurality of pixel positions of a plurality of texts of the sample picture and corresponding gray values thereof;

[0063] S220: based on the layout features of the sample picture as input and the layout element regions as a supervision condition, performing further training on a pre-trained object detection model.

[0064] Figure 3 A structure block diagram of a layout analysis device according to another embodiment of the present application. The layout analysis device comprises:

[0065] The recognition module 310 performs text recognition on the picture to be analyzed to obtain a plurality of texts.

[0066] The labeling module 320 labels a plurality of layout element labels for the plurality of texts based on the context semantic relationship between the plurality of texts.

[0067] The determining module 330 respectively determines a plurality of pixel grayscale values corresponding to the plurality of layout element labels.

[0068] The matching module 340 matches the plurality of pixel positions of the plurality of texts in the picture to be analyzed with the plurality of pixel grayscale values.

[0069] The analyzing module 350 inputs a matching relationship between the plurality of pixel positions and the plurality of pixel grayscale values into a pre-trained target detection model to obtain a layout element region of the picture to be analyzed.

[0070] In the scheme of the embodiment of the present application, the plurality of pixel grayscale values correspond to the plurality of layout element labels, and the matching between the pixel positions of the texts and the pixel grayscale values reflects the matching relationship between the layout element labels and the pixel positions, thereby reliably constructing the layout features and improving the compatibility of the existing target detection model and the expansion ability of the target detection model.

[0071] In some other examples, the labeling the plurality of layout element labels of the plurality of texts respectively based on the context semantic relationship between the plurality of texts comprises: inputting a text sequence formed by the plurality of texts into a pre-trained sequence labeling model to obtain the plurality of layout element labels of the plurality of texts.

[0072] In some other examples, the determining the plurality of pixel grayscale values corresponding to the plurality of layout element labels respectively comprises: determining a plurality of pixel grayscale value ranges corresponding to the plurality of layout element labels, the plurality of pixel grayscale value ranges do not overlap with each other; and selecting a plurality of pixel grayscale values in the plurality of pixel grayscale value ranges respectively.

[0073] In some other examples, the determining the plurality of pixel grayscale value ranges corresponding to the plurality of layout element labels comprises: determining a correlation degree between the plurality of layout element labels; and respectively determining a plurality of pixel grayscale value ranges corresponding to the plurality of layout element labels, so that the correlation degree between the plurality of layout element labels and a distance between the plurality of pixel grayscale value ranges are inversely related.

[0074] In some other examples, the determining the plurality of pixel grayscale value ranges corresponding to the plurality of layout element labels respectively comprises: dividing a pixel grayscale value range based on a number of the plurality of layout element labels to obtain a plurality of pixel grayscale value ranges; determining an order of the plurality of layout element labels based on a correlation degree between the plurality of layout element labels; and determining the plurality of pixel grayscale value ranges corresponding to the plurality of layout element labels based on the order of the plurality of layout element labels.

[0075] In some examples, the matching the pixel positions of the plurality of texts in the picture to be analyzed to the pixel gray scale values comprises: constructing a pixel coordinate system of the picture to be analyzed, a first direction and a second direction of the pixel coordinate system being orthogonal to each other; determining a first pixel coordinate of each text in the first direction and a second pixel coordinate of the text in the second direction as a pixel position of the text; and determining the first pixel coordinate, the second pixel coordinate, and a pixel gray scale value corresponding to the pixel position of the text as three parameter dimensions of the text.

[0076] In some examples, the plurality of layout element labels comprises at least two of a list, a title, a body, an author, and an illustration.

[0077] The apparatus of the embodiment is used to implement the corresponding method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here again. In addition, the function implementation of each module in the apparatus of the embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will not be described here again.

[0078] Figure 4 A structural block diagram of a model training apparatus according to another embodiment of the present application is shown. The model training apparatus of the embodiment comprises:

[0079] The obtaining module 410 obtains a training sample, the training sample comprising a layout feature and a layout element region of a sample picture, the layout feature comprising a plurality of pixel positions of a plurality of texts of the sample picture and corresponding gray scale values of the pixel positions.

[0080] The training module 420 performs further training on a pre-trained target detection model based on the layout feature of the sample picture as input and the layout element region as a supervision condition.

[0081] Referring to Figure 5 , a structural schematic diagram of an electronic device according to another embodiment of the present application is shown, and the specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0082] As Figure 5 shown, the electronic device can comprise a processor 502, a communications interface 504, a memory 506 storing a program 510, and a communications bus 508.

[0083] The processor, the communications interface, and the memory complete the communication among each other through the communications bus.

[0084] The communications interface is configured to communicate with other electronic devices or servers.

[0085] a processor configured to execute programs, which can specifically execute the related steps in the method embodiments described above.

[0086] Specifically, the programs can include program codes, which include at least one executable instruction.

[0087] The processor can be a processor CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device can be processors of the same type, such as one or more CPUs; or can be processors of different types, such as one or more CPUs and one or more ASICs.

[0088] a memory configured to store programs. The memory can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0089] The programs can specifically be used to cause the processor to execute the method of Figure 1 or Figure 2 .

[0090] In addition, the specific implementation of each step in the programs can refer to the corresponding description of the corresponding steps and units in the method embodiments described above, and will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the devices and modules described above can refer to the corresponding process description in the foregoing method embodiments, and will not be described here.

[0091] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into new components / steps, to achieve the purpose of the embodiments of the present application.

[0092] The above-described methods according to embodiments of the application can be implemented in hardware, firmware, or software, or any combination thereof, and can be stored in or implemented with the aid of one or more data storage media (e.g., removable storage media like CD-ROM, floppy disks, hard disks, optical disks, or memory such as ROM, RAM, flash memory, etc.), or implemented with the aid of one or more processors (e.g., general-purpose, special-purpose or programmable hardware) of a processing system, or any combination thereof. The methods described herein can be stored as software or computer code on a recording medium, such as a CD ROM, a RAM, a floppy disk, a hard disk, or an optical disk, or as computer code downloaded via a network, originally stored on a remote recording medium or non-transitory machine-readable medium and to be stored on a local recording medium, such that the methods described herein can be processed by such software using a general purpose computer, a special purpose processor, or a programmable or dedicated hardware (such as an ASIC or an FPGA). It will be appreciated that a computer, processor, microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. In addition, when a general purpose computer accesses code for implementing the methods shown herein, the execution of the code transforms the general purpose computer into a special purpose computer for executing the methods shown herein.

[0093] Those skilled in the art can appreciate that the units and method steps of the examples described in connection with the embodiments disclosed herein can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0094] The above embodiments are only used to illustrate the present application, but not to limit the present application. Those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions belong to the scope of the present application, and the patent protection scope of the present application should be defined by the claims.

Claims

1. A layout analysis method, comprising: Text recognition is performed on the image to be analyzed, resulting in multiple text entries; Based on the contextual semantic relationships between the multiple texts, multiple layout element tags are labeled for each of the multiple texts; Determine the grayscale values ​​of multiple pixels corresponding to the multiple layout element labels respectively; Match the multiple texts at multiple pixel positions in the image to be analyzed with the grayscale values ​​of the multiple pixels; The matching relationship between the multiple pixel positions and the multiple pixel gray values ​​is input into a pre-trained target detection model to obtain the layout element region of the image to be analyzed. Specifically, determining the multiple pixel grayscale values ​​corresponding to the multiple layout element labels includes: determining the range of multiple pixel grayscale values ​​corresponding to the multiple layout element labels; and selecting multiple pixel grayscale values ​​from the range of multiple pixel grayscale values.

2. The method according to claim 1, wherein, The step of labeling multiple layout element tags for the multiple texts based on the contextual semantic relationships between them includes: The text sequence formed by the multiple texts is input into a pre-trained sequence labeling model to obtain multiple page element labels for the multiple texts.

3. The method according to claim 1, wherein, The grayscale value ranges of the multiple pixels do not overlap with each other.

4. The method according to claim 3, wherein, Determining the range of multiple pixel grayscale values ​​corresponding to the multiple layout element labels includes: Determine the correlation between the multiple layout element tags; Each of the multiple pixel grayscale value ranges corresponding to the multiple layout element labels is determined such that the correlation between the multiple layout element labels is inversely correlated with the distance between the multiple pixel grayscale value ranges.

5. The method according to claim 4, wherein, The step of determining the range of multiple pixel grayscale values ​​corresponding to the multiple layout element labels includes: Based on the number of the multiple layout element tags, the range of pixel grayscale values ​​is divided to obtain multiple pixel grayscale value ranges. The order of the multiple page element tags is determined based on the correlation between them. Based on the sorting of the multiple layout element labels, the range of multiple pixel grayscale values ​​corresponding to the multiple layout element labels is determined.

6. The method according to claim 1, wherein, The step of matching the multiple texts with the multiple pixel positions in the image to be analyzed and the multiple pixel grayscale values ​​includes: Construct a pixel coordinate system for the image to be analyzed, wherein the first direction and the second direction of the pixel coordinate system are orthogonal to each other; Determine the first pixel coordinates of each text in the first direction and the second pixel coordinates in the second direction as the pixel position of the text; The first pixel coordinates, the second pixel coordinates, and the grayscale value corresponding to the pixel position of the text are determined as the three parameter dimensions of the text.

7. The method according to claim 1, wherein, The multiple layout element tags include at least two of the following: list, title, body text, author, and illustration.

8. A model training method, comprising: Obtain training samples, which include layout features and layout element regions of sample images. The layout features include multiple pixel positions of multiple texts in the sample images and their corresponding grayscale values. Using the layout features of the sample images as input and the layout element regions as supervision conditions, the pre-trained target detection model is further trained. The pre-trained object detection model is used to obtain the layout element region of the image to be analyzed based on the matching relationship between multiple pixel positions of multiple texts in the image to be analyzed and multiple pixel gray values ​​corresponding to multiple layout element labels. The matching relationship is obtained by matching the multiple pixel positions of the multiple texts in the image to be analyzed with the multiple pixel gray values. The multiple pixel gray values ​​corresponding to the multiple layout element labels are determined by selecting the multiple pixel gray values ​​from the range of multiple pixel gray values ​​corresponding to the multiple layout element labels. The multiple layout element labels are labeled based on the contextual semantic relationship between the multiple texts. The multiple texts are obtained by performing text recognition on the image to be analyzed.

9. A layout analysis device, comprising: The recognition module performs text recognition on the image to be analyzed, obtaining multiple texts. The annotation module, based on the contextual semantic relationships between the multiple texts, annotates multiple layout element tags for each of the multiple texts; The determination module determines the grayscale values ​​of multiple pixels corresponding to the multiple layout element labels; The matching module matches the multiple texts with the grayscale values ​​of the multiple pixels in the image to be analyzed. The analysis module inputs the matching relationship between the multiple pixel positions and the multiple pixel gray values ​​into a pre-trained target detection model to obtain the layout element region of the image to be analyzed; The determining module is configured to determine multiple pixel grayscale values ​​corresponding to the multiple layout element labels by performing the following steps: determining the range of multiple pixel grayscale values ​​corresponding to the multiple layout element labels; and selecting multiple pixel grayscale values ​​from the range of multiple pixel grayscale values.

10. A model training device, comprising: The acquisition module acquires training samples, which include the layout features and layout element regions of the sample image. The layout features include multiple pixel positions of multiple texts in the sample image and their corresponding grayscale values. The training module takes the layout features of the sample images as input and the layout element regions as supervision conditions to further train the pre-trained target detection model. The pre-trained object detection model is used to obtain the layout element region of the image to be analyzed based on the matching relationship between multiple pixel positions of multiple texts in the image to be analyzed and multiple pixel gray values ​​corresponding to multiple layout element labels. The matching relationship is obtained by matching the multiple pixel positions of the multiple texts in the image to be analyzed with the multiple pixel gray values. The multiple pixel gray values ​​corresponding to the multiple layout element labels are determined by selecting the multiple pixel gray values ​​from the range of multiple pixel gray values ​​corresponding to the multiple layout element labels. The multiple layout element labels are labeled based on the contextual semantic relationship between the multiple texts. The multiple texts are obtained by performing text recognition on the image to be analyzed.

11. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-8.

12. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-8.

Images