Method and device for identifying layout structure, equipment and medium

Abstract

The invention relates to a layout structure recognition method and device, equipment and a medium, information of a plurality of local ranges contained on a layout is obtained, the information of the local ranges comprises characters on the local ranges, the positions of the local ranges and visual information, and for any local range on the layout, the information of the local ranges is obtained; at least inputting the characters, the position and the visual information in the any local range into a preset identification model, and identifying to obtain a structure type of the any local range in a layout, and combining the any local range with at least one first local range in a preset range around the any local range to obtain a second local range, and taking the structure type of the any local range as the structure type of the second local range. And determining the position of the second local range in the layout based on the any local range and the position of the at least one first local range in the layout, so that the accuracy of layout structure identification can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of layout understanding technology, and in particular to a layout structure recognition method, device, equipment and medium. Background Art

[0002] Document intelligence is a software that can intelligently analyze the knowledge structure and knowledge relationship in data of any format. Layout structure recognition is a technology that uses a computer system to parse and understand the structure of a document (including the location of content blocks in the document and the structural type of content blocks in the document, such as titles, paragraphs, images, etc.). In practice, the implementation of document intelligence needs to rely on the recognition results of the layout structure. However, the layout forms are complex and diverse. For example, common layouts include single column, double column, text and image wrapping, complex tables, certificates, mind maps, etc. In this case, how to accurately identify the publication structure from the complex layout is a technical problem that needs to be solved. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, device, equipment and medium for identifying a layout structure.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for identifying a layout structure, the method comprising:

[0005] Acquire information of a plurality of local ranges contained in a layout, wherein the information of the local ranges includes characters on the local ranges, positions of the local ranges on the layout, and visual information;

[0006] For any local range on the layout, at least the characters on the local range, the position of the local range on the layout, and visual information are input into a preset recognition model, and the structural type of the local range in the layout is obtained based on the recognition model;

[0007] Merge any local range and at least one first local range within a preset range around any local range to obtain a second local range, and use the structure type of any local range as the structure type of the second local range, wherein the first local range refers to a local range with the same structure type as that of any local range;

[0008] Based on the positions of any one of the local ranges and the at least one first local range in the layout, the position of the second local range in the layout is determined.

[0009] In some implementations, for any local range on the layout, at least characters on the local range, the position of the local range on the layout, and visual information are input into a preset recognition model, and the structural type of the local range in the layout is obtained based on the recognition model, including:

[0010] Inputting characters on any local range, attribute information of the characters, position of the local range on the layout, and visual information into a preset recognition model, and identifying and obtaining a structural type of the any local range on the layout based on the recognition model;

[0011] The attribute information of the character includes at least one of the following information:

[0012] Font, color, size.

[0013] In some implementations, inputting the characters on any local range, the attribute information of the characters, the position of the local range on the layout, and the visual information into a preset recognition model, and obtaining the structural type of any local range on the layout based on the recognition model includes:

[0014] Input characters on any local range, attribute information of the characters, position of the local range on the page, and visual information into a semantic recognition network in the recognition model, and obtain a global vector of any local range based on processing by the semantic recognition network;

[0015] The global vector is input into the classification network in the recognition model, and the structural type of any local range in the layout is obtained based on the classification network identification.

[0016] In some implementations, inputting the characters on any local range, the attribute information of the characters, the position of the local range on the layout, and the visual information into a semantic recognition network in the recognition model, and obtaining a global vector of any local range based on the semantic recognition network processing includes:

[0017] Based on the first vectorization network in the semantic recognition network, convert the characters in any local range into corresponding first semantic vectors;

[0018] Based on the second vectorization network in the semantic recognition network, converting the attribute information of the character into a corresponding second semantic vector;

[0019] Based on the third vectorization network in the semantic recognition network, converting the position of any local range on the layout into a corresponding third semantic vector;

[0020] Based on a fourth vectorization network in the semantic recognition network, converting the visual information of any local range into a corresponding fourth semantic vector;

[0021] The first semantic vector, the second semantic vector, the third semantic vector and the fourth semantic vector are input into a five-vector quantization network in the semantic recognition network, and a global vector of any local range is obtained based on the five-vector quantization network.

[0022] In some embodiments, before converting the position of any local range on the layout into a corresponding third semantic vector based on the third vectorization network in the semantic recognition network, the method further includes:

[0023] Normalizing the position coordinates of any local range on the page to obtain normalized position coordinates;

[0024] The step of converting the position of any local range on the layout into a corresponding third semantic vector based on the third vectorization network in the semantic recognition network includes:

[0025] Based on the third vectorization network, the normalized position coordinates are converted into corresponding third semantic vectors.

[0026] In some embodiments, the visual information of any local range includes a background image of any local range;

[0027] The converting the visual information of any local range into a corresponding fourth semantic vector based on the fourth vectorization network in the semantic recognition network includes:

[0028] Based on the fourth vectorization network, the background image of any local range is converted into a fourth semantic vector.

[0029] In some embodiments, the structure type includes at least one of the following:

[0030] Headings, paragraphs, images, tables, code blocks, formulas, lists.

[0031] In a second aspect, an embodiment of the present disclosure provides a layout structure recognition device, comprising:

[0032] An acquisition module, used for acquiring information of a plurality of local ranges contained in a layout, wherein the information of the local ranges includes characters on the local ranges, positions of the local ranges on the layout, and visual information;

[0033] A recognition module, for inputting at least the characters on any local range on the layout, the position of any local range on the layout, and visual information into a preset recognition model, and identifying the structural type of any local range on the layout based on the recognition model;

[0034] a merging module, configured to merge the any local range and at least one first local range within a preset range around the any local range to obtain a second local range, and use the structure type of the any local range as the structure type of the second local range, wherein the first local range refers to a local range with the same structure type as the any local range;

[0035] A position determination module is used to determine the position of the second local range in the layout based on the positions of any local range and the at least one first local range in the layout.

[0036] In some embodiments, the identification module is used to:

[0037] Inputting characters on any local range, attribute information of the characters, position of the local range on the layout, and visual information into a preset recognition model, and identifying the structural type of any local range on the layout based on the recognition model;

[0038] The attribute information of the character includes at least one of the following information:

[0039] Font, color, size.

[0040] In some embodiments, the identification module is used to:

[0041] Input characters on any local range, attribute information of the characters, position of the local range on the page, and visual information into a semantic recognition network in the recognition model, and obtain a global vector of any local range based on processing by the semantic recognition network;

[0042] The global vector is input into the classification network in the recognition model, and the structural type of any local range in the layout is obtained based on the classification network identification.

[0043] In some embodiments, the identification module is used to:

[0044] Based on the first vectorization network in the semantic recognition network, convert the characters in any local range into corresponding first semantic vectors;

[0045] Based on the second vectorization network in the semantic recognition network, converting the attribute information of the character into a corresponding second semantic vector;

[0046] Based on the third vectorization network in the semantic recognition network, converting the position of any local range on the layout into a corresponding third semantic vector;

[0047] Based on a fourth vectorization network in the semantic recognition network, converting the visual information of any local range into a corresponding fourth semantic vector;

[0048] The first semantic vector, the second semantic vector, the third semantic vector and the fourth semantic vector are input into a five-vector quantization network in the semantic recognition network, and a global vector of any local range is obtained based on the five-vector quantization network.

[0049] In some implementations, the layout structure recognition device may further include:

[0050] A normalization module, used for normalizing the position coordinates of any local range on the layout to obtain normalized position coordinates;

[0051] Identification module for:

[0052] Based on the third vectorization network, the normalized position coordinates are converted into corresponding third semantic vectors.

[0053] In some embodiments, the visual information of any local range includes a background image of any local range;

[0054] Identification module for:

[0055] Based on the fourth vectorization network, the background image of any local range is converted into a fourth semantic vector.

[0056] In some embodiments, the structure type includes at least one of the following:

[0057] Headings, paragraphs, images, tables, code blocks, formulas, lists.

[0058] In a third aspect, an embodiment of the present disclosure provides a computer device, including:

[0059] Memory;

[0060] Processor; and

[0061] Computer programs;

[0062] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0063] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect.

[0064] The layout structure recognition method, device, equipment and medium provided by the embodiments of the present disclosure obtain information of multiple local ranges contained in the layout, wherein the local range information includes characters on the local range, the position of the local range on the layout and visual information. For any local range on the layout, at least the characters on any local range, the position of any local range on the layout and visual information are input into a preset recognition model, and the structure type of any local range in the layout is obtained based on the recognition model, so that the original information of the layout is not lost, and the accuracy of identifying the structure type of the local range on the layout is improved. The second local range is obtained by merging the any local range and at least one first local range (a local range with the same structure type as the any local range) within a preset range around the any local range, and the structure type of the any local range is used as the structure type of the second local range. Based on the positions of the any local range and the at least one first local range in the layout, the position of the second local range in the layout is determined. The accurate segmentation of the content on the layout can be achieved, and the accuracy of layout structure recognition is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0067] Figure 1 is a flow chart of a method for identifying a layout structure provided by an embodiment of the present disclosure;

[0068] Figure 2 is a schematic diagram of a page segmentation result provided by an embodiment of the present disclosure;

[0069] Figure 3 is a schematic diagram of a method for identifying a structural type of a local area provided by an embodiment of the present disclosure;

[0070] Figure 4 is a schematic diagram of a local range merging method provided by an embodiment of the present disclosure;

[0071] Figure 5 is a schematic diagram of a method for identifying a structural type in a local range provided by an embodiment of the present disclosure;

[0072] Figure 6 is a schematic diagram of a method for vectorizing input information provided by an embodiment of the present disclosure;

[0073] Figure 7 is a structural schematic diagram of a layout structure recognition device provided by an embodiment of the present disclosure;

[0074] Figure 8 A schematic diagram of the structure of a computer device embodiment provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0075] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0076] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0077] The layout structure recognition method provided by the related art uses the visual photo of the layout to be recognized and the text contained in the layout as the input of a pre-trained layout large language model (i.e., layout LLM), and recognizes the structure of the layout through the layout large language model, such as the area where the content blocks contained in the layout are located, and the structural type of the content contained in each area in the layout. At present, the structural types that can be recognized by the layout large language model include: title, paragraph, and image. However, since the input data used by the related art to recognize the layout structure is very limited (generally the visual picture and text of the layout), the accuracy is low when processing some layout structure recognition tasks that purely rely on text semantics. For example, txt data needs to rely on the meaning of the text to distinguish the layout structure. It is easy to make mistakes if it only relies on visual pictures and text to recognize the layout structure. In addition, it is impossible to recognize the layout structure of complex layouts such as mind maps by relying only on visual pictures and text.

[0078] In response to the above-mentioned problems existing in the related technologies, the embodiments of the present disclosure provide a method, device, equipment and medium for recognizing layout structure, so as to improve the accuracy of layout structure recognition, especially to improve the accuracy of some recognition tasks that purely rely on text semantics, and recognition tasks of complex data structures such as mind maps.

[0079] The solution provided by the embodiment of the present disclosure is described below in conjunction with exemplary embodiments.

[0080] Figure 1 1 is a flow chart of a layout structure recognition method provided by an embodiment of the present disclosure. The method can be performed by a computer device. The computer device can be specifically any device with computing and processing capabilities, such as a server, a distributed computing node, a laptop computer, a mobile phone, etc., but is not limited to the devices listed here. Figure 1 As shown, the layout structure recognition method provided by the embodiment of the present disclosure may include steps 101 to 104.

[0081] Step 101: Acquire information of multiple local ranges contained in a layout, wherein the information of the local ranges includes characters on the local ranges, positions of the local ranges on the layout, and visual information.

[0082] The layout referred to in the embodiments of the present disclosure can be exemplarily understood as an area or page that displays content in a certain layout structure. The content displayed on the layout may include at least one of a title, a paragraph, an image, a code block, a list, a formula, and a table.

[0083] The local range referred to in the embodiments of the present disclosure can be exemplarily understood as a geometric area on the page that contains part of the page content.

[0084] For example, in some implementations, the content contained in the layout can be segmented based on a preset segmentation method to obtain multiple local ranges containing the layout content. Figure 2 FIG. 1 is a schematic diagram of a layout segmentation result provided by an embodiment of the present disclosure. Figure 2 As shown, in some embodiments, the content contained in the layout can be traversed based on a window of a preset size and shape. The window range where each window position in the traversal process is located can be used as a local range, so as to obtain the following Figure 2 The segmentation result is shown. Figure 2 It is only an example and not the only limitation. It should be noted that the segmentation method in the embodiment of the present disclosure can be set as needed and is not limited to the above-mentioned segmentation method based on the preset window.

[0085] It should also be noted that in the embodiment of the present disclosure, the area where any paragraph, title, image, table, list, code block or formula on the layout is located can be divided into one or more local ranges.

[0086] In some embodiments, the information of the local scope referred to in the embodiments of the present disclosure may include characters located within the local scope, the position of the local scope on the layout, and visual information. Among them, the characters within the layout range may be at least one of text (text in any language), numbers, codes, and mathematical symbols. The position of the local scope on the layout can be understood as the position coordinates of the closed geometric figure corresponding to the local scope. The position coordinates may include the coordinates of the vertices of the closed geometric figure, or the center coordinates of the closed geometric figure and the shape and size of the closed geometric figure, or the coordinates of each edge that constitutes the closed geometric figure. The visual information of the local scope on the layout may include at least one of the background image on the local scope and the shape of the local scope.

[0087] In some exemplary embodiments, information of multiple local ranges included in the layout may be obtained from a preset storage device or file.

[0088] Step 102: for any local range on the layout, at least the characters on the local range, the position of the local range on the layout and the visual information are input into a preset recognition model, and the structural type of the local range in the layout is obtained based on the recognition model.

[0089] In the embodiment of the present disclosure, “at least inputting the characters in any local range, the position of any local range on the page, and the visual information into the preset recognition model” can be exemplarily understood as that in addition to inputting the characters in the local range, the position of the local range on the page, and the visual information into the preset recognition model, other information of the local range can also be input into the recognition model. For example, Figure 3 is a schematic diagram of a method for identifying a structural type of a local area provided by an embodiment of the present disclosure, such as Figure 3 As shown, in some embodiments, the characters in the local range, the attribute information of the characters, the position of the local range on the layout, and the visual information can be input into a preset recognition model to identify the structure type of the local range in the layout. The attribute information of the characters can include at least one of the following information: font, color, size. Of course, Figure 3 This is only an example and not a sole limitation. For example, in some other implementations, the other input information may not be limited to the attribute information of the characters but may be any other preset information of one or more dimensions.

[0090] The recognition model provided in the embodiments of the present disclosure can be understood as any model trained to recognize the structural type of a local scope in a layout, such as a large language model, but not limited to a large language model. The input of the model can be at least the characters on the local scope, the position of the local scope on the layout, and visual information, and the output is the structural type of the local scope on the layout, such as a paragraph, title, image, table, list, code block or formula. The training method of the recognition model referred to in the embodiments of the present disclosure can refer to the model training method provided by the relevant technology, which will not be repeated here.

[0091] Step 103: merge the any local range and at least one first local range within a preset range around the any local range to obtain a second local range, and use the structural type of the any local range as the structural type of the second local range, wherein the first local range refers to a local range with the same structural type as the any local range.

[0092] The size and shape of the preset range referred to in the embodiment of the present disclosure can be set as needed, and the embodiment of the present disclosure does not make any specific limitation.

[0093] for example, Figure 4 FIG. 1 is a schematic diagram of a local range merging method provided by an embodiment of the present disclosure. Figure 4 As shown, local range a, local range b, local range c, local range d, local range e, and local range f are six local ranges on the layout, among which the structure type corresponding to local range a is title, and the structure type corresponding to local range b, local range c, local range d, local range e, and local range f is paragraph. Figure 4 The dotted frame L1 in the figure is a preset range centered on the local range b. The local ranges b, c, d, and e are located within the dotted frame L1, and the local range f is located outside the dotted frame L1. Figure 4 In the example, local range b, local range c, local range d, and local range e can be merged, and the merged range is a solid-line frame L2, which only includes local range b, local range c, local range d, and local range e but does not include other local ranges. Among them, the solid-line frame L2 can be understood as the minimum rectangular frame containing the local range b, local range c, local range d, and local range e. The dotted-line frame can be determined based on the minimum and maximum horizontal coordinates, and the minimum and maximum vertical coordinates of the vertices in the local range b, local range c, local range d, and local range e. Of course Figure 4 It is only an explanation of the merging method in a local scope and not the only limitation.

[0094] Step 104: Determine the position of the second local range in the layout based on the positions of any local range and the at least one first local range in the layout.

[0095] by Figure 4 For example, Figure 4 The solid line frame L2 in the figure can be understood as the second local range obtained by merging the local ranges b, c, d, and e. The vertex coordinates of the second local range can be determined based on the minimum and maximum horizontal coordinates, and the minimum and maximum vertical coordinates of the vertices in the local ranges b, c, d, and e in the layout coordinate system. The position of the second local range in the layout can be determined based on the coordinates of the vertices of the second local range in the layout. Of course, the above is only an example of the method for determining the position of the second local range, and it is not the only limitation.

[0096] In the disclosed embodiment, by acquiring information of multiple local ranges contained in the layout, the information of the local ranges includes characters on the local ranges, the position of the local ranges on the layout, and visual information, for any local range on the layout, at least the characters on any local range, the position of any local range on the layout, and visual information are input into a preset recognition model, and the structural type of any local range in the layout is obtained based on the recognition model, which can ensure that the original information of the layout is not lost, and improve the accuracy of identifying the structural type of the local range on the layout. By merging the any local range and at least one first local range (a local range with the same structural type as the any local range) within a preset range around the any local range to obtain a second local range, the structural type of the any local range is used as the structural type of the second local range, and based on the positions of the any local range and the at least one first local range in the layout, the position of the second local range in the layout is determined. It is possible to achieve accurate segmentation of the content on the layout and improve the accuracy of layout structure recognition.

[0097] Figure 5 FIG. 1 is a schematic diagram of a method for identifying a local range structure type provided by an embodiment of the present disclosure. Figure 5 As shown, in some implementations, the above structure type identification method may include steps 501 and 502.

[0098] Step 501: Input characters on any local range, attribute information of the characters, the position of any local range on the page, and visual information into a semantic recognition network in a recognition model, and obtain a global vector of any local range based on the semantic recognition network processing.

[0099] In one embodiment, the characters on the same local range, the attribute information of the characters, the position of the local range on the layout (hereinafter referred to as the position of the local range) and the visual information can be vectorized simultaneously by the same vectorization network to obtain the semantic vectors corresponding to the characters, the attribute information of the characters, the position of the local range and the visual information, and then the global vector of the local range is determined based on the semantic vectors corresponding to the characters, the attribute information of the characters, the position of the local range and the visual information, and then the global vector is processed by the classification network to obtain the structural type corresponding to the local range. In this way, the vectorization results of each dimensional input information (such as characters, attribute information of characters, position of the local range and visual information) will be affected by the input information of other dimensions. For example, the vectorization results of characters will be affected by the attribute information of characters, the position of the local range and the visual information. By analogy, the attribute information of characters, the position of the local range and the visual information will also be affected by the input information of other dimensions.

[0100] In another embodiment, the semantic recognition network referred to in the present disclosure may include multiple mutually isolated vectorization networks. Input information of different dimensions (e.g., characters, character attribute information, local range position, visual information) may be vectorized by different vectorization networks. For example, Figure 6 is a schematic diagram of a method for vectorizing input information provided by an embodiment of the present disclosure, such as Figure 6 As shown, in one embodiment, the multiple mutually isolated vectorized networks contained in the semantic recognition network are respectively: a first vectorized network, a second vectorized network, a third vectorized network, a fourth vectorized network, and a fifth vectorized network. The characters on the local area can be vectorized by the first vectorized network to obtain the first semantic vector corresponding to the character, the attribute information of the character can be processed by the second vectorized network to obtain the second semantic vector corresponding to the attribute information of the character, the position of the local range can be vectorized by the third vectorized network to obtain the third semantic vector corresponding to the position, and the visual information (such as the background image) can be vectorized by the fourth vectorized network to obtain the fourth semantic vector corresponding to the visual information. Then the first semantic vector, the second semantic vector, the third semantic vector and the fourth semantic vector are input into the five vectorized networks, and the global vector of the local range is obtained based on the fifth vectorized network. By using multiple mutually isolated vectorized networks to vectorize input information of multiple different dimensions, it is possible to prevent the input information of different dimensions from affecting each other, thereby improving the accuracy of the vectorized results of the input information of each dimension. Among them, the training methods of the first vectorized network, the second vectorized network, the third vectorized network, the fourth vectorized network and the fifth vectorized network can be referred to in the relevant technology and will not be repeated here.

[0101] Step 502: input the global vector into the classification network in the recognition model, and obtain the structural type of any local range in the layout based on the classification network recognition.

[0102] The disclosed embodiment determines the global vector corresponding to the local scope through the semantic recognition network in the recognition model, and when determining the global vector, vectorizes the input information of multiple different dimensions through multiple mutually isolated vectorization networks, which can improve the accuracy of the vectorization of the input information of each dimension, and then improve the accuracy of the global vector of the local scope, thereby providing an accurate data basis for the identification of the local scope structure type.

[0103] In some embodiments, before the position of the local range is vectorized, a normalization step of the position may also be included. For example, in some embodiments, the position coordinates of the local range (for example, the coordinates of each vertex on the area corresponding to the local range) may be normalized based on a preset normalization method, and then the normalized position coordinates may be converted into a corresponding semantic vector, a third semantic vector. By normalizing the position coordinates of the local range, the influence of the error of the position coordinates on the structural type recognition can be reduced, thereby improving the accuracy of the structural type recognition.

[0104] Figure 7 is a schematic diagram of a layout structure recognition device provided by an embodiment of the present disclosure, such as Figure 7 As shown, in one implementation, the layout structure recognition device 70 provided by the embodiment of the present disclosure includes:

[0105] An acquisition module 71 is used to acquire information of multiple local ranges contained in the layout, wherein the information of the local ranges includes characters on the local ranges, positions of the local ranges on the layout, and visual information;

[0106] The recognition module 72 is used for inputting at least the characters on any local range on the layout, the position of any local range on the layout and the visual information into a preset recognition model, and obtaining the structural type of any local range on the layout based on the recognition model;

[0107] A merging module 73 is used to merge the any local range and at least one first local range within a preset range around the any local range to obtain a second local range, and use the structure type of the any local range as the structure type of the second local range, wherein the first local range refers to a local range with the same structure type as the any local range;

[0108] The position determination module 74 is used to determine the position of the second local range in the layout based on the positions of any local range and the at least one first local range in the layout.

[0109] In some embodiments, the identification module 72 is used to:

[0110] Inputting characters on any local range, attribute information of the characters, position of the local range on the layout, and visual information into a preset recognition model, and identifying the structural type of any local range on the layout based on the recognition model;

[0111] The attribute information of the character includes at least one of the following information:

[0112] Font, color, size.

[0113] In some embodiments, the identification module 72 is used to:

[0114] Input characters on any local range, attribute information of the characters, position of the local range on the page, and visual information into a semantic recognition network in the recognition model, and obtain a global vector of any local range based on processing by the semantic recognition network;

[0115] The global vector is input into the classification network in the recognition model, and the structural type of any local range in the layout is obtained based on the classification network identification.

[0116] In some embodiments, the identification module 72 is used to:

[0117] Based on the first vectorization network in the semantic recognition network, convert the characters in any local range into corresponding first semantic vectors;

[0118] Based on the second vectorization network in the semantic recognition network, converting the attribute information of the character into a corresponding second semantic vector;

[0119] Based on the third vectorization network in the semantic recognition network, converting the position of any local range on the layout into a corresponding third semantic vector;

[0120] Based on a fourth vectorization network in the semantic recognition network, converting the visual information of any local range into a corresponding fourth semantic vector;

[0121] The first semantic vector, the second semantic vector, the third semantic vector and the fourth semantic vector are input into a five-vector quantization network in the semantic recognition network, and a global vector of any local range is obtained based on the five-vector quantization network.

[0122] In some implementations, the layout structure recognition device may further include:

[0123] A normalization module, used for normalizing the position coordinates of any local range on the layout to obtain normalized position coordinates;

[0124] The identification module 72 is used to:

[0125] Based on the third vectorization network, the normalized position coordinates are converted into corresponding third semantic vectors.

[0126] In some embodiments, the visual information of any local range includes a background image of any local range;

[0127] The identification module 72 is used to:

[0128] Based on the fourth vectorization network, the background image of any local range is converted into a fourth semantic vector.

[0129] In some embodiments, the structure type includes at least one of the following:

[0130] Headings, paragraphs, images, tables, code blocks, formulas, lists.

[0131] The device provided in the embodiments of the present disclosure can execute the method of any of the above method embodiments, and its execution method and beneficial effects are similar, which will not be repeated here.

[0132] Figure 8 The schematic diagram of the structure of the computer device embodiment provided by the present disclosure embodiment. Figure 8 As shown, the computer device includes a memory 121 and a processor 122 .

[0133] The memory 121 is used to store programs. In addition to the above programs, the memory 121 can also be configured to store various other data to support operations on the computer device. Examples of these data include instructions, pictures, videos, etc. for any application or method operating on the computer device.

[0134] The memory 121 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0135] The processor 122 is coupled to the memory 121 and executes the program stored in the memory 121 to:

[0136] Acquire information of a plurality of local ranges contained in a layout, wherein the information of the local ranges includes characters on the local ranges, positions of the local ranges on the layout, and visual information;

[0137] For any local range on the layout, at least the characters on the local range, the position of the local range on the layout, and visual information are input into a preset recognition model, and the structural type of the local range in the layout is obtained based on the recognition model;

[0138] Merge any local range and at least one first local range within a preset range around any local range to obtain a second local range, and use the structure type of any local range as the structure type of the second local range, wherein the first local range refers to a local range with the same structure type as that of any local range;

[0139] Based on the positions of any one of the local ranges and the at least one first local range in the layout, the position of the second local range in the layout is determined.

[0140] In some implementations, for any local range on the layout, at least characters on the local range, the position of the local range on the layout, and visual information are input into a preset recognition model, and the structural type of the local range in the layout is obtained based on the recognition model, including:

[0141] Inputting characters on any local range, attribute information of the characters, position of the local range on the layout, and visual information into a preset recognition model, and identifying and obtaining a structural type of the any local range on the layout based on the recognition model;

[0142] The attribute information of the character includes at least one of the following information:

[0143] Font, color, size.

[0144] In some implementations, inputting the characters on any local range, the attribute information of the characters, the position of the local range on the layout, and the visual information into a preset recognition model, and obtaining the structural type of any local range on the layout based on the recognition model includes:

[0145] Input characters on any local range, attribute information of the characters, position of the local range on the page, and visual information into a semantic recognition network in the recognition model, and obtain a global vector of any local range based on processing by the semantic recognition network;

[0146] The global vector is input into the classification network in the recognition model, and the structural type of any local range in the layout is obtained based on the classification network identification.

[0147] In some implementations, inputting the characters on any local range, the attribute information of the characters, the position of the local range on the layout, and the visual information into a semantic recognition network in the recognition model, and obtaining a global vector of any local range based on the semantic recognition network processing includes:

[0148] Based on the first vectorization network in the semantic recognition network, convert the characters in any local range into corresponding first semantic vectors;

[0149] Based on the second vectorization network in the semantic recognition network, converting the attribute information of the character into a corresponding second semantic vector;

[0150] Based on the third vectorization network in the semantic recognition network, converting the position of any local range on the layout into a corresponding third semantic vector;

[0151] Based on a fourth vectorization network in the semantic recognition network, converting the visual information of any local range into a corresponding fourth semantic vector;

[0152] The first semantic vector, the second semantic vector, the third semantic vector and the fourth semantic vector are input into a five-vector quantization network in the semantic recognition network, and a global vector of any local range is obtained based on the five-vector quantization network.

[0153] In some embodiments, before converting the position of any local range on the layout into a corresponding third semantic vector based on the third vectorization network in the semantic recognition network, the method further includes:

[0154] Normalizing the position coordinates of any local range on the page to obtain normalized position coordinates;

[0155] The step of converting the position of any local range on the layout into a corresponding third semantic vector based on the third vectorization network in the semantic recognition network includes:

[0156] Based on the third vectorization network, the normalized position coordinates are converted into corresponding third semantic vectors.

[0157] In some embodiments, the visual information of any local range includes a background image of any local range;

[0158] The converting the visual information of any local range into a corresponding fourth semantic vector based on the fourth vectorization network in the semantic recognition network includes:

[0159] Based on the fourth vectorization network, the background image of any local range is converted into a fourth semantic vector.

[0160] In some embodiments, the structure type includes at least one of the following:

[0161] Headings, paragraphs, images, tables, code blocks, formulas, lists.

[0162] Further, if Figure 8 As shown, the computer device may also include: a communication component 123, a power component 124, an audio component 125, a display 126 and other components. Figure 8 Only some components are shown schematically, and it does not mean that the computer equipment only includes Figure 8 Components shown.

[0163] The communication component 123 is configured to facilitate wired or wireless communication between the computer device and other devices. The computer device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 123 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 123 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0164] The power supply component 124 provides power to various components of the computer device. The power supply component 124 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the computer device.

[0165] The audio component 125 is configured to output and / or input audio signals. For example, the audio component 125 includes a microphone (MIC), and when the computer device is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 121 or sent via the communication component 123. In some embodiments, the audio component 125 also includes a speaker for outputting audio signals.

[0166] The display 126 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0167] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the method described in any of the above method embodiments.

[0168] The exemplary embodiments of the present disclosure further provide a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute to implement the method described in any of the above method embodiments.

[0169] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0170] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying a layout structure, characterized in that: The method comprises: Acquire information of a plurality of local ranges contained in a layout, wherein the information of the local ranges includes characters on the local ranges, positions of the local ranges on the layout, and visual information; For any local range on the layout, at least the characters on the local range, the position of the local range on the layout, and visual information are input into a preset recognition model, and the structural type of the local range in the layout is obtained based on the recognition model; Merge any local range and at least one first local range within a preset range around any local range to obtain a second local range, and use the structure type of any local range as the structure type of the second local range, wherein the first local range refers to a local range with the same structure type as that of any local range; Based on the positions of any one of the local ranges and the at least one first local range in the layout, the position of the second local range in the layout is determined.

2. The method according to claim 1, characterized in that For any local range on the layout, at least inputting characters on the local range, the position of the local range on the layout, and visual information into a preset recognition model, and identifying the structural type of the local range in the layout based on the recognition model, includes: Inputting characters on any local range, attribute information of the characters, position of the local range on the layout, and visual information into a preset recognition model, and identifying the structural type of any local range on the layout based on the recognition model; The attribute information of the character includes at least one of the following information: Font, color, size.

3. The method according to claim 2, characterized in that The step of inputting the characters on any local range, the attribute information of the characters, the position of the local range on the layout, and the visual information into a preset recognition model, and obtaining the structural type of any local range on the layout based on the recognition model includes: Input characters on any local range, attribute information of the characters, position of any local range on the page, and visual information into a semantic recognition network in the recognition model, and obtain a global vector of any local range based on processing by the semantic recognition network; The global vector is input into the classification network in the recognition model, and the structural type of any local range in the layout is obtained based on the classification network identification.

4. The method according to claim 3, characterized in that The step of inputting the characters on any local range, the attribute information of the characters, the position of the local range on the layout, and the visual information into the semantic recognition network in the recognition model, and obtaining the global vector of any local range based on the processing of the semantic recognition network comprises: Based on the first vectorization network in the semantic recognition network, convert the characters in any local range into corresponding first semantic vectors; Based on the second vectorization network in the semantic recognition network, converting the attribute information of the character into a corresponding second semantic vector; Based on the third vectorization network in the semantic recognition network, converting the position of any local range on the layout into a corresponding third semantic vector; Based on a fourth vectorization network in the semantic recognition network, converting the visual information of any local range into a corresponding fourth semantic vector; The first semantic vector, the second semantic vector, the third semantic vector and the fourth semantic vector are input into a five-vector quantization network in the semantic recognition network, and a global vector of any local range is obtained based on the five-vector quantization network.

5. The method according to claim 4, characterized in that Before converting the position of any local range on the layout into a corresponding third semantic vector based on the third vectorization network in the semantic recognition network, the method further includes: Normalizing the position coordinates of any local range on the page to obtain normalized position coordinates; The step of converting the position of any local range on the layout into a corresponding third semantic vector based on the third vectorization network in the semantic recognition network includes: Based on the third vectorization network, the normalized position coordinates are converted into corresponding third semantic vectors.

6. The method according to claim 4, characterized in that The visual information of any local range includes a background image of any local range; The converting the visual information of any local range into a corresponding fourth semantic vector based on the fourth vectorization network in the semantic recognition network includes: Based on the fourth vectorization network, the background image of any local range is converted into a fourth semantic vector.

7. The method according to any one of claims 1 to 6, characterized in that The structure type includes at least one of the following: Headings, paragraphs, images, tables, code blocks, formulas, lists.

8. A layout structure recognition device, characterized in that: include: An acquisition module, used for acquiring information of a plurality of local ranges contained in a layout, wherein the information of the local ranges includes characters on the local ranges, positions of the local ranges on the layout, and visual information; A recognition module, for inputting at least the characters on any local range on the layout, the position of any local range on the layout, and visual information into a preset recognition model, and identifying the structural type of any local range on the layout based on the recognition model; a merging module, configured to merge the any local range and at least one first local range within a preset range around the any local range to obtain a second local range, and use the structure type of the any local range as the structure type of the second local range, wherein the first local range refers to a local range with the same structure type as the any local range; A position determination module is used to determine the position of the second local range in the layout based on the positions of any local range and the at least one first local range in the layout.

9. A computer device, wherein: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images