A layout image processing method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses a method, device, equipment and storage medium for processing layout images, wherein the method comprises: inputting the layout image into a pre-trained segmentation model, performing regional segmentation on the layout image based on the segmentation model, and when the segmented area contains a table area, performing text segmentation after linear elimination of the table area, and outputting the element position and element segmentation map obtained by the final segmentation of the layout image; performing text recognition on the element segmentation map belonging to the text type to obtain the text content; and displaying the element position of the layout image, the element segmentation map except the text type and the text content in a preset display mode. End-to-end layout image segmentation processing based on a segmentation model is realized, and compared with the traditional solution, the manual operation of data transmission between multiple models is reduced, and the processing efficiency of the layout image is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method, device, equipment and storage medium for processing layout images. Background Art

[0002] At present, the layout images of e-commerce product details pages contain rich content, such as image areas containing the product body, table areas, text introduction areas, etc. By automatically mining the layout information of product details pages and analyzing the layout images with visual quality or high product sales, it is of great significance to extract product information and automatically design the layout images of product details pages.

[0003] In the prior art, the method for processing layout images is usually to first use a layout recognition model to identify the various areas of the layout image and their positional relationships, then use a straight line detection model to perform straight line detection on the identified table area, and use an optical character recognition (OCR) model to perform text detection and recognition on the table area and text introduction area.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following technical problems in the prior art: the above-mentioned solution of using three models to process layout images has low processing efficiency. Summary of the invention

[0005] The embodiments of the present invention provide a layout image processing method, device, equipment and storage medium, which realize end-to-end layout image processing and improve the layout image processing efficiency.

[0006] In a first aspect, an embodiment of the present invention provides a method for processing a layout image, comprising:

[0007] Inputting a layout image into a pre-trained segmentation model, performing region segmentation on the layout image based on the segmentation model, and when the segmented region includes a table region, performing text segmentation after straight line elimination on the table region, and outputting the element position and element segmentation map obtained by the final segmentation of the layout image;

[0008] Perform text recognition on the element segmentation map belonging to the text type to obtain the text content;

[0009] According to a preset display method, the element position of the layout image, the element segmentation diagram except the text type and the text content are displayed.

[0010] In a second aspect, an embodiment of the present invention provides a layout image processing device, comprising:

[0011] A layout segmentation module is used to input a layout image into a pre-trained segmentation model, perform region segmentation on the layout image based on the segmentation model, and when the segmented region includes a table region, perform text segmentation after straight line elimination on the table region, and output the element position and element segmentation map obtained by the final segmentation of the layout image;

[0012] A text recognition module is used to perform text recognition on the element segmentation graph belonging to the text type to obtain the text content;

[0013] The result display module is used to display the element position of the layout image, the element segmentation diagram except the text type and the text content according to a preset display method.

[0014] In a third aspect, an embodiment of the present invention provides a terminal device, including:

[0015] one or more processors;

[0016] A memory for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the layout image processing method as described in any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for processing a layout image as described in any embodiment of the present invention.

[0019] The embodiments of the present invention provide a layout image processing method, device, equipment and storage medium, which inputs the layout image into a pre-trained segmentation model, performs regional segmentation on the layout image based on the segmentation model, and when the segmented area contains a table area, performs text segmentation after linear elimination of the table area, outputs the element position and element segmentation map of the layout image; performs text recognition on the element segmentation map belonging to the text type to obtain the text content; and displays the element position of the layout image, the element segmentation map except the text type and the text content in a preset display mode. End-to-end layout image segmentation processing based on a segmentation model is realized, which reduces the manual operation of data transmission between multiple models compared with the traditional solution, and improves the processing efficiency of layout images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flow chart of a method for processing a layout image provided by Embodiment 1 of the present invention is shown;

[0022] Figure 2 A model processing flow chart in a layout image processing method provided in Embodiment 1 of the present invention is shown;

[0023] Figure 3 A processing flow chart of a segmentation model in a method for processing a layout image provided in Embodiment 1 of the present invention is shown;

[0024] Figure 4 A flow chart of a method for processing a layout image provided by Embodiment 3 of the present invention is shown;

[0025] Figure 5 A schematic diagram of layout result output in a layout picture processing method provided in Embodiment 3 of the present invention is shown;

[0026] Figure 6 A schematic diagram of the structure of a layout image processing device provided by a fourth embodiment of the present invention is shown;

[0027] Figure 7 A schematic diagram of the hardware structure of a terminal device provided in Embodiment 5 of the present invention is shown. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described through implementation methods with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the following embodiments, each embodiment provides optional features and examples at the same time. The various features recorded in the embodiments can be combined to form multiple optional solutions. Each numbered embodiment should not be regarded as only one technical solution.

[0029] Embodiment 1

[0030] Figure 1The flowchart of a layout image processing method provided by the first embodiment of the present invention is shown. The layout image processing method provided by the embodiment of the present invention can be applied to the processing of layout images, such as the layout information analysis and mining of the layout images of the e-commerce product details page. The method can be executed by a layout image processing device, which is implemented in software and / or hardware, and is preferably configured in an electronic device, such as a computer. Figure 1 As shown, the method for processing a layout image provided in an embodiment of the present invention comprises the following steps:

[0031] S110. Input the layout image into a pre-trained segmentation model, perform region segmentation on the layout image based on the segmentation model, and when the segmented area includes a table area, perform text segmentation after eliminating straight lines in the table area, and output the element positions and element segmentation map obtained by the final segmentation of the layout image.

[0032] In an embodiment of the present invention, the layout image includes but is not limited to the layout image of a book, poster, magazine or web page, and the layout image may carry different types of element information, such as text, images, and tables. A segmentation model may be obtained through pre-end-to-end deep learning training. Based on the pre-trained segmentation model, the input layout image may be segmented into regions according to the type of element information, and when the segmented region includes a table region, the table region may be straight-line eliminated and then text segmentation may be performed, and the element segmentation map and element position information obtained by the final segmentation may be output. Among them, the element position information includes but is not limited to the position information of the element segmentation map, the position information of the element information in the layout image, etc., and the position information of the element information includes but is not limited to the position information of the table line and the position information of the text line, etc.

[0033] Compared with the traditional solution of using three models to process layout images, this embodiment is based on an end-to-end segmentation model to automatically segment each element of the layout image, reducing the manual operations required for data transmission between multiple models and improving the processing efficiency of the layout image.

[0034] S120, performing text recognition on the element segmentation graph belonging to the text type to obtain text content.

[0035] In this embodiment, the segmentation model can output the type of each element segmentation map while outputting the element segmentation map. Among them, the type of element information carried in the element segmentation map can be used as the type of the element segmentation map. For example, if the type of element information in the element segmentation map is an image, the type of the element segmentation map is an image type. If the element segmentation map output by the segmentation model contains an element segmentation map of the text type, the element segmentation map of the text type can be used for text recognition.

[0036] In a specific implementation, the element segmentation map of the text type can be identified by a text recognition model based on OCR. Among them, the text recognition model is, for example, an efficient and accurate scene text detection algorithm (An Efficient and Accuracy Scene Text Detector, EAST algorithm) + Convolutional Recurrent Neural Network (Convolutional Recurrent Neural Network, CRNN), or a shape robust text detection (Shape Robust Text Detection with Progressive Scale Expansion Network, PSENet) method + CRNN based on a progressive scale expansion network, or an EAST algorithm + an attention scene text recognizer with flexible correction function (An Attentional Scene Text Recognizer with Flexible Rectification, ASTER), or a CRNN + Convolutional Recurrent Neural Network (Convolutional Recurrent Neural Network, CTC) and other models, which are not exhaustive here.

[0037] By identifying the text on the text-type element segmentation graph, the text on the version image can be extracted and mined. When the version image is the version image of the product details page, it is helpful to mine the text introduction of the selling point of the hot-selling product, which has important guiding significance for generating selling point copy with product characteristics and generating layout images of other products based on the selling point copy.

[0038] S130: Display the element positions of the layout image, the element segmentation diagram except the text type, and the text content according to a preset display method.

[0039] In this embodiment, when the element positions of the layout image, the element segmentation diagram except the text type, and the text content are obtained through processing, the processing results can be displayed.

[0040] The preset display method may be, for example, a list display method in outline mode, which may specifically be to divide the element segmentation diagram and text content other than text type into main items and branch items according to the element position, and each branch item may be displayed or hidden according to the display / hide control of the main item. Such a display method may enable users to understand the layout structure more clearly and facilitate quick search for content of the corresponding layout structure.

[0041] In addition, the preset display mode can also be restored to the editable original layout display mode, which can be specifically rendered using the interface rendering algorithm according to the element position, element segmentation diagram except text type and text content to restore the structure and content of the original layout. This display mode can more intuitively display the layout processing results, making it easier for users to directly edit the layout and intuitively understand the editing effect.

[0042] Figure 2 The figure shows a model processing flow chart in a layout image processing method provided by the first embodiment of the present invention. Optionally, the segmentation model includes a semantic segmentation branch network, a line detection branch network and a text detection branch network; accordingly, see Figure 2 , based on the segmentation model, the layout image is segmented, and when the segmented area contains a table area, the table area is eliminated straight lines before text segmentation, and the element positions and element segmentation maps obtained by the final segmentation of the layout image are output, including:

[0043] S111. Input the layout image into the semantic segmentation branch network, perform region segmentation on the layout image based on the semantic segmentation branch network, and output at least one of the following region segmentation maps: an image region segmentation map, a table region segmentation map, and a text region segmentation map.

[0044] In this embodiment, the semantic segmentation branch network can be, for example, an end-to-end neural network, such as a segmentation network such as U-net or Deeplab v1-v3. The semantic segmentation branch network can be used to extract pixel-level layout segmentation information, that is, to determine the type of region to which each pixel belongs, and determine a region segmentation map based on pixels belonging to the same region type.

[0045] When the semantic segmentation branch network completes the segmentation of the image area, table area and text area, different processing operations can be performed on different areas. Specifically, the image area segmentation map and its position information can be directly output, the table area segmentation map can be input into the line detection branch network, and the text area segmentation map can be input into the text detection branch network to perform table line detection and text detection. In this way, the element position and element segmentation map of the layout image can be output based on a segmentation model, which improves the processing efficiency of the layout image.

[0046] S112. When the segmented area includes a table area, the table area segmentation map is input into a line detection branch network, and the position information of the table line is output based on the line detection branch network.

[0047] In this embodiment, the line detection branch network can be, for example, a neural network obtained through deep learning training. The line detection branch network can be used to directly segment the table lines and output the position information of the table lines. The position information of the table lines can be, for example, the position information of the segmentation map of the table lines. By segmenting the lines in the table and obtaining the position information of the lines, it is helpful to restore the display operation of the original layout.

[0048] S113. Eliminate the straight lines of the table area image based on the position information of the straight lines of the table.

[0049] In this embodiment, the table straight lines may be eliminated by subtracting the table straight line segmentation map from the table area segmentation map.

[0050] The text between table lines in an existing table area is usually small and densely arranged. When performing text detection on a table area, the text detection box may detect text in two or more small grids into one text box, that is, the text detection box crosses the straight lines of the table. This will cause the cropped text segmentation map to contain vertical or horizontal lines, which can easily affect the text recognition results. By suppressing the straight lines in the table area, it helps to improve the recognition accuracy of the text in the table area.

[0051] For example, suppose that in a table containing notebook parameters, there are adjacent column fields "refresh rate and color gamut", and the field data of the two column fields can be 60HZ, 72% NTSC respectively. Using the traditional recognition method, when performing text recognition on the table, it is easy to recognize it as "60HZ|72% NTSC". If the table area image is first linearly eliminated and then the text recognition is performed, the recognition result is "60HZ 72% NTSC", thereby improving the text recognition accuracy of the table area.

[0052] S114, inputting the table area segmentation map and the text area segmentation map with the straight lines eliminated into the text detection branch network, and outputting the position information of the text detection box and the text segmentation map based on the text detection branch network.

[0053] In this embodiment, there is no strict restriction on the timing of inputting the table area segmentation map and the text area segmentation map for eliminating straight lines into the text detection branch network. The table area segmentation map and the text area segmentation map for eliminating straight lines can be input into the text detection branch network at the same time, or the text area segmentation map can be input into the text detection branch network first after step S111, and the table area segmentation map for eliminating straight lines can be input into the text detection branch network after step S113.

[0054] The text detection branch network can perform text detection by combining the table area segmentation map and the text area segmentation map. The text detection branch network can perform more detailed segmentation of the text to obtain a text segmentation map, and can also regress the text detection frame of the text segmentation map and output the pixel coordinates of each corner point of the text detection frame.

[0055] Among them, when the text is regular text, the text detection box can be a rectangular box, and the position information of the text detection box can only include the pixel coordinates of the corner points of the top corners, such as the pixel coordinates of the upper left corner and the lower right corner of the rectangular box, or the pixel coordinates of the lower left corner and the upper right corner of the rectangular box. When the text is irregular text, the text detection box can be an irregular polygonal frame, and the position information of the text detection box needs to include the pixel coordinates of each corner point.

[0056] S115, outputting the image region segmentation map and the text segmentation map as element segmentation maps, and outputting the position information of the image region segmentation map, the position information of the table straight line, and the position information of the text detection box as element positions. Outputting the element positions and the element segmentation map is conducive to outputting a structured display of the output results.

[0057] Furthermore, the position information of the text detection box and the text segmentation map are output based on the text detection branch network, including: based on the text detection branch network, obtaining the text detection box of the table area segmentation map with the straight line eliminated according to the position information of the table straight line; determining the coordinates of the text detection box, and outputting the coordinates as the position information of the text detection box; segmenting the text in the text detection box, obtaining the text segmentation map and outputting it.

[0058] Among them, when performing text segmentation on a table area segmentation map with straight lines eliminated, the text can be first segmented according to the area enclosed by the table straight lines, and then the text detection box of the text segmentation map can be regressed, thereby avoiding the situation where the text detection box crosses the table straight lines.

[0059] Figure 3 A processing flow chart of a segmentation model in a layout image processing method provided in Embodiment 1 of the present invention is shown.

[0060] See also Figure 3The layout image is the layout image of the details page of the diamond ring product; the process part from inputting the layout image to outputting the image region segmentation map (region segmentation map filled with oblique lines), the table region segmentation map (region segmentation map filled with dots), and the text region segmentation map (region segmentation map filled with grid lines) is all completed by the semantic segmentation branch network; the process part from inputting the table region segmentation map to outputting the position information of the table straight line is all completed by the straight line detection branch network; the exposed part from inputting the table region segmentation map and the text region segmentation map with the straight line eliminated to outputting the text segmentation map and the position information of the text detection box is all completed by the text detection branch network.

[0061] See also Figure 3 , the semantic segmentation branch network is a deeplab v3 convolutional neural network that includes Atrous SpatialPyramid Pooling (APSS), and its backbone network can be replaced by the Split-Attention Network in ResnetST with higher accuracy. In the processing flow of the semantic segmentation branch network, Conv1+Pool1 is a convolution plus pooling process; Output stride is the output stride, and the output stride of Block1-Block4 is set to 4, 8, 16 and 16 respectively; Rate is the sampling rate; Concat+1*1Conv1 is used to realize data splicing and convolution; Upsampling Segmentation is upsampling segmentation. The semantic segmentation branch network can output the probability matrix of pixels belonging to each region segmentation map, and calculate the supervised segmentation error. Among them, the image region segmentation map and its position information can be directly output, and the table region segmentation map and the text region segmentation map can be passed to the subsequent branch networks for separate processing.

[0062] See also Figure 3 The line detection branch network is a U-net segmentation network, and its main network also uses a high-precision ResnetST network. Through the line detection branch network, the probability matrix of the table line and the segmentation map of the table line can be output, and the supervised segmentation error can be calculated. By subtracting the segmentation map of the table line from the segmentation map of the table area, the segmentation map after eliminating the line in the table area can be obtained, that is, the remaining text part.

[0063] See also Figure 3, the text detection branch network can also be a U-shape network (U-shape). The text detection branch network can comprehensively consider the text features of small receptive fields and large receptive fields, output the coordinates of the text detection box, the probability matrix and the text segmentation map according to the input pure text area segmentation map, and calculate the supervised segmentation error. Among them, the text segmentation map is a more refined segmentation output of the text based on the text area segmentation map and the table area segmentation map; Opencv can be used to obtain the minimum bounding box of each text segmentation map, and the text detection box and the text detection box coordinates can be obtained.

[0064] based on Figure 3 The segmentation model shown can complete the process from layout image input to image area segmentation map and its position information, table straight line position information, text segmentation map and text detection box position information output, with high processing efficiency.

[0065] The embodiment of the present invention provides a layout image processing method, which inputs the layout image into a pre-trained segmentation model, outputs the element position and element segmentation map of the layout image based on the segmentation model; performs text recognition on the element segmentation map belonging to the text type to obtain the text content; and displays the element position of the layout image, the element segmentation map except the text type, and the text content in a preset display mode. The end-to-end layout image segmentation processing based on a segmentation model is realized, which reduces the manual operation of data transmission between multiple models and improves the processing efficiency of layout images compared with traditional solutions.

[0066] Embodiment 2

[0067] Based on the above embodiment, this embodiment optimizes the training method of the segmentation model, and can improve the processing accuracy by transferring the loss function between the branch networks in the segmentation model. The processing method of the layout image proposed in this embodiment and the above embodiment belongs to the same inventive concept, and the technical details not described in detail in this embodiment can be referred to the above embodiment.

[0068] Optionally, the training method of the segmentation model includes: training at least one branch network of the segmentation model until convergence; back-propagating the error of at least one branch network to obtain the total error of the segmentation model, and training the segmentation model with the total error being less than a preset threshold as the goal.

[0069] In this embodiment, since the segmentation model contains many branch networks, each branch network can be trained to converge first, and then the whole end-to-end fine-tuning can be performed to complete the training of the segmentation model. Specifically, the end-to-end fine-tuning can be designed to adopt the back propagation (BP) training method to back propagate the error (i.e., the loss function Loss) to iteratively update the network parameters of the segmentation model. During the model training process, the loss function is transmitted between the branch networks to improve the model processing accuracy.

[0070] Furthermore, the error of at least one branch network is back-propagated to obtain the total error of the segmentation model, including:

[0071] The semantic segmentation branch network error, the line detection branch network error, the detection error of the position information of the text detection box of the text detection branch network, and the text segmentation error of the text detection branch network are multiplied by the corresponding preset constant coefficients, and the sum of the products is taken as the total error of the segmentation model.

[0072] In some specific implementations, the total error of the segmentation model may be calculated based on the following formula:

[0073] Loss=Loss seg +αLoss line +βLoss det +γLoss textseg ;

[0074] Wherein, α represents the first preset constant; β represents the second preset constant; γ represents the third preset constant; Loss seg Represents the semantic segmentation branch network error; Loss line Represents the error of the line detection branch network; Loss det Loss represents the detection error of the position information of the text detection box of the text detection branch network; textseg Represents the text segmentation error of the text detection branch network; Loss represents the total error of the segmentation model.

[0075] The errors of the four parts can be calculated using different error calculation methods according to the actual application scenarios. The coefficients of the four parts of the error can be set based on the semantic segmentation branch network error. The coefficient of the semantic segmentation branch network error can be set to 1, and the coefficients of the remaining three parts of the error can be set to α, β and γ according to empirical values ​​or experimental values, and α, β and γ are constants greater than 0. The error of the text detection branch network is equal to Loss det Add Loss textseg The sum.

[0076] In one embodiment, Loss segThe Dice Loss calculation method can be used, and the calculation formula can be shown as follows:

[0077]

[0078] Where n represents the number of segmented region segmentation maps; i represents the i-th region segmentation map; D(S i ,G i ) represents the Dice Loss of the i-th region segmentation map. The specific formula can be shown as follows:

[0079]

[0080] Where x and y represent the horizontal and vertical coordinates of the pixels in the i-th region segmentation map, respectively; S i represents the predicted classification result of the i-th region segmentation map, G i Represents the actual classification result of the i-th region segmentation map.

[0081] In one embodiment, Loss line and Loss textseg The cross-entropy loss calculation method of the balanced category segmentation error can be used, with Loss line For example, the calculation formula can be as follows:

[0082] Loss line =balanced-xent(Y,Y * )

[0083] =-aY * logY-(1-a)(1-Y * )log(1-Y)

[0084] Among them, Y represents the predicted probability matrix, Y * represents the actual value, a represents the ratio of the number of counterexample samples to the total number of samples, and

[0085] In one embodiment, Loss det The calculation method of QUAD Loss in the EAST algorithm can be used to calculate the detection error of the text detection box coordinates. Smoothed-L1 Loss can be used. The calculation method is as follows:

[0086]

[0087]

[0088] in, is the length of the shortest side among the i sides of the text detection box; CQ is the coordinate of the corner point of the text detection box. Since the angle of the text line is not fixed, C Q Can be the coordinates of the four corner points, specifically expressed as C Q ={x1,y1,x2,y2,x3,y3,x4,y4}; Q is the predicted value, Q * is the actual value; is Q with different vertex order * The set of equivalent quadrilaterals.

[0089] Based on the above embodiment, the embodiment of the present invention optimizes the training method of the segmentation model, and can improve the processing accuracy by transferring the loss function between the branch networks in the segmentation model. In addition, the processing method of the layout image proposed in the embodiment of the present invention and the above embodiment belongs to the same inventive concept, and the technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0090] Embodiment 3

[0091] Based on the above embodiments, this embodiment can further operate on the element position and element segmentation diagram after displaying the element position, element segmentation diagram and text content, and can realize operations such as extracting the element segmentation diagram and / or text content, editing the element position, element segmentation diagram and / or text content, and regenerating layout images according to the element position, element segmentation diagram and text content, etc., which is of great significance for extracting product information and automatically designing the layout images of product details pages. The layout image processing method proposed in this embodiment and the above embodiments belongs to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiments.

[0092] Figure 4 FIG. 1 is a flow chart of a method for processing a layout image provided by Embodiment 3 of the present invention. Figure 4 The method for processing a layout image provided in an embodiment of the present invention comprises the following steps:

[0093] S410, inputting the layout image into a pre-trained segmentation model, performing region segmentation on the layout image based on the segmentation model, and when the segmented area includes a table area, performing text segmentation after eliminating straight lines in the table area, and outputting the element positions and element segmentation map obtained by the final segmentation of the layout image.

[0094] S420: Perform text recognition on the element segmentation graph belonging to the text type to obtain text content.

[0095] S430: Display the element positions of the layout image, the element segmentation diagram except the text type, and the text content according to a preset display method.

[0096] Optionally, the element positions of the layout image, the element segmentation diagram and the text content other than the text type are displayed in a preset display manner, including: in the display interface, the element positions of the layout image are marked based on a position identification box in a preset format, and the element segmentation diagram and text content other than the text type are presented in a corresponding position identification box.

[0097] In this embodiment, the position identification frame of the preset format is, for example, an identification frame generated in a format of different colors, line types or effects, and the identification frame can be used to distinguish and identify different element positions in the layout image, for example, an orange frame is used to identify the position of the image area segmentation map, a green frame is used to identify the position of the table straight line, etc. Among them, an image control or a text control can be set in the identification frame to add an element segmentation map in the image control and add text content in the text control, so as to realize that the element segmentation map and text content other than the text type are presented in the corresponding position identification frame.

[0098] By utilizing the interface rendering algorithm, the element position identification box, element segmentation map and text content are rendered in the display interface to restore the original layout structure and content, allowing users to more intuitively display the layout processing results; and the content rendered on the interface is editable, making it convenient for users to directly edit the layout and intuitively understand the editing effect.

[0099] For example, Figure 5 FIG. 1 is a schematic diagram showing the layout result output in a layout image processing method provided in Embodiment 3 of the present invention. Figure 5 , the layout image is the layout image of the details page of the diamond ring product; the position identification box in the dotted line format is used to identify the position of the image area segmentation map, the position identification box in the solid line format is used to identify the position of the table straight line, and the position identification box in the dotted line format is used to identify the position information of the text detection box; the image area segmentation map of the diamond ring can be presented in the dotted line format position identification box, and the text content can be presented in the dotted line format position identification box, so that the structure and content of the original layout can be restored.

[0100] S440. In response to the layout operation instruction, operate the element position of the layout image, the element segmentation diagram other than the text type, and / or the text content.

[0101] In this embodiment, the layout operation instruction includes at least one of the following: content extraction instruction, editing instruction and layout image generation instruction. In addition, the layout operation instruction may also include other instructions for layout operation, and is not limited to the above instructions.

[0102] When a content extraction instruction is received, the image region segmentation map or text content may be extracted.

[0103] Among them, the editing instructions may include instructions such as move, replace, add and delete; when a move instruction is received, the position identification box can be moved, and the content presented in the position identification box can follow the movement; when a replace instruction is received, the image area segmentation map or text content can be replaced; when a add instruction is received, a new position identification box can be added, and an element segmentation map or text content can be added to the position identification box; when a add instruction is received, the position identification box can be deleted, and the content presented in the position identification box can be deleted, or the element segmentation map or text content in the position identification box can be deleted.

[0104] After editing the displayed layout, instructions can be generated according to the received layout image, and a new layout image can be generated using the edited display interface.

[0105] For example, when the layout image is the layout image of an e-commerce product details webpage, the text description of the selling points of the hot-selling product can be extracted through the extraction instruction, which has important guiding significance for generating selling point copy with product characteristics; editing the display interface through the editing instruction is conducive to replacing new product images, tables and text parameters according to the high-quality layout design when automatically designing the product details page image; through the layout image generation instruction, the edited display interface can be used to generate a new product layout image to reduce the manpower and time cost of the design.

[0106] Based on the above embodiments, the embodiments of the present invention can further operate on the element positions and element segmentation diagrams after displaying the element positions, element segmentation diagrams and text contents, and can realize operations such as extracting the element segmentation diagrams and / or text contents, editing the element positions, element segmentation diagrams and / or text contents, and regenerating layout images according to the element positions, element segmentation diagrams and text contents, etc., which is of great significance for extracting product information and automatically designing the layout images of product detail pages. In addition, the layout image processing methods proposed in the embodiments of the present invention and the above embodiments belong to the same inventive concept, and the technical details not described in detail in the present embodiments can be referred to the above embodiments, and the present embodiments have the same beneficial effects as the above embodiments.

[0107] Embodiment 4

[0108] Figure 6 The schematic diagram of the structure of a layout image processing device provided by the fourth embodiment of the present invention is shown. The embodiment of the present invention can be applied to the processing of layout images, such as the analysis and mining of layout information of the layout images of the e-commerce product details page. The layout image processing device provided by the present invention can realize the layout image processing method provided by the above embodiment.

[0109] like Figure 6As shown, the layout picture processing device in the embodiment of the present invention includes:

[0110] The layout segmentation module 610 is used to input the layout image into a pre-trained segmentation model, perform region segmentation on the layout image based on the segmentation model, and when the segmented region includes a table region, perform text segmentation after eliminating straight lines in the table region, and output the element positions and element segmentation map obtained by the final segmentation of the layout image;

[0111] The text recognition module 620 is used to perform text recognition on the element segmentation graph belonging to the text type to obtain the text content;

[0112] The result display module 630 is used to display the element positions of the layout image, the element segmentation diagram except the text type, and the text content according to a preset display method.

[0113] Optionally, the segmentation model includes a semantic segmentation branch network, a line detection branch network, and a text detection branch network;

[0114] Correspondingly, the layout segmentation module includes:

[0115] A region segmentation submodule is used to input the layout image into the semantic segmentation branch network, perform region segmentation on the layout image based on the semantic segmentation branch network, and output at least one of the following region segmentation maps: an image region segmentation map, a table region segmentation map, and a text region segmentation map;

[0116] A line detection submodule, for inputting the table area segmentation map into the line detection branch network when the segmented area includes the table area, and outputting the position information of the table line based on the line detection branch network;

[0117] A line elimination submodule is used to eliminate lines from the table area image based on the position information of the line of the table;

[0118] A text segmentation submodule is used to input the table area segmentation map and the text area segmentation map with straight lines eliminated into the text detection branch network, and output the position information of the text detection box and the text segmentation map based on the text detection branch network;

[0119] The output submodule is used to output the image region segmentation map and the text segmentation map as element segmentation maps, and output the position information of the image region segmentation map, the position information of the table straight line and the position information of the text detection box as element positions.

[0120] Optionally, a text segmentation submodule is specifically used to: based on the text detection branch network, obtain a text detection frame of the table area segmentation map that eliminates the straight lines according to the position information of the table straight lines; determine the coordinates of the text detection frame, and output the coordinates as the position information of the text detection frame; segment the text in the text detection frame, obtain a text segmentation map and output it.

[0121] Optionally, the layout image processing device also includes:

[0122] The model training module is used to train at least one branch network of the segmentation model until convergence; back-propagate the error of at least one branch network to obtain the total error of the segmentation model, and train the segmentation model with the total error being less than a preset threshold as the goal.

[0123] Optional, model training module, including:

[0124] The error calculation unit is used to multiply the semantic segmentation branch network error, the line detection branch network error, the detection error of the position information of the text detection box of the text detection branch network, and the text segmentation error of the text detection branch network with the corresponding preset constant coefficients, and use the sum of the products as the total error of the segmentation model.

[0125] Optionally, the result display module is specifically used to: in the display interface, mark the element positions of the layout image based on the position identification box of a preset format, and present the element segmentation diagram and text content other than the text type in the corresponding position identification box.

[0126] Optionally, the layout image processing device also includes:

[0127] The result operation module is used to respond to the layout operation instructions to operate the element position of the layout image, the element segmentation map other than the text type and / or the text content; wherein the layout operation instructions include at least one of the following: content extraction instructions, editing instructions and layout image generation instructions.

[0128] The layout image processing device provided in the embodiment of the present invention belongs to the same inventive concept as the layout image processing method provided in the above embodiment. The technical details not fully described in the embodiment of the present invention can be referred to the above embodiment, and the embodiment of the present invention has the same beneficial effects as the above embodiment.

[0129] Embodiment 5

[0130] Figure 7The hardware structure diagram of a terminal device provided by Embodiment 5 of the present invention is shown. The terminal device in the embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0131] like Figure 7 As shown, the terminal device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the terminal device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0132] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the terminal device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The terminal device 700 is shown to have various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0133] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the processing method of the layout picture provided by the embodiment of the present invention are executed.

[0134] The terminal provided in the embodiment of the present invention and the layout image processing method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in the embodiment of the present invention can be referred to the above embodiment, and the embodiment of the present invention has the same beneficial effects as the above embodiment.

[0135] Embodiment 6

[0136] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for processing a layout image provided by the above embodiment is implemented.

[0137] It should be noted that the computer-readable storage medium in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In an embodiment of the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0138] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0139] The computer-readable storage medium may be included in the terminal device, or may exist independently without being installed in the terminal device.

[0140] The terminal device stores and carries one or more programs. When the one or more programs are executed by the terminal device, the terminal device:

[0141] The layout image is input into a pre-trained segmentation model, and the layout image is segmented based on the segmentation model. When the segmented area contains a table area, the table area is eliminated by straight lines and then text segmentation is performed. The element position and element segmentation map obtained by the final segmentation of the layout image are output; the element segmentation map belonging to the text type is subjected to text recognition to obtain the text content; the element position of the layout image, the element segmentation map except the text type, and the text content are displayed in a preset display method.

[0142] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The units involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a unit does not limit the unit itself in some cases.

[0145] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0146] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for processing a layout image, characterized in that: include: Inputting a layout image into a pre-trained segmentation model, performing region segmentation on the layout image based on the segmentation model, and when the segmented region includes a table region, performing text segmentation after straight line elimination on the table region, and outputting the element position and element segmentation map obtained by the final segmentation of the layout image; Perform text recognition on the element segmentation map belonging to the text type to obtain the text content; Display the element positions of the layout image, the element segmentation diagram excluding the text type, and the text content in a preset display mode; The segmentation model includes a semantic segmentation branch network, a line detection branch network and a text detection branch network; The method of performing region segmentation on the layout image based on the segmentation model, and performing text segmentation after performing straight line elimination on the table area when the segmented area includes a table area, and outputting the element position and element segmentation map obtained by the final segmentation of the layout image, includes: Inputting the layout image into the semantic segmentation branch network, performing region segmentation on the layout image based on the semantic segmentation branch network, and outputting at least one of the following region segmentation maps: an image region segmentation map, a table region segmentation map, and a text region segmentation map; When the segmented area includes a table area, the table area segmentation map is input into the line detection branch network, and the position information of the table line is output based on the line detection branch network; wherein the line detection branch network directly segments the table line; Based on the position information of the straight line of the table, straight line elimination is performed on the table area image; Inputting the table area segmentation map and the text area segmentation map with straight lines eliminated into the text detection branch network, and outputting the position information of the text detection box and the text segmentation map based on the text detection branch network; The image region segmentation map and the text segmentation map are output as element segmentation maps, and the position information of the image region segmentation map, the position information of the table straight line and the position information of the text detection box are output as element positions.

2. The method according to claim 1, characterized in that The outputting of the position information of the text detection box and the text segmentation map based on the text detection branch network includes: Based on the text detection branch network, a text detection frame of the table area segmentation map with the straight line eliminated is obtained according to the position information of the table straight line; Determine the coordinates of the text detection frame, and output the coordinates as position information of the text detection frame; The text in the text detection box is segmented to obtain a text segmentation map and output it.

3. The method according to claim 1, characterized in that The training method of the segmentation model includes: Training at least one branch network of the segmentation model to convergence; The error of the at least one branch network is back-propagated to obtain a total error of the segmentation model, and the segmentation model is trained with the total error being less than a preset threshold as a goal.

4. The method according to claim 3, characterized in that The back-propagating the error of the at least one branch network to obtain the total error of the segmentation model includes: The semantic segmentation branch network error, the line detection branch network error, the detection error of the position information of the text detection box of the text detection branch network, and the text segmentation error of the text detection branch network are multiplied by the corresponding preset constant coefficients respectively, and the sum of the products is taken as the total error of the segmentation model.

5. The method according to claim 1, characterized in that The display of the element positions of the layout image, the element segmentation diagram excluding the text type, and the text content in a preset display manner includes: In the display interface, the element positions of the layout image are marked based on a position identification frame in a preset format, and the element segmentation diagram other than the text type and the text content are presented in the corresponding position identification frame.

6. The method according to any one of claims 1 to 5, characterized in that Also includes: In response to layout operation instructions, operations are performed on the element positions of the layout image, the element segmentation diagram other than the text type, and / or the text content; wherein the layout operation instructions include at least one of the following: content extraction instructions, editing instructions, and layout image generation instructions.

7. A layout image processing device, characterized in that: include: A layout segmentation module is used to input a layout image into a pre-trained segmentation model, perform region segmentation on the layout image based on the segmentation model, and when the segmented region includes a table region, perform text segmentation after straight line elimination on the table region, and output the element position and element segmentation map obtained by the final segmentation of the layout image; A text recognition module is used to perform text recognition on the element segmentation graph belonging to the text type to obtain the text content; A result display module, used to display the element positions of the layout image, the element segmentation diagram excluding the text type, and the text content according to a preset display method; The layout segmentation module includes a segmentation model, and the segmentation model includes a semantic segmentation branch network, a line detection branch network and a text detection branch network; The layout segmentation module also includes a region segmentation submodule, a straight line detection submodule, a straight line elimination submodule, a text segmentation submodule and an output submodule; The region segmentation submodule is used to input the layout image into the semantic segmentation branch network, perform region segmentation on the layout image based on the semantic segmentation branch network, and output at least one of the following region segmentation maps: an image region segmentation map, a table region segmentation map, and a text region segmentation map; The line detection submodule is used to input the table area segmentation map into the line detection branch network when the segmented area includes the table area, and output the position information of the table line based on the line detection branch network; wherein the line detection branch network directly segments the table line; The line elimination submodule is used to eliminate the lines of the table area image based on the position information of the line of the table; The text segmentation submodule is used to input the table area segmentation map and the text area segmentation map with straight lines eliminated into the text detection branch network, and output the position information of the text detection box and the text segmentation map based on the text detection branch network; The output submodule is used to output the image region segmentation map and the text segmentation map as element segmentation maps, and output the position information of the image region segmentation map, the position information of the table straight line and the position information of the text detection box as element positions.

8. A terminal device, characterized in that: The terminal comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the layout image processing method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for processing a layout image as described in any one of claims 1 to 6 is implemented.

Images